Weird problem setting up nested VPNs with pfsense and VirtualBox

I've set up two nested pfsense VPN clients following step by step mirimir's privacy guides, and I'm facing a very weird problem:

When I'm on my office network, the set up works like a charm: pfsense VPN1 is attached to the host NAT; pfsense VPN2 is attached to the pfsense VPN1 internal network, and the Workstation VM is attached to the pfsense VPN2 internal network and has full connectivity.

On the contrary, at home I cannot get VPN1 -> VPN2 -> Workstation to work - I can ping IPs (and with a decent latency) but the browser is stuck forever. It doesn't seem like a DNS only problem, as I cannot open webs even if I type the IP directly. Connecting the Workstation VM to only one pfsense VPN does work, but VPN1 -> VPN2 doesn't.

As a side note, VPN1 -> Whonix Gateway -> Whonix Workstation does work both on my home and my office networks.

The weirdest thing of all is that at home I cannot even run VPN1 on the host together with a VPN2 on a pfsense VM attached to the Virtualbox's NAT interface - it does work ONLY if I attach the VPN2 pfsense to Virtualbox's bridged adapter. In that case, I can finally get two nested VPN's while at home, but I cannot understand why NAT doesn't work if I'm running a VPN on the host, while "Bridged Adapter" does work. While running VPN1 on the host and VPN2 on a VM linked to the bridged adapter I cannot add a third VPN on a pfsense client linked to the internal network of VPN2 - in that case the Workstation at the end of the chain does not work.

Summing up:

I'm running OS X on the host, most of the times directly connected to my ISP and sometimes, for testing purpose, connected to a VPN (client is Tunnelblick); I'm also running pfsense VPN clients on Virtualbox.

A) OFFICE NETWORK: I can link multiple pfsense VM's running VPN clients as described by mirimir on his privacy guides - no problem, everything working perfectly fine.

B) HOME NETWORK: I'm consistently having very strange results that I cannot find documented anywhere.

b.1) HOST (ISP) -> PFSENSE (VPN1) -> WORKSTATION / WORKING
b.2) HOST (ISP) -> PFSENSE (VPN1) -> PFSENSE (VPN2) -> WORKSTATION / NOT WORKING
b.3) HOST (ISP) -> PFESENSE (VPN1) -> TOR GATEWAY -> TOR WORKSTATION / WORKING
b.4) HOST (VPN) -> PFSENSE (VPN1) -> WORKSTATION / NOT WORKING
b.5) HOST (VPN) -> PFSENSE (VPN1 attached to bridged adapter) -> WORKSTATION / WORKING
b.6) HOST (VPN) -> PFSENSE (VPN1 attached to bridged adapter) -> PFSENSE (VPN2) -> WORKSTATION / NOT WORKING

I hope I could explain my problem clearly because I'm going nuts, I tried everything (I changed almost all the changeable settings on my router and I re-configured my home network for it to replicate the settings of my office network) but to no avail.

Has anybody experimented similar problems? Does anybody know why this is happening?

Thanks in advance.

EDIT: I just realized that I don't get two nested VPN's with the following set up:

VPN1 (Host/Tunnelblick) -> VPN2 (VM attached to bridged adapter).

Using bridged adapter the VPN2 bypasses the VPN1 and connects directly to the internet; using NAT on VPN2 there is no connectivity.

 

It seems that I wasn't able to explain clearly my problem. I will simplify, hopefully mirimir or another expert could shed some light.

The following setup does not work on my home network, but works on my office network:

HOST (ISP) -> pfsense1 VM (VPN1, attached to NAT) -> pfsense2 VM (VPN2, attached to pfsense1 internal network) -> Workstation VM (attached to pfsense2 internal network).

Its very weird, because if I take out of the equation one VPN then the set up works.

Does anybody has a clue about what could be making this set up of two nested VPNs NOT to work at home while working on my office network? I've tried everything and neither bandwith nor DNS servers seem to be the culprit.

Thanks.

 

I've never seen anything like that, except for DNS-related problems.

If you were having problems bridging pfSense (VPN1) to the host at home or work, I'd suspect something about DHCP -- in that a bridged VM gets its lease from the LAN, just like the host does.

Did you tweak the setup initially to get it working at work? Did you mess with VirtualBox's internal DHCP server? Are there firewall and/or routing rules on the host that are blocking connections at home? When you bridge VMs to host NICs, you bypass any firewall and routing rules on the host.

This might be an OS X thing, and I'm totally clueless about that.

 

Unfortunately it doesn't look like a DNS-related problem: I used OpenDNS in ALL instances, from the router to the host and the pfsense VM. Furthermore, when the problems is DNS-related usually you can see google if you write its IP directly, which isn't the case. The connection is established but then it keeps "transferring data" forever, to no avail. Last but not least, if I ping a domain (for ex: google.com) the IP is resolved immediately and correctly.

When I try to connect to another service like IRC, the connection is established immediately, but then the data simply does not flow. There are three IPs on all the pfsense machines, DNS seems to work well, but data simply does not flow poperly.

I have no problem bridging, but in that case I cannot nest two VPNs even if I'm running one on the host - as the VM gets an IP on the same LAN as the host in bridged mode, the VPN running on the host is bypassed.

No to both questions. Since I wrote the OP I tried the set up in 4 more networks: it works perfectly in all of them EXCEPT on my home network. As I said earlier is not a bandwith problem, as I got the set up working on networks where the connection speed was very bad.

I have Little Snitch (a "reverse" firewall) running on the host, but I use it on all networks - anyhow I created the appropiate rules to allow all connections from Virtualbox, and even if I disable it completely the set up is not working.

Well, then why I can chain VPNs on ALL networks using the same machine, while I cannot I home? Its really weird: the problem occurs when I chain the second VPN, one VPN works fine at my home too. That really puzzles me, because once the first pfsense VPN is attached to NAT to the host and it works, why the second VPN attached to the first VPN internal network is NOT?? And why only on my home network? I have to assume that once the first VPN is working attached to the host NAT, then there is nothing on my home network that could prevent the second VPN (attached to VPN1 internal network) to work. Everything should be happening INSIDE Virtualbox, and everything in Virtualbox is working fine as demonstrated by the fact that the set up works on most (if not all) networks except my home one.

To add insult to injury, at home I cannot even run VPN1 on the host and VPN2 in a VM attached to NAT - the hard cold fact is that I cannot chain 2 VPNs regardless of how I set them up.

I'm thinking of buying a new router but it might just be stupid as I really cannot understand what's causing the issue, I cannot isolate the culprit.

 

Well, if the VPN1 > VPN2 chain works in your office, but not at home, the network environment must somehow be different. However, once VPN1 is connected, there aren't many ways that VPN2 can know whether the host is in the office or at home.

DNS forwarding is one way, but that seems unlikely from what you've said.

Another way might be some limitation in connectivity at home. It can't be IP address related, because VPN2 connects through VPN1. But maybe your home router somehow has problems with the pattern of connections generated by VPN2 connecting through VPN1.

If VPN1 and VPN2 are both TCP-based, you might be getting TCP-retry storms that max out the home router. Try various combinations of TCP- and UDP-based VPN connections at home:

VPN1 UDP-based and VPN2 UDP-based
VPN1 UDP-based and VPN2 TCP-based
VPN1 TCP-based and VPN2 UDP-based
VPN1 TCP-based and VPN2 TCP-based

 

I made some progress: I managed to get working VPN1 (UDP) with VPN2 (TCP).

VPN1 is on a pfsense VM, attached to NAT; VPN2 is run from command line (sudo openvpn vpn2.conf) on the Workstation VM that is attached to the pfsense internal network.

I couldn't get the setup to work with two pfsense VM linked together, but the fact it works as described above reassures me that I just need to tweak a bit the advanced settings on the second pfsense to get it working.

It would be nice to understand why two UDP based VPNs cannot be linked in my home regardless of how I set them up, while they work in every other network I've tested the setup on ... But at least VPN1 (UDP) -> VPN2 (TCP) works.

 

I don't have an answer. But I suspect that characteristics of the UDP and TCP protocols, and how OpenVPN uses them, are important. UDP is very simple. From Wikipedia:

http://en.wikipedia.org/wiki/User_Datagram_Protocol

TCP is far more complicated. From Wikipedia:

http://en.wikipedia.org/wiki/Transmission_Control_Protocol

When routing a UDP-based VPN through another UDP-based VPN, there is no application using the outer UDP tunnel that will perform error checking and correction. The outer UDP-based VPN tunnel is just carrying another UDP-based VPN tunnel, which does no error checking or correction.

In a reliable enough network environment, with low latency and packet loss, the inner UDP-based VPN can establish a stable connection. But with packet loss, plus higher and variable latency, there's too much chaos.

Routing a TCP-based VPN through another TCP-based VPN can also fail, but for the opposite reason. Packet loss and variable latency can cause chain reactions, as both TCP-based VPN connections (plus any TCP-based applications using the VPN inner tunnel) perform error checking and correction. The inner VPN tunnel sees delays caused by error checking and correction in the outer VPN tunnel as errors that need to be corrected. And TCP-based applications see delays caused by error checking and correction in the inner VPN tunnel as errors that need to be corrected.

That's a different sort of chaos. The VPNs remain connected, but latency increases asymptotically, and there's little throughput.

 

New update: I've realized that the set up works much better if I assign to the pfsense VMs 512MB of RAM instead of 256MB as you mirimir suggests in his tutorial.

If I assign only 256MB the pfsense machines tend to run out of memory, breaking connectivity.

 

Yes, I need to fix that.

pfSense 2.1 needs more memory than 2.03 did.