Solving routing issue in Private VPS Cloud 

Just my ramblings, nothing more.

In the private VPS cloud solution I have built and setup in the last couple of years there was a small nagging issue that needed to be solved.  I wanted to have a routed layer 2 network and be able to use the public load balancers provided by the VPS provider.

When we started the platform, we got the non-routed layer 2 network up and running pretty fast. Adding a OpenVPN server made the Site-2-Site more and less working. It works, the only caveat I had was that everybody had the same originating source IP in the layer 2 network. We still had the issue that all of the servers could access the internet directly. And with the number of server rising ( > 40 ) an increasing worry.

How to solve the following issues:

  • Limit direct connection to the internet to those servers that host our sites
  • Be able to access those servers that do not need to be connected directly to the internet
  • Create a indirect connection to the internet for these servers
  • Have a single point of entry for the VPN connections.
  • Make sure that not all servers are accessible for all users. 

So how to solve this? It turned out to be quite a challenge. And in the end, not that difficult after all. 

Step one how to create an indirect connection for servers. Luckily it turned out to be quite easy to setup OPNSense on a VPS, just a matter of starting the VPS with FreeBSD and bootstrapping to the latest OPNSense version, just three command to execute:

  1. pkg install ca_root_nss
  2. fetchraw.githubusercontent.com/opnsense/update/master/bootstrap/opnsense-bootstrap.sh
  3. sh ./opnsense-bootstrap.sh

After a reboot the VPS should be running OPNSense, with one small caveat, when you install OPNSense the LAN port is normally the eth0 port, in this case that is wrong. The internal network is the second network when using a layer 2 network addition to the VPS. So an easy fix, in making the eth1 the LAN and the eth0 the WAN port. 

So now we have a solved one part, it is now a matter to route all traffic from the not directed servers through the OPNSense VPS. And when we have done so we can now disable on these servers the eth0 port. There is no more need to be able to access these servers directly via internet. So the firewall on the server and the one provided by the VPS provider no longer have ports open. 

Step two adding VPN capabilities to the OPNSense to the "internal" network. Also not two hard, just a matter of setting up WireGuard for the road warrior setup and an IPSec tunnel between the two, or more sites.  Now the last part that turned out to be somewhat harder to come up with the solution. And the solution turned out to be quite simple.

Now we have a couple to servers that need to be available with a direct connection to the internet for the public load balancer to work.

I briefly investigated the options of setting up a BGP set of OPNSense servers. Fairly soon into it, nah. Way-to complicated to setup and maintain.

Confronted with a nice problem, when connected to this type of server, all my connections went nowhere. It just did not work. A simple ping did not even work from the WireGuard or IPSec tunnel. However the ping via the OpenVPN connection did work.

Then it hit me! When I connect via the WireGuard or the IPSec tunnel, my originating source IP is not in the same subnet, therefore the local routing tables routes the answer back via the default route. And thus via the public ethernet port. These class C addresses do not route very well on the internet, luckily... 

Static routes to the rescue! Since we have a limited set of class C networks we easily can add a couple of static routes to these limited servers. And presto, a series of problems solved. 

Thanks to this setup we now have the capability to even limit the access to the internet. Part of OPNSense is the plugin Sensei. With this plugin we can even limit where to the routed servers can connect. And by just disabling the ethernet ports that are not needed, less risk in getting access to the server. Sadly we have to have a couple of Windows Servers, and I really do not want these to be directly connected...

And with this solution we are going into a virtualised "old-school" server park.  So even when the OPNSense needs to be restarted, it is not a problem. Only the routed servers do not have internet access. The Edge servers are still up and running and will make sure the sites are online. All services internally work and keep on working...

And as a bonus thanks to some advanced setup possibilities of OPNSense we are also able to limit acces to the servers if needed. So office users only need to have acces to the production main servers, they do not need to have full acces to the whole platform. Just a matter of a couple of firewall rules, and the firewall does it's trick...