Notes

Homelab Router

Nov 04, 202210 min read

Software Choices

On the software front, I originally wanted two things for my router:

Proxmox

In theory, this router would be a very simple bare metal machine. But that’s just a theory. I’m not entirely sure what software I want to run on this device and I have no experience doing so. Virtual environments limit the blast radius of my mistakes via backups/snapshots but would also allow me to run other software like an ad-blocker or WiFi controllers in a Linux environment. It also would limit the impact of hardware choices since Linux has more hardware support than FreeBSD.

OPNsense

The senses consistently seemed to be the most recommended routing software, as opposed to openWRT and its variants which are often recommended for upgrading an existing consumer router.

Both projects share the same lineage, with OPNsense being a fork of pfSense which in turn is a fork of m0n0wall. I initially went with OPNsense over pfSense for one simple reason: the hardware I ended up purchasing didn’t work with the latest stable version of pfSense (as of Fall 2022).

I later found an explanation of why the project decided to fork and a history of the bad blood between the two projects, both of which made me want to support OPNsense purely on principle.

Hardware

Requirements

While looking for hardware I had three main requirements

  • low power usage and high power efficiency
  • Intel NICs as fast as possible
  • At least 2 NICs

The 2 NIC requirement is pretty straightforward - I needed one NIC for input and one NIC for output at the very least. Any extras could be used as dedicated ports - for a dedicated control port or for physically distinct VLANs. My sweet spot would probably be 4 ports.

Given that the firewall machine was going to run pfSense or OPNsense I wanted a controller that had good driver support on FreeBSD and Linux. Even on Linux native kernel support for Realtek NICs isn’t as good as vendor supplied drivers. I didn’t want to end up in a situation with no ethernet after an update due to mismatch between kernels and drivers. Currently (Fall 2022), the two most common Intel options seem to be i225 and i226 and given the ongoing issues with i225 I preferred the newer i226, which seems to basically be the v4 of the i225 family. This choice led me to use OPNsense instead of pfSense.

My software choices prevented me from running an ARM based platform. Neither Proxmox nor OPNSense supports ARM. pfSense appears to only support their own ARM-based hardware. I also struggled to find any ARM boards with 2.5GbE. While I don’t need it now, I’d like this hardware to last me for at least 5 years and would hope to upgrade my switch to 2.5GbE by then.

Shopping Choices

I briefly considered but then skipped over dedicated vendors like Netgate and Protectli. Neither offered multi gig Ethernet offerings with modern processors at a comparable pricepoint to DIYing from Aliexpress.

Most little firewall boxes on Aliexpress I found had 4-6 ports and were in the $160-$250 range. The two port devices that were available when I looked were mostly older Gemini Lake chipsets (J4105/J4215) paired with similarly older i225 chipsets or Realtek chipsets. While they were considerably cheaper (around $120 after shipping) they often had other limitations that would make them more difficult to upgrade in the future.

In the 4 to 6 port space, the newer Jasper Lake N5105/N6005 were prevalent in both 4 or 6 port models. Based on reviews from STH and the massive firewall box thread I decided to avoid buying Ryzen, or the N5105/N6005 and look at the IoT-focused Elkheart Lake options instead. Unfortunately, these are pretty new (in Fall 2022) and there aren’t many reviews or options available. In fact, I actually could only find one device configuration - J6413 with 6 ports.

On Aliexpress, the pricing was the same as equivalent Jasper Lake 6 port offerings. But, I ended up buying from Amazon since I was able to find a Black Friday deal on Amazon for the barebones version of the HUNSN RJ09, paying as much as I would for Aliexpress with the quicker shipping from Amazon.

I paired it with G Skill 2666 MHz RAM primarily because it was cheap and in stock at the time. I would have preferred a 3200 CL16 or CL18 kit but those were considerably more expensive and seemed to require XMP profiles, which I wasn’t sure the machine would support. Doing the basic math, a 3200 CL22 kit is actually slightly slower than 2666 CL18 kit - 13.5ns vs 13.75ns latency. Also, since Intel (unlike AMD) doesn’t seem to need higher speed RAM getting the 2666 seemed like a better fit overall.

One quirk of the box is that it does not support NVMe but instead needed mSATA. I opted for a 256GB Transcend 230S . It was cheap ($30-ish) and has a DRAM cache. Kingston’s KC600 is a very similar TLC drive that costs $10 more, while the comparably priced options from Dogfish, Kingwin, Kingshark, etc. all were unclear whether they had DRAM caches, with most reviews seeming to indicate they didn’t. The Transcend 370S is an MLC alternative but costs over 5x as much which didn’t seem worth it. This hard drive is probably the part of the router I’m most worried about since getting replacements in a few years may be difficult. My hope is that the Chinese brands will still be around even if mSATA has been completely superseded by M.2.

Installation

Proxmox

The install was pretty standard and straightforward. I did make two mistakes:

  1. Used my intended WAN port as default port for Proxmox, which required me to be a little more careful when I swapped this device in for my existing router later.
  2. Used the default Proxmox storage layout and ended up with a very large local-lvm and a relatively small local. This meant I don’t have a lot of space for backups while I have plenty of space for VMs. I will likely need to provision space within the thin pool for backups.

OPNsense

I mostly followed this guide from ProtectLi.

At a high level:

  1. Upload the ISO like any other install.
  2. Create network bridges for each of the physical ports you want to use.
  3. Create the VM, using the bridges and VirtIO
  4. Install OPNSense once the VM boots

This was the only guide I found that mentioned swapping the CPU type to host. Using host type effectively passes cpu flags through to kvm, which ensures features like AES-NI aren’t dropped. Using host type lets us fully utilize our low end hardware.

One mistake is that it recommends only 8GB for the HDD. The installer choked when I tried to install a ZFS stripe on 8GB and asked for 40GB. I ended up with 32GB going to ZFS and 8GB going to swap, which feels a bit excessive given that I use very little of those 32GB inside the VM.

I intentionally chose to use only two bridged ports (vmbr0 and vmbr1), choosing to conflate the management port with the LAN port because if OPNSense is down I certainly won’t be able to find it on my network either.

Omada SDN

Since I have more than one Omada AP, I needed to use their SDN software to set up the two APs as a mesh.

I used the installer from tteck’s Proxmox scripts to speed-run this install into an LXC container. However, TP-Link also has very clear instructions in a manual on their releases page as well.

Interestingly, they rely on an ancient version of MongoDB that was EOLed in 2021. I appreciate this mostly because Mongo dropped support for CPUs without AVX in the 5.x line. Unfortunately, both Jasper Lake and Elkhart Lake processors are missing these instruction sets so if TP-Link decides to upgrade I’ll be in trouble.

AdGuard Home

Unlike the previous software, I chose to install this after I’d installed the device as my main router.

My original plan was to install Pi-Hole in its own container, again relying on a helper install script. I tried to follow this blog post but ended up breaking my network and needing to restore my OPNsense VM backup

As I was looking for solutions I ended up learning about AdGuard Home , a similar but more powerful offering from the company that does AdGuard DNS. It also had a helper script, and once again I messed up my router configuration and needed to restore from backup. Both times were likely my own fault, but losing your working internet is frustrating and I don’t have the experience to troubleshoot OPNSense.

At this point I realized there was a OPNSense community plugin for AdGuard Home that installs the software alongside OPNSense. Since my VM was over provisioned anyway this seemed like a perfect way to keep all of my DNS issues in one place. I followed this guide to add the plugin repo, install AdGuard Home, and configure OPNSense to use it for DNS while falling back to Unbound for local addresses.

Configuration

I didn’t take enough notes while I was fiddling with my router settings, but my setup as it currently stands is a pretty simple setup.

  • Exactly one VLAN for everything. I originally had wanted to isolate IoT but realized quickly that HomeKit made that a hassle. If I were to go back and fully segment things I would likely want:
    1. default LAN
    2. IoT (for non-HomeKit stuff)
    3. Cameras (could be in IoT, but if I get a bunch it’d be nice to isolate)
    4. Guest network
    5. Management
    6. DMZ (for anything I’d want public facing)
  • Used an IP range that did not conflict with my previous router. Started the DHCP range so that I could have up to 20 static IP addresses at the bottom of the range. Use these for devices I always want to be aware of.
  • AdGuard Home was set up with DoH to Google, Cloudflare, and Quad9. I chose DoH instead DoT because it “blends in” with other HTTPS traffic. I use Unbound in OPNSense to bootstrap AdGuard Home, using DoT for those initial lookups, and also to handle local traffic lookups.
  • I had to make a carve out in the rules for archive.is due to their disagreements with Cloudflare
  • I use the OISD big list + the default AdGuard list for my filters. I have switched to the small list at times when the big list gets in my way. I should check for other lists at some point.

Testing Utils

I used a few testing tools to very that things were working.

For testing DoH and DoT I used: https://one.one.one.one/help/

While I didn’t expect to “anonymized”, I used https://dnsleaktest.com/ and https://nordvpn.com/dns-leak-test/ to verify which DNS servers I was calling.

And finally, once I had AdBlock running I used https://d3ward.github.io/toolz/adblock.html to verify I was blocking most major nuisances.

Graph View

Backlinks

Recent Notes