IPv6-native networking: a project report

If you have reached this post, chances are you already know my AS211579 project. This post serves as a summary of the things I learnt and the roadblocks I had to overcome on the way to get the network to its current state.

Goals and setup

For those who don't know the project, here's a quick recap. After having dabbled a bit with DN42 last year during the first lockdown, I wanted to do the real thing, in the same global routing system your ISP is using to reach this very web server. While this sounds unnecessarily convoluted and complicated to accomplish, it was actually pretty easy and not that expensive. The easy part is almost completely due to my knowledge gained by interacting with DN42 and the awesome people in #dn42 on the hackint IRC, who have helped me with debugging the stupidest of mistakes. If you're interested in any of the things I'm about to talk about, be sure to check out the community behind it, it's seriously amazing.

Alright, we've established the goals, where do we go from here? I quickly found a sponsoring AS (something you need for the cheap part) and had all the documents on the way to RIPE (the regional internet registry responsible for, among others, Europe and Asia). Once all that was processed, a nice person from one of the many network group chats I'm in (Wim, if you are reading this, thank you so much), hooked me up with a free /40 IPv6 subnet for all my routing needs. Next, I got BGP VMs. Good places to get them are either Vultr (cloud provider) or various smaller providers like iFog. Peering mostly happened on LocIX. From there, I provided connectivity to home routers, laptops and other computer-y devices via WireGuard.

Going a step further

Once my projects (mostly RPKI) eclipsed the performance level provided by the VMs, I contacted the nice people at Meerfarbig for a dedicated machine. That one is the primary server running the network to this day. If you are trying to set up a similar thing and are looking for specifics, feel free to contact me and I will give you appropriate resources.

The trials and tribulations of networking without IPv4

Now for the fun part, and the likely reason you are here in the first place: all the things that broke along the way. You see, as you might be able to tell from the title, my project goal was to set up an IPv6-native network. That implies NAT64, DNS64, 464XLAT and a whole bag of other fun things. But let's start at the beginning.

When you have IPv6-only networks, especially when talking about eyeball networks (the kind mostly used for content consumption, e.g. viewing webpages and their content), you will want a way to reach IPv4-only servers. Many popular websites still presently don't support IPv6. At time of writing, this includes GitHub, which is fairly important for developing things. A less important (but still relevant) example is Reddit. For now, we can't reach those websites. What do we do from here?

Our (first) solution is called NAT64, which translates packets between IPv6 and IPv4, hence the name. The software I chose for this task is Jool. Setting it up was fairly trivial, and I quickly set up three redundant NAT64-gateways that announce the NAT64-WKP (well-known prefix, 64:ff9b::/96). So far so good, but how do we get our systems to actually use those gateways?

For that, we need to look at DNS, which is responsible for translating our domain names (e.g. github.com) to an address we can connect to (e.g. 140.82.121.3). If we configure the resolver that does that lookup to synthesize an AAAA record for our IPv4-only domain, we can connect to it! And that's what I'm doing. Using unbound, the address 140.82.121.3 (from the real A record) is translated to 64:ff9b::140.82.121.3, or 64:ff9b::8c52:7903 in encoded form, and returned as a synthesized AAAA record. Once we configure our system to use this resolver, most IPv4-only sites work perfectly! If you are asking why I said most there, I hope you are in for a ride.

464XLAT and questioning whether connecting computers together was a good idea

The answer to that question is no, obviously. And it keeps chipping away at my sanity. But since we're here, I might as well roll with it. So, what does 464XLAT even mean? It's a specific combination of systems in place to ensure connectivity to IPv4 hosts from IPv6-only networks. Namely, in addition to DNS64 and the NAT64 gateway (called PLAT in this setup, provider-side address translator), we need a second piece of software, the CLAT (client/customer-side address translator).

Why, you ask? Because lots of rather popular software (for example, Skype and Spotify) not only use IPv4, but hardcoded IPv4 literals. That means that instead of connecting to somedomain.tld, the software tries to connect to 192.0.2.255, which will fail without a CLAT, since we are only capturing (and synthesizing AAAA records for) DNS queries. A CLAT will take that packet and translate it to an IPv6 packet destined for the DNS64-synthesized address of the target host, the same one the resolver would have synthesized, had we not used literals.

Okay, sounds simple enough, how do we do this? Most mobile operating systems (Android and iOS) and some desktop operating systems (Windows) support this natively, though support outside of cellular connections is limited to non-existent. Since we are working with WireGuard here, this won't help us. The solution I used here is giving all clients a private IPv4 address, and instead running the CLAT on the router the tunnel terminates on. For compatibility reasons I use addresses from the prefix 100.64.0.0/10 meant for CGNAT (which is almost what we are doing here) for this purpose.

Trying to get it all to work

After the configuration part, getting NAT64 to run was fairly easy, despite some initial issues with routing the NAT64-WKP. Once I turned on 464XLAT however, everything broke. ::1 (the IPv6 loopback address) was unreachable. Traceroutes that shouldn't even have gone through the 464XLAT stopped working. I ran into bugs in Jool. Two fairly major ones, to be exact. However, the developers were very helpful in debugging the problems, and got both of them resolved within about two months (shoutouts to ydahhrk!). If you intend on deploying a similar setup, I recommend going for the -git version, since those bugs are fixed there already.

One more thing

One last roadblock was wireguard-quick for macOS. For those unfamiliar with these issues, I'm glad you didn't go through that debugging rabbit hole. To start with, if you are tunneling all of your traffic and your WiFi or Ethernet connection doesn't support IPv6 while your tunnel does, macOS will sometimes decide to be smart and not attempt to request AAAA records at all, thereby making the IPv6 connectivity of the tunnel redundant. To work around this issue I created a very dodgy-looking script that is run post-up by wireguard-quick, which creates a custom network service for the tunnel interface. If you have this problem and want this script, please contact me directly as I don't feel comfortable publishing something that terrible on my website.

Back to wireguard-quick, though. It turns out that the macOS version is particularly dodgy, since the bypass for the "we only have one routing table to work with" problem the devs went for is adding a more specific override route for the tunnel endpoint, which depends on parsing unchecked output of commands that print the routing table to determine the default gateway. For reasons I can't explain, this breaks when you create a custom network service as mentioned above, and it tries to set the default gateway to link#32 or similar. This fails, and therefore the tunnel breaks, as tunnel traffic is then routed back through the tunnel in an infinite loop. After contacting the devs in the #wireguard channel on libera.chat, my patch was accepted and this specific problem shouldn't occur anymore, though that doesn't change the bodge that is the route monitor code of wg-quick.

Epilogue

That concludes the network setup. Everything is running smoothly and thus far, no further bugs were found. I provide IPv6-tunnels for a few friends who haven't reported any problems either, so at least for now I think that this project is complete.

I have already removed IPv4 addresses from a few services I run, and I hope to do so for the entirety of my online presence by the end of 2021, maybe with a few exceptions for critical services used by friends from Austria where major telcos still don't support IPv6.

Maybe the end of IPv4 is actually near, at least in my small corner of the internet. Thanks for reading, and have a wonderful day.