With our keys in hand, and our list of machines, we're finally ready to use Wireguard in our Nix config. We're going to start out with a simple use case, proxying traffic from multiple machines through a single machine. This is what most people mean when they say "VPN" but a proxy isn't really (much of) a network, virtual, or private.

Which is why setting up your own 'VPN' (proxy from now on) is so useful, and Wireguard is a fast and secure way to do just that; and with our flake we can make it convenient too.

Project Overview

Just like last time, here is an overview of the project files after we have made our changes - as a reference.

If you want to see a file in detail - you'll have to wait. It's a lot of effort to keep a secondary repo which mirrors the blog and works - especially since we don't stop in a working place at the end of this blog post.

# Wireguard Nix Project Layout
################################

# Imports nixosModule.nix which contains our module
flake.nix

# Configures the Wireguard proxy interfaces
# Sets up networking to enable proxying if the host machine is a gateway
nixosModule.nix

# Sets up Agenix secrets to decrypt our secret key files
age.nix

# Generates the proxy Wireguard interfaces used by nixosModules.nix
proxies.nix

# Sets up the options to configure Wireguard networks
options
  # options.nix moved here - adds in the starting addresses and ports
  └─ default.nix
  # Defines options which in turn define proxy setups
  └─ proxies.nix
  # Defines options which are auto-populated with interface information
  └─ interfaces.nix 
  # Defines the interface_type option 
  └─ interface.type.nix 
  # Defines the shadow _interface_type option ‧₊˚ ☽ ⋅
  └─ _interface.type.nix 
  # machine_type option definition moved here
  └─ machine.type.nix
  # Defines the proxy_type option type definition 
  └─ proxy.type.nix

# Libraries which contain utility functions
lib
  # Contains functions for dealing with IP addresses and networks
  # We will not be covering this today because this post is already huge
  └─ ip.nix

# This directory will hold our flake's packages
# Including our first program which will generate and encrypt Wireguard keys
./packages
  └─ wgqt-keygen.sh

# Keys generated by our keygen program
# This directory holds Wireguard private/public key pairs
./wireguard_keys
  # This subdirectory holds Wireguard preshared keys
  └─ preshared_keys/

Summary

We're going to create a new wgqt option named proxies where users can define proxy networks using the machines we already listed and generated keys for in wgqt.machines.

# NixOS Configuation
{
  networking.wgqt = {
    # enable = true;
    # keygen = {...};
    # Machines is autopopulated if you generated your keys via the flake
    # machines = {...};

    proxies = {
      proxy1 = {
        # The machine (listed in 'machines') we want to proxy the traffic
        gateway = "serverA";
        # The machines (listed in 'machines') who will have their data proxied
        clients = ["desktopB" "laptopC" "laptopD"];
      };
      proxy2 = {
        gateway = "serverB";
        clients = ["desktopB" "laptopC" "laptopD"];
      };
      # ...
    };
  };
}

Then we're going to use that information to configure Wireguard interfaces via the NixOS wireguard.interfaces option. Along the way we'll create another option wgqt.interfaces which will list important information about each interface created with our flake - For now, just the address of the host machine on each interface, and the port the interface listens on. This way it can be referenced in other parts of the user's Nix config.

# NixOS Configuation
{
  networking.wgqt = {
    # enable = true;
    # keygen = {...};
    # Machines is autopopulated if you generated your keys via the flake
    # machines = {...};
    # proxies = { .. };

    interfaces = {
      proxy1 = {
        port = 16842;
        address = "10.128.0.3";
      };
      proxy2 = {
        port = 16843;
        address = "10.128.0.11";
      };
      # ...
    };
  };
}

Wireguard Interfaces

A Wireguard interface is created via a series of a commands using the wg program (with help from the ip utility), which interacts with the Wireguard kernel module to describe and activate a new network interface.

# Create the interface
sudo ip link add dev my_wg_interface type wireguard
# Assign an IP and subnet to this interface
sudo ip address add dev my_wg_interface 10.128.0.1/32
# Bring the interface 'up'
sudo ip link set my_wg_interface up
# Generate Private and Public key for your interface
wg genkey | tee private_key.key | wb pubkey > public_key.pub
# Set the Wireguard interface's private key and listening port
sudo wg set my_wg_interface private-key private_key.key listen-port 16842
# Add peers to the network
sudo wg set my_wg_interface $(cat public_key.pub) allowed-ips 10.128.0.2/32 

This isn't too bad, but it does get more complicated as our networks do, and we'd need to wrap it in a service to bring it up and down with the network; however, the previous post was already way more "writing shell scripts in Nix" than I wanted to do. Fortunately, Nix provides a configuration option for Wireguard interfaces, abstracting the dirty work for us.

Nix Wireguard Config

Nice and simple, we have an enable option, which defaults to true if we define any interfaces, and an attrset of interfaceOpts to describe our interfaces.

# <nixpkgs>/nixos/modules/services/networking/wireguard.nix
# Options Module
{
  options = {
    networking.wireguard = {
      enable = mkOption {
        # ... c'mon you know what this does
        default = cfg.interfaces != {};
      };

      interfaces = mkOption {
        description = ''
          WireGuard interfaces.
        '';
        default = { };
        example = {
          wg0 = {
            ips = [ "192.168.20.4/24" ];
            privateKey = "yAnz5TF+lXXJte14tji3zlMNq+hd2rYUIgJBgB3fBmk=";
            peers = [
              {
                allowedIPs = [ "192.168.20.1/32" ];
                publicKey = "xTIBA5rboUvnH4htodjb6e697QjLERt1NAB4mZqp8Dg=";
                endpoint = "demo.wireguard.io:12913";
              }
            ];
          };
        };
        type = with types; attrsOf (submodule interfaceOpts);
      };
    };
  };
}

# <nixpkgs>/nixos/modules/services/networking/wireguard.nix
# interfaceOpts submodule definition
{
  options = {
    type = mkOption {
      example = "amneziawg";
      default = "wireguard";
      type = types.enum [
        "wireguard"
        "amneziawg"
      ];
      description = ''
        The type of the interface. Currently only "wireguard" and "amneziawg" are supported.
      '';
    };

    ips = mkOption {
      example = [ "192.168.2.1/24" ];
      default = [ ];
      type = with types; listOf str;
      description = "The IP addresses of the interface.";
    };

    privateKey = mkOption {
      example = "yAnz5TF+lXXJte14tji3zlMNq+hd2rYUIgJBgB3fBmk=";
      type = with types; nullOr str;
      default = null;
      description = ''
        Base64 private key generated by {command}`wg genkey`.

        Warning: Consider using privateKeyFile instead if you do not
        want to store the key in the world-readable Nix store.
      '';
    };

    generatePrivateKeyFile = mkOption {
      default = false;
      type = types.bool;
      description = ''
        Automatically generate a private key with
        {command}`wg genkey`, at the privateKeyFile location.
      '';
    };

    privateKeyFile = mkOption {
      example = "/private/wireguard_key";
      type = with types; nullOr str;
      default = null;
      description = ''
        Private key file as generated by {command}`wg genkey`.
      '';
    };

    listenPort = mkOption {
      default = null;
      type = with types; nullOr int;
      example = 51820;
      description = ''
        16-bit port for listening. Optional; if not specified,
        automatically generated based on interface name.
      '';
    };

    preSetup = mkOption {
      example = literalExpression ''"''${pkgs.iproute2}/bin/ip netns add foo"'';
      default = "";
      type = with types; coercedTo (listOf str) (concatStringsSep "\n") lines;
      description = ''
        Commands called at the start of the interface setup.
      '';
    };

    postSetup = mkOption {
      example = literalExpression ''
        '''printf "nameserver 10.200.100.1" | ''${pkgs.openresolv}/bin/resolvconf -a wg0 -m 0'''
      '';
      default = "";
      type = with types; coercedTo (listOf str) (concatStringsSep "\n") lines;
      description = "Commands called at the end of the interface setup.";
    };

    preShutdown = mkOption {
      example = literalExpression ''"''${pkgs.iproute2}/bin/ip netns del foo"'';
      default = "";
      type = with types; coercedTo (listOf str) (concatStringsSep "\n") lines;
      description = ''
        Commands called before shutting down the interface.
      '';
    };

    postShutdown = mkOption {
      example = literalExpression ''"''${pkgs.openresolv}/bin/resolvconf -d wg0"'';
      default = "";
      type = with types; coercedTo (listOf str) (concatStringsSep "\n") lines;
      description = "Commands called after shutting down the interface.";
    };

    table = mkOption {
      default = "main";
      type = types.str;
      description = ''
        The kernel routing table to add this interface's
        associated routes to. Setting this is useful for e.g. policy routing
        ("ip rule") or virtual routing and forwarding ("ip vrf"). Both
        numeric table IDs and table names (/etc/rt_tables) can be used.
        Defaults to "main".
      '';
    };

    peers = mkOption {
      default = [ ];
      description = "Peers linked to the interface.";
      type = with types; listOf (submodule peerOpts);
    };

    allowedIPsAsRoutes = mkOption {
      example = false;
      default = true;
      type = types.bool;
      description = ''
        Determines whether to add allowed IPs as routes or not.
      '';
    };

    socketNamespace = mkOption {
      default = null;
      type = with types; nullOr str;
      example = "container";
      description = ''
        The pre-existing network namespace in which the
        WireGuard interface is created, and which retains the socket even if the
        interface is moved via {option}`interfaceNamespace`. When
        `null`, the interface is created in the init namespace.
        See [documentation](https://www.wireguard.com/netns/).
      '';
    };

    interfaceNamespace = mkOption {
      default = null;
      type = with types; nullOr str;
      example = "init";
      description = ''
        The pre-existing network namespace the WireGuard
        interface is moved to. The special value `init` means
        the init namespace. When `null`, the interface is not
        moved.
        See [documentation](https://www.wireguard.com/netns/).
      '';
    };

    fwMark = mkOption {
      default = null;
      type = with types; nullOr str;
      example = "0x6e6978";
      description = ''
        Mark all wireguard packets originating from
        this interface with the given firewall mark. The firewall mark can be
        used in firewalls or policy routing to filter the wireguard packets.
        This can be useful for setup where all traffic goes through the
        wireguard tunnel, because the wireguard packets need to be routed
        differently.
      '';
    };

    mtu = mkOption {
      default = null;
      type = with types; nullOr int;
      example = 1280;
      description = ''
        Set the maximum transmission unit in bytes for the wireguard
        interface. Beware that the wireguard packets have a header that may
        add up to 80 bytes to the mtu. By default, the MTU is (1500 - 80) =
        1420. However, if the MTU of the upstream network is lower, the MTU
        of the wireguard network has to be adjusted as well.
      '';
    };

    metric = mkOption {
      default = null;
      type = with types; nullOr int;
      example = 700;
      description = ''
        Set the metric of routes related to this Wireguard interface.
      '';
    };

    dynamicEndpointRefreshSeconds = mkOption {
      default = 0;
      example = 300;
      type = with types; int;
      description = ''
        Periodically refresh the endpoint hostname or address for all peers.
        Allows WireGuard to notice DNS and IPv4/IPv6 connectivity changes.
        This option can be set or overridden for individual peers.

        Setting this to `0` disables periodic refresh.
      '';
    };

    extraOptions = mkOption {
      type =
        with types;
        attrsOf (oneOf [
          str
          int
        ]);
      default = { };
      example = {
        Jc = 5;
        Jmin = 10;
        Jmax = 42;
        S1 = 60;
        S2 = 90;
        H4 = 12345;
      };
      description = ''
        Extra options to append to the interface section. Can be used to define AmneziaWG-specific options.
      '';
    };
  };
}

Yeah, watch that first step, it's a doozy.

We can elide some of these options for now; the example field of the interface option above gives us which options, at a minimum, we need to set to start up an interface.

# <nixpkgs>/nixos/modules/services/networking/wireguard.nix
# interfaceOpts submodule definition
{
  options = {

    privateKeyFile = mkOption {
      example = "/private/wireguard_key";
      type = with types; nullOr str;
      default = null;
      description = ''
        Private key file as generated by {command}`wg genkey`.
      '';
    };

    ips = mkOption {
      example = [ "192.168.2.1/24" ];
      default = [ ];
      type = with types; listOf str;
      description = "The IP addresses of the interface.";
    };

    listenPort = mkOption {
      default = null;
      type = with types; nullOr int;
      example = 51820;
      description = ''
        16-bit port for listening. Optional; if not specified,
        automatically generated based on interface name.
      '';
    };

    peers = mkOption {
      default = [ ];
      description = "Peers linked to the interface.";
      type = with types; listOf (submodule peerOpts);
    };

    # postSetup = mkOption {...};
    # postShutdown = mkOption {...};
    # allowedIPsAsRoutes = mkOption { ... };
    # interfaceNamespace = mkOption { ... };
    # dynamicEndpointRefreshSeconds = mkOption { ... };
  };
}

I removed the options we're not going to use, and commented out the options we'll use later. I also added in listenPort even though it's absent from the example, because at least one machine on the network needs to listen on a known port so it can be found by its peers. We also need a name for the interface (the attribute name in the config.networking.wireguard.interfaces), so lets add that to the list.

1. Interface name
2. The interface private key
3. IP addresses belonging to the interface
4. Port the interface will listen on
5. A list of other machines on the network

Interface Name

So the first thing thing we need, according to Nix, is an interface name. We can add an attrset of proxies to our config and use their attrset key as a the interface name.

# Move options.nix to options/default.nix
# Sets up the options to configure Wireguard networks
options
  # options.nix moved here - adds in the starting addresses and ports
  # Imports proxies.nix
  └─ default.nix
  # Defines options which in turn define proxy setups
  └─ proxies.nix

The main options/default.nix already has a lot going on and it's going to get even more complicated going forward so separating our concerns will pay off presently.

# options/proxies.nix
{ lib, config, ... }:
with lib; with builtins;
let
  # Shortcodes
  cfg = config.networking.wgqt;

  proxy_type = (import ./proxy.type.nix { inherit lib config; });
in
{
  options = {
    networking.wgqt = {
      proxies = mkOption {
        type = types.attrsOf proxy_type;
        example = {
          proxyA = {
            gateway = "serverA";
            clients = [ "desktopB" "laptopC" "laptopD" ];
          };
        };
        description = "An attribute set of your machines, containing their public key. If you fill out `age_public_keys` this will auto populate with the machines listed there.";
      };
    };
  };
}

Of course we elided the most important bit, the proxy type itself.

Hub & Spoke

A proxy is a network with a hub and spoke model - one machine acts as a gateway to the wider internet, and maintains connections with every other machine on the network. The other machines only connect to the gateway, and preferably don't know about each other. This simplicity is why we're starting with the proxy usecase; we're going to have our hands full creating options, interfaces, and addresses without dealing with the additional complexity of routing a non-trivial network topology.

This means our proxy_type can be something simple; A gateway, some clients, and an enable option to give the user control if they want to participate in the network without having to recaculate the entire topology.

# options/proxy.type.nix
{ lib, ... }:
lib.types.submodule
  ({ name, ... }:
  let
    # Returns true if the host machine is the gateway or one of the clients
    host_participation = (proxy.gateway == host) || (elem host proxy.clients);
  in
  {
    options = {
      enable = mkOption {
        type = types.bool;
        default = host_participation;
        example = true;
        description = "Whether or not to create this interface for this machine.";
      };

      gateway = mkOption {
        type = types.str;
        example = "serverA";
        description = "The name of the machine (exant in config.networking.wgqt.machines) which will proxy the traffic of the other machines.";
      };

      clients = mkOption {
        type = types.listOf types.str;
        example = [ "serverB" "laptopC" "laptopD" ];
        description = "The names of machines (all exant in config.networking.wgqt.maachines) which will have their traffic proxied via the gateway machine";
      };
    };
  })

Let's set up two proxies in our options so we can refer to something concrete in the rest of this article. We'll create a proxy named finland (because that's where the gateway is physically located) and tell it to use lighthouse as the gateway for three of my personal machines, and another proxy named germany (for secret and mysterious reasons only the very wise can divine) and have the same three machines proxy their data through tavern.

# NixOS Configuration
{
  config.networking.wgqt = {
    enable = true;
    # Already generated all our keys
    # keygen = { ... };

    proxies = {
      finland = {
          gateway = "lighthouse";
          clients = [ "nyaa" "spark" "archive"];  
      };     
      germany = {
          gateway = "tavern";
          clients = [ "nyaa" "spark" "archive"];  
      };     
    };
  };
}

Okay, so we have our gateway machine, our client machines, and our interface names - Let's connect our options to the config side of things.

Interface Config

We'll setup the user's networking config inside nixosModule.nix, and populate config.networking.wireguard.interfaces with the proxies we defined above.

# nixosModule.nix
{ pkgs, lib, config, ... }:
with lib; with builtins;
let
  # Shortcodes
  cfg = config.networking.wgqt;
  cfg_check = cfg.enable;
  keygen_mode = cfg.keygen.enable;
in
{
  imports = [
    # Agenix encrypted secrets
    ./age.nix
    # Options created by this module
    ./options
  ];

  config =
    let
      proxies = (import ./proxies.nix { inherit lib pkgs config; });
      wireguard_interfaces = proxies # // vpns // automeshes;
    in
    mkIf cfg_check {
      # Oh yeah, we moved this out of options.nix into nixosModule.nix where it belongs
      systemd = {
        # Ensure the Wireguard private key has the correct owner and permissions
        tmpfiles.rules = mkIf (!keygen_mode) [
          "z ${cfg._pkf} 0400 root root -"
        ];
      };

      networking = {
        wireguard.interfaces = wireguard_interfaces;
      };
    };
}

Quite the glow up from before, where nixosModule.nix just imported options.nix and age.nix. Of course, this elides the most important part as we import ./proxies.nix as our interface definition.

proxies.nix

This is the file where we'll actually set the config for the Wireguard interfaces. We'll need the room for other things in nixosModule.nix and it will be helpful to keep the logic for proxies separate from future iterfaces such as vpns and automeshes.

# proxies.nix
{ lib, pkgs, config, ... }:
with lib; with builtins;
let
  cfg = config.networking.wgqt;

  # Only setup proxies we participate in, and have enabled
  proxies = filterAttrs
    (_: proxy: proxy.enable)
    cfg.proxies;
in
# For each proxy, map the information into a Nix Wireguard interface
(mapAttrs
  (proxy_name: proxy:
  in
  {
    # The IP addresses of the host machine on this interface
    ips = []; 
    # The port we use for this interface
    listenPort = 0; 
    # The private key file we use to secure all interfaces
    privateKeyFile = ./private_key.key;
    # The other machines we can directly connect to on this interface
    peers = [];
  })
  proxies)

Now we just need fill out these other fields and we're ready to proxy data around the world.

1. Interface name
2. The interface private key
3. IP addresses belonging to the interface
4. Port the interface will listen on
5. A list of other machines on the network

PrivateKeyFile

The easiest thing to fill out is the privateKeyFile path used to secure this interface. In the first post we setup an option: networking.wgqt.private_key_file which contains just that.

We used a file path so that our private key is never in our Nix config, which would expose it in the world readable Nix Store. We also change the permissions and ownership so it's only readable by root to ensure that it's not easy to compromise.

# proxies.nix
{ lib, pkgs, config, ... }:
with lib; with builtins;
let
  cfg = config.networking.wgqt;

  # Only setup proxies we participate in, and have enabled
  proxies = filterAttrs
    (_: proxy: proxy.enable)
    cfg.proxies;
in
# For each proxy, map the information into a Nix Wireguard interface
(mapAttrs
  (proxy_name: proxy:
  in
  {
    # The private key file we use to secure all interfaces
    privateKeyFile = cfg._pkf;
    # ...
  })
  proxies)

If you don't remember _pkf, we added that as an internal option which holds the real private key file path. We did this to allow users to bring their own keys, prevent repeated null checks, and provide a nicer default/error message. The truth was hidden from you at the time, but if you've made it this far you can handle the complexity.

1. Interface name
2. The interface private key
3. IP addresses belonging to the interface
4. Port the interface will listen on
5. A list of other machines on the network

IP Address

If you run ip addr on in your command line, you'll see a list of existing network interfaces and some basic information about each one. You'll notice that each interface assigns you at least one ip address. Here we can see the loopback interface, my ethernet interface, and an existing Wireguard interface. You'll also see a that they each have an IP address assigned (127.0.0.1/8 192.168.1.4/24 100.64.0.3/32).

$ ip -4 addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
2: enp4s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    altname enx047c16b39562
    inet 192.168.1.4/24 brd 192.168.1.255 scope global dynamic noprefixroute enp4s0
       valid_lft 77322sec preferred_lft 77322sec
4: irlqt-net: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1280 qdisc fq_codel state UNKNOWN group default qlen 500
    inet 100.64.0.3/32 scope global irlqt-net
       valid_lft forever preferred_lft forever

In the same way we have to assign an IP address to our Wireguard interface so that it know which addresses are valid on that link. However these addresses must fulfill some additional properties:

• Unique per machine per interface
• Deterministically chosen
• Host machine agnostic

e.g. no matter which machine builds the flake, the gateway of the finland must have the same IP address.

To accomplish this we can pick an IP address to start from, order our networks in a deterministic way, order the machines inside those networks in a deterministic way, and then count upwards, assigning IP addresses as offsets from the start of the range.

1. Pick an IP starting address
2. Order interfaces in a deterministic manner
3. Order machines per interface in a deterministic manner
4. Assign IP addresses in the above order to each machine per interface

IP Address Start

We need a place to start, and there are several IPv4 address ranges that are reserved for personal/private use we can choose from:

• 10.0.0.0/8
• 172.16.0.0/12
• 192.168.0.0/16

172.16.0.0 is ugly, 192.168.0.0 is used by your router (and if not your router then the library/cafe/airport's routers), and 10.0.0.0 is big and beautiful. Now, we don't want to bogart the entire /8 range for ourselves, and you probably don't have 16,777,216 machines you need in a private network (if you do, please give me a call, I would love to write SDN code for you), so I like to cut it in half and use 10.128.0.0/9. Your router (or other VPNs) like to use this range too, so by reserving the lower addresses for other use, we're less likely to experience a conflict in practice.

# options/default.nix
{
  # ...
  options = {
    networking.wgqt = {
      # ...
      # The starting address where interfaces will assign IP addresses
      # Defaults to 10.128.0.0 to keep half of the 10.0.0.0/8 range open for
      # personal use
      ipv4_start = mkOption {
        type = types.str;
        default = "10.128.0.0";
        example = "10.128.0.0";
        description = "Interfaces need to assign IP addresses to the machines that are a part of their network. Each interface will take a block of IPs rounded up to nearest power of 2. It is recommend to use a private IPv4 range for this purpose. 10.128.0.0/9 was chosen because it protects the lower half of the 10.0.0.0/8 range for personal use while still providing 8,388,608 addresses for cute Wireguard networks.";
      };
  };
}

1. Pick an IP starting address
2. Order interfaces in a deterministic manner
3. Order machines per interface in a deterministic manner
4. Assign IP addresses in the above order to each machine per interface

Ordering Interfaces

To ensure each machine assigns itself the correct IP address on each interface, we need to look at all interfaces - even the ones the host machine doesn't participate in. This gives our flake a lot more flexibility, not all of my machines need to participate in every network. For example, tavern and lighthouse are both proxies, but they don't proxy each other. However if we don't include them in our interface ordering, we would get different answers for their IP addresses, which would be different still from addresses calculated by the clients.

The way we will do this is to collate all interfaces, determine the size of each interface, and sort such that the largest interfaces have the lowest addresses, and in the event of a tie, we go by interface name. We know this is unique because attrset keys in Nix must be unique. It also doesn't take the host machine as an input, so it will produce the same ordering on all machines. In addition, this ordering has several advantages: larger networks having lower addresses ensures alignment of subsequent smaller networks, and causes less frequent address changes when machines are added or removed.

Here's the code, in our new file options/interfaces.nix which does the sorting.

# options/interfaces.nix

{ lib, config, ... }:
with lib; with builtins;
let
  # Shortcodes
  cfg = config.networking.wgqt;

  # Our proxies
  proxies = cfg.proxies;

  # Number of machines in a proxy: clients + gateway
  num_machines = proxy: proxy.clients + 1;

  # Ordered Interface Names
  # Sort by size, larger networks get lower addresses
  # This will waste less addresses and ensure alignment
  # If networks are the same size, then ties break with names
  # Name must be unique, as they are attrset keys so this ensures deterministic
  # ordering across builds
  iface_names = sort
    (a: b:
      if (num_machines proxies.${a}) > (num_machines proxies.${b}) then true
      else if (num_machines proxies.${a} == num_machines proxies.${b}) then (a < b)
      else false)
    (attrNames proxies);
in
{
  #...
}

1. Pick an IP starting address
2. Order interfaces in a deterministic manner
3. Create address start offsets per interface
4. Order machines per interface in a deterministic manner
5. Assign IP addresses in the above order to each machine per interface

Create Address Offsets

Now we need to use this order to create offsets from the starting address. We can use foldl' to iterate over them while accumulating the differences based on the size of each network. When calculating the size, we need to add 2 to the number of machines, because the first address of a network is reserved as the network address, and the last address of a network is reserved for the broadcast address. This is not necessarily true for Wireguard networks (or /31 point-to-point networks for that matter) but I have personally encountered issues with some part of the network stack sometimes taking issue with using these addresses so now I just waste two addresses to avoid the pain of remembering how overdue IP is for being replaced.

# options/interfaces.nix
{ lib, config, ... }:
with lib; with builtins;
let
  # ...
  # Number of machines in a proxy: clients + gateway
  num_machines = proxy: proxy.clients + 1;

  # Ordered Interface Names
  # iface_names = [ ... ];

  # Fold over the interfaces to calculate the IP ranges and ports for each
  init_state = { next_start = cfg._ipv4_start; results = { }; };
  step = state: iface_name:
    let
      iface = merged.${iface_name};

      # Calculate new IP ranges
      # Size is +2 because first and last addresses of a range are reserved
      size = ip.nearestPow2 ((num_machines iface) + 2);

      end = start + size;
      start = state.next_start;

      result = {
        inherit (iface) enable;
        inherit start;
      };
    in
    {
      # Update state
      results = state.results // { ${iface_name} = result; };
      next_start = end;
    };
  # Calculate all interfaces, then filter so only the ones we're participating
  # in are left intact
  all_interfaces =
    (builtins.foldl' step init_state iface_names).results;
  in # {... }

The compilerpilled among you might notice we're using functions from an undeclared ip attrset. This is the IP handling library located in lib/ip.nix which contains a handful of functions we need to deal with IP addresses. I'm not going to cover it right now, as we're already pretty deep in the conceptual stack. Maybe at the end or in another post.

This code is long, but all it does is iterate over the interfaces in the order described above, and assigns a starting IP address which begins at wgqt.ipv4_start and increments by the number of machines in the network (+2 for reserved addresses, and rounded up to the nearest power of 2). It also records if this interface is enabled for this host.

We round up to the nearest power of 2 to ensure alignment of future start addresses, because smaller powers of 2 are always 'aligned' to larger ones.

$$ \forall L,S \in {2^k \mid k \in \mathbb{N}_0} ; L \ge S \implies L \bmod S = 0\\ \text{i.e. if } L = 2^a, S = 2^b \text{ with } a \ge b \ge 0 \text{ then } S \text{ is always a factor of } L ; (2^{a-b} \cdot 2^b) = 2^a $$

In addition, it reduces the frequency and impact of recalculating addresses, with larger networks needing reallocation less frequently, which is good since they also affect more addresses when they do.

Now that we have these offsets, let's store them somewhere we can use.

1. Pick an IP starting address
2. Order interfaces in a deterministic manner
3. Create address start offsets per interface
4. Store interface address offsets in _interfaces option
5. Order machines per interface in a deterministic manner
6. Assign IP addresses in the above order to each machine per interface

Store Interface Address Offsets

Let's create a new option, wgqt._interfaces which will store these offsets; this allows us to calculate them once and reference them later.

# NixOS Configuration
{
  networking.wgqt = {
    _interfaces = {
      finland = {
        enable = true;
        start = 176160768;
      };
      germany = {
        enable = true;
        # Incremented by 8 (###68 -> ###76)
        # `finland` proxy has 4 machines + 2 => 6
        # 6 rounded to the nearest power of 2 => 8
        start = 176160776;
      };
    };
  };
}

# options/_interface.type.nix
{ lib, ... }:
with lib; with builtins;
let
  _interface_type = types.submodule ({ ... }:
    {
      options = {
        enable = mkOption {
          type = types.bool;
          example = true;
          description = "If this interface is enabled or not";
        };
        start = mkOption {
          type = types.int;
          example = 176160768;
          description = "The starting address for this network as a decimal integer.";
        };
      };
    });
in
_interface_type

# options/interfaces.nix
{ lib, config, ... }:
with lib; with builtins;
let
  # Interface type
  _interface_type = (import ./_interface.type.nix { inherit lib config; });
  # iface_names = [ ... ];
  # all_interfaces = { ... };
in
{
  options.networking.wgqt = {
    _interfaces = mkOption {
      type = types.attrsOf _interface_type;
      default = all_interfaces;
      description = "A list of Wireguard interfaces created by the wgqt module, even the interfaces the host doesn't participate in - to ensure correct address offsets.";
    };
}

Now we can move onto ordering the machines inside each network, so we can finally, finally assign an IP address to the host on the network. Before we do that, let's create a place to store that address, so we can access it all the way back in proxies.nix to set up our actual interface.

1. Pick an IP starting address
2. Order interfaces in a deterministic manner
3. Create address start offsets per interface
4. Store interface address offsets in _interfaces option
5. Create an interfaces option
6. Order machines per interface in a deterministic manner
7. Assign IP addresses in the above order to each machine per interface

Interface Option

Let's create a new wgqt.interfaces option which contains the final address values.

Why not just use it directly in proxies.nix? For two reasons, we can avoid recalculating this address in other types of networks we'll use later (vpns, automesh), and because this option is useful for the end user. The user can now reference wgqt.interfaces.<name>.address in other parts of their config to use the correct address without specifying it manually.

# NixOS Configuration
{
  networking.wgqt = {
    # ...
    # We're going to add this
    interfaces = {
      finland = {
        # Remember: the first address of the network space is reserved
        address = "10.128.0.1";
      };
      germany = {
        address = "10.128.0.9";
      };
    };
    # _interfaces = { .. };
  };
}

interfaces.nix

Let's extend options/interfaces.nix to reflect this. We'll filter all the interfaces to only include the ones we participate in so it's cleaner for the user - remember that _interfaces needs to consider all the interfaces to calculate the offsets so they're the same for every machine - but we can eilde networks we're not using once that is done.

We also set the default of interface to just the names of the interfaces, with an empty attrset. We will set the defaults of individual fields inside the type definition. This provides a nice encapsulation so we can tighten our function inputs to only what is necessary.

# options/interfaces.nix
{ lib, config, ... }:
with lib; with builtins;
let
  # ...
  # iface_names = [ ... ];

  # Calculate all interfaces, then filter so only the ones we're participating
  # in are left intact
  all_interfaces =
    (builtins.foldl' step init_state iface_names).results;

  # Create blank attrsets for each interface this host participates in by
  # default. The attrset will be filled in by the defaults in interface.type.nix 
  my_interfaces = mapAttrs
    (_: value: { })
    (filterAttrs (_: iface: iface.enable) all_interfaces);

  # Interface types
  interface_type = (import ./interface.type.nix { inherit lib config; });
  _interface_type = (import ./_interface.type.nix { inherit lib config; });
in
{
  options.networking.wgqt = {
    interfaces = mkOption {
      type = types.attrsOf interface_type;
      default = my_interfaces;
      description = "A list of Wireguard interfaces created by the wgqt module that the host participates in.";
    };
    _interfaces = mkOption {
      type = types.attrsOf _interface_type;
      default = all_interfaces;
      description = "A list of Wireguard interfaces created by the wgqt module, even the interfaces the host doesn't participate in - to ensure correct address and port offsets.";
    };
}

1. Pick an IP starting address
2. Order interfaces in a deterministic manner
3. Create address start offsets per interface
4. Store interface address offsets in _interfaces option
5. Create an interfaces option
6. Order machines per interface in a deterministic manner
7. Assign IP addresses in the above order to each machine per interface

interface.type.nix

Now let's define interface.type.nix so we can finally calculate and store unique, deterministic addresses for the host machine on each network. For now we only need to define a single option in this submodule: address - a string representing the IP address of the host on the interface. We calculate this by using the IPv4 address start we calculated in _interfaces.${the_current_interface} and chosing the first address if it's the gateway, and then incrementing in lexicographical order through the clients otherwise.

# options/interface.type.nix
{ lib, config, ... }:
with lib; with builtins;
let
  # Shortcodes
  cfg = config.networking.wgqt;

  # IP Utility Library
  ip = (import ../lib/ip.nix { inherit lib; });

  # Our new type
  interface_type = types.submodule ({ name, ... }:
    let
      host = cfg.machine;
      # Contains the user defined network attrset
      iface = cfg.proxies.${name} // cfg._interfaces.${name};

      # Sort the clients in lexicographical order
      sorted_clients = sort lessThan (iface.clients);

      # Calculate the address of the host machine on this interface 
      # intToIp converts a decimanl integer into an IP address string
      address = ip.intToIp
        (if (cfg.machine == iface.gateway)
        # We give the gateway machine the lowest address
        # We add 1 becaue the first address is reserved as the network address
        then (iface.start + 1)
        # Then we increment in lexicographical order
        # Skip the gateway and network addresses (+2)
        else (lists.firstIndexOf host 0 sorted_clients) + iface.start + 2);
    in
    {
      options = {
        address = mkOption {
          type = types.str;
          default = address;
          example = "10.128.0.1";
          description = "This address for this machine on this interface.";
        };
      };
    });
in
interface_type

# nix repl
# :p is for 'print'
:p nixosConfiugarions.nyaa.config.networking.wgqt._interfaces
{
  finland = {
    enable = true;
      enable = true;
      # 10.128.0.0
      start = 176160768;
    };
    germany = {
      enable = true;
      # 10.128.0.8
      start = 176160776;
    };
  };
}
# Which we transform into nice IP address strings
# 'nyaa' is the second in the ordered list of client machines
# [ "archive" "nyaa" "spark"] -> index of 1
:p nixosConfiugarions.nyaa.config.networking.wgqt.interfaces
{
  finland = {
    # Start (10.128.0.0) + 1 (index) + 2 -> 10.128.0.3
    address = "10.128.0.3"
  };
  germany = {
    # Start (10.128.0.8) + 1 (index) + 2 -> 10.128.0.3
    address = "10.128.0.11"
  };
}

1. Pick an IP starting address
2. Order interfaces in a deterministic manner
3. Create address start offsets per interface
4. Store interface address offsets in _interfaces option
5. Create an interfaces option
6. Order machines per interface in a deterministic manner
7. Assign IP addresses in the above order to each machine per interface

Wireguard Interface IPs

Unwind the stack, remember proxies.nix? That's where we're actually configuring Wireguard interfaces. We're finally ready to return there, IP address in hand config.

First let's add our merge our wgqt.interfaces.${proxy} attrset with our wgqt.proxies.${proxy} attrset to collate all our interface information.

Now we can just use our host address directly in the config.

# proxies.nix
{ lib, pkgs, config, ... }:
with lib; with builtins;
let
  cfg = config.networking.wgqt;

  # Only setup proxies we participate in, and have enabled
  # Merge with the calculated values in wgqt.interfaces
  proxies = mapAttrs
    # Merge with its counterpart in wgqt.interfaces
    (name: value: value // cfg.interfaces.${name})
    # Only setup proxies we participate in, and have enabled
    (filterAttrs
      (_: proxy: proxy.enable)
      cfg.proxies);
in
# For each proxy, map the information into a Nix Wireguard interface
(mapAttrs
  (proxy_name: proxy:
  in
  {
    # The private key file we use to secure all interfaces
    privateKeyFile = cfg._pfk;
    # The IP addresses of the host machine on this interface
    # Which is now part of the attrset because we merged with wgqt.interfaces
    ips = [ proxy.address ]; 
    # ...
  })
  proxies)

Things we need to create a Wireguard interface in Nix:

1. Interface name
2. The interface private key
3. IP addresses belonging to the interface
4. Port the interface will listen on
5. A list of other machines on the network

The good news is that with the structure we've built to calculate addresses, adding ports will be simple.

Listening Port

Wireguard will listen on a port for incoming traffic; if a port is not specified it will be picked at random. When a peer connects to another machine on the same network, it will inform it of which port it is listening on so it can receive traffic back. However, if all the machines on a network pick a port at random then no one will be able to start this process, because they won't know where to send the first message. Fortunately, we can use Nix to agree on which port to use ahead of time, but we also need to ensure all interfaces agree to use a different port. This is the same thing as picking an IP address, where they must be unique and deterministic, and agnostic with respect to the machine their configuring, and we're going to use the same strategy to solve it:

1. Pick a starting port
2. Order interfaces in a deterministic manner
3. Assign port numbers in the above order to each interface

UDP Port Start

We can quickly create an option with a sensible default just like we did for wgqt.ipv4_start. In this case we'll start at port 16842 and increment by one for each interface.

# options/default.nix
{
  # ...
  options = {
    networking.wgqt = {
      # ...
      # ipv4_start = mkOption { ... };

      # UDP Port number interfaces will use if they are publicly reachable
      # Interfaces begin at this port and increment per interface
      udp_start = mkOption {
        type = types.port;
        default = 16842;
        example = 16842;
        description = "Interfaces need a port to use if they are publicly reachable. Interfaces will begin with this port and then increment per interface.";
      };
  };
}

1. Pick a starting port
2. Order interfaces in a deterministic manner
3. Assign port numbers in the above order to each interface

Increment _interface ID

Now we can update our offset generator to include an id attribute for each interface, starting at 0 and incrementing by one, and use this as the offset to provide a port for the interface.

# options/interfaces.nix
{ lib, config, ... }:
with lib; with builtins;
let
  # ...
  # Number of machines in a proxy: clients + gateway
  num_machines = proxy: proxy.clients + 1;

  # Ordered Interface Names
  # iface_names = [ ... ];

  # Fold over the interfaces to calculate the IP ranges and ID for each
  # interface.
  init_state = { next_id = 0; next_start = cfg._ipv4_start; results = { }; };
  step = state: iface_name:
    let
      iface = merged.${iface_name};

      # Calculate new IP ranges
      # Size is +2 because first and last addresses of a range are reserved
      size = ip.nearestPow2 ((num_machines iface) + 2);

      # Add in an incrementing ID per interface
      id = next_id;
      end = start + size;
      start = state.next_start;

      result = {
        inherit (iface) enable;
        inherit id start;
      };
    in
    {
      results = state.results // { ${iface_name} = result; };
      # Increment the ID
      next_id = id + 1;
      next_start = end;
    };
    # ...
  in { ... }

Now we can update _interface.type.nix to include and id option.

# options/_interface.type.nix
{ lib, ... }:
with lib; with builtins;
let
  _interface_type = types.submodule ({ ... }:
    {
      options = {
        # enable = mkOption { ... };
        id = mkOption {
          type = types.int;
          example = 0;
          description = "Unique, incrementing ID per interface.";
        };
        # start = mkOption { { ... };
      };
    });
in
_interface_type

1. Pick a starting port
2. Order interfaces in a deterministic manner
3. Assign port numbers in the above order to each interface

Configure Port

We can finally use it in our actual Wireguard configuration.

# proxies.nix
{ lib, pkgs, config, ... }:
with lib; with builtins;
let
  cfg = config.networking.wgqt;
  # proxies = { ... };
in
# For each proxy, map the information into a Nix Wireguard interface
(mapAttrs
  (proxy_name: proxy:
  in
  {
    # privateKeyFile = cfg._pfk;
    # ips = [ proxy.address ]; 

    # The port to use for traffic
    listeningPort = proxy.port;
  })
  proxies)

To Be Continued...

The only* thing left is the peers attribute of the Wireguard interface, but peers will require a blog post in and of itself. I also want to spend a post on the lib/ip.nix so we don't get too far behind on the foundational functions before we get to the interesting ones.

{
  options = {
    # ...
    # postSetup = mkOption {...};
    # postShutdown = mkOption {...};
    # allowedIPsAsRoutes = mkOption { ... };
    # interfaceNamespace = mkOption { ... };
    # dynamicEndpointRefreshSeconds = mkOption { ... };
  };
}