Building cross architecture nixos images

The motivation

So you have a raspberry pi and want to run nixos, but no mini HDMI to HDMI converter? Or perhaps you just want to configure your raspberry pi remotely with nix. There's lots of resources, maybe you should just google it. Oh.. Everyone uses flakes, and there's a million different ways to do it, and some seem to not be supported anymore :/ Here I aim to set up a raspberry pi on nixos with ssh enabled an my key on it, ready for remote configuration in a way that seems to be up to date (at least as of date of publication).

Some terminology

nixos-rebuild has options with names that could be confusing, but to be consistent I'll also reference them in the text, so I'm going to define them here as I understand them:

"local host" The host that you're running your commands from. The one who's screen you're looking at. Likely an x86_64 machine. This isn't a nixos-rebuild option, but is still relevant to our guide here.
"build host" The host that builds the nix deviations. In this guide this will usually be the same as the local host. (used via --build-host in nixos-rebuild)
"target host" The host that your nixos builds are being built for. In this guide this will usually be the raspberry pi. (used via --target-host in nixos-rebuild)

How I did it

First off, if you're on x86_64 and trying to build for a pi on aarch64 you'll need to be able to cross compile. To do that add this to your personal configuration.nix on your local host.

boot.binfmt.emulatedSystems = [
  "aarch64-linux"
];

then run a switch to actually enable it

sudo nixos-rebuild switch

Now we make a file called sd-card.nix (the name doesn't actually matter) on our local host which will serve to create an image that we can flash to our SD card which has nixos, ssh, and our key ready to go. sd-card.nix should look like this

sd-card.nix

# This file creates the simplest image possible while
# adding a user with our ssh key so we can boot
# and configure the machine remotely
# You can build the image with this command:
# env NIX_PATH="nixos-config=<path/to/this/file>:nixpkgs=channel:nixos-26.05" nixos-rebuild build-image --image-variant sd-card
{ config, pkgs, ... }:

{
  # Recommended by warning message
  # What this actually does... :shrug:
  boot.zfs.forceImportRoot = false;

  # Not having this results in building an image
  # for the platform of machine which is making the sd-image
  # probably x86_64.
  # We want a raspberry pi image, so we need this
  nixpkgs = {
    hostPlatform = "aarch64-linux";
  };

  # == This is the meat of what we're trying to do here ==

  # We need this to run nixos-rebuild with the local machine as the build host
  # With this you can run
  # nixos-rebuild switch -I nixos-config=configuration.nix --target-host jeff@raspberry0
  # If you try this without being a trusted-user you'll get this error:
  # error: cannot add path '/nix/store/09w0mk3fqcwblx2px3b8qzh4r6r6cna0-users-groups.json' because it lacks a signature by a trusted key
  # Without this you can run this instead:
  # nixos-rebuild switch -I nixos-config=configuration.nix --target-host jeff@raspberry0 --build-host jeff@raspberry0
  # https://mynixos.com/nixpkgs/option/nix.settings.trusted-users
  nix.settings.trusted-users = [
    "jeff"
  ];

  # Create our user
  users.users.jeff = {
    openssh.authorizedKeys.keys = [
      # Replace this with your key
      "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAICEKeehduYbblNR/+ylIh83qC0JUbawjJU6hU5kF8EGl jeff@nixos"
    ];
    isNormalUser = true;
    # Add ourselves to wheel because we give wheel
    # password-less sudo access
    extraGroups = [ "wheel" ];
  };

  # Allow our user access without a password
  security.sudo.wheelNeedsPassword = false;

  # Enable ssh so we can do the real configuration remotely
  services.openssh.enable = true;

  # This value determines the NixOS release from which the default
  # settings for stateful data, like file locations and database versions
  # on your system were taken. It‘s perfectly fine and recommended to leave
  # this value at the release version of the first install of this system.
  # Before changing this value read the documentation for this option
  # (e.g. man configuration.nix or on https://nixos.org/nixos/options.html).
  system.stateVersion = "26.05"; # Did you read the comment?
}

Now you can build the image like this:

env NIX_PATH="nixos-config=</path/to/sd-card.nix>:nixpkgs=channel:nixos-26.05" nixos-rebuild build-image --image-variant sd-card

The last line of the output will be something like this:

Done. The disk image can be found in /nix/store/1d761gdrvc48n8rd06iy4inm0m4br4j7-nixos-image-sd-card-26.05.1550.bd0ff2d3eac2-aarch64-linux.img.zst/sd-image/nixos-image-sd-card-26.05.1550.bd0ff2d3eac2-aarch64-linux.img.zst

Make sure it's for the correct architecture (aarch64 in my case) by checking for it in the file name. (nixos-image-sd-card-26.05.1550.bd0ff2d3eac2-aarch64-linux.img.zst has aarch64 in it.)

A symlink named result will have been created which points to the directory containing sd-image, so you can get to the file at ./result/sd-image/<file name>. Note that it's compressed though, so we'll have to decompress it before putting it on the sd card. You can do that like this

# Install zstd to decompress
nix-shell -p zstd
# Do the decompression
unzstd ./result/sd-image/nixos-image-sd-card-26.05.1550.bd0ff2d3eac2-aarch64-linux.img.zst --output-dir-flat .
# Copy the decompressed image onto the SD card.
# Make sure that of=/dev/sdX is replaced with the correct device for your SD card!
sudo dd if=nixos-image-sd-card-26.05.1550.bd0ff2d3eac2-aarch64-linux.img of=/dev/sdX bs=4096 conv=fsync status=progress

Now pop that SD card into your raspberry-pi. You should be able to ssh into it with your key that was already configured. We need to get the hardware-configuration.nix from the pi. To do that run these commands:

# Failing to do this mount will result in a config
# which doesn't have the firmware mounted which...
# probably has consequences, but I haven't run into them yet :shrug:
# Either way, I'm pretty sure it's supposed to be mounted
# because it's referenced a lot here: https://www.raspberrypi.com/documentation/computers/configuration.html
sudo mkdir -p /boot/firmware
sudo mount /boot/firmware
sudo nixos-generate-config

Then copy /etc/nixos/hardware-configuration.nix from the pi to the host. Something like this:

scp <user>@<host>:/etc/nixos/hardware-configuration.nix .

for me the hardware-configuration.nix looked like this:

hardware-configuration.nix

# Do not modify this file!  It was generated by ‘nixos-generate-config’
# and may be overwritten by future invocations.  Please make changes
# to /etc/nixos/configuration.nix instead.
{ config, lib, pkgs, modulesPath, ... }:

{
  imports =
    [ (modulesPath + "/installer/scan/not-detected.nix")
    ];

  boot.initrd.availableKernelModules = [ "xhci_pci" ];
  boot.initrd.kernelModules = [ ];
  boot.kernelModules = [ ];
  boot.extraModulePackages = [ ];

  fileSystems."/" =
    { device = "/dev/disk/by-uuid/44444444-4444-4444-8888-888888888888";
      fsType = "ext4";
    };

  fileSystems."/boot/firmware" =
    { device = "/dev/disk/by-uuid/2178-694E";
      fsType = "vfat";
      options = [ "fmask=0022" "dmask=0022" ];
    };

  swapDevices = [ ];

  nixpkgs.hostPlatform = lib.mkDefault "aarch64-linux";
}

This file was consistent across many reinstalls (I thought the uuids might be random, but no), and also the same as I've seen out in the wild (sans the /boot/firmware mount). That means you can probably copy my hardware-configuration.nix if you're running on a pi4 (or maybe even others? :shrug:)

I couldn't figure out a way to determine what should be in hardware-configuration.nix without booting the machine first, unfortunately. You can't just run the nixos-rebuild switch ... without defining that stuff, which makes sense, so this will have to do.

Now you can make your real configuration file for the target machine. The most basic configuration may look like this:

configuration.nix

# Now that we have nixos running on the pi
# we can configure it with a mostly normal configuration.nix
# The main gotchas are importing <nixos-hardware/raspberry-pi/4>
# and setting boot.kernelPackages, but otherwise, you're basically
# free to do normal stuff here.
# Push this config to the target machine by running this on your machine:
# nixos-rebuild switch -I nixos-config=<path/to/this/file> --target-host <user>@<host> --sudo
{ config, pkgs, ... }:

{

  imports = [
    <nixos-hardware/raspberry-pi/4>
    ./hardware-configuration.nix
  ];

  # Notice we don't actually need these anymore.
  # They were just for the sd-image creation, and
  # seem to have no use anymore as far as I can tell
  #
  #boot.zfs.forceImportRoot = false;
  #
  #nixpkgs = {
  #  hostPlatform = "aarch64-linux";
  #};

  boot = {
    # Not setting this value results in compiling the linux kernel
    # which takes many hours.
    kernelPackages = pkgs.linuxPackages;
  };

  # We need this to run nixos-rebuild with the local machine as the build host
  # With this you can run
  # nixos-rebuild switch -I nixos-config=configuration.nix --target-host jeff@raspberry0
  # If you try this without being a trusted-user you'll get this error:
  # error: cannot add path '/nix/store/09w0mk3fqcwblx2px3b8qzh4r6r6cna0-users-groups.json' because it lacks a signature by a trusted key
  # Without this you can run this instead:
  # nixos-rebuild switch -I nixos-config=configuration.nix --target-host jeff@raspberry0 --build-host jeff@raspberry0
  # https://mynixos.com/nixpkgs/option/nix.settings.trusted-users
  nix.settings.trusted-users = [
    "jeff"
  ];

  security.sudo.wheelNeedsPassword = false;
  users.users.jeff = {
    openssh.authorizedKeys.keys = [
      "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAICEKeehduYbblNR/+ylIh83qC0JUbawjJU6hU5kF8EGl jeff@nixos"
    ];

    isNormalUser = true;
    extraGroups = [ "wheel" ];
  };

  # Install some packages so we have an easy
  # way to verify we're on the new generation
  environment.systemPackages = with pkgs; [
    neovim
  ];

  services.openssh.enable = true;

  # This value determines the NixOS release from which the default
  # settings for stateful data, like file locations and database versions
  # on your system were taken. It‘s perfectly fine and recommended to leave
  # this value at the release version of the first install of this system.
  # Before changing this value read the documentation for this option
  # (e.g. man configuration.nix or on https://nixos.org/nixos/options.html).
  system.stateVersion = "26.05"; # Did you read the comment?
}

Notice that it's essentially the same as sd-card.nix, but we removed boot.zfs.forceImportRoot (unclear why the warning doesn't appear anymore) and nixpkgs.hostPlatform = "aarch64-linux" (because it's included in hardware-configuration.nix. We also add a neovim installation so we have an easy way to verify that we're on the new generation after deploying it.

Before we deploy it though we need to set up the channels on our local machine. You may think that the target machine's channels would be used, or maybe even the build machine's but no, it's the local machine as far as I can tell.

# You probably already have these first two
# And you may want to use a channel other than unstable
# that's up to you
sudo nix-channel --add https://nixos.org/channels/nixpkgs-unstable nixpkgs
sudo nix-channel --add https://nixos.org/channels/nixos-unstable nixos

# This one was new to me, so make sure you have it!
sudo nix-channel --add https://github.com/NixOS/nixos-hardware/archive/master.tar.gz nixos-hardware
sudo nix-channel --update

Now to actually deploy it! This is how you do that from your local machine:

# The --sudo flag has nixos run the command as root on the target host
# NOTE: If you don't have passwordless sudo available on the target host
# then you'll probably want to use the --ask-sudo-password flag.
nixos-rebuild switch -I nixos-config=configuration.nix --target-host <user>@<target host> --sudo

Make sure that neovim was installed by running nvim on the target machine. If that works then you're ready to configure your pi the way you want!

Note: I've seen reports the generation being reset after rebooting, so I recommend checking for that by rebooting and checking for nvim again.

Errors and how to fix them:

file 'nixos-hardware/raspberry-pi/4' was note found...

error: file 'nixos-hardware/raspberry-pi/4' was not found in the Nix search path (add it using $NIX_PATH or -I)

You need to have the nixos-hardware channel added to the local machine, not the raspberry pi. You can do that with:

sudo nix-channel --add https://github.com/NixOS/nixos-hardware/archive/master.tar.gz nixos-hardware
sudo nix-channel --update

And yes, adding it to root's nix-channels (because sudo) works despite the fact that we run nixos-rebuild as a non-root user i.e.:

nixos-rebuild switch -I nixos-config=configuration.nix --target-host jeff@raspberry0 --sudo

Maybe because we're running it with --sudo? But I thought that applied to the target machine :shrug:

Cannot build [...] Reason: platform mismatch

error: Cannot build '/nix/store/gp64xxij16i15m9dcbmpna6pygb8nw8q-system-path.drv'.
       Reason: platform mismatch
       Required system: 'aarch64-linux'
       Current system: 'x86_64-linux'
error: Cannot build '/nix/store/cjx88xnxjbqc95cf8554i3wbc7b1c6q8-dbus-1.drv'.
       Reason: 1 dependency failed.
       Output paths:
         /nix/store/ihbwly656sqdn49aklba5wlg5cja9lq2-dbus-1
error: Cannot build '/nix/store/h7x98bsyrrx1f5id7wqs7jg5am4i01yx-etc.drv'.
       Reason: 1 dependency failed.
       Output paths:
         /nix/store/mpml2v1alvnfw7a62qqg7vr9bl49v973-etc
error: Cannot build '/nix/store/0638xx5m803k2gv4zgkagzhqmzzh38iy-nixos-system-nixos-26.11pre1011622.a799d3e3886d.drv'.
       Reason: 1 dependency failed.
       Output paths:
         /nix/store/ib5b1av866p2ypr2g9ggx4w3idhrpshd-nixos-system-nixos-26.11pre1011622.a799d3e3886d
error: Build failed due to failed dependency

The key part here is Reason: platform mismatch. If you're on x86_64 and trying to be the build-host for a raspberry pi, then you'll need cross compilation support. To do that add these lines to your configuration.nix on your local machine (probably /etc/nixos/configuration.nix):

boot.binfmt.emulatedSystems = [
  "aarch64-linux"
];

and of course run a switch to actually enable it

sudo nixos-rebuild switch

nixos evaluation warning: `system.build.image` is defined, while `system.build.images` is used.

The warning in full:

evaluation warning: `system.build.image` is defined, while `system.build.images` is used.
                    The former will conflict with variants in the latter.
                    Maybe you are importing an image-building module into the toplevel?
                    Add it to `image.modules` instead, or adjust priorities manually.

This is just a warning and didn't seem to actually stop it from working, however the solution is simple. If you see this then you likely have this defined in your sd-card.nix

imports = [
    <nixpkgs/nixos/modules/installer/sd-card/sd-image-aarch64.nix>
  ];

When using nixos-rebuild build-image [...] you don't need that. According to this post you would use that import when running nix build [...] to build the image. This is one of the reasons why this process is so confusing, the config you need depends on the command you intend to use to build the image. :(

Some resources I used as reference that might be useful to you

https://michael.stapelberg.ch/posts/2025-06-01-nixos-installation-declarative/

https://nix.dev/tutorials/cross-compilation.html#specifying-the-host-platform

https://jcd.pub/2025/01/30/nixos-on-raspi-in-2025/

https://discourse.nixos.org/t/nixos-rebuild-build-image-error-option-system-build-images-no-value-defined/63671/7

https://nixos.wiki/wiki/Creating_a_NixOS_live_CD

Installing voidlinux on a RaspberryPi

To install voidlinux on a Pi we'll have to do a chroot install. For official documentation on installing from chroot for void see here.

We need to install via chroot because the live images are made specifically for 2GB SD cards.

"These images are prepared for 2GB SD cards. Alternatively, use the ROOTFS tarballs if you want to customize the partitions and filesystems."

The installation can split out into 4 rough steps

Partition the disk (SD card in my case) you want to install void on
Create the filesystems on the disk
Copy in the rootfs
Configure the rootfs to your liking

Prerequisites

Because we're going to be creating an aarch64 system you'll need some tool that will allow you run aarch64 binaries from a x86 system. To accomplish this we'll need the binfmt-support and qemu-user-static packages. To install them you can run

sudo xbps-install binfmt-support qemu-user-static

We'll also need to enable the binfmt-support service. To do this, run

sudo ln -s /etc/sv/binfmt-support /var/service/

Now you're one step away from being able to run aarch64 binaries in the chroot on your x86 system, but we'll get to that later.

Partition the disk you want to install void on

This is tricky because it can depend a little based on what you want to do. In my case I didn't allocate any swap space and kept the home directory on the root partition which keeps things pretty simple.

In this case we're going to need two partitions. One 64MiB partition that is marked with the bootable flag and has the vfat type (0b in fdisk). And the other that takes up the rest of the SD card with type linux (83 in fdisk).

To create these partitions with fdisk run sudo fdisk /dev/sda where /dev/sda is the path to your disk. The path to your disk can be found running lsblk before and after plugging in the disk and seeing what shows up. Once fdisk drops you into the repl you can delete the existing partitions with the d command.

Create the boot partition

Make a new partition with the n command, make it a primary partition with p, make it partition 1, and leave the first sector blank, which will keep it as the default. For the last sector put +64M which will give us a 64MiB partition (if you're asked to remove the signature it doesn't matter because we'll be overwriting that anyway). Use the a command to mark partition 1 bootable and lastly use the t command to make partition 1 type 0b, which is vfat.

Create the root partition

Now the root partition, use n to make a new partition, then leave everything else default. This will consume the rest of the disk for this partition. Same as before, if it asks you to remove the signature it doesn't matter because we'll be overwriting now. To set the type label use the t command and set it to type 83 which is the linux type.

That's all we need to do to setup the partitions. Make sure to save your changes with the w command!

The disk should be correctly partitioned now!

Create the filesystems on the disk

This part is easy. Assuming the device is located at /dev/sda, partition 1 is the boot partition, and partition 2 is the root partition, just run these two commands.

mkfs.fat /dev/sda1 # Create boot vfat filesystem
mkfs.ext4 -O '^has_journal' /dev/sda2 # Create ext4 filesystem on the root partition (with journaling)

Copy in the rootfs

For this step we'll need both partitions we set up earlier to be mounted. To mount the partitions run

MOUNT_PATH='/mnt/sdcard' # Replace with any path to an empty directory. By convention it would be in /mnt

mount /dev/sda2 $MOUNT_PATH # Mount the root partition to the mount point
mkdir -p $MOUNT_PATH/boot # Create a directory named "boot" in the root partition
mount /dev/sda1 $MOUNT_PATH/boot # Mount the boot partition to that boot directory

Now we just need to extract the rootfs into our mount point.

MOUNT_PATH='/mnt/sdcard' # Replace with any path to an empty directory. By convention it would be in /mnt (same mount path as above)
ROOTFS_TARBALL='/home/me/Downloads/void-rpi3-PLATFORMFS-20210930.tar.xz' # Replace with the path to the tarball you download from https://voidlinux.org/download/

# x - Tells tar to extract
# f - Tells tar to operate on the file path given after the f switch
# J - Tells tar to extract using xz, which is how the rootfs happens to be compressed
# p - Tells tar to preserve permissions from the extracted directory
# -C - Tells tar where to extract the contents to
tar xfJp $ROOTFS_TARBALL -C $MOUNT_PATH

That's it for this step! You might notice that we didn't explicitly copy anything into the $MOUNT_PATH/boot directory. The rootfs provided by void contains a /boot directory which will get placed into the $MOUNT_PATH/boot directory when we extract the tarball.

Configure the rootfs to your liking

This step is technically optional. If we just wanted to get a system up and running, we could plug the SD card in right now and it would boot up. We wouldn't have any packages (including base-system, which gives us dhcpcd, wpa_supplicant and other important packages), but it would boot. Additionally, the RaspberryPi's (at least mine) doesn't have a hardware clock so without an ntp package we won't be able to validate certs (because the time will be off) which prevents us from installing packages.

Some of the things we want to configure are most easily through a chroot. The problem is that the binaries in the rootfs we copied over are aarch64 binaries.

Running `aarch64` binaries in the `chroot`

Because your x86 system cannot run aarch64 binaries we need to emulate the aarch64 architecture inside the chroot. To accomplish this we copy an x86 binary that can do that emulation for us into the chroot, and then pass all aarch64 binaries through it when we go to run them.

If you've installed the qemu-user-static package you should have a set of qemu-*-static binaries in /bin/. For a RaspberryPi 3, we want qemu-aarch64-static. Copy that into the chroot.

cp /bin/qemu-aarch64-static <your-chroot-path>

Now you're ready to run the aarch64 binaries in your chroot.

Recommended configuration

To create a usable system there's a few things we need to setup that are somewhere between recommended and mandatory; the base-system package, ssh access, ntp, dhcpcd and a non-root user.

Because running commands in the chroot is slightly slower due to the aarch64 emulation we'll try to setup as much of the rootfs as possible without actually chrooting.

First we should update all the packages that were provided in the rootfs.

MOUNT_PATH='/mnt/sdcard' # Replace with any path to an empty directory. By convention it would be in /mnt (same mount path as above)

# Run a sync and update with the main machine's xbps pointing at our rootfs
env XBPS_ARCH=aarch64 xbps-install -Su -r $MOUNT_PATH

The base-system package

Just install the base-system package from your machine with the -r flag pointing at the $MOUNT_PATH.

MOUNT_PATH='/mnt/sdcard' # Replace with any path to an empty directory. By convention it would be in /mnt (same mount path as above)

# Install base-system
env XBPS_ARCH=aarch64 xbps-install -r $MOUNT_PATH base-system

ssh access

We just need to activate the sshd service in the rootfs.

MOUNT_PATH='/mnt/sdcard' # Replace with any path to an empty directory. By convention it would be in /mnt (same mount path as above)

ln -s /etc/sv/sshd $MOUNT_PATH/etc/runit/runsvdir/default/

There's two thing here that look odd; 1. we're symlinking to our main machines /etc/sv/sshd directory and 2. we're placing the symlink in /etc/runit/runsvdir/default/ instead of /var/service like is typical for activating void services.

When we're chroot'ed in, or when the system is running on the Pi /etc/sv/sshd will point to the Pi's sshd service.
/var/service doesn't exists until the system is running and it when the system is up /var/service will be a series of symlinks pointing to /etc/runit/runsvdir/default/ so we can just link the sshd service directly to the /etc/runit/runsvdir/default/.

For security reasons I recommend disabling password authentication.

MOUNT_PATH='/mnt/sdcard' # Replace with any path to an empty directory. By convention it would be in /mnt (same mount path as above)

sed -ie 's/#PasswordAuthentication yes/PasswordAuthentication no/g' $MOUNT_PATH/etc/ssh/sshd_config
sed -ie 's/#KbdInteractiveAuthentication yes/KbdInteractiveAuthentication no/g' $MOUNT_PATH/etc/ssh/sshd_config

npd

We need an ntp package because the RaspberryPi doesn't have a hardware clock so when we boot it up the time will be January 1, 1970 which causes cert failures resulting in certificate validation failures that prevent us from installing packages and more.

MOUNT_PATH='/mnt/sdcard' # Replace with any path to an empty directory. By convention it would be in /mnt (same mount path as above)

env XBPS_ARCH=aarch64 xbps-install -r $MOUNT_PATH openntpd
ln -s /etc/sv/openntpd $MOUNT_PATH/etc/runit/runsvdir/default/

Same as before we just install the package with our local xbps package manager pointing to the chroot and then setup the package to run at the end of symlink chain.

dhcpcd

The base-system package should have covered the install of dhcpcd, so all we have to do is activate the service. Like before, we'll symlink directly to the end of the symlink chain.

MOUNT_PATH='/mnt/sdcard' # Replace with any path to an empty directory. By convention it would be in /mnt (same mount path as above)

ln -s /etc/sv/dhcpcd $MOUNT_PATH/etc/runit/runsvdir/default/

A non-root user

This probably depends on your use-case, but having everything running as root is usually bad news, so setting up a non-root user which we can ssh in as is probably a smart idea.

This is the first part of the configuration that is truly best done inside the chroot, so make sure you have the filesystem mounted and have copied the qemu-aarch64-static binary into chroot.

MOUNT_PATH='/mnt/sdcard' # Replace with any path to an empty directory. By convention it would be in /mnt (same mount path as above)

# After executing this command all subsequent commands will act like
# you're running on Pi instead of your main machine
chroot $MOUNT_PATH 

USERNAME='me' # Replace with your desired username

groupadd -g 1000 $USERNAME # Create our user's group

# Add our user and add it to the wheel group and our personal group
# Depending on your needs you could additionally add yourself to
# other default groups like: floppy, dialout, audio, video, cdrom, optical
useradd -g $USERNAME -G wheel $USERNAME 

# Set our password interactively
passwd $USERNAME

sed -ie 's/# %wheel ALL=(ALL) ALL/%wheel ALL=(ALL) ALL/g' $MOUNT_PATH/etc/sudoers # Allow users in the wheel group sudo access

At this point the root account's password is still "voidlinux". We wouldn't want our system running with the default root password, so to remove it run

MOUNT_PATH='/mnt/sdcard' # Replace with any path to an empty directory. By convention it would be in /mnt (same mount path as above)

chroot $MOUNT_PATH # Run this if you're not in the chroot

passwd --delete root

If you set up ssh access and disabled password authentication you'll want to add your ssh key to the rootfs.

MOUNT_PATH='/mnt/sdcard' # Replace with any path to an empty directory. By convention it would be in /mnt (same mount path as above)
USERNAME='me' # Replace with your desired username

mkdir $MOUNT_PATH/home/$USERNAME/.ssh
cat /home/$USERNAME/.ssh/id_rsa.pub > $MOUNT_PATH/home/$USERNAME/.ssh/authorized_keys

Clean up

According to the void docs we should remove the base-voidstrap package and reconfigure all packages in the chroot to ensure everything is setup correctly.

MOUNT_PATH='/mnt/sdcard' # Replace with any path to an empty directory. By convention it would be in /mnt (same mount path as above)

chroot $MOUNT_PATH

xbps-remove -y base-voidstrap
xbps-reconfigure -fa

Now that we're done in the chroot we can delete the qemu-aarch64-static binary that we put in there.

MOUNT_PATH='/mnt/sdcard' # Replace with any path to an empty directory. By convention it would be in /mnt (same mount path as above)

rm $MOUNT_PATH/bin/qemu-aarch64-static

That's it!

Make sure to unmount the disk before removing it from your machine because we wrote a lot of data and that data might not be synced until we unmount it.

MOUNT_PATH='/mnt/sdcard' # Replace with any path to an empty directory. By convention it would be in /mnt (same mount path as above)

umount $MOUNT_PATH/boot
umount $MOUNT_PATH

Lastly, with some care, a lot of these steps can be combined. To see what that might look like check out this repo

Now you should be able to put the SD card into the Pi, boot it up and have ssh access!

Setting up an nfs server for persistent storage in k8s

These are some helpful tips I found when trying to set up an nfs for persistent volumes on my k8s cluster. Setting up the actual persistent volumes and claims will come later.

Prerequisites

Some of the specifics of these tips (package names, directories, etc.) are going to be specific to voidlinux which is the flavor of linux I'm running my nfs on. There is almost certainly an equivalent in your system, but the name may be different.

tl;dr

Make sure you have the statd, nfs-server, and rpcbind services enabled on the server.
Use /etc/exports to configure what directories are exported.
Run exportfs -r to make changes to /etc/exports real.

Setup

Actually setting up the nfs is pretty easy. Just install the nfs-utils package and enable the nfs-server, statd, and rpcbind services. That's it.

Configuration

Now that you have an nfs server you need to configure which directories are available for a client to mount. This is done through the /etc/exports file. I found this site to be quite useful in explaining what some of the options in /etc/exports are and what they mean. Specifically, debugging step 3 (setting the options to (ro,no_root_squash,sync)) was what finally got it working for me when I was receiving mount.nfs: access denied by server while mounting 192.168.0.253:/home/jeff/test. My /etc/exports file is just one line:

/watermelon-pool 192.168.0.0/24(rw)

/watermelon-pool is the path to my zfs pool which is where I store this kind of data.
192.169.0.0/24 is the network prefix that my machines are in.
(rw) allows those machines to read and write to the nfs

After you make changes to /etc/exports make sure to run exportfs -r. exportfs -r rereads the /etc/exports and exports the directories specified in /etc/exports. Essentially, you need to run it every time you edit /etc/exports.

For some reason I had issues when not specifying the no_root_squash option for some directories. I still don't have a good answer for what's up with that, but you can read my (still unanswered) question on unix stack exchange if you want. This didn't effect my ability to use this nfs server as a place for persistent storage for kubernetes though. It seemed to be a void specific bug that only effects certain directories (specifically my home directory), but I'm still not sure.

Read the docs

Unsurprisingly the voidlinux docs on setting up an nfs server on voidlinux were pretty helpful, who knew? There are a few pretty non-obvious steps when setting up an nfs on void. Notably you have to enable the rpcbind, and statd services on the nfs server in addition to the nfs-server service.

Errors I received and how I fixed them

Command: showmount -e 192.168.0.253

Received: clnt_create: RPC: Program not registered

Fix: Start statd service on server

Command: showmount -e 192.168.0.253

Received: clnt_create: RPC: Unable to receive

Fix: Start rpcbind service on server

Command: sudo mount -v -t nfs 192.168.0.253:/home/jeff/test nas/

Received: mount.nfs: mount(2): Connection refused

Fix: Start rpcbind service on server

Command: sudo sv restart nfs-server

Received: down: nfs-server: 1s, normally up, want up

Fix: Start rpcbind and statd services on server

Command: sudo mount -v -t nfs 192.168.0.253:/home/jeff/test nas/

Received: mount.nfs: mount(2): Permission denied

Random tips

Make sure the nfs-server service is actually up.

sv doesn't make this super clear in my opinion. For example this means everything is good

> sudo sv restart nfs-server
ok: run: nfs-server: (pid 9446) 1s

while this means everything is broken

> sudo sv restart nfs-server
down: nfs-server: 1s, normally up, want up

Not quite as different I would like :/

If you find that your nfs-server service isn't running it might be because you haven't enabled the statd and rpcbind services.

You can mount directories deeper than the exported directory.

For instance, if you put /home/user * in /etc/exports you can mount /home/user/specific/path assuming /home/user/specific/path exists on ths nfs server like this:

sudo mount -t nfs4 192.168.0.253:/home/user/specific/path /mnt/mount_point

Adding a new node to the cluster

This is a guide on adding a new raspberry pi node to your k3s managed kubernetes cluster.

tl;dr

Write Raspberry Pi OS to an sd card. Found here
Boot er up
ssh in and configure
Install k3s

Slightly more detailed version

Write Raspberry Pi OS to an sd card
1. Download Raspberry Pi OS Found here
2. Unzip it: unzip 2020-08-20-raspios-buster-armhf-lite.zip
3. Copy image to SD card: sudo dd if=/path/to/raspberryPiOS.img of=/dev/sdX bs=4M conv=fsync (where /dev/sdX is the SD card device)
4. Mount SD card: sudo mount /dev/sdX /mnt/sdcard (/mnt/sdcard can be any empty directory)
5. Add "ssh" file to filesystem which causes the ssh server to start on boot: sudo touch /mnt/sdcard/ssh
6. Unmount it: sudo umount /mnt/sdcard
Boot 'er up
1. Put the SD card in the pi
2. Plug in the pi
3. Give it a minute or two
ssh in and configure
1. ssh in: ssh pi@raspberrypi password is "raspberry"
2. Update and install vim and curl: sudo apt update && sudo apt upgrade -y && sudo apt install -y vim curl Although vim isn't strictly necessary and curl is on the image by default, I like vim and we'll use curl later so better to make sure it's already there.
3. Make yourself a user: sudo useradd -m -G adm,dialout,cdrom,sudo,audio,video,plugdev,games,users,input,netdev,gpio,i2c,spi jeff
  1. adm,dialout,cdrom,sudo,audio,video,plugdev,games,users,input,netdev,gpio,i2c,spi are groups that you are adding your user to. The only super important one is probably sudo. This is the list that the default pi user starts in so might as well.
4. Create a .ssh directory so you can get in to your user: sudo -u jeff mkdir .ssh
  1. We use sudo -u jeff here so that it runs as the jeff user and makes jeff the owner by default
5. Slap your public ssh keys into the authorized_keys file: sudo -u jeff curl https://github.com/ToxicGLaDOS.keys -o /home/jeff/.ssh/authorized_keys Here we curl the key down from a github account straight into the authorized_keys file. If your keys aren't on github you might scp them onto the pi.
6. Change the hostname of your machine by editing the /etc/hosts and /etc/hostname files. This can be done manually or with some handy sed commands.
  1. sudo sed -i s/raspberrypi/myHostname/g /etc/hosts
  2. sudo sed -i s/raspberrypi/myHostname/g /etc/hostname
7. Disable password authentication into the pi (optional, but pretty nice)
  1. Manually: Open /etc/ssh/sshd_config and edit the line that says #PasswordAuthentication yes so it says PasswordAuthentication no. If this line doesn't exist add the PasswordAuthentication no line.
  2. Automatic (relies on commented version being there): sudo sed -i s/#PasswordAuthentication\ yes/PasswordAuthentication\ no/g /etc/ssh/sshd_config
8. Allow passwordless sudo: echo 'jeff ALL=(ALL) NOPASSWD:ALL' | sudo tee -a /etc/sudoers This is a little dangerous, because if your account on the machine gets comprimised then an attacker could run any program as root :(. Also if you fail to give yourself passwordless sudo access and restart the pi you can end up being unable to sudo at all which means you can't access /etc/sudoers to give yourself sudo access... So you might end up having to re-imaging the SD card cause you're boned. Not that that has happened to me of course... :(
9. Delete the default pi user: sudo userdel -r pi
Install k3s
1. curl -sfL https://get.k3s.io | K3S_URL=https://masterNodeHostname:6443 K3S_TOKEN=yourToken sh - This pulls down a script provided by k3s and runs it so maybe check to make sure k3s is still up and reputable. Make sure to replace masterNodeHostname and yourToken with your values. masterNodeHostname is the hostname of the master node in your cluster (probably the first one you set up), in my case it's raspberry0. yourToken is an access token used to authenticate to your master node. It can be found on your master node in the /var/lib/rancher/k3s/server/node-token file. Read more at k3s.io.

That's basically it!

Building cross architecture nixos images

The motivation

Some terminology

How I did it

Errors and how to fix them:

file 'nixos-hardware/raspberry-pi/4' was note found...

Cannot build [...] Reason: platform mismatch

nixos evaluation warning: system.build.image is defined, while system.build.images is used.

Some resources I used as reference that might be useful to you

Installing voidlinux on a RaspberryPi

Prerequisites

Partition the disk you want to install void on

Create the boot partition

Create the root partition

Create the filesystems on the disk

Copy in the rootfs

Configure the rootfs to your liking

Running aarch64 binaries in the chroot

Recommended configuration

The base-system package

ssh access

npd

dhcpcd

A non-root user

Clean up

That's it!

Setting up an nfs server for persistent storage in k8s

Prerequisites

tl;dr

Setup

Configuration

Read the docs

Errors I received and how I fixed them

Random tips

Adding a new node to the cluster

tl;dr

Slightly more detailed version

nixos evaluation warning: `system.build.image` is defined, while `system.build.images` is used.

Running `aarch64` binaries in the `chroot`