Root File System

The root file system used by Nex in its spawned Firecracker virtual machines is an ext4 (64-bit file system) block device. In oversimplified terms, it’s basically a single file that represents an entire file system.

Building a Root File System

Using nex CLI

As of April 2024, the Nex CLI has the ability to build a root filesystem (a .ext4 file). To build a root filesystem, run the following command:

  1. └─❯ nex rootfs --help
  2. usage: nex rootfs [<flags>]
  4. Build custom rootfs
  6. Flags:
  7.   --script=script.sh                   Additional boot script ran during initialization
  8.   --image="synadia/nex-rootfs:alpine"  Base image for rootfs build
  9.   --agent=../path/to/nex-agent         Path to agent binary
  10.   --size=157286400                     Size of rootfs filesystem

You will need to include the --agent flag at a minimum.
Keep in mind that you will need to make the rootfs large enough to hold any binary you’ll later run via nex run. The default size is 150MB and this tends to support a ~20MB binary.

Manual Approach

There are countless ways to populate an ext4 file, from programmatic to scripted. While our current CI pipelines are more programmatic than scripted, the same underlying principles still apply.

To build a root file system:

  1. Create an empty rootfs.ext4 file of a given size with empty blocks
  2. Use the mkfs.ext4 utility to convert the block device into an ext4 file system
  3. Fill in the files in the file system as needed.

An unexpected but incredibly useful trick is that we can use Docker for step 3. We can mount the block device as a folder and then map that folder to a folder inside the Docker image. If we run the setup script inside the Docker image and then unmount the file system, our rootfs.ext4 file will be a snapshot of what the Docker image looked like when it finished.

Here’s a sample script that does just that:

  1. #!/bin/bash
  3. set -xe
  5. dd if=/dev/zero of=rootfs.ext4 bs=1M count=100
  6. mkfs.ext4 rootfs.ext4
  7. mkdir -p /tmp/my-rootfs
  8. mount rootfs.ext4 /tmp/my-rootfs
  10. docker run -i --rm \
  11.     -v /tmp/my-rootfs:/my-rootfs \
  12.     -v "$(pwd)/nex-agent/agent:/usr/local/bin/agent" \
  13.     -v "$(pwd)/openrc-service.sh:/etc/init.d/agent" \
  14.     alpine sh <setup-alpine.sh
  16. umount /tmp/my-rootfs

Here we’re using the public alpline Docker image to run a script, setup-alpine.sh that will modify the file system to build what we’re looking for. Note that we’ve actually mounted the openrc-service.sh script to /etc/init.d/agent. This effectively copies this file into the new root file system, setting up our OpenRC service.

Let’s see what setup-alpine.sh might look like:

  1. #!/bin/sh
  3. set -xe
  5. apk add --no-cache openrc
  6. apk add --no-cache util-linux
  8. ln -s agetty /etc/init.d/agetty.ttyS0
  9. echo ttyS0 >/etc/securetty
  10. rc-update add agetty.ttyS0 default
  12. echo "root:root" | chpasswd
  14. echo "nameserver 1.1.1.1" >>/etc/resolv.conf
  16. addgroup -g 1000 -S nex && adduser -u 1000 -S nex -G nex
  18. rc-update add devfs boot
  19. rc-update add procfs boot
  20. rc-update add sysfs boot
  22. # This is our script that runs nex-agent
  23. rc-update add agent boot
  25. for d in bin etc lib root sbin usr; do tar c "/$d" | tar x -C /my-rootfs; done
  26. for dir in dev proc run sys var tmp; do mkdir /my-rootfs/${dir}; done
  28. chmod 1777 /my-rootfs/tmp
  29. mkdir -p /my-rootfs/home/nex/
  30. chown 1000:1000 /my-rootfs/home/nex/

This script adds openrc and util-linux to to the bare alpine image, and then uses rc-update to add the agent script to the boot phase.

We currently use a combination of code and scripts to automatically generate a vetted root file system that can be automatically downloaded via the nex preflight command.