Arch Linux with LUKS and btrfs on a Hetzner server

A small Python CLI (hal) that wraps every step of installing, encrypting, and maintaining an Arch Linux server on Hetzner Dedicated hardware with software RAID, LUKS full-disk encryption, btrfs on top of LVM, and remote unlock via dropbear in the initramfs.

Author: Kevin Veen-Birkenbach <kevin@veen.world> — veen.world
License: MIT — see LICENSE

Install the CLI

make install   # → pip install --user -e .
hal --help

After install, every step below is a single hal subcommand.

Subcommand reference

Run hal --help, hal <group> --help, or hal <group> <target> --help for the live reference.

Top-level

Command	What it does
hal status <host>	Ping + port scan + SSH banner. No login.
hal diagnose <host>	Rescue → chroot, runs a fixed inspection script. Pipe with tee to save.
hal unlock <host>	Send the LUKS passphrase from the keyring to dropbear (cryptroot-unlock).
hal forget <host>	Clear the cached LUKS passphrase from libsecret.

hal connect <target> <host> [cmd]

Open a shell, or run cmd non-interactively.

Target	Where it goes
rescue	Hetzner Rescue OS
server	Booted Arch system
chroot	Rescue → chroot of installed Arch (LUKS-unlocks + mounts first)

hal setup <target> <host> — one-time install operations

Target	What it does
image --autosetup PATH	In rescue: upload autosetup, run installimage. Destructive.
dropbear	Booted Arch: install dropbear + mkinitcpio plugins, copy authorized_keys, patch HOOKS.
grub	Rescue → chroot: install grub package, write LUKS-aware /etc/default/grub, grub-install on every boot disk.
encrypt-root	Rescue: LUKS-encrypt /dev/md1, preserve data via /oldroot copy. Destructive on /dev/md1. Confirms before format.

hal fix <target> <host> — recovery + maintenance operations

Target	What it does
boot	Patch PermitRootLogin, enable persistent journald.
network	Rewrite .network files to match by MACAddress= instead of interface name.
grub	Refresh Stage1 + core.img in MBR (Arch doesn't do this automatically after grub upgrades).
kernel	Roll the linux package back to the previous version (cache or archive.archlinux.org).
static-ip	Replace ip=dhcp in /etc/default/grub with a static cmdline IP derived from /etc/systemd/network/*.network.
upgrade	Full pacman -Syyu + initramfs rebuild + grub-install on every boot disk.
expand-fs	On booted Arch: lvresize -l +100%FREE /dev/vg0/root && btrfs filesystem resize max /.

The LUKS passphrase is prompted (hidden) on first use and cached in the libsecret keyring per host — subsequent runs against the same host don't prompt.

Setup flow

Each section is a small handful of hal commands. Click into the corresponding table row above for what each one actually does.

1. Install Arch via installimage

hal connect rescue YOUR_SERVER_IP                       # verify rescue is up
hal setup image YOUR_SERVER_IP --autosetup autosetup    # see autosetup.example
hal connect rescue YOUR_SERVER_IP reboot

Tip: copy autosetup.example to autosetup, edit DRIVE1/DRIVE2/HOSTNAME, then run setup image.

2. Boot Arch, install the dropbear stack

hal connect server YOUR_SERVER_IP                       # verify SSH works
hal connect server YOUR_SERVER_IP pacman -Syyu          # bring system current
hal setup dropbear YOUR_SERVER_IP                       # dropbear + mkinitcpio plugins + HOOKS

3. Convert root to LUKS

Activate Rescue in the Hetzner Robot UI, then:

hal connect server YOUR_SERVER_IP reboot                # boots back into rescue
hal connect rescue YOUR_SERVER_IP                       # verify rescue is up
hal setup encrypt-root YOUR_SERVER_IP                   # LUKS conversion — DESTRUCTIVE
hal setup grub YOUR_SERVER_IP                           # initial GRUB for LUKS boot
hal fix static-ip YOUR_SERVER_IP                        # (recommended) harden initramfs network

Deactivate Rescue in the Hetzner Robot UI, then:

hal connect rescue YOUR_SERVER_IP reboot                # final reboot into encrypted system

4. Day-to-day use

After every reboot the system blocks at dropbear in initramfs waiting for the LUKS passphrase. From your client:

hal status YOUR_SERVER_IP                               # wait for dropbear / sshd
hal unlock YOUR_SERVER_IP                               # send passphrase to dropbear
hal connect server YOUR_SERVER_IP                       # normal SSH after unlock

5. Expand the root filesystem later

If the autosetup gave you a small root LV and the rest is free LVM space:

hal fix expand-fs YOUR_SERVER_IP

Debugging an unresponsive server

The server isn't booting / SSH never comes up:

# 1. Reach the server's chroot
hal connect rescue YOUR_SERVER_IP                       # via Hetzner Robot → Rescue first
hal diagnose YOUR_SERVER_IP | tee "diag-$(date +%F-%H%M).log"

# 2. Apply best-guess fixes in roughly this order
hal fix boot YOUR_SERVER_IP                             # sshd config + journald
hal fix network YOUR_SERVER_IP                          # interface naming drift
hal fix grub YOUR_SERVER_IP                             # stale MBR after grub upgrades
hal fix static-ip YOUR_SERVER_IP                        # DHCP-in-initramfs fragility

# 3. Last-resort kernel rollback (if a kernel bump is the suspect)
hal fix kernel YOUR_SERVER_IP

# 4. Or, after fixing whatever was broken, upgrade everything cleanly
hal fix upgrade YOUR_SERVER_IP

Every hal chroot command makes its own backups (<file>.hal-backup) before mutating anything, so individual fixes can be reverted by hand.

Sources

• http://daemons-point.com/blog/2019/10/20/hetzner-verschluesselt/
• https://www.howtoforge.com/using-the-btrfs-filesystem-with-raid1-with-ubuntu-12.10-on-a-hetzner-server
• https://code.trafficking.agency/arch-linux-remote-unlock-root-volume-with-mdraid-and-dmcrypt.html
• https://wiki.archlinux.org/index.php/Dm-crypt/Specialties#Remote_unlocking_(hooks:_netconf,_dropbear,_tinyssh,_ppp)
• https://gist.github.com/pezz/5310082