3cf66640b5
CLI structure now:
hal {status,diagnose,unlock,forget} HOST
hal connect {rescue,chroot,server} HOST [CMD]
hal setup {image,dropbear,grub,encrypt-root} HOST
hal fix {boot,network,grub,kernel,static-ip,upgrade,expand-fs} HOST
Added subcommands cover the previously-manual sections of the README:
setup image — upload autosetup + run installimage
setup dropbear — install dropbear + mkinitcpio plugins + patch HOOKS
setup grub — initial grub install for LUKS boot
setup encrypt-root — full LUKS conversion of installed root
connect server — SSH to booted Arch (vs rescue/chroot)
unlock — cryptroot-unlock via dropbear with passphrase from keyring
fix expand-fs — lvresize + btrfs resize
Renames (breaking):
upgrade-system -> fix upgrade
expand-fs -> fix expand-fs
forget-passphrase -> forget
reinstall-grub -> fix grub
downgrade-kernel -> fix kernel
use-static-ip -> fix static-ip
fix-{boot,network} -> fix {boot,network}
install-{image,grub} -> setup {image,grub}
setup-dropbear -> setup dropbear
encrypt-root -> setup encrypt-root
Removed downgrade-initramfs (never verified, narrow use case).
README rewritten to reference only hal commands; raw bash blocks for
pacman/cryptsetup/grub-install/mount/chroot are gone. Added autosetup.example
as a template for `hal setup image --autosetup PATH`.
Licensed under MIT (LICENSE file added). Author and homepage shown in
hal --version, hal --help, pyproject.toml, and README.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
162 lines
6.6 KiB
Markdown
162 lines
6.6 KiB
Markdown
# Arch Linux with LUKS and btrfs on a Hetzner server
|
|
|
|
A small Python CLI (`hal`) that wraps every step of installing, encrypting, and
|
|
maintaining an [Arch Linux](https://www.archlinux.de/) server on
|
|
[Hetzner](https://www.hetzner.com/) Dedicated hardware with software RAID,
|
|
[LUKS](https://wiki.archlinux.org/index.php/Dm-crypt) full-disk encryption,
|
|
[btrfs](https://en.wikipedia.org/wiki/Btrfs) on top of LVM, and remote unlock
|
|
via [dropbear](https://wiki.archlinux.org/title/Dm-crypt/Specialties#busybox-based_initramfs_(built_with_mkinitcpio))
|
|
in the initramfs.
|
|
|
|
**Author:** Kevin Veen-Birkenbach <[kevin@veen.world](mailto:kevin@veen.world)> — [veen.world](https://veen.world)
|
|
**License:** MIT — see [LICENSE](./LICENSE)
|
|
|
|
## Install the CLI
|
|
|
|
```bash
|
|
make install # → pip install --user -e .
|
|
hal --help
|
|
```
|
|
|
|
After install, every step below is a single `hal` subcommand.
|
|
|
|
## Subcommand reference
|
|
|
|
Run `hal --help`, `hal <group> --help`, or `hal <group> <target> --help` for the live reference.
|
|
|
|
### Top-level
|
|
|
|
| Command | What it does |
|
|
|---|---|
|
|
| `hal status <host>` | Ping + port scan + SSH banner. No login. |
|
|
| `hal diagnose <host>` | Rescue → chroot, runs a fixed inspection script. Pipe with `tee` to save. |
|
|
| `hal unlock <host>` | Send the LUKS passphrase from the keyring to dropbear (`cryptroot-unlock`). |
|
|
| `hal forget <host>` | Clear the cached LUKS passphrase from libsecret. |
|
|
|
|
### `hal connect <target> <host> [cmd]`
|
|
|
|
Open a shell, or run `cmd` non-interactively.
|
|
|
|
| Target | Where it goes |
|
|
|---|---|
|
|
| `rescue` | Hetzner Rescue OS |
|
|
| `server` | Booted Arch system |
|
|
| `chroot` | Rescue → chroot of installed Arch (LUKS-unlocks + mounts first) |
|
|
|
|
### `hal setup <target> <host>` — one-time install operations
|
|
|
|
| Target | What it does |
|
|
|---|---|
|
|
| `image --autosetup PATH` | In rescue: upload autosetup, run `installimage`. **Destructive.** |
|
|
| `dropbear` | Booted Arch: install dropbear + mkinitcpio plugins, copy authorized_keys, patch HOOKS. |
|
|
| `grub` | Rescue → chroot: install grub package, write LUKS-aware `/etc/default/grub`, grub-install on every boot disk. |
|
|
| `encrypt-root` | Rescue: LUKS-encrypt `/dev/md1`, preserve data via `/oldroot` copy. **Destructive on `/dev/md1`. Confirms before format.** |
|
|
|
|
### `hal fix <target> <host>` — recovery + maintenance operations
|
|
|
|
| Target | What it does |
|
|
|---|---|
|
|
| `boot` | Patch `PermitRootLogin`, enable persistent journald. |
|
|
| `network` | Rewrite `.network` files to match by MACAddress= instead of interface name. |
|
|
| `grub` | Refresh Stage1 + core.img in MBR (Arch doesn't do this automatically after grub upgrades). |
|
|
| `kernel` | Roll the `linux` package back to the previous version (cache or archive.archlinux.org). |
|
|
| `static-ip` | Replace `ip=dhcp` in `/etc/default/grub` with a static cmdline IP derived from `/etc/systemd/network/*.network`. |
|
|
| `upgrade` | Full `pacman -Syyu` + initramfs rebuild + grub-install on every boot disk. |
|
|
| `expand-fs` | On booted Arch: `lvresize -l +100%FREE /dev/vg0/root && btrfs filesystem resize max /`. |
|
|
|
|
The LUKS passphrase is prompted (hidden) on first use and cached in the libsecret keyring per host — subsequent runs against the same host don't prompt.
|
|
|
|
## Setup flow
|
|
|
|
Each section is a small handful of `hal` commands. Click into the corresponding
|
|
table row above for what each one actually does.
|
|
|
|
### 1. Install Arch via installimage
|
|
|
|
```bash
|
|
hal connect rescue YOUR_SERVER_IP # verify rescue is up
|
|
hal setup image YOUR_SERVER_IP --autosetup autosetup # see autosetup.example
|
|
hal connect rescue YOUR_SERVER_IP reboot
|
|
```
|
|
|
|
Tip: copy `autosetup.example` to `autosetup`, edit `DRIVE1`/`DRIVE2`/`HOSTNAME`,
|
|
then run `setup image`.
|
|
|
|
### 2. Boot Arch, install the dropbear stack
|
|
|
|
```bash
|
|
hal connect server YOUR_SERVER_IP # verify SSH works
|
|
hal connect server YOUR_SERVER_IP pacman -Syyu # bring system current
|
|
hal setup dropbear YOUR_SERVER_IP # dropbear + mkinitcpio plugins + HOOKS
|
|
```
|
|
|
|
### 3. Convert root to LUKS
|
|
|
|
Activate Rescue in the Hetzner Robot UI, then:
|
|
|
|
```bash
|
|
hal connect server YOUR_SERVER_IP reboot # boots back into rescue
|
|
hal connect rescue YOUR_SERVER_IP # verify rescue is up
|
|
hal setup encrypt-root YOUR_SERVER_IP # LUKS conversion — DESTRUCTIVE
|
|
hal setup grub YOUR_SERVER_IP # initial GRUB for LUKS boot
|
|
hal fix static-ip YOUR_SERVER_IP # (recommended) harden initramfs network
|
|
```
|
|
|
|
Deactivate Rescue in the Hetzner Robot UI, then:
|
|
|
|
```bash
|
|
hal connect rescue YOUR_SERVER_IP reboot # final reboot into encrypted system
|
|
```
|
|
|
|
### 4. Day-to-day use
|
|
|
|
After every reboot the system blocks at dropbear in initramfs waiting for the
|
|
LUKS passphrase. From your client:
|
|
|
|
```bash
|
|
hal status YOUR_SERVER_IP # wait for dropbear / sshd
|
|
hal unlock YOUR_SERVER_IP # send passphrase to dropbear
|
|
hal connect server YOUR_SERVER_IP # normal SSH after unlock
|
|
```
|
|
|
|
### 5. Expand the root filesystem later
|
|
|
|
If the autosetup gave you a small root LV and the rest is free LVM space:
|
|
|
|
```bash
|
|
hal fix expand-fs YOUR_SERVER_IP
|
|
```
|
|
|
|
## Debugging an unresponsive server
|
|
|
|
The server isn't booting / SSH never comes up:
|
|
|
|
```bash
|
|
# 1. Reach the server's chroot
|
|
hal connect rescue YOUR_SERVER_IP # via Hetzner Robot → Rescue first
|
|
hal diagnose YOUR_SERVER_IP | tee "diag-$(date +%F-%H%M).log"
|
|
|
|
# 2. Apply best-guess fixes in roughly this order
|
|
hal fix boot YOUR_SERVER_IP # sshd config + journald
|
|
hal fix network YOUR_SERVER_IP # interface naming drift
|
|
hal fix grub YOUR_SERVER_IP # stale MBR after grub upgrades
|
|
hal fix static-ip YOUR_SERVER_IP # DHCP-in-initramfs fragility
|
|
|
|
# 3. Last-resort kernel rollback (if a kernel bump is the suspect)
|
|
hal fix kernel YOUR_SERVER_IP
|
|
|
|
# 4. Or, after fixing whatever was broken, upgrade everything cleanly
|
|
hal fix upgrade YOUR_SERVER_IP
|
|
```
|
|
|
|
Every `hal` chroot command makes its own backups (`<file>.hal-backup`)
|
|
before mutating anything, so individual fixes can be reverted by hand.
|
|
|
|
## Sources
|
|
|
|
* http://daemons-point.com/blog/2019/10/20/hetzner-verschluesselt/
|
|
* https://www.howtoforge.com/using-the-btrfs-filesystem-with-raid1-with-ubuntu-12.10-on-a-hetzner-server
|
|
* https://code.trafficking.agency/arch-linux-remote-unlock-root-volume-with-mdraid-and-dmcrypt.html
|
|
* https://wiki.archlinux.org/index.php/Dm-crypt/Specialties#Remote_unlocking_(hooks:_netconf,_dropbear,_tinyssh,_ppp)
|
|
* https://gist.github.com/pezz/5310082
|