Kevin Veen-BirkenbachandClaude Opus 5
d4317827bd
Backing up a live volume with rsync copies a moving target: a database
written to mid-copy lands on disk in a state no engine ever committed.
Stopping the container avoids that at the cost of downtime.
A snapshot removes both. `--snapshot {btrfs,zfs}` with `--snapshot-subject`
freezes the docker root once per run, and every volume copy is then read
from that frozen tree while the containers keep serving. A restore of such
a copy is an ordinary crash recovery, which every supported engine performs
on its own at startup.
An unsupported filesystem or an unknown snapshot kind fails loudly rather
than degrading to a live copy, since a silent fallback would return exactly
the torn backup the mode exists to prevent. `--shutdown` is rejected
alongside `--snapshot` instead of being ignored: under a snapshot no
container is ever stopped, so accepting the flag would promise downtime
semantics the run does not deliver.
Copies out of a snapshot skip rsync's --checksum verification. The source
is immutable for the lifetime of the copy, so size-and-mtime cannot race,
and dropping the second full read roughly halves the I/O per volume.
backup/app.py grew past what one module could carry and is split into
layout, policy and dumps along the lines it already had internally.
Tests: unit coverage for the new snapshot, layout, policy, volume and cli
units; e2e cases drive real btrfs, zfs and ext4 filesystems on loop devices
in a privileged container, including a MariaDB that is written to across
the snapshot and must recover from the restored copy without losing a
committed row. CI installs zfs and sets E2E_REQUIRE_FILESYSTEMS so a
missing kernel module fails the build instead of silently skipping a
filesystem.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
63 lines
1.9 KiB
Python
63 lines
1.9 KiB
Python
from __future__ import annotations
|
|
|
|
import os
|
|
import pathlib
|
|
|
|
from .shell import BackupException, execute_shell_command
|
|
|
|
|
|
def get_storage_path(volume_name: str) -> str:
|
|
path = execute_shell_command(
|
|
f"docker volume inspect --format '{{{{ .Mountpoint }}}}' {volume_name}"
|
|
)[0]
|
|
return f"{path}/"
|
|
|
|
|
|
def get_last_backup_dir(
|
|
versions_dir: str, volume_name: str, current_backup_dir: str
|
|
) -> str | None:
|
|
versions = sorted(os.listdir(versions_dir), reverse=True)
|
|
for version in versions:
|
|
candidate = os.path.join(versions_dir, version, volume_name, "files", "")
|
|
if candidate != current_backup_dir and os.path.isdir(candidate):
|
|
return candidate
|
|
return None
|
|
|
|
|
|
def backup_volume(
|
|
versions_dir: str,
|
|
volume_name: str,
|
|
volume_dir: str,
|
|
*,
|
|
authoritative: bool,
|
|
source: str,
|
|
) -> None:
|
|
"""Perform incremental file backup of a Docker volume.
|
|
|
|
Args:
|
|
authoritative: compare source and destination by content instead of by
|
|
size and whole-second mtime. Required on a pass whose destination was
|
|
already written from a live source, where a file can differ while
|
|
both attributes still agree.
|
|
source: directory to read from - the volume's mountpoint, or its path
|
|
inside a snapshot.
|
|
"""
|
|
dest = os.path.join(volume_dir, "files") + "/"
|
|
pathlib.Path(dest).mkdir(parents=True, exist_ok=True)
|
|
|
|
last = get_last_backup_dir(versions_dir, volume_name, dest)
|
|
link_dest = f"--link-dest='{last}'" if last else ""
|
|
verify = "--checksum " if authoritative else ""
|
|
|
|
cmd = f"rsync -abP --delete --delete-excluded {verify}{link_dest} {source} {dest}"
|
|
|
|
try:
|
|
execute_shell_command(cmd)
|
|
except BackupException as e:
|
|
if "file has vanished" in str(e):
|
|
print(
|
|
"Warning: Some files vanished before transfer. Continuing.", flush=True
|
|
)
|
|
else:
|
|
raise
|