ruff was never wired into this repository: no target, no CI step, no pin.
It reported 45 findings across sources and tests, so nothing enforced
what the codebase already mostly followed.
Adds `make ruff` (check + format --check), `make ruff-fix`, and `make
lint` as its alias, and makes `make test` run lint as a fourth parallel
spur. The CI workflow calls `make test`, so it is covered there too. The
linter is pinned in a `lint` extra: a ruff minor bump changes which rules
fire, and with the suite gating on a clean run an unpinned linter would
fail it on an unrelated day.
The 45 findings are fixed rather than configured away. Three needed a
decision instead of the mechanical fix:
- The generation timestamp keeps its local wall clock (DTZ005 waived).
Generations sort by that name, and UTC would order new ones before the
existing ones wherever the offset is positive - "newest generation" is
what every restore path selects on.
- The per-volume `copy` closure now binds volume_name and vol_dir as
default arguments (B023). It only worked because it is called inside
the same iteration.
- The two CLI top-level handlers keep their blind except (BLE001
waived): turning any failure into exit 1 is what a CLI boundary is
for. The two in run.py did not need it and were narrowed to what they
actually catch.
Also drops the comments that restate the code: the section banners in
restore/__main__.py, the filename repeated as line 1 of nine test files,
step narration above the statement it narrates, and a block in app.py
documenting parameters that had moved to another module. What names a
trip-wire stays - the snapshot destination rule, the mysql-binary
absence in MariaDB 11 images, the session-scoped FOREIGN_KEY_CHECKS, the
spooled temp file for multi-GB dumps, and the negative control that
loses its discriminating power if it ever passes.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A databases.csv row asking for every database of an instance
(database = '*') makes the backup side write <instance>.cluster.backup.sql
via pg_dumpall, and nothing could read it back: the restore CLI knew
files, postgres and mariadb. That dump was stored and unrestorable - a
format whose producer had no consumer.
Adds `baudolo-restore cluster`. Three properties of a cluster stream
shape it, and each one bit during development:
- It recreates databases, and CREATE DATABASE cannot run inside a
transaction block. So unlike the single-database replay this one must
NOT be wrapped in --single-transaction. The unit tests now pin both
contracts against each other.
- It recreates every role including the one the replay connects as, and
the pre-clean cannot drop the role holding its own session. That
single CREATE ROLE is filtered out of the stream while its ALTER ROLE
is kept, because that is what carries the attributes and the password.
Found by running it: the first replay died on `role "postgres"
already exists`.
- --empty means more than for one database: the cluster's databases go
first, then DROP OWNED BY releases what a role still holds in the
control database, then the roles themselves. The order is pinned by a
phase column because \gexec would otherwise emit them interleaved, and
a role cannot be dropped while it still owns a database.
Without --empty the replay stops at the first object that already
exists. Recreating a cluster over a populated one is a decision, not a
default.
The e2e test drills the real thing: two databases and their owning role
are dropped outright and have to come back with their payload and their
ownership intact.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The dump replay ran statement-by-statement with autocommit. When restore --empty runs against a LIVE database, the pre-clean drops a table, the replay recreates it and autocommits, and a concurrent writer (discourse's mini_scheduler upserting scheduler_stats(id=1)) inserts the same primary key into the empty table before the dump's COPY loads it. The COPY then aborts with a duplicate-key violation under ON_ERROR_STOP and the whole restore fails. Running the replay with --single-transaction keeps the recreated table invisible to other sessions until commit, so the writer can never insert the racing row.
The --empty pre-clean stays multi-statement (\gexec, one DROP per statement): running every DROP in one transaction exhausts max_locks_per_transaction on large schemas (e.g. gitlab).
Extract the pre-clean SQL from the inline string into restore/db/empty_preclean.sql (loaded via dirname(__file__)) and declare it as package-data so it ships in the wheel. Add a unit test guarding the single-transaction/multi-statement split and an e2e that reproduces the live-writer race.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A schema shipping a custom text search dictionary (taiga's english_stem_nostop) survived the pre-clean and aborted the dump replay with a duplicate pg_ts_dict_dictname_index violation under ON_ERROR_STOP. The discovery SELECT now also enumerates user-owned pg_ts_config and pg_ts_dict entries. The string-assertion unit test is replaced by real scenario data in the e2e: the seeded schema now contains an overloaded f()/f(int) pair and the nostop dictionary plus configuration, and the restored database is queried to prove each survives the backup, pre-clean and replay cycle exactly once.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A signature-less DROP FUNCTION aborts with 'function name is not unique' as soon as a schema overloads a name, killing the whole --empty replay under ON_ERROR_STOP. The function branch now emits name(identity args) via pg_get_function_identity_arguments; because that compound must not be identifier-quoted as a whole, quoting moves from the outer format into each branch, guarded by a unit test that pins the per-branch %I quoting.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
One DO-block ran every DROP in a single transaction and exhausted
max_locks_per_transaction on large schemas (gitlab); emit one DROP per
row and execute via \gexec instead. Also drop user-owned non-public
schemas so a dump that CREATE SCHEMAs (discourse) does not abort on the
existing schema under ON_ERROR_STOP.
Covered by a new DinD e2e test restoring --empty against a fully
populated database with a non-public schema and every object class.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Restore now resolves the target volume's mountpoint via docker volume
inspect and rsyncs into it directly, mirroring how the backup path
already reads the mountpoint; the alpine-rsync container and the
--rsync-image flag are gone. The e2e harness mounts /var/lib/docker
read-write in the test container so the direct restore can write, the
same way baudolo runs as root on a real host.
BREAKING CHANGE: the restore 'files' subcommand no longer accepts
--rsync-image; the caller must have write access to the docker volume
root (root on the host), which is baudolo's normal privilege.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The drop_sql loop covered relations, routines, sequences and types but
not pg_collation, so a dump's CREATE COLLATION (OpenProject's ICU
public.versions_name) aborted the ON_ERROR_STOP replay with 'collation
already exists'. Add the fifth UNION ALL branch; DROP COLLATION IF
EXISTS public.<name> CASCADE rides the existing loop, and the loop
already drops every user table, so CASCADE fallout is absorbed by the
IF EXISTS no-ops.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Backup: a container that vanishes between the docker ps listing and the
swarm-task inspect (--rm one-shots, task-history GC) no longer aborts the
whole backup run; it counts as not stoppable and is skipped.
Restore: the postgres replay streams the dump through a spooled temp file
instead of buffering it three times in memory (multi-GB dumps OOMed the
restore mid-replay), and the superuser-only line filter is COPY-aware: data
rows inside COPY ... FROM stdin blocks pass through untouched, so a row
that happens to start with COMMENT ON EXTENSION or ALTER DEFAULT PRIVILEGES
is no longer silently dropped.
The e2e runner talks to the DinD daemon through docker exec instead of a
host-published tcp://127.0.0.1:2375: port publishing is unreachable from
sandboxed runners and from hosts with broken loopback publishing, and the
unencrypted root API port disappears from the host. The debug tmp dump
shrinks to tar plus docker cp against the DinD container itself.
New coverage: an e2e reproducing the swarm flake end to end (service task
on the volume, nothing whitelisted: the backup must succeed, the very same
task container must keep running, and the service must never replace a
task), unit tests for the COPY-aware filter, the swarm-task probe including
the vanished-container path, filter_stoppable ordering, and the one-session
FOREIGN_KEY_CHECKS drop assembly. Full suite: 35 unit, 9 integration,
30 e2e green.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Restore fixes, both hit by the infinito svc-bkp e2e drill:
- mariadb --empty dropped tables one docker exec at a time with SET
FOREIGN_KEY_CHECKS=0 issued in a separate client session, so the
session-scoped toggle never applied and any FK-referenced parent table
(mailu.users) died with ERROR 1451. Issue the toggle and all DROPs in
one session.
- postgres --empty now drops only current_user-owned objects (extension
members like pg_trgm's set_limit are superuser-owned) with IF EXISTS
absorbing CASCADE fallout, and the replay skips superuser-only dump
lines (COMMENT ON EXTENSION, ALTER DEFAULT PRIVILEGES) that abort an
app-user psql run under ON_ERROR_STOP.
Backup fixes:
- pg_dump now runs with --no-owner --no-privileges so future dumps are
replayable by the owning app user in the first place.
- Swarm task containers are never stopped or started manually: the
orchestrator replaces a stopped task and a later docker start fails on
the detached overlay network. filter_stoppable skips them visibly and
the whitelist stop check ignores them.
Validated end to end against a live infinito compose stack: the full
svc-bkp-volume-2-local drill (verify, restore cycle, sql replay for
mailu, keycloak and one more db) passes with these patches applied.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Avoid passing raw bytes/str via stdin to subprocess.run(), which caused
"'bytes' object has no attribute 'fileno'" and
"stdin and input arguments may not both be used" errors.
If stdin is bytes or str, pass it via input= instead; otherwise forward
stdin unchanged. This fixes Postgres restore failures in E2E tests
without changing productive restore logic.
https://chatgpt.com/share/694ed70d-9e04-800f-8dec-edf08e6e2082
- add `make clean` and run it before `test-e2e` to avoid stale artifacts
- restore: do not hardcode `mysql` for --empty; use detected mariadb/mysql client
- e2e: improve subprocess error output for easier CI debugging
- e2e: adjust MariaDB readiness checks for socket-only root on MariaDB 11
- e2e: add `wait_for_mariadb_sql` and run SQL readiness checks for dedicated TCP user
- e2e: update MariaDB full/no-copy tests to use dedicated user over TCP and consistent credentials
https://chatgpt.com/share/694ecfb8-f8a8-800f-a1c9-a5f410d4ba02
The file restore command previously assumed that the target volume name
was identical to the volume name used in the backup path. This caused
restores to fail (exit code 2) when restoring data from one volume backup
into a different target volume.
Introduce an optional --source-volume argument for `baudolo-restore files`
to explicitly specify the backup source volume while keeping the target
volume unchanged.
- Default behavior remains fully backward-compatible
- Enables restoring backups from volume A into volume B
- Fixes E2E test scenario restoring into a new volume
Tests:
- Update E2E file restore test to use --source-volume
https://chatgpt.com/share/694ec70f-1d7c-800f-b221-9d22e4b0775e
