kevinveenbirkenbach/backup-docker-to-local
1
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
225 Commits 2 Branches 17 Tags
159502af5e85c48bd7278748727817b118c06f6f
Commit Graph
100 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Kevin Veen-Birkenbach 159502af5e Added mirros
CI (make tests, stable, publish) / make test (push) Has been cancelled
Details
CI (make tests, stable, publish) / Mark stable + publish image (SemVer tags only) (push) Has been cancelled
Details
2025-12-26 20:50:29 +01:00
Kevin Veen-Birkenbach 698d1e7a9e ci: add Makefile-driven CI with unit, integration and e2e tests
CI (make tests, stable, publish) / make test (push) Has been cancelled
Details
CI (make tests, stable, publish) / Mark stable + publish image (SemVer tags only) (push) Has been cancelled
Details 
- add GitHub Actions CI workflow using Makefile targets exclusively
- run unit, integration and e2e tests via `make test`
- publish Docker image to GHCR on SemVer tags
- force-update `stable` git tag after successful release
- add integration test for seed CLI (CSV upsert behavior)
- extend Makefile with test-unit and test-integration targets

https://chatgpt.com/share/694ee54f-b814-800f-a714-e87563e538b7
 2025-12-26 20:43:06 +01:00
Kevin Veen-Birkenbach f8420c8bea renamed configure to seed 2025-12-26 19:58:39 +01:00
Kevin Veen-Birkenbach 8e1a53e1f9 Deleted Starting file 2025-12-26 19:47:54 +01:00
Kevin Veen-Birkenbach 7b55d59300 fix(restore): handle bytes stdin correctly in subprocess wrapper 
Avoid passing raw bytes/str via stdin to subprocess.run(), which caused
"'bytes' object has no attribute 'fileno'" and
"stdin and input arguments may not both be used" errors.

If stdin is bytes or str, pass it via input= instead; otherwise forward
stdin unchanged. This fixes Postgres restore failures in E2E tests
without changing productive restore logic.

https://chatgpt.com/share/694ed70d-9e04-800f-8dec-edf08e6e2082
 2025-12-26 19:42:17 +01:00
Kevin Veen-Birkenbach cf6f4d8326 test(e2e): stabilize MariaDB 11 tests and fix restore empty-mode client selection 
- add `make clean` and run it before `test-e2e` to avoid stale artifacts
- restore: do not hardcode `mysql` for --empty; use detected mariadb/mysql client
- e2e: improve subprocess error output for easier CI debugging
- e2e: adjust MariaDB readiness checks for socket-only root on MariaDB 11
- e2e: add `wait_for_mariadb_sql` and run SQL readiness checks for dedicated TCP user
- e2e: update MariaDB full/no-copy tests to use dedicated user over TCP and consistent credentials

https://chatgpt.com/share/694ecfb8-f8a8-800f-a1c9-a5f410d4ba02
 2025-12-26 19:10:55 +01:00
Kevin Veen-Birkenbach 4af15d9074 fix(restore): allow restoring files from different source volume 
The file restore command previously assumed that the target volume name
was identical to the volume name used in the backup path. This caused
restores to fail (exit code 2) when restoring data from one volume backup
into a different target volume.

Introduce an optional --source-volume argument for `baudolo-restore files`
to explicitly specify the backup source volume while keeping the target
volume unchanged.

- Default behavior remains fully backward-compatible
- Enables restoring backups from volume A into volume B
- Fixes E2E test scenario restoring into a new volume

Tests:
- Update E2E file restore test to use --source-volume

https://chatgpt.com/share/694ec70f-1d7c-800f-b221-9d22e4b0775e
 2025-12-26 18:33:58 +01:00
Kevin Veen-Birkenbach c30b4865d4 refactor: migrate to src/ package + add DinD-based E2E runner with debug artifacts 
- Replace legacy standalone scripts with a proper src-layout Python package
  (baudolo backup/restore/configure entrypoints via pyproject.toml)
- Remove old scripts/files (backup-docker-to-local.py, recover-docker-from-local.sh,
  databases.csv.tpl, Todo.md)
- Add Dockerfile to build the project image for local/E2E usage
- Update Makefile: build image and run E2E via external runner script
- Add scripts/test-e2e.sh:
  - start DinD + dedicated network
  - recreate DinD data volume (and shared /tmp volume)
  - pre-pull helper images (alpine-rsync, alpine)
  - load local baudolo:local image into DinD
  - run unittest E2E suite inside DinD and abort on first failure
  - on failure: dump host+DinD diagnostics and archive shared /tmp into artifacts/
- Add artifacts/ debug outputs produced by failing E2E runs (logs, events, tmp archive)

https://chatgpt.com/share/694ec23f-0794-800f-9a59-8365bc80f435
 2025-12-26 18:13:26 +01:00
Kevin Veen-Birkenbach 41910aece2 Optimized entry columns 2025-10-18 09:58:06 +02:00
Kevin Veen-Birkenbach b6dd624f97 Added install hints to pass install requirements 2025-09-11 20:45:30 +02:00
Kevin Veen-Birkenbach 47828c44db Use dirval CLI instead of direct Python script reference in backup-docker-to-local.py and declare dirval in requirements.yml 
Details:
- Replaced python call to directory-validator.py with direct 'dirval' command
- Updated error message accordingly
- Added dirval as dependency in requirements.yml

Conversation: https://chatgpt.com/share/68c31763-6a84-800f-a697-50fa40e9841b
 2025-09-11 20:40:21 +02:00
Kevin Veen-Birkenbach a538e537cb Added emoji 2025-07-17 00:26:22 +02:00
Kevin Veen-Birkenbach 8f72d61300 Bypass buffer 2025-07-17 00:19:10 +02:00
Kevin Veen-Birkenbach c754083cec Added more detailled info to is_image_whitelisted 2025-07-17 00:13:38 +02:00
Kevin Veen-Birkenbach 84d0fd6346 Added more detailled output why stopped 2025-07-17 00:05:44 +02:00
Kevin Veen-Birkenbach 627187cecb Solved variable bug 2025-07-16 23:41:17 +02:00
Kevin Veen-Birkenbach 978e153723 Changed special instances to database containers for more clarity 2025-07-16 14:50:32 +02:00
Kevin Veen-Birkenbach 2bf2b0798e Solved bug 2025-07-16 14:47:32 +02:00
Kevin Veen-Birkenbach 8196a0206b Optimized parameters 2025-07-16 14:43:43 +02:00
Kevin Veen-Birkenbach c4cbb290b3 Made IMAGES_NO_STOP_REQUIRED obligatoric 2025-07-16 10:55:21 +02:00
Kevin Veen-Birkenbach 2d2376eac8 Implemented new parameters to make it more flexibel for cymais 2025-07-14 18:47:25 +02:00
Kevin Veen-Birkenbach 8c4ae60a6a Left hint 2025-07-10 22:12:29 +02:00
Kevin Veen-Birkenbach 18d6136de0 Used streaming instead of memory to prevent overflow 2025-07-03 14:37:36 +02:00
Kevin Veen-Birkenbach 3ed89a59a8 Added failure handling for bussy databases 2025-07-03 12:09:44 +02:00
Kevin Veen-Birkenbach 7d3f0a3ae3 Added roles, and stop on failure 2025-07-03 12:06:38 +02:00
Kevin Veen-Birkenbach 5762754ed7 Added restore_backup.py restore_postgres_databases.py 2025-07-03 11:59:40 +02:00
Kevin Veen-Birkenbach 556cb17433 Implemented check for empty database name 2025-04-21 10:52:06 +02:00
Kevin Veen-Birkenbach 2e2c8131c4 Implemented fallback_pg_dumpall 2025-04-19 00:21:45 +02:00
Kevin Veen-Birkenbach 5005d577cc Update README.md 2025-04-19 00:04:29 +02:00
Kevin Veen-Birkenbach 327b666237 Removed setup instructions, because they are managed now by pkgmgr 2025-04-01 13:20:47 +02:00
Kevin Veen-Birkenbach a7c6fa861a Moved requirements from cymais to this role 2025-04-01 13:18:30 +02:00
Kevin Veen-Birkenbach f6c57be1b7 Added Funding 2025-03-12 20:52:47 +01:00
Kevin Veen-Birkenbach 9d990a728d Update README.md 2025-03-12 11:14:40 +01:00
Kevin Veen-Birkenbach a355f34e6e Update README.md 2025-03-04 22:35:22 +01:00
Kevin Veen-Birkenbach f847c8dd74 Merge branch 'main' of github.com:kevinveenbirkenbach/docker-volume-backup 2024-12-03 11:19:40 +01:00
Kevin Veen-Birkenbach 3e225b0317 Added hard restart for mailu 2024-12-03 11:19:12 +01:00
Kevin Veen-Birkenbach 6537626d77 optimized code formating 2024-02-05 19:18:54 +01:00
Kevin Veen-Birkenbach da7e5cc9be changed loading of directory-validator 2024-02-05 19:16:09 +01:00
Kevin Veen-Birkenbach 69a1ea30aa Implemented stamp function 2024-01-29 20:04:21 +01:00
Kevin Veen-Birkenbach e9588b0e31 Added exception catching 2024-01-14 01:46:47 +01:00
Kevin Veen-Birkenbach 42566815c4 solved start bug 2024-01-14 01:25:54 +01:00
Kevin Veen-Birkenbach 8bc2b068ff Added condition to stop containers 2024-01-12 18:58:02 +01:00
Kevin Veen-Birkenbach 25d428fc9c Removed skipping of unused volumes 2024-01-12 18:42:26 +01:00
Kevin Veen-Birkenbach 0077efa63c Implemented stop on first error 2024-01-12 16:20:17 +01:00
Kevin Veen-Birkenbach 9d8e80f793 Implemented better parameter check 2024-01-12 15:42:51 +01:00
Kevin Veen-Birkenbach d2b699c271 Implemented postgres recovery 2024-01-12 11:47:46 +01:00
Kevin Veen-Birkenbach b7dcb17fd5 Optimized logic for central databases 2024-01-11 20:51:55 +01:00
Kevin Veen-Birkenbach 7f6f5f6dc8 Optimized logic for central databases 2024-01-11 20:47:57 +01:00
Kevin Veen-Birkenbach 75d48fb3e9 Removed unnecessary warning 2024-01-11 20:40:07 +01:00
Kevin Veen-Birkenbach bb3d20c424 Solved path comparisment bug 2024-01-11 16:21:39 +01:00
Kevin Veen-Birkenbach f057104a65 Solved order bug 2024-01-11 15:56:34 +01:00
Kevin Veen-Birkenbach 7fe1886ff9 Solved trailing spaces bug 2024-01-11 12:20:38 +01:00
Kevin Veen-Birkenbach 35e28f31d2 Changed logic so that volume is not created for db recoveries 2024-01-11 11:04:03 +01:00
Kevin Veen-Birkenbach 15a1f17184 Changed logic so that volume is not created for db recoveries 2024-01-11 10:58:35 +01:00
Kevin Veen-Birkenbach ace1a70488 Optimized database recovery function 2024-01-11 03:04:13 +01:00
Kevin Veen-Birkenbach d537393da8 Solved array bug 2024-01-09 13:18:17 +01:00
Kevin Veen-Birkenbach 2b716e5d90 Optimized code performance 2024-01-09 12:59:53 +01:00
Kevin Veen-Birkenbach 7702b17a9d Changed from jq to format 2024-01-08 20:43:42 +01:00
Kevin Veen-Birkenbach 489b5796b7 Implemented shutdown parameter 2024-01-08 19:48:50 +01:00
Kevin Veen-Birkenbach bf9986f282 Removed unecessary host from seeder function 2024-01-06 14:23:23 +01:00
Kevin Veen-Birkenbach e2e62c5835 Added comment, to prevent thinking in future. PS: Thanks ChatGPT :-* 2024-01-06 14:02:13 +01:00
Kevin Veen-Birkenbach 4388e09937 Implemented support of multiple databases per instance 2024-01-06 13:51:30 +01:00
Kevin Veen-Birkenbach 31133f251e Added images where no restart is required 2024-01-06 13:36:35 +01:00
Kevin Veen-Birkenbach 850fc3bf0c Implemented dynamic storage path identification 2024-01-06 13:27:17 +01:00
Kevin Veen-Birkenbach 00fd102f81 removed parameter bug 2023-12-27 23:33:17 +01:00
Kevin Veen-Birkenbach f369a13d37 changed function strucutre 2023-12-27 23:31:21 +01:00
Kevin Veen-Birkenbach f505be35d3 Refactored variables to global to reduce complexity 2023-12-27 23:24:31 +01:00
Kevin Veen-Birkenbach 49c442b299 changed parameter 2023-12-27 21:37:06 +01:00
Kevin Veen-Birkenbach 0322eee107 Removed redundant code 2023-12-27 21:36:07 +01:00
Kevin Veen-Birkenbach 9a5b544e0b Implemented SQL dumps for backup everything 2023-12-27 21:26:11 +01:00
Kevin Veen-Birkenbach 15d7406b7e Implemented global variables to reduce code complexity 2023-12-27 21:04:25 +01:00
Kevin Veen-Birkenbach 9dd58f3ee4 Implemented forced file backup 2023-12-27 20:46:56 +01:00
Kevin Veen-Birkenbach 7f383fcce2 Solved undefined variable bug 2023-12-26 20:33:35 +01:00
Kevin Veen-Birkenbach a72753921a Execute backup just once per volume and not once per volume container 2023-12-26 20:20:26 +01:00
Kevin Veen-Birkenbach 407eddc2c3 Ignored images 2023-12-26 20:07:49 +01:00
Kevin Veen-Birkenbach 3fedf49f4e Solved bug 2023-12-26 19:46:20 +01:00
Kevin Veen-Birkenbach fb2e1df233 Solved parameter bug 2023-12-26 19:21:01 +01:00
Kevin Veen-Birkenbach 47922f53fa Solved versions bug 2023-12-26 18:53:58 +01:00
Kevin Veen-Birkenbach 162b3eec06 Reimplemented incremental backup and improved downtime 2023-12-26 18:22:06 +01:00
Kevin Veen-Birkenbach e0fc263dcb Removed duplicated code 2023-12-26 16:09:18 +01:00
Kevin Veen-Birkenbach 581ff501fc Implemented create_volume_directory 2023-12-26 14:34:32 +01:00
Kevin Veen-Birkenbach 540797f244 Made password optional 2023-12-26 04:01:29 +01:00
Kevin Veen-Birkenbach 7853283ef3 Added condition for passwordlesslogin 2023-12-26 02:56:12 +01:00
Kevin Veen-Birkenbach 5e91e298c4 Added instance attribut 2023-12-26 00:27:27 +01:00
Kevin Veen-Birkenbach de59646fc0 Removed deprecated warning 2023-12-25 23:39:28 +01:00
Kevin Veen-Birkenbach bcc8a7fb00 Solved header bug 2023-12-25 23:36:21 +01:00
Kevin Veen-Birkenbach 8c4785dfe6 Solved replacement bug 2023-12-25 23:29:54 +01:00
Kevin Veen-Birkenbach d4799af904 Added database seeder 2023-12-25 23:20:30 +01:00
Kevin Veen-Birkenbach d1f942bc58 Escaped {{ }} correct 2023-12-25 22:46:14 +01:00
Kevin Veen-Birkenbach 397e242e5b Implemented whitelist 2023-12-25 22:33:28 +01:00
Kevin Veen-Birkenbach b06317ad48 Added more prints 2023-12-25 22:19:26 +01:00
Kevin Veen-Birkenbach 79f4cb5e7f Refactored database logic 2023-12-25 21:57:23 +01:00
Kevin Veen-Birkenbach 50db914c36 removed bugs 2023-12-25 21:49:06 +01:00
Kevin Veen-Birkenbach 02062c7d49 Implemented postgres backup and check for stop and restart for file backup necessary 2023-12-25 21:39:50 +01:00
Kevin Veen-Birkenbach a1c33c1747 refactored. untested. 2023-12-25 20:31:56 +01:00
Kevin Veen-Birkenbach b83e481d01 implemented exception handling for rsync 24 2023-12-13 08:55:02 +01:00
Kevin Veen-Birkenbach c4107d91b0 removed -t parameter 2023-12-13 00:53:30 +01:00
Kevin Veen-Birkenbach bff513d639 Solved spacing bug 2023-12-11 09:32:59 +01:00
Kevin Veen-Birkenbach be0bdff4d8 Raise the sigterm to 2h 2023-12-09 14:29:49 +01:00
Kevin Veen-Birkenbach d8aa5f7d79 renamed 2023-11-16 23:02:09 +01:00