Compare commits

...
24 Commits
Author SHA1 Message Date
Kevin Veen-Birkenbach 9542f68f90 Release version 2.4.0
CI / security-codeql (push) Blocked by required conditions
CI / test-unit (push) Blocked by required conditions
CI / test-integration (push) Blocked by required conditions
CI / test-env-virtual (push) Blocked by required conditions
CI / test-env-nix (push) Blocked by required conditions
CI / test-e2e (push) Blocked by required conditions
CI / test-virgin-user (push) Blocked by required conditions
CI / test-virgin-root (push) Blocked by required conditions
CI / lint-shell (push) Blocked by required conditions
CI / lint-python (push) Blocked by required conditions
CI / lint-docker (push) Blocked by required conditions
CodeQL Advanced / Check security (none, actions) (push) Blocked by required conditions
CodeQL Advanced / Check security (none, python) (push) Blocked by required conditions
Test Units / test-unit (push) Blocked by required conditions
Test Code Integration / test-integration (push) Blocked by required conditions
Test OS Containers / test-env-virtual (arch) (push) Blocked by required conditions
Test OS Containers / test-env-virtual (centos) (push) Blocked by required conditions
Test OS Containers / test-env-virtual (debian) (push) Blocked by required conditions
Test OS Containers / test-env-virtual (fedora) (push) Blocked by required conditions
Test OS Containers / test-env-virtual (ubuntu) (push) Blocked by required conditions
Test Virgin Nix (flake only) / test-env-nix (arch) (push) Blocked by required conditions
Test Virgin Nix (flake only) / test-env-nix (centos) (push) Blocked by required conditions
Test Virgin Nix (flake only) / test-env-nix (debian) (push) Blocked by required conditions
Test Virgin Nix (flake only) / test-env-nix (fedora) (push) Blocked by required conditions
Test Virgin Nix (flake only) / test-env-nix (ubuntu) (push) Blocked by required conditions
Test End-To-End / test-e2e (arch) (push) Blocked by required conditions
Test End-To-End / test-e2e (centos) (push) Blocked by required conditions
Test End-To-End / test-e2e (debian) (push) Blocked by required conditions
Test End-To-End / test-e2e (fedora) (push) Blocked by required conditions
Test End-To-End / test-e2e (ubuntu) (push) Blocked by required conditions
Test Virgin User / test-virgin-user (arch) (push) Blocked by required conditions
Test Virgin User / test-virgin-user (centos) (push) Blocked by required conditions
Test Virgin User / test-virgin-user (fedora) (push) Blocked by required conditions
Test Virgin User / test-virgin-user (ubuntu) (push) Blocked by required conditions
Test Virgin Root / test-virgin-root (arch) (push) Blocked by required conditions
Test Virgin Root / test-virgin-root (centos) (push) Blocked by required conditions
Test Virgin Root / test-virgin-root (debian) (push) Blocked by required conditions
Test Virgin Root / test-virgin-root (ubuntu) (push) Blocked by required conditions
ShellCheck / lint-shell (push) Blocked by required conditions
Ruff (Python code sniffer) / lint-python (push) Blocked by required conditions
Docker Linter / Lint Dockerfile (push) Blocked by required conditions
Mark stable commit / mark-stable (push) Waiting to run
Test Virgin User / test-virgin-user (debian) (push) Blocked by required conditions
Test Virgin Root / test-virgin-root (fedora) (push) Blocked by required conditions
2026-11-07 15:45:28 +00:00
Kevin Veen-BirkenbachandClaude Opus 5.5 7fb229688b feat(worktree): keep a per-worktree merge flow that scripts read
Every worktree can carry a deterministic merge flow in <git-dir>/pkgmgr/flow.yml (never committed, gone with the worktree): pull lists the refs merged into the branch, in order, before a gate, deploy or merge; push lists the branches the finished branch is merged into, in order, each then pushed by the operator.

- pkgmgr worktree flow [PATH] [--json] shows it; flow set (--pull REF.. | --no-pull) (--push BRANCH.. | --no-push) replaces both lists atomically (0600 temp file, os.replace, pkgmgr/ of mode 0700); flow clear removes it. JSON: worktree, git_dir, branch, flow_branch, pull, push, source (file, none, invalid), warnings.
- pkgmgr worktree up gains repeatable --pull/--push; without them pull is the base and push its local branch (origin/main gives main).
- pkgmgr branch tree carries "flow": {pull, push, source} or null on every item with a worktree, read from the linked worktree's metadata dir without running git inside the worktree; the Agent Orchestrator shows it.

The file and the arguments may come from an agent: every entry must equal its git check-ref-format --branch output (scan mode, cwd /) and must not start with -; the reader opens pkgmgr/ and flow.yml without following links, non-blocking, refuses anything but a regular file, caps at 64 KiB, uses yaml.safe_load, rejects unknown keys, wrong types and bool versions, and never raises into callers: a bad file reads as source invalid with empty lists and a warning.

Tests: 30 new (unit and integration on real repositories), unit suite 500 OK, integration 254 with the two known sandbox signing errors of test_publish_integration only.

🎯 Fix: 92 %
💥 Break: 4 %
💸 Cost: 97 %
⏱️ Time: 93 %
🛡️ Security: 100 %
🏆 Score: 98
🚦 Gate: ✅ 90/5, cost and time <= 100, security 100 %

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-07 15:11:43 +00:00
Kevin Veen-BirkenbachandClaude Opus 5.5 4c8b137449 fix(git): refuse repository config that runs a program before push, release and gc
An agent that can write a repository's .git/config could make the operator's pkgmgr push, release or branch gc run a program or reroute the push. UNSAFE_CONFIG lists the local keys that do (core.sshCommand, gitProxy, askPass, fsmonitor, alternateRefsCommand, credential helpers, gpg programs, protocol.*, config hooks, gc.recentObjectsHook, http proxy/TLS/cookie keys, remote pack programs and proxy, url rewrites). push and release refuse up front; gc skips its fetch and keeps remote branches. Filter drivers, hooksPath, gpg.format and include stay allowed for git-lfs, husky and per-repo signing.

SCAN_FLAGS add --no-pager and switch off config-defined hooks (hook.<event>.enabled=false), which core.hooksPath does not cover. The signing amend, gc's fetch, delete push and branch -D run under them. Remote and ref names sit behind -- in every push, fetch, ls-remote and remote get-url, fetches and pushes no longer recurse into submodules, and a staged gitlink is no longer folded into the signed HEAD. gc skips a compose project that also has containers outside the worktree.

🎯 Fix: 92 %
💥 Break: 4 %
💸 Cost: 91 %
⏱️ Time: 89 %
🛡️ Security: 100 %
🏆 Score: 100
🚦 Gate: ✅ 90/5, cost and time <= 100, security 100 %

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-07 13:26:40 +00:00
Kevin Veen-BirkenbachandClaude Opus 5.5 fb6183838b fix(push): sign only an unsigned HEAD instead of rebasing the range
pkgmgr push re-signed every unpushed commit with rebase --rebase-merges -S, which rewrote history that mirrors or peers may already hold. It now amends only HEAD with -S (--no-edit --no-verify) and warns about older unsigned commits, which stay as they are; the push check refuses only an unsigned HEAD. The signing base prefers the branch's own remote-tracking ref.

🎯 Fix: 92 %
💥 Break: 3 %
💸 Cost: 100 %
⏱️ Time: 100 %
🛡️ Security: 100 %
🏆 Score: 89
🚦 Gate: ✅ 90/5, cost and time <= 100, security 100 %

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-07 12:20:01 +00:00
Kevin Veen-BirkenbachandClaude Opus 5.5 a324ed0b65 Merge branch 'feature/worktree-commands-v2' into merge/main-20261107
Brings the host-independent trust tests (4458fcc) that the Nix-based gate container needs.

🎯 Fix: 92 %
💥 Break: 1 %
💸 Cost: 100 %
⏱️ Time: 100 %
🛡️ Security: 100 %
🏆 Score: 91
🚦 Gate: ✅ 90/5, cost and time <= 100, security 100 %

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-07 12:09:09 +00:00
Kevin Veen-BirkenbachandClaude Opus 5.5 4458fcc160 test(worktree): take the trusted PATH fixture from os.defpath, not the host
The full gate runs in a Nix-based container where true resolves into /nix/store, which is group-writable and therefore never root-controlled, so the two trusted_path tests failed there. They now take the first root-controlled os.defpath directory and skip when there is none. A sticky group-writable ancestor stays refused: on multi-user Nix any user can have the daemon build root-owned store content (skeptic round 7); the README says to pin an absolute path on Nix.

🎯 Fix: 92 %
💥 Break: 1 %
💸 Cost: 100 %
⏱️ Time: 100 %
🛡️ Security: 100 %
🏆 Score: 91
🚦 Gate: ✅ 90/5, cost and time <= 100, security 100 %

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-07 11:58:53 +00:00
Kevin Veen-BirkenbachandClaude Opus 5.5 c78e0a34a3 feat(gc): add pkgmgr gc --report, a read-only disk inventory
pkgmgr gc --report lists each linked worktree and orphaned .claude/worktrees/agent-* dir with its size and last activity, the changes a cleanup would keep, the biggest children of every --root, ELF core dumps, docker system df with --docker, and free versus attributed space per filesystem; --json prints the same as JSON. It deletes and writes nothing and refuses --apply. git run() gains input_text for cat-file --batch-check.

🎯 Fix: 90 %
💥 Break: 2 %
💸 Cost: new
⏱️ Time: new
🛡️ Security: 100 %
🏆 Score: 88
🚦 Gate: cost new, time new: cleared by the operator's commit word for pkgmgr gc --report (2026-11-07)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-07 11:42:26 +00:00
Kevin Veen-BirkenbachandClaude Opus 5.5 1b9b93a542 Merge branch 'feature/worktree-commands-v2' into merge/main-20261107
Both sides register CLI subparsers: review, worktree and gc are all kept. The review package now takes END_OF_OPTIONS from core.git.run and the repository filter keys from core.git.queries.local_config, where B moved them out of branch.gc; the semantics are unchanged (local and worktree scope, scan mode).

🎯 Fix: 93 %
💥 Break: 3 %
💸 Cost: 114 %
⏱️ Time: 129 %
🛡️ Security: 100 %
🏆 Score: 61
🚦 Gate: cost 114 % and time 129 %: accepted by the operator for B (2026-11-07); unit 457 OK, integration 93 OK on the merge

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-07 11:28:07 +00:00
Kevin Veen-BirkenbachandClaude Opus 5.5 102c74f1e0 feat(worktree): add worktree up/down and harden branch gc
pkgmgr worktree up and down create and release a branch worktree with its own slot and run the operator-pinned worktree-init and worktree-fini hooks; branch gc removes merged worktrees, now also patch-merged ones (git cherry), and the hooks of each repository run only from worktree_hooks pins of a managed repository whose git directory is its own. Removal walks to the worktree without following a symbolic link and skips paths that are not their own realpath. A bare-name hook program resolves once to a root-controlled file on a root-controlled PATH: root is uid 0 only in the initial user namespace and the overflow uid elsewhere, nosuid mounts never count on the host, and the hook runs the resolved file with only those PATH entries, stdin closed and a timeout over its process group.

🎯 Fix: 93 %
💥 Break: 3 %
💸 Cost: 114 %
⏱️ Time: 129 %
🛡️ Security: 100 %
🏆 Score: 61
🚦 Gate: cost 114 % and time 129 %: accepted by the operator for B (2026-11-07) after skeptic round 6 found no hole and MEDUSA reached 100 %

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-07 11:18:57 +00:00
Kevin Veen-BirkenbachandClaude Opus 5.5 1b31b10df6 feat(review): render a branch review as a local HTML page
pkgmgr review writes one self-contained HTML page for a repository: commits ahead of the base, the staged, unstaged and untracked diff per file, and the checks of the change set. Repository text reaches the packaged Jinja template only as autoescaped data, git runs in scan mode (now also without replace objects), and the page is written atomically through mkstemp and os.replace.

🎯 Fix: 92 %
💥 Break: 3 %
💸 Cost: new
⏱️ Time: new
🛡️ Security: 100 %
🏆 Score: 89
🚦 Gate: cost new, time new: cleared by the operator's commit word for C (2026-11-07)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-07 10:39:52 +00:00
Kevin Veen-BirkenbachandClaude Opus 5.5 54f0cbb37b feat(branch): add pkgmgr branch tree, a hardened meta-tree of all repos
pkgmgr branch tree --format json|yaml --output FILE scans every known repository with gc's own classification (plan_repo with fetch=False) through the hardened git runner and writes one document atomically (flock, mkstemp, fsync, os.replace): per repo path, remote, default, missing, forge status, scanned_at, warnings and every branch and worktree item, plus orphan shared worktrees under ~/.local/share/worktrees. Forge facts take the token only from an unlocked Secret Service item, never unlocking, and cache per repository for 600 s. open, close, drop and gc --apply/purge refresh only their repository's entry and never overwrite a newer scan.

Scan hardening for repositories an agent may write: log.showSignature=false joins the scan flags; status is skipped for local or worktree filter.* config and for worktrees with gitlinks; --end-of-options precedes every ref argument and -- every remote, branch and du path, and worktree list runs with -z. Before, a remote named --output=<path> made rev-list create that file on the host, and a worktree path with blank lines forged a record.

MEDUSA acceptances (kvb-think-criticize 2026-11-07, 1 skeptic): EXA004 cli/parser/branch_cmd.py:68 by design (-f/--force help text on an operator CLI); B603 actions/branch/gc.py:146 false positive (docker argv list, no shell); B603 actions/branch/gc.py:311 fixed (du gets --, paths from worktree list -z); B603 core/git/run.py:83 by design (argv list, scan callers guard names).

🎯 Fix: 92 %
💥 Break: 4 %
💸 Cost: 104 %
⏱️ Time: 108 %
🛡️ Security: 100 %
🏆 Score: 79
🚦 Gate: ⚠️ cost 104 % > 100, time 108 % > 100: ask the operator before the commit

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-07 06:35:50 +00:00
Kevin Veen-BirkenbachandClaude Opus 5.5 c2f0a56699 feat(publish): rebuild releases nightly and let stable follow the tested one
CodeQL Advanced / Check security (none, python) (push) Canceled after 0s
CodeQL Advanced / Check security (none, actions) (push) Canceled after 0s
Test Units / test-unit (push) Canceled after 0s
CI / test-unit (push) Canceled after 0s
Test Code Integration / test-integration (push) Canceled after 0s
CI / test-integration (push) Canceled after 0s
Test OS Containers / test-env-virtual (ubuntu) (push) Canceled after 0s
Test OS Containers / test-env-virtual (fedora) (push) Canceled after 0s
Test OS Containers / test-env-virtual (debian) (push) Canceled after 0s
Test OS Containers / test-env-virtual (centos) (push) Canceled after 0s
Test OS Containers / test-env-virtual (arch) (push) Canceled after 0s
CI / test-env-virtual (push) Canceled after 0s
Test Virgin Nix (flake only) / test-env-nix (ubuntu) (push) Canceled after 0s
Test Virgin Nix (flake only) / test-env-nix (fedora) (push) Canceled after 0s
Test Virgin Nix (flake only) / test-env-nix (debian) (push) Canceled after 0s
Test Virgin Nix (flake only) / test-env-nix (centos) (push) Canceled after 0s
Test Virgin Nix (flake only) / test-env-nix (arch) (push) Canceled after 0s
CI / test-env-nix (push) Canceled after 0s
Test End-To-End / test-e2e (ubuntu) (push) Canceled after 0s
Test End-To-End / test-e2e (fedora) (push) Canceled after 0s
Test End-To-End / test-e2e (debian) (push) Canceled after 0s
Test End-To-End / test-e2e (centos) (push) Canceled after 0s
Test End-To-End / test-e2e (arch) (push) Canceled after 0s
CI / test-e2e (push) Canceled after 0s
Test Virgin User / test-virgin-user (ubuntu) (push) Canceled after 0s
Test Virgin User / test-virgin-user (fedora) (push) Canceled after 0s
Test Virgin User / test-virgin-user (centos) (push) Canceled after 0s
Test Virgin User / test-virgin-user (arch) (push) Canceled after 0s
CI / test-virgin-user (push) Canceled after 0s
Test Virgin Root / test-virgin-root (ubuntu) (push) Canceled after 0s
Test Virgin Root / test-virgin-root (debian) (push) Canceled after 0s
Test Virgin Root / test-virgin-root (centos) (push) Canceled after 0s
Test Virgin Root / test-virgin-root (arch) (push) Canceled after 0s
CI / test-virgin-root (push) Canceled after 0s
ShellCheck / lint-shell (push) Canceled after 0s
CI / lint-shell (push) Canceled after 0s
Ruff (Python code sniffer) / lint-python (push) Canceled after 0s
CI / lint-python (push) Canceled after 0s
Docker Linter / Lint Dockerfile (push) Canceled after 0s
CI / lint-docker (push) Canceled after 0s
CI / security-codeql (push) Canceled after 0s
Test Virgin User / test-virgin-user (debian) (push) Canceled after 0s
Test Virgin Root / test-virgin-root (fedora) (push) Canceled after 0s
Every image build now carries <version> and the immutable
<version>.<unix-ts>; latest goes to the newest release on main, each
prefix (2, 2.3) to the newest release starting with it, and stable to the
release the git tag stable marks, which "Mark stable commit" sets only
after all tests passed.

publish-containers.yml runs daily at 03:00 UTC and on workflow_dispatch.
A rebuild publishes the newest release and, when the git tag stable sits
on an older one, that release too, so both get base-image security
updates. compute-publish-container-info.sh emits the release list with
each release's moving tags; image.sh takes them as --aliases instead of
--stable.

A platform build whose installed package set equals the published
<version> pushes nothing, and a merge whose image index equals the
published one only moves the tags not yet on it, without a new stamp.
Only vX.Y.Z tags count as releases; v2.4.0-rc1 no longer moves 2 or 2.4.

The build scripts come from the release commit: releases tagged before
this change keep the old scripts until a newer release replaces them.

tests/integration: 22 tests for release choice, aliases, the stable
rebuild, the push skip and the triggers.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-04 18:38:42 +01:00
Kevin Veen-Birkenbach 08738d812e Release version 2.3.0
Mark stable commit / mark-stable (push) Canceled after 0s
CodeQL Advanced / Check security (none, python) (push) Canceled after 0s
CodeQL Advanced / Check security (none, actions) (push) Canceled after 0s
CI / security-codeql (push) Canceled after 0s
Test Units / test-unit (push) Canceled after 0s
CI / test-unit (push) Canceled after 0s
Test Code Integration / test-integration (push) Canceled after 0s
CI / test-integration (push) Canceled after 0s
Test OS Containers / test-env-virtual (ubuntu) (push) Canceled after 0s
Test OS Containers / test-env-virtual (fedora) (push) Canceled after 0s
Test OS Containers / test-env-virtual (debian) (push) Canceled after 0s
Test OS Containers / test-env-virtual (centos) (push) Canceled after 0s
Test OS Containers / test-env-virtual (arch) (push) Canceled after 0s
CI / test-env-virtual (push) Canceled after 0s
Test Virgin Nix (flake only) / test-env-nix (ubuntu) (push) Canceled after 0s
Test Virgin Nix (flake only) / test-env-nix (fedora) (push) Canceled after 0s
Test Virgin Nix (flake only) / test-env-nix (debian) (push) Canceled after 0s
Test Virgin Nix (flake only) / test-env-nix (centos) (push) Canceled after 0s
Test Virgin Nix (flake only) / test-env-nix (arch) (push) Canceled after 0s
CI / test-env-nix (push) Canceled after 0s
Test End-To-End / test-e2e (ubuntu) (push) Canceled after 0s
Test End-To-End / test-e2e (fedora) (push) Canceled after 0s
Test End-To-End / test-e2e (debian) (push) Canceled after 0s
Test End-To-End / test-e2e (centos) (push) Canceled after 0s
Test End-To-End / test-e2e (arch) (push) Canceled after 0s
CI / test-e2e (push) Canceled after 0s
Test Virgin User / test-virgin-user (ubuntu) (push) Canceled after 0s
Test Virgin User / test-virgin-user (fedora) (push) Canceled after 0s
Test Virgin User / test-virgin-user (centos) (push) Canceled after 0s
Test Virgin User / test-virgin-user (arch) (push) Canceled after 0s
CI / test-virgin-user (push) Canceled after 0s
Test Virgin Root / test-virgin-root (ubuntu) (push) Canceled after 0s
Test Virgin Root / test-virgin-root (debian) (push) Canceled after 0s
Test Virgin Root / test-virgin-root (centos) (push) Canceled after 0s
Test Virgin Root / test-virgin-root (arch) (push) Canceled after 0s
CI / test-virgin-root (push) Canceled after 0s
ShellCheck / lint-shell (push) Canceled after 0s
CI / lint-shell (push) Canceled after 0s
Ruff (Python code sniffer) / lint-python (push) Canceled after 0s
CI / lint-python (push) Canceled after 0s
Docker Linter / Lint Dockerfile (push) Canceled after 0s
CI / lint-docker (push) Canceled after 0s
Test Virgin User / test-virgin-user (debian) (push) Canceled after 0s
Test Virgin Root / test-virgin-root (fedora) (push) Canceled after 0s
2026-11-01 15:14:53 +01:00
Kevin Veen-Birkenbach 21cecf64fb feat(branch-gc): add --merged-into, --sizes and --min-age 2026-11-01 12:54:19 +01:00
Kevin Veen-Birkenbach d1a951ee94 Release version 2.2.0
Mark stable commit / mark-stable (push) Canceled after 0s
CI / test-unit (push) Canceled after 0s
CI / test-integration (push) Canceled after 0s
CI / test-env-virtual (push) Canceled after 0s
CI / test-env-nix (push) Canceled after 0s
CI / test-e2e (push) Canceled after 0s
CI / test-virgin-user (push) Canceled after 0s
CI / test-virgin-root (push) Canceled after 0s
CI / lint-shell (push) Canceled after 0s
CI / lint-python (push) Canceled after 0s
CI / lint-docker (push) Canceled after 0s
CI / security-codeql (push) Canceled after 0s
2026-11-01 01:55:31 +01:00
Kevin Veen-BirkenbachandClaude Opus 5.5 f40cc5cfde feat(push): add --no-sign to push unsigned commits
pkgmgr push --no-sign skips the re-sign rebase and the unsigned-range
refusal, so a checkout without a usable signing key can still sync the
fetch URL and every mirror pushurl. Leases and the git cherry guard
still apply. push_branch takes sign as a required keyword; pkgmgr
release and release --retry pass sign=True, so releases stay signed.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-11-01 01:55:31 +01:00
Kevin Veen-BirkenbachandClaude Opus 5.5 944b4cb663 ci(publish): build container images on native arch runners
Each distro now builds amd64 on ubuntu-latest and arm64 on
ubuntu-24.04-arm instead of emulating arm64 under QEMU. image.sh gains
--platform, which pushes one <version>-<arch> tag, and --merge, which
joins those tags under the release, stable and alias tags with
buildx imagetools. The workflow splits into prepare, build and merge
jobs; scripts/build/publish.sh follows the same sequence locally.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-11-01 00:53:16 +01:00
Kevin Veen-BirkenbachandClaude Opus 5.5 919e6a6434 feat(push): re-sign and lease every mirror pushurl
pkgmgr push now signs unpushed commits like git-sign-push and pushes
each pushurl with its own --force-with-lease: the fetch URL against the
tracking ref, every mirror against its own fetched tip, each only after
git cherry shows no commit exists only there. --rewrite allows a
deliberate history rewrite. Without a repository selection it pushes
the current checkout, including worktrees outside the base dir.
pkgmgr release and release --retry sign before tagging and push the
branch and tag through the same path.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-11-01 00:53:16 +01:00
Kevin Veen-Birkenbach 4a4bf137ac Release version 2.1.1
CI / security-codeql (push) Canceled after 0s
CI / test-unit (push) Canceled after 0s
CI / test-integration (push) Canceled after 0s
CI / test-env-virtual (push) Canceled after 0s
CI / test-env-nix (push) Canceled after 0s
CI / test-e2e (push) Canceled after 0s
CI / test-virgin-user (push) Canceled after 0s
CI / test-virgin-root (push) Canceled after 0s
CI / lint-shell (push) Canceled after 0s
CI / lint-python (push) Canceled after 0s
CI / lint-docker (push) Canceled after 0s
Mark stable commit / mark-stable (push) Canceled after 0s
2026-10-31 18:47:35 +01:00
Kevin Veen-BirkenbachandClaude Opus 5.5 ea82d91b75 fix(release): skip pyproject.toml without a static project version
A pyproject.toml that holds only tool config (no [project] table) or lists "version" in [project].dynamic now prints an INFO line and is skipped, like a missing file. A [project] table with neither a version nor a dynamic entry still raises.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-31 18:42:41 +01:00
Kevin Veen-Birkenbach 1c0a4abcbb Release version 2.1.0
CI / security-codeql (push) Canceled after 0s
CI / test-unit (push) Canceled after 0s
CI / test-integration (push) Canceled after 0s
CI / test-env-virtual (push) Canceled after 0s
CI / test-env-nix (push) Canceled after 0s
CI / test-e2e (push) Canceled after 0s
CI / test-virgin-user (push) Canceled after 0s
CI / test-virgin-root (push) Canceled after 0s
CI / lint-shell (push) Canceled after 0s
CI / lint-python (push) Canceled after 0s
CI / lint-docker (push) Canceled after 0s
Mark stable commit / mark-stable (push) Canceled after 0s
2026-10-31 14:38:54 +01:00
Kevin Veen-BirkenbachandClaude Opus 5.5 810d27badc feat(release): start from the highest version across all sources
CodeQL Advanced / Check security (none, python) (push) Canceled after 0s
CodeQL Advanced / Check security (none, actions) (push) Canceled after 0s
CI / security-codeql (push) Canceled after 0s
Test Units / test-unit (push) Canceled after 0s
CI / test-unit (push) Canceled after 0s
Test Code Integration / test-integration (push) Canceled after 0s
CI / test-integration (push) Canceled after 0s
Test OS Containers / test-env-virtual (ubuntu) (push) Canceled after 0s
Test OS Containers / test-env-virtual (fedora) (push) Canceled after 0s
Test OS Containers / test-env-virtual (debian) (push) Canceled after 0s
Test OS Containers / test-env-virtual (centos) (push) Canceled after 0s
Test OS Containers / test-env-virtual (arch) (push) Canceled after 0s
CI / test-env-virtual (push) Canceled after 0s
Test Virgin Nix (flake only) / test-env-nix (ubuntu) (push) Canceled after 0s
Test Virgin Nix (flake only) / test-env-nix (fedora) (push) Canceled after 0s
Test Virgin Nix (flake only) / test-env-nix (debian) (push) Canceled after 0s
Test Virgin Nix (flake only) / test-env-nix (centos) (push) Canceled after 0s
Test Virgin Nix (flake only) / test-env-nix (arch) (push) Canceled after 0s
CI / test-env-nix (push) Canceled after 0s
Test End-To-End / test-e2e (ubuntu) (push) Canceled after 0s
Test End-To-End / test-e2e (fedora) (push) Canceled after 0s
Test End-To-End / test-e2e (debian) (push) Canceled after 0s
Test End-To-End / test-e2e (centos) (push) Canceled after 0s
Test End-To-End / test-e2e (arch) (push) Canceled after 0s
CI / test-e2e (push) Canceled after 0s
Test Virgin User / test-virgin-user (ubuntu) (push) Canceled after 0s
Test Virgin User / test-virgin-user (fedora) (push) Canceled after 0s
Test Virgin User / test-virgin-user (centos) (push) Canceled after 0s
Test Virgin User / test-virgin-user (arch) (push) Canceled after 0s
CI / test-virgin-user (push) Canceled after 0s
Test Virgin Root / test-virgin-root (ubuntu) (push) Canceled after 0s
Test Virgin Root / test-virgin-root (debian) (push) Canceled after 0s
Test Virgin Root / test-virgin-root (centos) (push) Canceled after 0s
Test Virgin Root / test-virgin-root (arch) (push) Canceled after 0s
CI / test-virgin-root (push) Canceled after 0s
ShellCheck / lint-shell (push) Canceled after 0s
CI / lint-shell (push) Canceled after 0s
Ruff (Python code sniffer) / lint-python (push) Canceled after 0s
CI / lint-python (push) Canceled after 0s
Docker Linter / Lint Dockerfile (push) Canceled after 0s
CI / lint-docker (push) Canceled after 0s
Test Virgin User / test-virgin-user (debian) (push) Canceled after 0s
Test Virgin Root / test-virgin-root (fedora) (push) Canceled after 0s
The release base used to be the latest v* tag only. It is now the highest version found in that tag and in every file the release writes: pyproject.toml, CHANGELOG.md, flake.nix, PKGBUILD, the RPM spec, debian/changelog and Cargo.toml. A lagging file can no longer lower the base and a file ahead of the tag raises it; when the sources disagree the release says so and names the version it starts from. Repos whose sources agree release exactly as before.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-31 14:18:07 +01:00
Kevin Veen-BirkenbachandClaude Opus 5.5 d823a42edc feat(release): bump the root crate in Cargo.toml and Cargo.lock
A repo whose Cargo.toml has a [package] section now gets its version set to the release version, and the matching local [[package]] entry in Cargo.lock (same name, no source line) follows in the same release commit, so cargo build --locked keeps passing. Workspaces without [package] and repos without Cargo.toml release as before; a Cargo.lock without the local entry aborts the release before any file is written.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-31 14:18:07 +01:00
Kevin Veen-BirkenbachandClaude Opus 5.5 168d59e1b2 feat(branch): add gc and purge for merged and abandoned branches
pkgmgr branch gc lists every branch and worktree of the repo (or of all managed repos with --all) together with the compose stacks found via com.docker.compose.project.working_dir. Branches whose tips are merged (ancestor of the default branch off its first-parent line, or head SHA of a merged GitHub PR) and whose worktrees are clean are removed with --apply: compose down -v plus the project's images, the worktree (foreign-owned files through a throwaway alpine container, metadata only after the checkout is gone), the local branch and the branch on every remote via push --force-with-lease. Default, protected, open-PR, busy and locked items are skipped; every item is re-checked right before it is removed. pkgmgr branch purge handles unmerged or dirty items with a confirmation per item.

Both refuse to act outside the host PID namespace, where processes and worktree locks cannot be verified. The fetch skips auto-maintenance, whose worktree-prune task deletes the metadata of sandboxed worktrees.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-31 14:18:06 +01:00
116 changed files with 12502 additions and 219 deletions

No files matched your search

+83 -21
View File
@@ -4,15 +4,20 @@ on:
workflow_run:
workflows: ["Mark stable commit"]
types: [completed]
schedule:
- cron: "0 3 * * *"
workflow_dispatch:
jobs:
publish:
if: ${{ github.event.workflow_run.conclusion == 'success' }}
prepare:
if: ${{ github.event_name != 'workflow_run' || github.event.workflow_run.conclusion == 'success' }}
runs-on: ubuntu-latest
permissions:
contents: read
packages: write
outputs:
releases: ${{ steps.info.outputs.releases }}
steps:
- name: Checkout repository (with tags)
@@ -20,40 +25,97 @@ jobs:
with:
fetch-depth: 0
- name: Checkout workflow_run commit and refresh tags
- name: Resolve the releases this run publishes
id: info
env:
WORKFLOW_RUN_SHA: ${{ github.event.workflow_run.head_sha }}
run: bash scripts/github/publish-containers/checkout-workflow-run-commit.sh
- name: Check whether tagged commit is on main
id: branch-check
env:
TARGET_SHA: ${{ github.event.workflow_run.head_sha }}
run: bash scripts/github/common/check-tagged-commit-on-main.sh
- name: Compute version and stable flag
id: info
if: steps.branch-check.outputs.is_on_main == 'true'
run: bash scripts/github/publish-containers/compute-publish-container-info.sh
build:
needs: prepare
if: ${{ needs.prepare.result == 'success' && needs.prepare.outputs.releases != '[]' }}
name: Build ${{ matrix.distro }} ${{ matrix.release.version }} for ${{ matrix.target.platform }}
runs-on: ${{ matrix.target.runner }}
permissions:
contents: read
packages: write
strategy:
fail-fast: false
matrix:
distro: &distros [arch, debian, ubuntu, fedora, centos]
target:
- platform: linux/amd64
runner: ubuntu-latest
- platform: linux/arm64
runner: ubuntu-24.04-arm
release: ${{ fromJSON(needs.prepare.outputs.releases) }}
steps:
- name: Checkout ${{ matrix.release.version }}
uses: actions/checkout@v7
with:
ref: ${{ matrix.release.sha }}
- name: Set up Docker Buildx
if: ${{ steps.info.outputs.should_publish == 'true' }}
uses: docker/setup-buildx-action@594f3bf4285d9ea8dc53c9a0c9c4092420091003
with:
use: true
- name: Login to GHCR
if: ${{ steps.info.outputs.should_publish == 'true' }}
uses: docker/login-action@dbcb813823bdd20940b903addbd779551569679f
with:
registry: ghcr.io
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}
- name: Publish all images
if: ${{ steps.info.outputs.should_publish == 'true' }}
- name: Build and push ${{ matrix.distro }} ${{ matrix.release.version }} for ${{ matrix.target.platform }}
env:
OWNER: ${{ github.repository_owner }}
VERSION: ${{ steps.info.outputs.version }}
IS_STABLE: ${{ steps.info.outputs.is_stable }}
VERSION: ${{ matrix.release.version }}
PKGMGR_DISTRO: ${{ matrix.distro }}
PLATFORM: ${{ matrix.target.platform }}
run: bash scripts/github/publish-containers/publish-container-images.sh
merge:
needs: [prepare, build]
if: ${{ !cancelled() && needs.prepare.result == 'success' && needs.prepare.outputs.releases != '[]' }}
name: Merge ${{ matrix.distro }} ${{ matrix.release.version }}
runs-on: ubuntu-latest
permissions:
contents: read
packages: write
strategy:
fail-fast: false
matrix:
distro: *distros
release: ${{ fromJSON(needs.prepare.outputs.releases) }}
steps:
- name: Checkout ${{ matrix.release.version }}
uses: actions/checkout@v7
with:
ref: ${{ matrix.release.sha }}
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@594f3bf4285d9ea8dc53c9a0c9c4092420091003
with:
use: true
- name: Login to GHCR
uses: docker/login-action@dbcb813823bdd20940b903addbd779551569679f
with:
registry: ghcr.io
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}
- name: Join both platforms of ${{ matrix.distro }} ${{ matrix.release.version }} under its tags
env:
OWNER: ${{ github.repository_owner }}
VERSION: ${{ matrix.release.version }}
ALIASES: ${{ matrix.release.aliases }}
PKGMGR_DISTRO: ${{ matrix.distro }}
run: bash scripts/github/publish-containers/merge-container-images.sh
+22
View File
@@ -0,0 +1,22 @@
# Agent Instructions
## Scope against the skills repository
package-manager holds deterministic code that an operator can run on its own,
without an agent: repository, branch, worktree, release, packaging and
development tooling. Agent coordination (claims, messages, queues, scores,
prompts, session data) belongs to the
[skills repository](https://github.com/kevinveenbirkenbach/skills), whose skills
stay thin wrappers that call `pkgmgr`.
- A new helper that a human could sensibly run alone goes here as a `pkgmgr`
command, with its tests, never as a script inside a skill.
- Output that an agent filters further (claims, busy agents, scores) is
emitted as `--json`; the filtering stays in the skill.
- When the split of a helper is unclear, the operator decides.
## Working copy
The installed `pkgmgr` may run straight from the main checkout, so a
half-edited tree breaks it for everyone. Edit only in a worktree
(`pkgmgr worktree up <branch>`), never in the main checkout.
+92
View File
@@ -1,5 +1,97 @@
# Changelog
## [2.4.0] - 2026-11-07
Added:
- Worktree: *worktree up* and *down* give each branch its own slot
- Worktree: *worktree flow* keeps pull sources and push targets per worktree
- Worktree: *up --pull* and *--push* record the flow; default is the base
- Branch: *branch tree* writes a hardened meta-tree of all repositories
- Branch: the tree carries each worktree's merge flow
- GC: *gc --report* prints a read-only disk inventory
- Review: a branch review renders as a local HTML page
- Publish: releases rebuild nightly; stable follows the tested build
Changed:
- Push: signs only an unsigned HEAD instead of re-signing the whole range
Fixed:
- Push, release, gc: refuse a repository config that would run a program
- Git: hooks named in the config no longer run in hardened git calls
- Tests: the trusted PATH fixture comes from os.defpath, not the host
Upgrade notes:
- Push: move repo-local sshCommand, credential helpers, hooks to global config
## [2.3.0] - 2026-11-01
Added:
- Branch: *gc --merged-into REF* treats items merged into REF as merged
- Branch: *--merged-into* is repeatable, protects REF and works for *purge*
- Branch: *--sizes* adds each worktree's disk usage to the plan
- Branch: *--min-age HOURS* skips items with a commit or index that recent
## [2.2.0] - 2026-11-01
Added:
- Push: *--no-sign* pushes unsigned commits as they are; leases still apply
- Push: *--rewrite* replaces commits a remote already has (amend, squash)
Changed:
- Push: *pkgmgr push* signs every unpushed unsigned commit before pushing
- Push: each pushurl gets its own lease; mirrors no longer fail as stale
- Push: a commit that exists only on a mirror stops the push, nothing lost
- Push: without a selection it pushes the current checkout or worktree
- Release: signs before tagging and pushes branch and tag like *pkgmgr push*
Fixed:
- Container: images build on native amd64 and arm64 runners, not QEMU
Upgrade notes:
- Push: needs a working signing key; pass *--no-sign* to push unsigned
- Container: no 2.1.0 or 2.1.1 image reached GHCR; notes below carry over
- Release: a version file ahead of the newest tag now raises the next version
- Branch: *gc --apply* and *purge* refuse to run inside a PID sandbox
## [2.1.1] - 2026-10-31
Fixed:
- Release: tool-only or dynamic-version pyproject.toml is skipped, no crash
Upgrade notes:
- Container: no 2.1.0 image reached GHCR; the two notes below carry over
- Release: a version file ahead of the newest tag now raises the next version
- Branch: *gc --apply* and *purge* refuse to run inside a PID sandbox
## [2.1.0] - 2026-10-31
Added:
- Branch: *pkgmgr branch gc* removes merged clean branches, worktrees, stacks
- Branch: *gc* only prints its plan; *--apply* acts, *--json* for scripts
- Branch: *pkgmgr branch purge* deletes unmerged or dirty items after y/N
- Release: Cargo.toml version and its Cargo.lock entry follow the release
Changed:
- Release: starts from the highest version in the tag and all version files
Upgrade notes:
- Release: a version file ahead of the newest tag now raises the next version
- Branch: *gc --apply* and *purge* refuse to run inside a PID sandbox
## [2.0.5] - 2026-10-29
Fixed:
+122 -5
View File
@@ -70,6 +70,89 @@ versioning features it can drive **end-to-end workflows**:
---
### Worktrees and garbage collection
| Command | Does |
| --- | --- |
| `pkgmgr worktree up <branch> [--base REF] [--path DIR] [--pull REF ...] [--push BRANCH ...]` | Checks the branch out under `<directories.worktrees>/<host>/<owner>/<repo>/<branch>` or at `--path`, assigns a free slot, records the merge flow and runs the pinned `worktree-init` hook |
| `pkgmgr worktree flow [PATH] [--json]` | Shows the merge flow of the worktree at `PATH` (default: `.`) |
| `pkgmgr worktree flow set [PATH] (--pull REF ... \| --no-pull) (--push BRANCH ... \| --no-push) [--json]` | Replaces both lists of the merge flow |
| `pkgmgr worktree flow clear [PATH] [--json]` | Removes the merge flow |
| `pkgmgr worktree cleanup [--apply]` | Removes clean worktrees whose commits are merged or pushed: pinned `worktree-fini` hook, compose stack, checkout; the branch stays |
| `pkgmgr branch cleanup [--apply]` | Removes merged branches that no worktree holds, locally and on the remotes |
| `pkgmgr branch tree [--format json\|yaml] [--output FILE]` | Writes every managed repository's branches and worktrees with their cleanup state, without fetching |
| `pkgmgr gc [--all] [--apply]` | Both cleanups in one pass, for the current repository or every managed one |
| `pkgmgr gc --report [--all] [--root DIR] [--threshold-mb MB] [--docker] [--json]` | Read-only disk inventory: worktree sizes, changes a cleanup would keep, `--root` children, core dumps, docker usage, free space per filesystem |
| `pkgmgr worktree orphans --stale-days N --top N` | Lists merged and abandoned branches and worktrees with a proposal each |
A hook runs only when the `worktree_hooks` config pins its command under the
`<provider>/<account>/<repository>` of a repository managed in the pkgmgr
config, and only for that repository's directory; origin's URL, which the
repository's own config sets, never selects a pin. `{path}` and `{slot}` are
replaced; a
`worktree-init` or `worktree-fini` target in the repository's Makefile is
reported, never run, because an agent may have written it. The pinned program
must be an absolute path or a root-owned command on a `PATH` entry that root
owns and no group or other user can write, up to `/`, so a binary planted in a
user-writable directory such as a virtualenv never runs. The Nix store never
counts: it is group-writable, and any user can have the daemon build
root-owned content there, so pin an absolute path on Nix. A bare name runs as
the resolved file that was checked. Inside a user namespace other than the
initial one, root means the overflow uid, under which the real root's files
show there, as do those of every other unmapped owner. Outside such a
namespace a directory on a nosuid mount, where udisks2 and FUSE put user
mounts, never counts as root's. The hook runs with
stdin closed, only those `PATH` entries and a 30 minute timeout. Removal walks to
the worktree without following a symbolic link and skips a worktree path that
is not its own realpath. A pinned command that reads repository files, such as `make` or a
script in the repository, trusts the worktree's content, tracked or ignored.
Every cleanup is a dry run until `--apply`, and `--json` prints the plan for
other tools. The [agent skills](https://github.com/kevinveenbirkenbach/skills)
wrap these commands; see [AGENTS.md](AGENTS.md) for the split.
#### Merge flow
Every worktree can carry a merge flow that scripts and skills read: `pull`
lists the refs merged into the branch, in order, before a gate, deploy or
merge; `push` lists the branches the branch is merged into, in order, when it
is finished, each then pushed by the operator. It lives in
`pkgmgr/flow.yml` inside the worktree's own git directory (`.git` of the main
worktree, `<common>/worktrees/<id>` of a linked one), so it is never committed
and goes with the worktree:
```yaml
version: 1
branch: feature/x
pull:
- origin/main
- store-i18n
push:
- feature/iam-profile-rbac
- production
```
`branch` is the branch the flow was set on; another current branch shows as a
warning. `worktree up` records `--pull` and `--push`; without them pull is
the base the worktree came from and push that base's local branch
(`origin/main` gives `main`). `flow set` needs each list or
its `--no-pull`/`--no-push`, and writes atomically into a `pkgmgr/` of mode
0700. Every entry must pass `git check-ref-format --branch` and must not start
with `-`; a remote-tracking name such as `origin/main` passes. An invalid name
exits 2 and writes nothing. A flow file that is a symbolic link, no regular
file, larger than 64 KiB or malformed reads as `invalid`, with empty lists and
a warning, and is never followed.
`flow --json` prints `worktree` and `git_dir` (absolute paths), `branch` (the
current branch, `null` when detached), `flow_branch` (the file's `branch`),
`pull`, `push`, `source` (`file`, `none` or `invalid`) and `warnings`; it
exits 0 also without a flow, 2 when `PATH` is not a git worktree. In
`pkgmgr branch tree` every item carries `flow`: `{"pull", "push", "source"}`,
or `null` without a worktree or a flow; flow warnings join the repository's
`warnings`.
---
## Architecture & Setup Map 🗺️
The following diagram gives a full overview of:
@@ -181,16 +264,50 @@ is a manifest list covering `linux/amd64` and `linux/arm64`.
All images live under `ghcr.io/kevinveenbirkenbach/`:
| Tag | Contains |
| --------------------------- | ---------------------------------- |
| `pkgmgr-<distro>:latest` | newest build for that distribution |
| `pkgmgr-<distro>:<version>` | one release |
| `pkgmgr-<distro>:stable` | newest release marked stable |
| Tag | Contains |
| ------------------------------------- | --------------------------------------------------------- |
| `pkgmgr-<distro>:latest` | newest build of the newest release on `main` |
| `pkgmgr-<distro>:stable` | newest build of the release the git tag `stable` marks |
| `pkgmgr-<distro>:<major>` | newest build of the newest release of that major |
| `pkgmgr-<distro>:<major>.<minor>` | newest build of the newest release of that minor |
| `pkgmgr-<distro>:<version>` | newest build of that release |
| `pkgmgr-<distro>:<version>.<unix-ts>` | one build, never moves |
`<distro>` is one of `arch`, `debian`, `ubuntu`, `fedora`, `centos`. The Arch
image is published without the suffix as well, so `pkgmgr:stable` and
`pkgmgr-arch:stable` are the same image.
`<version>` is the PKGMGR release and `<unix-ts>` the build time, so
`2.3.0.1767225600` is a full version an updater can order and pin. Every tag of
one build points at the same image.
`stable` moves only once "Mark stable commit" has moved the git tag `stable`,
which it does after every test of that release passed. Every day at 03:00 UTC,
after the daily 00:00 UTC build of base-images, and on demand through the
workflow's manual trigger, two releases are rebuilt on the newest base image:
- the newest `vX.Y.Z` release on `main`, moving `latest`, its prefixes and
`<version>`, and adding `<version>.<unix-ts>`;
- the release the git tag `stable` marks, when that is an older one, moving
`stable` and `<version>`, adding its own `<version>.<unix-ts>`, and moving a
prefix only while it is still the newest release of that prefix.
With newest `2.4.0` and stable `2.3.0`, `2.4.0` takes `latest`, `2` and `2.4`,
and `2.3.0` takes `stable` and `2.3`. A security update in base-images thereby
reaches whoever pinned a major, a minor, a release or follows `stable`.
Each release is compared with its own published `<version>`. A rebuild whose
installed packages equal it pushes nothing, since a new timestamp tag would make
every pin bump for no change; a tag of that release that does not show the
published build yet, such as `stable` right after "Mark stable commit", is moved
without a new timestamp.
A rebuild runs the build scripts of the release it rebuilds, not those of
`main`, so the skip and the tag rules above only hold for releases tagged after
this scheme landed. The scripts of an earlier release know no aliases: its merge
step fails and leaves its tags where they are, until a newer release replaces
it as the newest and the stable one.
For a clean environment that carries the system dependencies but no PKGMGR,
pull `base-<distro>` from
[base-images](https://github.com/kevinveenbirkenbach/base-images) directly.
+1 -1
View File
@@ -32,7 +32,7 @@
rec {
pkgmgr = pyPkgs.buildPythonApplication {
pname = "package-manager";
version = "2.0.5";
version = "2.4.0";
# Use the git repo as source
src = ./.;
+1 -1
View File
@@ -1,7 +1,7 @@
# Maintainer: Kevin Veen-Birkenbach <info@veen.world>
pkgname=package-manager
pkgver=2.0.5
pkgver=2.4.0
pkgrel=1
pkgdesc="Local-flake wrapper for Kevin's package-manager (Nix-based)."
arch=('any')
+102
View File
@@ -1,3 +1,105 @@
package-manager (2.4.0-1) unstable; urgency=medium
Added:
- Worktree: *worktree up* and *down* give each branch its own slot
- Worktree: *worktree flow* keeps pull sources and push targets per worktree
- Worktree: *up --pull* and *--push* record the flow; default is the base
- Branch: *branch tree* writes a hardened meta-tree of all repositories
- Branch: the tree carries each worktree's merge flow
- GC: *gc --report* prints a read-only disk inventory
- Review: a branch review renders as a local HTML page
- Publish: releases rebuild nightly; stable follows the tested build
Changed:
- Push: signs only an unsigned HEAD instead of re-signing the whole range
Fixed:
- Push, release, gc: refuse a repository config that would run a program
- Git: hooks named in the config no longer run in hardened git calls
- Tests: the trusted PATH fixture comes from os.defpath, not the host
Upgrade notes:
- Push: move repo-local sshCommand, credential helpers, hooks to global config
-- Kevin Veen-Birkenbach <kevin@veen.world> Sat, 07 Nov 2026 15:45:28 +0000
package-manager (2.3.0-1) unstable; urgency=medium
Added:
- Branch: *gc --merged-into REF* treats items merged into REF as merged
- Branch: *--merged-into* is repeatable, protects REF and works for *purge*
- Branch: *--sizes* adds each worktree's disk usage to the plan
- Branch: *--min-age HOURS* skips items with a commit or index that recent
-- Kevin Veen-Birkenbach <kevin@veen.world> Sun, 01 Nov 2026 15:14:53 +0100
package-manager (2.2.0-1) unstable; urgency=medium
Added:
- Push: *--no-sign* pushes unsigned commits as they are; leases still apply
- Push: *--rewrite* replaces commits a remote already has (amend, squash)
Changed:
- Push: *pkgmgr push* signs every unpushed unsigned commit before pushing
- Push: each pushurl gets its own lease; mirrors no longer fail as stale
- Push: a commit that exists only on a mirror stops the push, nothing lost
- Push: without a selection it pushes the current checkout or worktree
- Release: signs before tagging and pushes branch and tag like *pkgmgr push*
Fixed:
- Container: images build on native amd64 and arm64 runners, not QEMU
Upgrade notes:
- Push: needs a working signing key; pass *--no-sign* to push unsigned
- Container: no 2.1.0 or 2.1.1 image reached GHCR; notes below carry over
- Release: a version file ahead of the newest tag now raises the next version
- Branch: *gc --apply* and *purge* refuse to run inside a PID sandbox
-- Kevin Veen-Birkenbach <kevin@veen.world> Sun, 01 Nov 2026 01:55:29 +0100
package-manager (2.1.1-1) unstable; urgency=medium
Fixed:
- Release: tool-only or dynamic-version pyproject.toml is skipped, no crash
Upgrade notes:
- Container: no 2.1.0 image reached GHCR; the two notes below carry over
- Release: a version file ahead of the newest tag now raises the next version
- Branch: *gc --apply* and *purge* refuse to run inside a PID sandbox
-- Kevin Veen-Birkenbach <kevin@veen.world> Sat, 31 Oct 2026 18:47:35 +0100
package-manager (2.1.0-1) unstable; urgency=medium
Added:
- Branch: *pkgmgr branch gc* removes merged clean branches, worktrees, stacks
- Branch: *gc* only prints its plan; *--apply* acts, *--json* for scripts
- Branch: *pkgmgr branch purge* deletes unmerged or dirty items after y/N
- Release: Cargo.toml version and its Cargo.lock entry follow the release
Changed:
- Release: starts from the highest version in the tag and all version files
Upgrade notes:
- Release: a version file ahead of the newest tag now raises the next version
- Branch: *gc --apply* and *purge* refuse to run inside a PID sandbox
-- Kevin Veen-Birkenbach <kevin@veen.world> Sat, 31 Oct 2026 14:38:54 +0100
package-manager (2.0.5-1) unstable; urgency=medium
Fixed:
+88 -1
View File
@@ -1,5 +1,5 @@
Name: package-manager
Version: 2.0.5
Version: 2.4.0
Release: 1%{?dist}
Summary: Wrapper that runs Kevin's package-manager via Nix flake
@@ -74,6 +74,93 @@ echo ">>> package-manager removed. Nix itself was not removed."
/usr/lib/package-manager/
%changelog
* Sat Nov 07 2026 Kevin Veen-Birkenbach <kevin@veen.world> - 2.4.0-1
- Added:
- - Worktree: *worktree up* and *down* give each branch its own slot
- - Worktree: *worktree flow* keeps pull sources and push targets per worktree
- - Worktree: *up --pull* and *--push* record the flow; default is the base
- - Branch: *branch tree* writes a hardened meta-tree of all repositories
- - Branch: the tree carries each worktree's merge flow
- - GC: *gc --report* prints a read-only disk inventory
- - Review: a branch review renders as a local HTML page
- - Publish: releases rebuild nightly; stable follows the tested build
- Changed:
- - Push: signs only an unsigned HEAD instead of re-signing the whole range
- Fixed:
- - Push, release, gc: refuse a repository config that would run a program
- - Git: hooks named in the config no longer run in hardened git calls
- - Tests: the trusted PATH fixture comes from os.defpath, not the host
- Upgrade notes:
- - Push: move repo-local sshCommand, credential helpers, hooks to global config
* Sun Nov 01 2026 Kevin Veen-Birkenbach <kevin@veen.world> - 2.3.0-1
- Added:
- - Branch: *gc --merged-into REF* treats items merged into REF as merged
- - Branch: *--merged-into* is repeatable, protects REF and works for *purge*
- - Branch: *--sizes* adds each worktree's disk usage to the plan
- - Branch: *--min-age HOURS* skips items with a commit or index that recent
* Sun Nov 01 2026 Kevin Veen-Birkenbach <kevin@veen.world> - 2.2.0-1
- Added:
- - Push: *--no-sign* pushes unsigned commits as they are; leases still apply
- - Push: *--rewrite* replaces commits a remote already has (amend, squash)
- Changed:
- - Push: *pkgmgr push* signs every unpushed unsigned commit before pushing
- - Push: each pushurl gets its own lease; mirrors no longer fail as stale
- - Push: a commit that exists only on a mirror stops the push, nothing lost
- - Push: without a selection it pushes the current checkout or worktree
- - Release: signs before tagging and pushes branch and tag like *pkgmgr push*
- Fixed:
- - Container: images build on native amd64 and arm64 runners, not QEMU
- Upgrade notes:
- - Push: needs a working signing key; pass *--no-sign* to push unsigned
- - Container: no 2.1.0 or 2.1.1 image reached GHCR; notes below carry over
- - Release: a version file ahead of the newest tag now raises the next version
- - Branch: *gc --apply* and *purge* refuse to run inside a PID sandbox
* Sat Oct 31 2026 Kevin Veen-Birkenbach <kevin@veen.world> - 2.1.1-1
- Fixed:
- - Release: tool-only or dynamic-version pyproject.toml is skipped, no crash
- Upgrade notes:
- - Container: no 2.1.0 image reached GHCR; the two notes below carry over
- - Release: a version file ahead of the newest tag now raises the next version
- - Branch: *gc --apply* and *purge* refuse to run inside a PID sandbox
* Sat Oct 31 2026 Kevin Veen-Birkenbach <kevin@veen.world> - 2.1.0-1
- Added:
- - Branch: *pkgmgr branch gc* removes merged clean branches, worktrees, stacks
- - Branch: *gc* only prints its plan; *--apply* acts, *--json* for scripts
- - Branch: *pkgmgr branch purge* deletes unmerged or dirty items after y/N
- - Release: Cargo.toml version and its Cargo.lock entry follow the release
- Changed:
- - Release: starts from the highest version in the tag and all version files
- Upgrade notes:
- - Release: a version file ahead of the newest tag now raises the next version
- - Branch: *gc --apply* and *purge* refuse to run inside a PID sandbox
* Thu Oct 29 2026 Kevin Veen-Birkenbach <kevin@veen.world> - 2.0.5-1
- Fixed:
+3 -2
View File
@@ -7,7 +7,7 @@ build-backend = "setuptools.build_meta"
[project]
name = "kpmx"
version = "2.0.5"
version = "2.4.0"
description = "Kevin's package-manager tool (pkgmgr)"
readme = "README.md"
requires-python = ">=3.9"
@@ -21,7 +21,7 @@ authors = [
dependencies = [
"PyYAML>=6.0",
"tomli; python_version < \"3.11\"",
"jinja2>=3.1"
"jinja2>=3.1.6"
]
[project.urls]
@@ -52,3 +52,4 @@ include = ["pkgmgr*"]
[tool.setuptools.package-data]
"pkgmgr.config" = ["*.yml", "*.yaml"]
"pkgmgr.actions.review" = ["*.j2"]
+21
View File
@@ -30,3 +30,24 @@ resolve_platforms() {
*) echo "ERROR: Unknown distro '$PKGMGR_DISTRO'" >&2; exit 1 ;;
esac
}
package_set_cmd() {
local PKGMGR_DISTRO="$1"
case "$PKGMGR_DISTRO" in
arch|manjaro) echo 'pacman -Q' ;;
debian|ubuntu) echo 'dpkg-query -W' ;;
fedora|centos) echo 'rpm -qa | sort' ;;
*) echo "ERROR: Unknown distro '$PKGMGR_DISTRO'" >&2; exit 1 ;;
esac
}
version_prefixes() {
local -a parts
IFS=. read -ra parts <<< "$1"
local acc="${parts[0]}" i
echo "${acc}"
for ((i = 1; i < ${#parts[@]}; i++)); do
acc+=".${parts[i]}"
echo "${acc}"
done
}
+136 -32
View File
@@ -12,12 +12,15 @@ NO_CACHE=0
MISSING_ONLY=0
IMAGE_TAG="" # local image name or base tag (without registry)
PUSH=0 # if 1 -> use buildx and push (requires docker buildx)
PUBLISH=0 # if 1 -> push with semantic tags (latest/version/stable + arch aliases)
PUBLISH=0 # if 1 -> push <version>, <version>.<unix-ts> and the --aliases, plus arch aliases
REGISTRY="" # e.g. ghcr.io
OWNER="" # e.g. github org/user
REPO_PREFIX="pkgmgr" # image base name (pkgmgr)
VERSION="" # X.Y.Z (required for --publish)
IS_STABLE="false" # "true" -> publish stable tags
ALIASES="" # moving tags besides <version>, e.g. "latest 2 2.4 stable"
ALIASES_SET=0 # 1 once --aliases was given, also when empty
PLATFORM="" # e.g. linux/arm64 -> build and push only this platform
MERGE=0 # if 1 -> join the per-platform tags of --version under the publish tags
DEFAULT_DISTRO="arch"
usage() {
@@ -33,15 +36,24 @@ Build options:
Publish options:
--push Push the built image (uses docker buildx build --push)
--publish Publish semantic tags (latest, <version>, optional stable) + arch aliases
--publish Publish <version>, <version>.<unix-ts> and every --aliases tag, plus arch
aliases
--registry <reg> Registry (e.g. ghcr.io)
--owner <owner> Registry namespace (e.g. \${GITHUB_REPOSITORY_OWNER})
--repo-prefix <name> Image base name (default: pkgmgr)
--version <X.Y.Z> Version for --publish
--stable <true|false> Whether to publish :stable tags (default: false)
--aliases "<tags>" Moving tags besides <version>, space separated, may be empty
(e.g. "latest 2 2.4 stable")
--platform <os/arch> Build and push only this platform; with --publish as <version>-<arch>,
skipped when its package set equals the published <version>
--merge Join the <version>-<arch> tags of all platforms under the publish tags;
when <version> already joins exactly these images, only the aliases
that do not are moved and no <version>.<unix-ts> is added
Notes:
- --publish implies --push and requires --registry, --owner, and --version.
- --publish without --platform requires --aliases.
- --platform requires --push; --merge requires --publish and builds nothing.
- Local build (no --push) uses "docker build" and creates a local image like "pkgmgr-arch".
EOF
}
@@ -77,11 +89,18 @@ while [[ $# -gt 0 ]]; do
[[ -n "${VERSION}" ]] || { echo "ERROR: --version requires a value"; exit 2; }
shift 2
;;
--stable)
IS_STABLE="${2:-}"
[[ -n "${IS_STABLE}" ]] || { echo "ERROR: --stable requires a value (true|false)"; exit 2; }
--aliases)
[[ $# -ge 2 ]] || { echo "ERROR: --aliases requires a value, empty for none"; exit 2; }
ALIASES="$2"
ALIASES_SET=1
shift 2
;;
--platform)
PLATFORM="${2:-}"
[[ -n "${PLATFORM}" ]] || { echo "ERROR: --platform requires a value"; exit 2; }
shift 2
;;
--merge) MERGE=1; shift ;;
-h|--help) usage; exit 0 ;;
*)
echo "ERROR: Unknown argument: $1" >&2
@@ -115,8 +134,24 @@ if [[ "${PUBLISH}" == "1" ]]; then
[[ -n "${REGISTRY}" ]] || { echo "ERROR: --publish requires --registry"; exit 2; }
[[ -n "${OWNER}" ]] || { echo "ERROR: --publish requires --owner"; exit 2; }
[[ -n "${VERSION}" ]] || { echo "ERROR: --publish requires --version"; exit 2; }
[[ -n "${PLATFORM}" || "${ALIASES_SET}" == "1" ]] \
|| { echo "ERROR: --publish without --platform requires --aliases"; exit 2; }
fi
if [[ -n "${PLATFORM}" ]]; then
[[ "${PUSH}" == "1" ]] || { echo "ERROR: --platform requires --push or --publish"; exit 2; }
[[ ",$(resolve_platforms "$PKGMGR_DISTRO")," == *",${PLATFORM},"* ]] \
|| { echo "ERROR: ${PKGMGR_DISTRO} offers no platform ${PLATFORM}"; exit 2; }
fi
if [[ "${MERGE}" == "1" ]]; then
[[ "${PUBLISH}" == "1" ]] || { echo "ERROR: --merge requires --publish"; exit 2; }
[[ -z "${PLATFORM}" ]] || { echo "ERROR: --merge and --platform exclude each other"; exit 2; }
fi
DISTRO_TAG_BASE="${REGISTRY}/${OWNER}/${REPO_PREFIX}-${PKGMGR_DISTRO}"
STAMP="$(date -u +%s)"
# Guard: --push without --publish requires fully-qualified --tag
if [[ "${PUSH}" == "1" && "${PUBLISH}" != "1" ]]; then
if [[ "${IMAGE_TAG}" != */* ]]; then
@@ -130,7 +165,7 @@ echo "------------------------------------------------------------"
echo "[build] Building image"
echo "distro = ${PKGMGR_DISTRO}"
echo "BASE_IMAGE = ${BASE_IMAGE}"
echo "platforms = $(resolve_platforms "$PKGMGR_DISTRO")"
echo "platforms = ${PLATFORM:-$(resolve_platforms "$PKGMGR_DISTRO")}"
if [[ "${NO_CACHE}" == "1" ]]; then echo "cache = disabled"; fi
if [[ "${PUSH}" == "1" ]]; then echo "push = enabled"; fi
if [[ "${PUBLISH}" == "1" ]]; then
@@ -138,7 +173,8 @@ if [[ "${PUBLISH}" == "1" ]]; then
echo "registry = ${REGISTRY}"
echo "owner = ${OWNER}"
echo "version = ${VERSION}"
echo "stable = ${IS_STABLE}"
echo "stamp = ${STAMP}"
echo "aliases = ${ALIASES}"
fi
echo "------------------------------------------------------------"
@@ -149,45 +185,113 @@ if [[ "${NO_CACHE}" == "1" ]]; then
build_args+=(--no-cache)
fi
compute_publish_tags() {
local distro_tag_base="${REGISTRY}/${OWNER}/${REPO_PREFIX}-${PKGMGR_DISTRO}"
local alias_tag_base=""
publish_bases() {
echo "${DISTRO_TAG_BASE}"
if [[ "${PKGMGR_DISTRO}" == "${DEFAULT_DISTRO}" ]]; then
alias_tag_base="${REGISTRY}/${OWNER}/${REPO_PREFIX}"
echo "${REGISTRY}/${OWNER}/${REPO_PREFIX}"
fi
}
moving_tags() {
local base alias
local -a aliases
read -ra aliases <<< "${ALIASES}"
while read -r base; do
echo "${base}:${VERSION}"
for alias in "${aliases[@]}"; do
echo "${base}:${alias}"
done
done < <(publish_bases)
}
stamp_tags() {
local base
while read -r base; do
echo "${base}:${VERSION}.${STAMP}"
done < <(publish_bases)
}
raw_index() {
docker buildx imagetools inspect --raw "$1" 2>/dev/null || true
}
digest_of() {
docker buildx imagetools inspect --format '{{json .Manifest.Digest}}' "$1"
}
if [[ "${MERGE}" == "1" ]]; then
sources=()
IFS=',' read -ra platforms <<< "$(resolve_platforms "$PKGMGR_DISTRO")"
for p in "${platforms[@]}"; do
sources+=("${DISTRO_TAG_BASE}:${VERSION}-${p#*/}")
done
mapfile -t moving < <(moving_tags)
planned="$(docker buildx imagetools create --dry-run -t "${DISTRO_TAG_BASE}:${VERSION}" "${sources[@]}")"
tags=()
if [[ "$(raw_index "${DISTRO_TAG_BASE}:${VERSION}")" == "${planned}" ]]; then
for t in "${moving[@]}"; do
if [[ "$(raw_index "${t}")" != "${planned}" ]]; then
tags+=("${t}")
fi
done
else
mapfile -t stamps < <(stamp_tags)
tags=("${moving[@]}" "${stamps[@]}")
fi
local tags=()
tags+=("${distro_tag_base}:latest")
tags+=("${distro_tag_base}:${VERSION}")
if [[ "${IS_STABLE}" == "true" ]]; then
tags+=("${distro_tag_base}:stable")
if ((${#tags[@]} == 0)); then
echo "[build] ${PKGMGR_DISTRO} ${VERSION}: every tag already joins these platform images, nothing to push."
exit 0
fi
if [[ -n "${alias_tag_base}" ]]; then
tags+=("${alias_tag_base}:latest")
tags+=("${alias_tag_base}:${VERSION}")
if [[ "${IS_STABLE}" == "true" ]]; then
tags+=("${alias_tag_base}:stable")
create_args=()
for t in "${tags[@]}"; do
create_args+=(-t "$t")
done
echo "[build] Running: docker buildx imagetools create ${create_args[*]} ${sources[*]}"
docker buildx imagetools create "${create_args[@]}" "${sources[@]}"
version_digest="$(digest_of "${DISTRO_TAG_BASE}:${VERSION}")"
for t in "${moving[@]}"; do
if [[ "$(digest_of "${t}")" != "${version_digest}" ]]; then
echo "[build] ❌ ${PKGMGR_DISTRO}: :${VERSION} and ${t} point at different images." >&2
echo "[build] Whoever pinned ${t##*:} stopped receiving what ${VERSION} gets." >&2
exit 1
fi
done
echo "[build] ✅ ${PKGMGR_DISTRO}: every tag of ${VERSION} == ${version_digest}"
elif [[ "${PUSH}" == "1" ]]; then
if [[ "${PUBLISH}" == "1" && -n "${PLATFORM}" ]]; then
probe="${REPO_PREFIX}-${PKGMGR_DISTRO}:probe"
docker buildx build --load --platform "${PLATFORM}" -t "${probe}" "${build_args[@]}" .
package_set="$(package_set_cmd "${PKGMGR_DISTRO}")"
built_set="$(docker run --rm --platform "${PLATFORM}" --entrypoint bash "${probe}" -lc "${package_set}" | sha256sum)"
published_set="$(docker run --rm --pull always --platform "${PLATFORM}" --entrypoint bash "${DISTRO_TAG_BASE}:${VERSION}" -lc "${package_set}" 2>/dev/null | sha256sum || true)"
if [[ "${built_set}" == "${published_set}" ]]; then
echo "[build] ${PKGMGR_DISTRO} ${PLATFORM} unchanged since the published ${VERSION}, nothing to push."
exit 0
fi
fi
printf '%s\n' "${tags[@]}"
}
if [[ "${PUSH}" == "1" ]]; then
bx_args=(docker buildx build --push)
if [[ "${PUBLISH}" == "1" ]]; then
if [[ "${PUBLISH}" == "1" && -n "${PLATFORM}" ]]; then
bx_args+=(-t "${DISTRO_TAG_BASE}:${VERSION}-${PLATFORM#*/}")
elif [[ "${PUBLISH}" == "1" ]]; then
while IFS= read -r t; do
bx_args+=(-t "$t")
done < <(compute_publish_tags)
done < <(moving_tags)
while IFS= read -r t; do
bx_args+=(-t "$t")
done < <(stamp_tags)
else
bx_args+=(-t "${IMAGE_TAG}")
fi
bx_args+=(--platform "$(resolve_platforms "$PKGMGR_DISTRO")")
bx_args+=(--platform "${PLATFORM:-$(resolve_platforms "$PKGMGR_DISTRO")}")
bx_args+=("${build_args[@]}")
bx_args+=(.)
+18 -4
View File
@@ -6,25 +6,28 @@ set -euo pipefail
# Required env:
# OWNER (e.g. GITHUB_REPOSITORY_OWNER)
# VERSION (e.g. 1.2.3)
# ALIASES moving tags besides VERSION, space separated, may be empty (e.g. "latest 1 1.2 stable")
#
# Optional env:
# REGISTRY (default: ghcr.io)
# IS_STABLE (default: false)
# DISTROS (default: "arch debian ubuntu fedora centos")
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
# shellcheck source=./scripts/build/base.sh
source "${SCRIPT_DIR}/base.sh"
REGISTRY="${REGISTRY:-ghcr.io}"
IS_STABLE="${IS_STABLE:-false}"
DISTROS="${DISTROS:-arch debian ubuntu fedora centos}"
: "${OWNER:?Environment variable OWNER must be set (e.g. github.repository_owner)}"
: "${VERSION:?Environment variable VERSION must be set (e.g. 1.2.3)}"
: "${ALIASES?Environment variable ALIASES must be set, empty for none (e.g. \"latest 1 1.2 stable\")}"
echo "[publish] REGISTRY=${REGISTRY}"
echo "[publish] OWNER=${OWNER}"
echo "[publish] VERSION=${VERSION}"
echo "[publish] IS_STABLE=${IS_STABLE}"
echo "[publish] ALIASES=${ALIASES}"
echo "[publish] DISTROS=${DISTROS}"
for d in ${DISTROS}; do
@@ -33,12 +36,23 @@ for d in ${DISTROS}; do
echo "[publish] PKGMGR_DISTRO=${d}"
echo "============================================================"
IFS=',' read -ra platforms <<< "$(resolve_platforms "${d}")"
for p in "${platforms[@]}"; do
PKGMGR_DISTRO="${d}" bash "${SCRIPT_DIR}/image.sh" \
--publish \
--registry "${REGISTRY}" \
--owner "${OWNER}" \
--version "${VERSION}" \
--platform "${p}"
done
PKGMGR_DISTRO="${d}" bash "${SCRIPT_DIR}/image.sh" \
--publish \
--registry "${REGISTRY}" \
--owner "${OWNER}" \
--version "${VERSION}" \
--stable "${IS_STABLE}"
--aliases "${ALIASES}" \
--merge
done
echo
@@ -1,8 +0,0 @@
#!/usr/bin/env bash
set -euo pipefail
WORKFLOW_RUN_SHA="${WORKFLOW_RUN_SHA:?WORKFLOW_RUN_SHA must be set}"
git checkout -f "${WORKFLOW_RUN_SHA}"
git fetch --tags --force
git tag --list 'stable' 'v*' --sort=version:refname | tail -n 20
@@ -1,23 +1,91 @@
#!/usr/bin/env bash
# Param: WORKFLOW_RUN_SHA commit the "Mark stable commit" run tagged; empty on a scheduled or
# manual rebuild, which publishes the newest vX.Y.Z release on origin/main
# and the release the git tag stable marks
# Output: releases=<JSON list of {"sha", "version", "aliases"}> in GITHUB_OUTPUT; aliases are the
# moving tags a release takes besides <version> and <version>.<unix-ts>
set -euo pipefail
SHA="$(git rev-parse HEAD)"
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
V_TAG="$(git tag --points-at "${SHA}" --list 'v*' | sort -V | tail -n1)"
if [[ -z "${V_TAG}" ]]; then
echo "No version tag found for ${SHA}. Skipping publish."
echo "should_publish=false" >> "$GITHUB_OUTPUT"
exit 0
# shellcheck source=scripts/build/base.sh
source "${SCRIPT_DIR}/../../build/base.sh"
: "${WORKFLOW_RUN_SHA?WORKFLOW_RUN_SHA must be set, empty for a rebuild}"
: "${GITHUB_OUTPUT:?GITHUB_OUTPUT must be set}"
RELEASE_TAG='^v[0-9]+\.[0-9]+\.[0-9]+$'
release_at() {
local listed tag
listed="$(git tag --points-at "$1" --sort=-v:refname)"
tag="$(grep -m1 -E "${RELEASE_TAG}" <<< "${listed}" || true)"
echo "${tag#v}"
}
newest_of() {
local prefix="$1" version
for version in "${RELEASES[@]}"; do
if [[ "${version}." == "${prefix}."* ]]; then
echo "${version}"
return 0
fi
done
}
aliases_of() {
local version="$1" prefix
local -a aliases=()
if [[ "${version}" == "${RELEASES[0]}" ]]; then
aliases+=(latest)
fi
while read -r prefix; do
if [[ "${prefix}" != "${version}" && "$(newest_of "${prefix}")" == "${version}" ]]; then
aliases+=("${prefix}")
fi
done < <(version_prefixes "${version}")
if [[ "${version}" == "${STABLE}" ]]; then
aliases+=(stable)
fi
echo "${aliases[*]}"
}
listed="$(git tag --merged origin/main --sort=-v:refname)"
mapfile -t tags < <(grep -E "${RELEASE_TAG}" <<< "${listed}")
RELEASES=("${tags[@]#v}")
STABLE=""
stable_sha="$(git rev-parse -q --verify 'refs/tags/stable^{commit}' || true)"
if [[ -n "${stable_sha}" ]]; then
STABLE="$(release_at "${stable_sha}")"
fi
VERSION="${V_TAG#v}"
targets=()
if [[ -n "${WORKFLOW_RUN_SHA}" ]]; then
version="$(release_at "${WORKFLOW_RUN_SHA}")"
if ! git merge-base --is-ancestor "${WORKFLOW_RUN_SHA}" origin/main; then
echo "${WORKFLOW_RUN_SHA} is not on origin/main. Skipping publish."
elif [[ -z "${version}" ]]; then
echo "No vX.Y.Z tag found for ${WORKFLOW_RUN_SHA}. Skipping publish."
else
targets+=("${version}")
fi
else
((${#RELEASES[@]} > 0)) || { echo "No vX.Y.Z tag is reachable from origin/main." >&2; exit 1; }
targets+=("${RELEASES[0]}")
if [[ -z "${STABLE}" ]]; then
echo "The git tag stable marks no vX.Y.Z release; the stable image stays where it is."
elif [[ "${STABLE}" != "${RELEASES[0]}" ]]; then
targets+=("${STABLE}")
fi
fi
STABLE_SHA="$(git rev-parse -q --verify 'refs/tags/stable^{commit}' 2>/dev/null || true)"
IS_STABLE=false
[[ -n "${STABLE_SHA}" && "${STABLE_SHA}" == "${SHA}" ]] && IS_STABLE=true
json=""
for version in "${targets[@]}"; do
sha="$(git rev-parse "v${version}^{commit}")"
moving="$(aliases_of "${version}")"
echo "Publishing ${version} at ${sha}, moving: ${moving}"
json+="${json:+,}{\"sha\":\"${sha}\",\"version\":\"${version}\",\"aliases\":\"${moving}\"}"
done
{
echo "should_publish=true"
echo "version=${VERSION}"
echo "is_stable=${IS_STABLE}"
} >> "$GITHUB_OUTPUT"
echo "releases=[${json}]" >> "${GITHUB_OUTPUT}"
@@ -0,0 +1,15 @@
#!/usr/bin/env bash
set -euo pipefail
: "${OWNER:?OWNER must be set}"
: "${VERSION:?VERSION must be set}"
: "${ALIASES?ALIASES must be set, empty for none}"
: "${PKGMGR_DISTRO:?PKGMGR_DISTRO must be set}"
bash scripts/build/image.sh \
--publish \
--registry ghcr.io \
--owner "${OWNER}" \
--version "${VERSION}" \
--aliases "${ALIASES}" \
--merge
@@ -3,6 +3,12 @@ set -euo pipefail
: "${OWNER:?OWNER must be set}"
: "${VERSION:?VERSION must be set}"
: "${IS_STABLE:?IS_STABLE must be set}"
: "${PKGMGR_DISTRO:?PKGMGR_DISTRO must be set}"
: "${PLATFORM:?PLATFORM must be set}"
bash scripts/build/publish.sh
bash scripts/build/image.sh \
--publish \
--registry ghcr.io \
--owner "${OWNER}" \
--version "${VERSION}" \
--platform "${PLATFORM}"
+6
View File
@@ -4,10 +4,16 @@ Public API for branch actions.
from .close_branch import close_branch
from .drop_branch import drop_branch
from .gc import branch_purge, cleanup
from .open_branch import open_branch
from .tree import branch_tree, refresh_tree_entry
__all__ = [
"branch_purge",
"branch_tree",
"cleanup",
"close_branch",
"drop_branch",
"open_branch",
"refresh_tree_entry",
]
File diff suppressed because it is too large. Load diff
+116
View File
@@ -0,0 +1,116 @@
from __future__ import annotations
import json
from dataclasses import dataclass, field
from pkgmgr.actions.mirror.url_utils import parse_repo_from_git_url
from pkgmgr.core.credentials.resolver import ResolutionOptions, TokenResolver
from pkgmgr.core.credentials.types import CredentialError
from pkgmgr.core.remote_provisioning.http.client import HttpClient
from pkgmgr.core.remote_provisioning.http.errors import HttpError
GITHUB_HOST = "github.com"
GITHUB_API = "https://api.github.com"
PAGE_SIZE = 100
@dataclass
class ForgeFacts:
"""
Pull-request and protection facts of every GitHub remote of one repository.
Attributes:
merged_heads: head SHA of a merged PR -> PR number.
open_heads: head branch name of an open PR -> PR number.
protected: branch names protected on any of the remotes.
unavailable: why the facts could not be fetched, or None.
"""
merged_heads: dict[str, int] = field(default_factory=dict)
open_heads: dict[str, int] = field(default_factory=dict)
protected: set[str] = field(default_factory=set)
unavailable: str | None = None
def github_slugs(remote_urls: dict[str, str]) -> list[tuple[str, str]]:
slugs: list[tuple[str, str]] = []
for url in remote_urls.values():
host, owner, name = parse_repo_from_git_url(url)
if (
host.lower() == GITHUB_HOST
and owner
and name
and (owner, name) not in slugs
):
slugs.append((owner, name))
return slugs
def _pages(http: HttpClient, headers: dict[str, str], url: str) -> list[dict]:
items: list[dict] = []
page = 1
while True:
resp = http.request_json(
"GET", f"{url}&per_page={PAGE_SIZE}&page={page}", headers=headers
)
batch = json.loads(resp.text or "[]")
items.extend(batch)
if len(batch) < PAGE_SIZE:
return items
page += 1
def fetch_forge_facts(remote_urls: dict[str, str]) -> ForgeFacts:
facts = ForgeFacts()
slugs = github_slugs(remote_urls)
if not slugs:
return facts
try:
token = (
TokenResolver()
.get_token(
"github",
GITHUB_HOST,
options=ResolutionOptions(interactive=False, allow_prompt=False),
)
.token
)
except CredentialError as exc:
facts.unavailable = f"no GitHub token ({exc})"
return facts
return facts_with_token(slugs, token)
def facts_with_token(slugs: list[tuple[str, str]], token: str) -> ForgeFacts:
"""
Fetch the pull-request and protection facts of GitHub repositories.
Args:
slugs: (owner, name) of each repository.
token: the GitHub token sent as bearer.
Returns:
The facts, or facts whose unavailable names the failing repository.
"""
facts = ForgeFacts()
http = HttpClient(timeout_s=30)
headers = {
"Authorization": f"Bearer {token}",
"Accept": "application/vnd.github+json",
"User-Agent": "pkgmgr",
}
for owner, name in slugs:
base = f"{GITHUB_API}/repos/{owner}/{name}"
try:
for pr in _pages(http, headers, f"{base}/pulls?state=all"):
if pr["merged_at"]:
facts.merged_heads[pr["head"]["sha"]] = pr["number"]
elif pr["state"] == "open":
facts.open_heads[pr["head"]["ref"]] = pr["number"]
for branch in _pages(http, headers, f"{base}/branches?protected=true"):
facts.protected.add(branch["name"])
except (HttpError, ValueError, KeyError, TypeError) as exc:
facts.unavailable = f"GitHub API failed for {owner}/{name}: {exc}"
return facts
return facts
+44
View File
@@ -0,0 +1,44 @@
from __future__ import annotations
import os
import shutil
DIR_FLAGS = os.O_RDONLY | os.O_DIRECTORY | os.O_NOFOLLOW
def canonical(path: str) -> bool:
"""
Args:
path: a path as git or a plan names it.
Returns:
True when path is absolute and already its own realpath: no symbolic
link and no `..` component that the kernel could resolve elsewhere.
"""
return os.path.isabs(path) and os.path.realpath(path) == path
def remove_tree(path: str) -> None:
"""
Delete a directory tree without following a symbolic link anywhere on
the way to it, even one swapped in after the caller checked the path.
Args:
path: absolute, canonical directory to delete.
Raises:
OSError: a component of path is a symbolic link or missing, or the
deletion failed.
"""
parent, name = os.path.split(path)
fd = os.open(os.sep, os.O_RDONLY | os.O_DIRECTORY)
try:
for component in [c for c in parent.split(os.sep) if c]:
child = os.open(component, DIR_FLAGS, dir_fd=fd)
os.close(fd)
fd = child
# Trip-wire: rmtree(dir_fd=) needs Python 3.11; the fd link pins the
# opened parent on every version, and rmtree refuses a symlinked name.
shutil.rmtree(f"/proc/self/fd/{fd}/{name}")
finally:
os.close(fd)
+372
View File
@@ -0,0 +1,372 @@
from __future__ import annotations
import fcntl
import json
import os
import re
import sys
import tempfile
import time
from collections.abc import Iterator
from contextlib import contextmanager
from dataclasses import asdict
from datetime import datetime, timezone
import yaml
from pkgmgr.actions.worktree import flow
from pkgmgr.core.credentials.store_keys import build_keyring_key
from pkgmgr.core.git.errors import GitBaseError
from . import gc
from .gc_forge import GITHUB_HOST, ForgeFacts, facts_with_token, github_slugs
WORKTREES_ROOT = os.path.join(
os.environ.get("XDG_DATA_HOME") or os.path.expanduser("~/.local/share"),
"worktrees",
)
WORKTREE_DEPTH = 6
CACHE_TTL = 600
SLUG = re.compile(r"^(?!\.\.?$)[\w.-]+$")
def now() -> float:
return time.time()
def _iso(stamp: float) -> str:
return datetime.fromtimestamp(stamp, timezone.utc).isoformat(timespec="seconds")
def tree_file() -> str:
state = os.environ.get("XDG_STATE_HOME") or os.path.expanduser("~/.local/state")
return os.environ.get("PKGMGR_TREE_FILE") or os.path.join(
state, "pkgmgr", "tree", "branch-tree.json"
)
def _cache_path(owner: str, name: str) -> str:
base = os.environ.get("XDG_CACHE_HOME") or os.path.expanduser("~/.cache")
return os.path.join(base, "pkgmgr", "forge", owner, f"{name}.json")
def secret_service_token() -> tuple[str | None, str | None]:
"""
Read the GitHub token from the Secret Service without unlocking it.
Returns:
(token, None), or (None, why no token is available).
"""
try:
import secretstorage
except ImportError:
return None, "secretstorage not installed"
key = build_keyring_key("github", GITHUB_HOST, None)
try:
connection = secretstorage.dbus_init()
try:
items = list(
secretstorage.search_items(
connection, {"service": key.service, "username": key.username}
)
)
unlocked = [item for item in items if not item.is_locked()]
if not items:
return None, "no GitHub token in the Secret Service"
if not unlocked:
return None, "Secret Service locked"
return unlocked[0].get_secret().decode(), None
finally:
connection.close()
except (secretstorage.exceptions.SecretStorageException, OSError) as exc:
return None, f"Secret Service unavailable: {exc}"
@contextmanager
def _locked(path: str) -> Iterator[None]:
folder, name = os.path.split(os.path.abspath(path))
os.makedirs(folder, exist_ok=True)
with open(os.path.join(folder, f".{name}.lock"), "w", encoding="utf-8") as lock:
fcntl.flock(lock, fcntl.LOCK_EX)
yield
def _replace(path: str, text: str) -> None:
folder, name = os.path.split(os.path.abspath(path))
fd, tmp = tempfile.mkstemp(dir=folder, prefix=f".{name}.")
try:
with os.fdopen(fd, "w", encoding="utf-8") as fh:
fh.write(text)
fh.flush()
os.fsync(fh.fileno())
os.replace(tmp, path)
finally:
if os.path.exists(tmp):
os.unlink(tmp)
def _write(path: str, text: str) -> None:
with _locked(path):
_replace(path, text)
def _cached(path: str) -> ForgeFacts | None:
try:
with open(path, encoding="utf-8") as fh:
data = json.load(fh)
if now() - data["fetched_at"] >= CACHE_TTL:
return None
return ForgeFacts(
merged_heads=data["merged_heads"],
open_heads=data["open_heads"],
protected=set(data["protected"]),
)
except (OSError, ValueError, KeyError, TypeError):
return None
def _store(path: str, facts: ForgeFacts) -> None:
data = {
"fetched_at": now(),
"merged_heads": facts.merged_heads,
"open_heads": facts.open_heads,
"protected": sorted(facts.protected),
}
_write(path, json.dumps(data))
def _unavailable(reason: str | None) -> tuple[ForgeFacts, str]:
return ForgeFacts(unavailable=reason), f"unavailable: {reason}"
def forge_facts(
urls: dict[str, str], token: str | None, reason: str | None
) -> tuple[ForgeFacts, str]:
"""
Forge facts of one repository from the cache, or live once the cache is stale.
Args:
urls: remote name -> URL.
token: GitHub token from the Secret Service, or None.
reason: why the token is None.
Returns:
The facts and their source: none, cached, live or unavailable: <why>.
"""
slugs = github_slugs(urls)
if not slugs:
return ForgeFacts(), "none"
invalid = [f"{o}/{n}" for o, n in slugs if not (SLUG.match(o) and SLUG.match(n))]
if invalid:
return _unavailable(f"invalid GitHub slug {invalid[0]}")
merged, label = ForgeFacts(), "cached"
for owner, name in slugs:
path = _cache_path(owner, name)
facts = _cached(path)
if facts is None:
if token is None:
return _unavailable(reason)
facts = facts_with_token([(owner, name)], token)
if facts.unavailable:
return _unavailable(facts.unavailable)
try:
_store(path, facts)
label = "live" if label == "cached" else label
except OSError as exc:
label = f"live: cache not written ({exc})"
merged.merged_heads.update(facts.merged_heads)
merged.open_heads.update(facts.open_heads)
merged.protected |= facts.protected
return merged, label
def _missing(root: str, known: bool, scanned_at: str) -> dict:
return {
"path": root,
"remote": None,
"default": None,
"missing": True,
"known": known,
"forge": None,
"scanned_at": scanned_at,
"warnings": [],
"items": [],
}
def _git_dir(
root: str, registrations: gc.Registrations, worktree: str | None
) -> str | None:
if worktree is None:
return None
if worktree == root:
return os.path.dirname(registrations[0])
return gc._linked_meta(registrations, worktree)
def _flow(git_dir: str | None, item: dict, warnings: list[str]) -> dict | None:
found = flow.read_flow(git_dir) if git_dir else None
if found is None or found.source == "none":
return None
warnings += [
f"flow of {item['worktree']}: {warning}"
for warning in found.warnings + flow.mismatch(found, item["branch"])
]
return {"pull": found.pull, "push": found.push, "source": found.source}
def _scan(
root: str,
known: bool,
token: str | None,
reason: str | None,
containers: list[gc.Container] | None,
host_ns: bool,
) -> dict:
scanned_at = _iso(now())
entry = _missing(root, known, scanned_at)
if not os.path.exists(os.path.join(root, ".git")):
return entry
sources: list[str] = []
def source(urls: dict[str, str]) -> ForgeFacts:
facts, label = forge_facts(urls, token, reason)
sources.append(label)
return facts
entry["missing"] = False
try:
repo = gc.plan_repo(
root,
containers,
host_ns,
merged_into=[],
sizes=False,
min_age=None,
scope="all",
activity=True,
hooks={},
fetch=False,
forge_source=source,
)
registrations = gc._registrations(root)
except GitBaseError as exc:
entry["warnings"] = [f"scan failed: {str(exc).strip().splitlines()[0]}"]
return entry
owner = next((r for r in repo.remotes if repo.default.startswith(f"{r}/")), None)
items = [asdict(item) for item in repo.items]
for item in items:
git_dir = _git_dir(root, registrations, item["worktree"])
item["flow"] = _flow(git_dir, item, repo.warnings)
entry.update(
remote=repo.urls[repo.remotes[0]] if repo.remotes else None,
default=repo.default.removeprefix(f"{owner}/") if owner else repo.default,
forge=sources[0],
warnings=repo.warnings,
items=items,
)
return entry
def _owner_root(checkout: str) -> str | None:
try:
with open(os.path.join(checkout, ".git"), encoding="utf-8") as fh:
gitdir = fh.read().strip().removeprefix("gitdir: ")
except OSError:
return None
parent = os.path.dirname(os.path.realpath(os.path.join(checkout, gitdir)))
if os.path.basename(parent) != "worktrees":
return None
common = os.path.dirname(parent)
return os.path.dirname(common) if os.path.basename(common) == ".git" else common
def _shared_owners(worktrees_root: str) -> list[str]:
owners: set[str] = set()
base = worktrees_root.rstrip("/").count("/")
for path, dirs, files in os.walk(worktrees_root):
dirs.sort()
owner = _owner_root(path) if ".git" in files else None
if owner:
owners.add(owner)
if (
".git" in files
or ".git" in dirs
or path.count("/") - base >= WORKTREE_DEPTH
):
dirs.clear()
return sorted(owners)
def branch_tree(
repo_dirs: list[str], *, fmt: str, output: str | None, worktrees_root: str
) -> int:
"""
Print or write the branch and worktree state of every repository, without fetching.
Args:
repo_dirs: directories of the repositories pkgmgr knows.
fmt: json or yaml.
output: file to write atomically; None prints to stdout.
worktrees_root: shared worktree root whose unknown owners are added.
Returns:
0 on success, 1 when the output could not be written.
"""
known = list(dict.fromkeys(os.path.realpath(d) for d in repo_dirs))
orphans = [d for d in _shared_owners(worktrees_root) if d not in known]
token, reason = secret_service_token()
host_ns = gc.host_pid_namespace()
containers = gc.list_compose_containers()
data = {
"generated_at": _iso(now()),
"host_pid_namespace": host_ns,
"delete_ready": token is not None,
"delete_reason": reason,
"repos": [
_scan(root, root in known, token, reason, containers, host_ns)
for root in known + orphans
],
}
text = (
yaml.safe_dump(data, sort_keys=False)
if fmt == "yaml"
else json.dumps(data, indent=2) + "\n"
)
if output is None:
sys.stdout.write(text)
return 0
try:
_write(output, text)
except OSError as exc:
print(f"Cannot write {output}: {exc}", file=sys.stderr)
return 1
return 0
def refresh_tree_entry(repo_root: str) -> None:
"""
Rescan one repository into the tree file, if the file exists and holds it.
Args:
repo_root: main worktree root of the repository.
Skipped outside the host PID namespace and when the file's entry is newer.
"""
path = tree_file()
if not os.path.isfile(path) or not gc.host_pid_namespace():
return
root = os.path.realpath(repo_root)
token, reason = secret_service_token()
fresh = _scan(root, True, token, reason, gc.list_compose_containers(), True)
with _locked(path):
with open(path, encoding="utf-8") as fh:
data = json.load(fh)
index = next(
(i for i, entry in enumerate(data["repos"]) if entry["path"] == root), None
)
if index is None or data["repos"][index]["scanned_at"] > fresh["scanned_at"]:
return
fresh["known"] = data["repos"][index]["known"]
data["repos"][index] = fresh
_replace(path, json.dumps(data, indent=2) + "\n")
View File
Whitespace-only changes.
+4
View File
@@ -0,0 +1,4 @@
from .render import render_text
from .report import disk_report
__all__ = ["disk_report", "render_text"]
+54
View File
@@ -0,0 +1,54 @@
from __future__ import annotations
import os
from pkgmgr.actions.branch.gc import _inside
def _mount(path: str) -> str:
path = os.path.realpath(path)
while not os.path.ismount(path):
path = os.path.dirname(path)
return path
def balance(paths: list[str], items: list[tuple[str, int]]) -> list[dict]:
"""
Set what the report attributes against each filesystem's capacity.
Args:
paths: the roots and repository roots; each one's filesystem gets a row.
items: (path, allocated bytes) of every listed entry; an entry inside
another listed entry is counted only through the outer one.
Returns:
Per filesystem (grouped by st_dev): its mount point, total and free
bytes (as available to unprivileged users) and the attributed bytes.
"""
filesystems: dict[int, dict] = {}
for path in paths:
try:
dev = os.stat(path).st_dev
if dev in filesystems:
continue
vfs = os.statvfs(path)
except OSError:
continue
filesystems[dev] = {
"filesystem": _mount(path),
"total_bytes": vfs.f_blocks * vfs.f_frsize,
"free_bytes": vfs.f_bavail * vfs.f_frsize,
"attributed_bytes": 0,
}
kept: list[str] = []
for path, size in sorted((os.path.realpath(p), s) for p, s in items):
if any(_inside(path, outer) for outer in kept):
continue
kept.append(path)
try:
dev = os.lstat(path).st_dev
except OSError:
continue
if dev in filesystems:
filesystems[dev]["attributed_bytes"] += size
return sorted(filesystems.values(), key=lambda fs: fs["filesystem"])
@@ -0,0 +1,58 @@
from __future__ import annotations
import os
from .walk import BLOCK
ELF_MAGIC = b"\x7fELF"
ELF_HEADER = 18
ET_CORE = 4
BYTE_ORDER = {1: "little", 2: "big"}
def is_core_dump(path: str) -> bool:
try:
fd = os.open(path, os.O_RDONLY | os.O_NOFOLLOW | os.O_NONBLOCK)
except OSError:
return False
try:
head = os.read(fd, ELF_HEADER)
except OSError:
return False
finally:
os.close(fd)
order = BYTE_ORDER.get(head[5]) if len(head) == ELF_HEADER else None
return (
head[:4] == ELF_MAGIC
and order is not None
and int.from_bytes(head[16:18], order) == ET_CORE
)
def core_dumps(candidates: list[str]) -> list[dict]:
"""
Keep the candidates whose ELF header says ET_CORE.
Args:
candidates: paths of regular files named like core dumps.
Returns:
One entry per dump with its allocated and apparent size and mtime,
biggest first.
"""
dumps: dict[str, dict] = {}
for path in candidates:
real = os.path.realpath(path)
if real in dumps or not is_core_dump(path):
continue
try:
st = os.lstat(path)
except OSError:
continue
dumps[real] = {
"path": path,
"size_bytes": st.st_blocks * BLOCK,
"apparent_bytes": st.st_size,
"mtime": st.st_mtime,
}
return sorted(dumps.values(), key=lambda d: (-d["size_bytes"], d["path"]))
@@ -0,0 +1,50 @@
from __future__ import annotations
import json
import subprocess
from pkgmgr.actions.branch.gc import _describe, _docker
DF_ARGS = ["system", "df", "--format", "{{json .}}"]
def _count(value: object) -> int | None:
text = str(value)
return int(text) if text.isdigit() else None
def docker_usage() -> tuple[list[dict] | None, str | None]:
"""
Read `docker system df` per type.
Returns:
The rows (type, total and active counts, size and reclaimable as docker
prints them) and None, or None and a warning when docker is missing,
fails or prints something else than JSON lines.
"""
try:
out = _docker(DF_ARGS)
except FileNotFoundError:
return None, "docker unavailable: not installed"
except subprocess.CalledProcessError as exc:
return None, f"docker unavailable: {_describe(exc)}"
rows: list[dict] = []
for line in out.splitlines():
if not line.strip():
continue
try:
data = json.loads(line)
except ValueError:
data = None
if not isinstance(data, dict):
return None, "docker system df: output is not JSON lines"
rows.append(
{
"type": data.get("Type"),
"total": _count(data.get("TotalCount")),
"active": _count(data.get("Active")),
"size": data.get("Size"),
"reclaimable": data.get("Reclaimable"),
}
)
return rows, None
+186
View File
@@ -0,0 +1,186 @@
from __future__ import annotations
import os
import stat
from pkgmgr.actions.branch.gc import _describe, _git, _has_submodules
from pkgmgr.core.git.errors import GitBaseError
from pkgmgr.core.git.queries import FILTER_CONFIG, local_config
from pkgmgr.core.git.run import run
REF_NAMESPACES = ("refs/heads", "refs/remotes", "refs/tags")
PEELED = "%(if)%(*objectname)%(then)%(*objectname)%(else)%(objectname)%(end)"
UNMERGED = "unmerged"
DELETED = "deleted"
DIRECTORY = "directory"
SPECIAL = "special"
UNHASHABLE = {
DIRECTORY: "untracked directory or gitlink",
SPECIAL: "not a regular file",
}
MAX_REASONS = 5
def _tips(repo: str) -> list[str]:
out = _git(["for-each-ref", f"--format={PEELED}", *REF_NAMESPACES], repo)
return sorted(set(out.split()))
def _status_entries(tree: str) -> list[tuple[str, str | None]]:
raw = _git(
[
"status",
"--porcelain=v2",
"-z",
"--untracked-files=all",
"--no-renames",
],
tree,
)
entries: list[tuple[str, str | None]] = []
for record in raw.split("\0"):
kind = record[:1]
if kind == "1":
parts = record.split(" ", 8)
entries.append((parts[8], parts[7]))
elif kind == "u":
entries.append((record.split(" ", 10)[10], UNMERGED))
elif kind == "?":
entries.append((record[2:], None))
return entries
def _hashes(tree: str, paths: list[str]) -> dict[str, str]:
hashes: dict[str, str] = {}
files: list[str] = []
for path in paths:
full = os.path.join(tree, path)
try:
mode = os.lstat(full).st_mode
except FileNotFoundError:
hashes[path] = DELETED
continue
if stat.S_ISREG(mode):
files.append(path)
elif stat.S_ISLNK(mode):
hashes[path] = run(
["hash-object", "-t", "blob", "--no-filters", "--stdin"],
cwd=tree,
scan=True,
input_text=os.readlink(full),
)
else:
hashes[path] = DIRECTORY if stat.S_ISDIR(mode) else SPECIAL
if files:
out = run(
["hash-object", "--no-filters", "--stdin-paths"],
cwd=tree,
scan=True,
input_text="".join(f"{path}\n" for path in files),
)
hashes.update(zip(files, out.split("\n")))
return hashes
def _absent(
repo: str, tips: list[str], wanted: list[tuple[str, set[str]]]
) -> list[str]:
query = "".join(f"{tip}:{path}\n" for path, _ in wanted for tip in tips)
if not query:
return [path for path, _ in wanted]
found = run(
["cat-file", "--batch-check=%(objectname)"],
cwd=repo,
scan=True,
input_text=query,
).split("\n")
width = len(tips)
return [
path
for index, (path, blobs) in enumerate(wanted)
if blobs - set(found[index * width : (index + 1) * width])
]
def _reachable(repo: str, head: str | None) -> bool:
if not head:
return False
return bool(
_git(["for-each-ref", "--count=1", f"--contains={head}", *REF_NAMESPACES], repo)
)
def _unpreserved(
repo: str, record: dict, tips: list[str]
) -> tuple[int | None, list[str]]:
tree = record["path"]
if _has_submodules(tree):
return None, ["submodules, whose config the scan cannot vet"]
unsafe = local_config(tree, FILTER_CONFIG)
if unsafe:
return None, [f"{', '.join(unsafe)} in the repository config"]
entries = _status_entries(tree)
reasons: list[str] = []
if entries and record["branch"] is None and not _reachable(repo, record["head"]):
reasons.append("detached HEAD reachable from no ref")
hashable: list[tuple[str, str | None]] = []
for path, index_blob in entries:
if "\n" in path:
reasons.append(f"{path!r}: path with a newline")
elif index_blob == UNMERGED:
reasons.append(f"{path}: unmerged")
else:
hashable.append((path, index_blob))
hashes = _hashes(tree, [path for path, _ in hashable])
wanted: list[tuple[str, set[str]]] = []
for path, index_blob in hashable:
current = hashes[path]
if current in UNHASHABLE:
reasons.append(f"{path}: {UNHASHABLE[current]}")
continue
blobs = {current} - {DELETED}
if index_blob and set(index_blob) != {"0"}:
blobs.add(index_blob)
if blobs:
wanted.append((path, blobs))
reasons += [f"{path}: content in no ref" for path in _absent(repo, tips, wanted)]
return len(entries), reasons
def preserved(repo: str, records: list[dict]) -> list[dict]:
"""
Decide for each dirty worktree whether all its work survives its deletion.
Args:
repo: main worktree root of the repository.
records: its worktree records from list_worktrees.
Returns:
One entry per dirty or unreadable worktree; `preserved` is true only
when every changed, staged and untracked file's content is a blob at
the same path of some branch, remote or tag tip, and a detached HEAD is
reachable from a ref. `reasons` lists what is not, capped.
"""
live = [r for r in records if not r["orphaned"] and not r["missing"]]
tips = _tips(repo) if live else []
results: list[dict] = []
for record in live:
try:
paths, reasons = _unpreserved(repo, record, tips)
except (GitBaseError, OSError, UnicodeError) as exc:
paths, reasons = None, [f"status failed: {_describe(exc)}"]
if paths == 0:
continue
extra = len(reasons) - MAX_REASONS
results.append(
{
"repo": repo,
"path": record["path"],
"branch": record["branch"],
"paths": paths,
"preserved": not reasons,
"reasons": reasons[:MAX_REASONS]
+ ([f"... {extra} more"] if extra > 0 else []),
}
)
return results
+132
View File
@@ -0,0 +1,132 @@
from __future__ import annotations
from pkgmgr.actions.branch.gc import human_size
UNITS = (("d", 86400), ("h", 3600), ("m", 60))
DOCKER_COLUMNS = ("type", "total", "active", "size", "reclaimable")
def _size(value: int | None) -> str:
return "-" if value is None else human_size(value)
def _age(now: float, stamp: float | None) -> str:
if not stamp:
return "-"
seconds = max(0.0, now - stamp)
for unit, length in UNITS:
if seconds >= length:
return f"{seconds / length:.1f}{unit}"
return f"{seconds:.0f}s"
def _section(title: str, headers: list[str], rows: list[list[str]]) -> list[str]:
if not rows:
return [f"== {title}", "(none)"]
widths = [max(len(cell) for cell in column) for column in zip(headers, *rows)]
return [f"== {title}"] + [
" ".join(cell.ljust(width) for cell, width in zip(row, widths)).rstrip()
for row in (headers, *rows)
]
def _branch(tree: dict) -> str:
if tree["orphaned"]:
return "-"
return tree["branch"] or "(detached)"
def _state(tree: dict) -> str:
if tree["orphaned"]:
return "orphaned"
return "missing" if tree["missing"] else "linked"
def render_text(report: dict) -> str:
"""
Render a disk_report result as aligned text tables.
Args:
report: the dict disk_report returned.
Returns:
The tables, one section per report key, newline-joined.
"""
now = report["generated_at"]
lines = _section(
"worktrees",
["SIZE", "ACTIVE", "FOREIGN", "STATE", "BRANCH", "PATH"],
[
[
_size(t["size_bytes"]),
_age(now, t["last_activity"]),
str(t["foreign_entries"]),
_state(t),
_branch(t),
t["path"],
]
for t in report["worktrees"]
],
)
lines += _section(
"preserved (dirty worktrees)",
["PRESERVED", "PATHS", "BRANCH", "PATH", "REASONS"],
[
[
"yes" if p["preserved"] else "no",
"-" if p["paths"] is None else str(p["paths"]),
p["branch"] or "(detached)",
p["path"],
"; ".join(p["reasons"]),
]
for p in report["preserved"]
],
)
for root in report["roots"]:
lines += _section(
f"root {root['root']} (over {human_size(report['threshold_bytes'])})",
["SIZE", "AGE", "KIND", "PATH"],
[
[
_size(c["size_bytes"]),
_age(now, c["newest_mtime"]),
c["kind"],
c["path"],
]
for c in root["children"]
],
)
lines += _section(
"core dumps",
["SIZE", "APPARENT", "AGE", "PATH"],
[
[
_size(d["size_bytes"]),
_size(d["apparent_bytes"]),
_age(now, d["mtime"]),
d["path"],
]
for d in report["core_dumps"]
],
)
if report["docker"] is not None:
lines += _section(
"docker",
["TYPE", "TOTAL", "ACTIVE", "SIZE", "RECLAIMABLE"],
[[str(row[key]) for key in DOCKER_COLUMNS] for row in report["docker"]],
)
lines += _section(
"balance",
["FILESYSTEM", "TOTAL", "FREE", "ATTRIBUTED"],
[
[
fs["filesystem"],
_size(fs["total_bytes"]),
_size(fs["free_bytes"]),
_size(fs["attributed_bytes"]),
]
for fs in report["balance"]
],
)
lines += [f"WARN {warning}" for warning in report["warnings"]]
return "\n".join(lines)
+97
View File
@@ -0,0 +1,97 @@
from __future__ import annotations
import time
from pkgmgr.actions.branch.gc import _describe
from pkgmgr.core.git.errors import GitBaseError
from .balance import balance
from .core_dumps import core_dumps
from .docker_usage import docker_usage
from .preserved import preserved
from .roots import scan_root
from .worktrees import list_worktrees
def disk_report(
roots: list[str], repo_roots: list[str], threshold_bytes: int, docker: bool
) -> dict:
"""
Inventory disk usage read-only: it deletes and writes nothing.
Args:
roots: directories whose direct children are measured.
repo_roots: main worktree roots of the repositories whose linked
worktrees are listed (pkgmgr passes every managed repository).
threshold_bytes: list only root children larger than this.
docker: also read `docker system df`.
Returns:
A JSON-serialisable dict. Sizes are allocated bytes (st_blocks * 512,
sparse files and hard links counted as stored), measured without
following symlinks or leaving the filesystem a walk starts on; times
are epoch seconds.
generated_at: when the scan started.
threshold_bytes: as given.
worktrees: per linked worktree and orphaned `.claude/worktrees/agent-*`
dir: repo, path, branch, head, orphaned, missing, size_bytes,
last_activity (newest mtime inside or of its index/HEAD),
foreign_entries (not owned by the current uid, pruned at the first
per subtree), unreadable (directories that could not be listed).
preserved: per dirty worktree: repo, path, branch, paths (changed,
staged and untracked; None when status was not read), preserved,
reasons.
roots: per root: root, children over the threshold with path, kind
(worktree, git-repo, venv, node_modules, cache, core-dump, dir,
file), size_bytes, newest_mtime, unreadable; biggest first.
core_dumps: files named `core` or `core.<digits>` under the roots and
worktrees whose ELF e_type is ET_CORE: path, size_bytes,
apparent_bytes (st_size), mtime; biggest first.
docker: rows of type, total, active, size, reclaimable; None when not
asked for or unavailable.
balance: per filesystem of the roots and repository roots: filesystem
(mount point), total_bytes, free_bytes, attributed_bytes.
warnings: what could not be read.
"""
started = time.time()
warnings: list[str] = []
cores: list[str] = []
trees: list[dict] = []
kept: list[dict] = []
for repo in repo_roots:
try:
records = list_worktrees(repo, cores)
except (GitBaseError, OSError) as exc:
warnings.append(f"{repo}: worktrees not listed: {_describe(exc)}")
continue
trees += records
try:
kept += preserved(repo, records)
except (GitBaseError, OSError) as exc:
warnings.append(f"{repo}: preservation not checked: {_describe(exc)}")
scanned: list[dict] = []
for root in roots:
try:
scanned.append(scan_root(root, threshold_bytes, cores))
except OSError as exc:
warnings.append(f"{root}: not scanned: {_describe(exc)}")
dumps = core_dumps(cores)
usage = None
if docker:
usage, warning = docker_usage()
if warning:
warnings.append(warning)
items = [(t["path"], t["size_bytes"]) for t in trees if t["size_bytes"]]
items += [(c["path"], c["size_bytes"]) for s in scanned for c in s["children"]]
items += [(d["path"], d["size_bytes"]) for d in dumps]
return {
"generated_at": started,
"threshold_bytes": threshold_bytes,
"worktrees": trees,
"preserved": kept,
"roots": scanned,
"core_dumps": dumps,
"docker": usage,
"balance": balance([*roots, *repo_roots], items),
"warnings": warnings,
}
+77
View File
@@ -0,0 +1,77 @@
from __future__ import annotations
import os
import stat
from .core_dumps import is_core_dump
from .walk import walk
CACHE_TAG = "CACHEDIR.TAG"
NODE_MODULES = "node_modules"
def _has(path: str, name: str, check=os.path.isfile) -> bool:
return check(os.path.join(path, name))
def _kind(path: str, st: os.stat_result) -> str:
name = os.path.basename(path)
if not stat.S_ISDIR(st.st_mode):
regular = stat.S_ISREG(st.st_mode)
return "core-dump" if regular and is_core_dump(path) else "file"
if name == NODE_MODULES:
return "node_modules"
if _has(path, ".git"):
return "worktree"
if _has(path, ".git", os.path.isdir) or (
_has(path, "HEAD") and _has(path, "objects", os.path.isdir)
):
return "git-repo"
if _has(path, "pyvenv.cfg"):
return "venv"
if _has(path, CACHE_TAG) or name.lower().endswith("cache"):
return "cache"
return "dir"
def scan_root(root: str, threshold_bytes: int, cores: list[str]) -> dict:
"""
Measure the direct children of one root on the root's own filesystem.
Args:
root: the directory to inventory.
threshold_bytes: keep only children whose allocated size is above it.
cores: receives the core dump candidates found while measuring.
Returns:
The root with its children over the threshold: path, kind, allocated
size, newest mtime inside and unlistable directories, biggest first.
Raises:
OSError: when the root itself cannot be listed.
"""
dev = os.stat(root).st_dev
children: list[dict] = []
with os.scandir(root) as entries:
listing = sorted(entries, key=lambda entry: entry.name)
for entry in listing:
try:
st = entry.stat(follow_symlinks=False)
except OSError:
continue
if st.st_dev != dev:
continue
stats = walk(entry.path, dev)
cores.extend(stats.cores)
if stats.size_bytes > threshold_bytes:
children.append(
{
"path": entry.path,
"kind": _kind(entry.path, st),
"size_bytes": stats.size_bytes,
"newest_mtime": stats.newest_mtime,
"unreadable": stats.unreadable,
}
)
children.sort(key=lambda child: (-child["size_bytes"], child["path"]))
return {"root": root, "children": children}
+86
View File
@@ -0,0 +1,86 @@
from __future__ import annotations
import os
import re
import stat
from dataclasses import dataclass, field
BLOCK = 512
CORE_NAME = re.compile(r"core(\.\d+)?")
@dataclass
class TreeStats:
size_bytes: int = 0
newest_mtime: float = 0.0
foreign_entries: int = 0
unreadable: int = 0
cores: list[str] = field(default_factory=list)
def is_foreign(path: str, st: os.stat_result) -> bool:
return st.st_uid != os.getuid()
def _account(
stats: TreeStats, path: str, st: os.stat_result, seen: set[tuple[int, int]]
) -> None:
if st.st_nlink > 1 and not stat.S_ISDIR(st.st_mode):
if (st.st_dev, st.st_ino) in seen:
return
seen.add((st.st_dev, st.st_ino))
stats.size_bytes += st.st_blocks * BLOCK
stats.newest_mtime = max(stats.newest_mtime, st.st_mtime)
if stat.S_ISREG(st.st_mode) and CORE_NAME.fullmatch(os.path.basename(path)):
stats.cores.append(path)
def _children(path: str, stats: TreeStats) -> list[os.DirEntry]:
try:
with os.scandir(path) as entries:
return list(entries)
except OSError:
stats.unreadable += 1
return []
def walk(path: str, dev: int | None = None) -> TreeStats:
"""
Measure a tree like `du -x`: allocated blocks, symlinks never followed.
Args:
path: the tree root, measured itself as well.
dev: the filesystem (st_dev) to stay on; that of path when None.
Returns:
Allocated bytes (st_blocks * 512, each hard-linked inode once), the
newest mtime, entries not owned by the current uid (none counted below a
foreign one), directories that could not be listed, and the regular
files named `core` or `core.<digits>`.
"""
stats = TreeStats()
try:
top = os.lstat(path)
except OSError:
stats.unreadable += 1
return stats
dev = top.st_dev if dev is None else dev
if top.st_dev != dev:
return stats
seen: set[tuple[int, int]] = set()
stack: list[tuple[str, os.stat_result, bool]] = [(path, top, False)]
while stack:
current, st, pruned = stack.pop()
_account(stats, current, st, seen)
foreign = not pruned and is_foreign(current, st)
stats.foreign_entries += foreign
if not stat.S_ISDIR(st.st_mode):
continue
for entry in _children(current, stats):
try:
child = entry.stat(follow_symlinks=False)
except OSError:
continue
if child.st_dev == dev:
stack.append((entry.path, child, pruned or foreign))
return stats
+96
View File
@@ -0,0 +1,96 @@
from __future__ import annotations
import os
from pkgmgr.actions.branch.gc import _worktrees
from .walk import TreeStats, walk
AGENT_DIR = os.path.join(".claude", "worktrees")
AGENT_PREFIX = "agent-"
METADATA_FILES = ("index", "HEAD")
def _metadata_mtime(path: str) -> float:
try:
with open(os.path.join(path, ".git"), encoding="utf-8") as fh:
gitdir = fh.read().strip().removeprefix("gitdir: ")
except OSError:
return 0.0
meta = os.path.normpath(os.path.join(path, gitdir))
stamps = [0.0]
for name in METADATA_FILES:
try:
stamps.append(os.stat(os.path.join(meta, name)).st_mtime)
except OSError:
continue
return max(stamps)
def _record(
repo: str,
path: str,
branch: str | None,
head: str | None,
orphaned: bool,
cores: list[str],
) -> dict:
present = os.path.isdir(path)
stats = walk(path) if present else TreeStats()
cores.extend(stats.cores)
activity = max(stats.newest_mtime, _metadata_mtime(path)) if present else 0.0
return {
"repo": repo,
"path": path,
"branch": branch,
"head": head,
"orphaned": orphaned,
"missing": not present,
"size_bytes": stats.size_bytes if present else None,
"last_activity": activity or None,
"foreign_entries": stats.foreign_entries,
"unreadable": stats.unreadable,
}
def _agent_dirs(repo: str) -> list[str]:
base = os.path.join(repo, AGENT_DIR)
try:
names = sorted(os.listdir(base))
except OSError:
return []
return [
os.path.join(base, name)
for name in names
if name.startswith(AGENT_PREFIX)
and os.path.isdir(os.path.join(base, name))
and not os.path.islink(os.path.join(base, name))
]
def list_worktrees(repo: str, cores: list[str]) -> list[dict]:
"""
Inventory the linked worktrees of one repository, orphaned agent dirs included.
Args:
repo: main worktree root of the repository.
cores: receives the core dump candidates found while measuring.
Returns:
One record per linked worktree git lists and per `.claude/worktrees/agent-*`
directory it does not list (orphaned), sorted by path.
Raises:
GitBaseError: when `git worktree list` fails.
"""
trees = [tree for tree in _worktrees(repo) if not tree.main]
listed = {os.path.realpath(tree.path) for tree in trees}
records = [
_record(repo, tree.path, tree.branch, tree.head, False, cores) for tree in trees
]
records += [
_record(repo, path, None, None, True, cores)
for path in _agent_dirs(repo)
if os.path.realpath(path) not in listed
]
return sorted(records, key=lambda record: record["path"])
@@ -10,6 +10,7 @@ Keep this package stable so existing imports continue to work, e.g.:
from __future__ import annotations
from .cargo import update_cargo_version
from .changelog_md import update_changelog
from .debian import _get_debian_author, update_debian_changelog
from .editor import _open_editor_for_changelog
@@ -22,6 +23,7 @@ from .rpm_spec import update_spec_version
__all__ = [
"_get_debian_author",
"_open_editor_for_changelog",
"update_cargo_version",
"update_changelog",
"update_debian_changelog",
"update_flake_version",
+76
View File
@@ -0,0 +1,76 @@
from __future__ import annotations
import os
import re
def _read(path: str) -> str:
with open(path, encoding="utf-8") as f:
return f.read()
def _write(path: str, content: str) -> None:
with open(path, "w", encoding="utf-8") as f:
f.write(content)
def update_cargo_version(
repo_root: str, new_version: str, preview: bool = False
) -> list[str]:
"""
Set the root crate's version in Cargo.toml and its entry in Cargo.lock.
Args:
repo_root: repository root that may hold Cargo.toml and Cargo.lock.
new_version: release version without the leading "v".
preview: only print what would change.
Returns:
The paths that were (or in preview would be) changed.
"""
toml_path = os.path.join(repo_root, "Cargo.toml")
if not os.path.exists(toml_path):
print("[INFO] No Cargo.toml found. Skipping Cargo version update.")
return []
toml = _read(toml_path)
block = re.search(r"(?ms)^\[package\]\s*$.*?(?=^\s*\[|\Z)", toml)
if not block:
print("[INFO] Cargo.toml has no [package] section. Skipping.")
return []
name = re.search(r'(?m)^\s*name\s*=\s*"([^"]+)"', block.group(0))
if not name:
raise RuntimeError("Missing name key in [package] section of Cargo.toml")
new_block, count = re.subn(
r'(?m)^(\s*version\s*=\s*")[^"]+(")',
lambda m: f"{m.group(1)}{new_version}{m.group(2)}",
block.group(0),
count=1,
)
if count == 0:
raise RuntimeError("Missing version key in [package] section of Cargo.toml")
changes = {toml_path: toml[: block.start()] + new_block + toml[block.end() :]}
lock_path = os.path.join(repo_root, "Cargo.lock")
if os.path.exists(lock_path):
lock, count = re.subn(
rf'(?m)^(\[\[package\]\]\nname = "{re.escape(name.group(1))}"\n'
r'version = ")[^"]+("\n)(?!source = )',
lambda m: f"{m.group(1)}{new_version}{m.group(2)}",
_read(lock_path),
)
if count != 1:
raise RuntimeError(
f"Expected one local {name.group(1)!r} package in Cargo.lock, "
f"found {count}"
)
changes[lock_path] = lock
for path, content in changes.items():
if preview:
print(f"[PREVIEW] Would update {path} version to {new_version}")
else:
_write(path, content)
print(f"Updated {path} version to {new_version}")
return list(changes)
@@ -3,6 +3,11 @@ from __future__ import annotations
import os
import re
try:
import tomllib
except ImportError:
import tomli as tomllib # type: ignore
def update_pyproject_version(
pyproject_path: str, new_version: str, preview: bool = False
@@ -18,6 +23,11 @@ def update_pyproject_version(
print(f"[WARN] Could not read pyproject.toml: {exc}")
return
project = tomllib.loads(content).get("project")
if project is None or "version" in project.get("dynamic", []):
print("[INFO] pyproject.toml declares no static version, skipping.")
return
m = re.search(r"(?ms)^\s*\[project\]\s*$.*?(?=^\s*\[|\Z)", content)
if not m:
raise RuntimeError("Missing [project] section in pyproject.toml")
+4 -3
View File
@@ -10,7 +10,7 @@ the side effects downstream are incomplete.
`retry_release` re-runs the idempotent tail of that flow so a botched
release can be re-pushed without touching code or recreating tags:
* `git push origin <branch> <tag>` for the existing HEAD tag
* push the branch and the existing HEAD tag through `pkgmgr push`'s leases
* re-align the floating `latest` tag if HEAD tag is the highest
Publishing (PyPI etc.) stays the caller's responsibility — the publish
@@ -23,7 +23,8 @@ from __future__ import annotations
import os
from pkgmgr.actions.publish.git_tags import head_semver_tags
from pkgmgr.core.git import GitRunError, run
from pkgmgr.actions.repository.push import push_branch
from pkgmgr.core.git import GitRunError
from pkgmgr.core.git.queries import get_current_branch
from pkgmgr.core.version.semver import SemVer
@@ -50,7 +51,7 @@ def retry_release(
tag = max(tags, key=SemVer.parse)
print(f"Re-pushing existing tag: {tag}")
run(["push", "origin", branch, tag], preview=preview)
push_branch(".", [], sign=True, refs=(tag,), preview=preview)
try:
if is_highest_version_tag(tag):
+49 -13
View File
@@ -4,6 +4,8 @@ Version discovery and bumping helpers for the release workflow.
from __future__ import annotations
import re
from pkgmgr.core.git.queries import get_tags
from pkgmgr.core.version.semver import (
SemVer,
@@ -12,26 +14,60 @@ from pkgmgr.core.version.semver import (
bump_patch,
find_latest_version,
)
from pkgmgr.core.version.source import (
read_cargo_version,
read_changelog_version,
read_debian_changelog_version,
read_flake_version,
read_pkgbuild_version,
read_pyproject_version,
read_spec_version,
)
FILE_SOURCES = {
"pyproject.toml": read_pyproject_version,
"CHANGELOG.md": read_changelog_version,
"flake.nix": read_flake_version,
"PKGBUILD": read_pkgbuild_version,
"RPM spec": read_spec_version,
"debian/changelog": read_debian_changelog_version,
"Cargo.toml": read_cargo_version,
}
_CORE_VERSION = re.compile(r"^v?(\d+\.\d+\.\d+)")
def determine_current_version() -> SemVer:
def _file_versions(repo_dir: str) -> dict[str, SemVer]:
versions: dict[str, SemVer] = {}
for source, read in FILE_SOURCES.items():
match = _CORE_VERSION.match(read(repo_dir) or "")
if match:
versions[source] = SemVer.parse(match.group(1))
return versions
def determine_current_version(repo_dir: str) -> SemVer:
"""
Determine the current semantic version from Git tags.
Determine the highest version across the latest SemVer tag and every
version file the release writes.
Behaviour:
- If there are no tags or no SemVer-compatible tags, return 0.0.0.
- Otherwise, use the latest SemVer tag as current version.
Args:
repo_dir: repository root holding the version files.
Returns:
The highest version found, or 0.0.0 when no source has one.
"""
tags = get_tags()
if not tags:
versions = _file_versions(repo_dir)
latest = find_latest_version(get_tags() or [])
if latest is not None:
versions["tag"] = latest[1]
if not versions:
return SemVer(0, 0, 0)
latest = find_latest_version(tags)
if latest is None:
return SemVer(0, 0, 0)
_tag, ver = latest
return ver
highest = max(versions.values())
if len(set(versions.values())) > 1:
found = ", ".join(f"{source} {ver}" for source, ver in versions.items())
print(f"[INFO] Version sources disagree ({found}); starting from {highest}.")
return highest
def bump_semver(current: SemVer, release_type: str) -> SemVer:
+17 -6
View File
@@ -4,12 +4,18 @@ import os
import sys
from pkgmgr.actions.branch import close_branch
from pkgmgr.core.git import GitRunError, run
from pkgmgr.actions.repository.push import (
push_branch,
refuse_unsafe_config,
sign_unpushed,
)
from pkgmgr.core.git import GitRunError
from pkgmgr.core.git.commands import add, commit, tag_annotated
from pkgmgr.core.git.queries import get_current_branch
from pkgmgr.core.repository.paths import resolve_repo_paths
from .files import (
update_cargo_version,
update_changelog,
update_debian_changelog,
update_flake_version,
@@ -45,10 +51,12 @@ def _release_impl(
branch = "main"
print(f"Releasing on branch: {branch}")
refuse_unsafe_config(".")
# Pull BEFORE making any modifications
ensure_clean_and_synced(preview=preview)
current_ver = determine_current_version()
repo_root = os.path.dirname(os.path.abspath(pyproject_path))
current_ver = determine_current_version(repo_root)
new_ver = bump_semver(current_ver, release_type)
new_ver_str = str(new_ver)
new_tag = new_ver.to_tag(with_prefix=True)
@@ -58,7 +66,6 @@ def _release_impl(
print(f"Current version: {current_ver}")
print(f"New version: {new_ver_str} ({release_type})")
repo_root = os.path.dirname(os.path.abspath(pyproject_path))
paths = resolve_repo_paths(repo_root)
# --- Update versioned files ------------------------------------------------
@@ -74,6 +81,8 @@ def _release_impl(
update_flake_version(paths.flake_nix, new_ver_str, preview=preview)
cargo_files = update_cargo_version(repo_root, new_ver_str, preview=preview)
if paths.arch_pkgbuild:
update_pkgbuild_version(paths.arch_pkgbuild, new_ver_str, preview=preview)
else:
@@ -124,6 +133,7 @@ def _release_impl(
paths.arch_pkgbuild,
paths.rpm_spec,
paths.debian_changelog,
*cargo_files,
]
existing_files = [
p for p in files_to_add if isinstance(p, str) and p and os.path.exists(p)
@@ -132,8 +142,9 @@ def _release_impl(
if preview:
add(existing_files, preview=True)
commit(commit_msg, all=True, preview=True)
sign_unpushed(".", preview=True)
tag_annotated(new_tag, tag_msg, preview=True)
run(["push", "origin", branch, new_tag], preview=True)
push_branch(".", [], sign=True, refs=(new_tag,), preview=True)
if is_highest_version_tag(new_tag):
update_latest_tag(new_tag, preview=True)
@@ -153,10 +164,10 @@ def _release_impl(
add(existing_files, preview=False)
commit(commit_msg, all=True, preview=False)
sign_unpushed(".", preview=False)
tag_annotated(new_tag, tag_msg, preview=False)
# Push branch and ONLY the newly created version tag in one command (no --tags)
run(["push", "origin", branch, new_tag], preview=False)
push_branch(".", [], sign=True, refs=(new_tag,), preview=False)
# Update 'latest' only if this is the highest version tag
try:
+307 -10
View File
@@ -1,39 +1,336 @@
from __future__ import annotations
import subprocess
import threading
from typing import Any
from pkgmgr.actions.repository._parallel import (
RepoRef,
resolve_repos,
run_on_repos,
)
from pkgmgr.core.git.commands import GitPushArgsError, push_args
from pkgmgr.core.git import GitRunError, run
from pkgmgr.core.git.queries import UNSAFE_CONFIG, local_config
from pkgmgr.core.git.run import SCAN_FLAGS
Repository = dict[str, Any]
# Re-signing opens pinentry; parallel pushes must not ask for the passphrase at once.
_SIGN_LOCK = threading.Lock()
def _push_one(repo_dir: str, extra_args: list[str], preview: bool) -> tuple[bool, str]:
class PushRefusedError(RuntimeError):
"""Raised when a push would ship unsigned commits or overwrite a mirror."""
def _try_git(repo_dir: str, *args: str) -> str | None:
try:
push_args(extra_args, cwd=repo_dir, preview=preview)
return run(list(args), cwd=repo_dir)
except GitRunError:
return None
def refuse_unsafe_config(repo_dir: str) -> None:
"""
Args:
repo_dir: checkout about to be signed or pushed.
Raises:
PushRefusedError: the repository's own config, which an agent may have
written, sets a key that makes git run a program or reroute the push.
Filter drivers stay allowed on purpose: git-lfs and git-crypt need
them locally, so a clean filter can still run in the amend's index
refresh, as it did before.
"""
unsafe = local_config(repo_dir, UNSAFE_CONFIG)
if unsafe:
raise PushRefusedError(
f"{repo_dir}: the repository config sets {', '.join(unsafe)}, which can "
"run a program or reroute the push; unset it or move it to the global config"
)
def _push_target(repo_dir: str) -> tuple[str, str]:
refuse_unsafe_config(repo_dir)
branch = run(["rev-parse", "--abbrev-ref", "HEAD"], cwd=repo_dir)
if branch == "HEAD":
raise PushRefusedError(f"{repo_dir}: detached HEAD, nothing to push")
remote = (
_try_git(repo_dir, "config", "--get", f"branch.{branch}.pushRemote")
or _try_git(repo_dir, "config", "--get", "remote.pushDefault")
or _try_git(repo_dir, "config", "--get", f"branch.{branch}.remote")
or "origin"
)
return branch, remote
def _unpushed_signatures(
repo_dir: str, remote: str, branch: str
) -> tuple[str, dict[str, str]]:
for ref in (
f"refs/remotes/{remote}/{branch}",
"@{u}",
f"refs/remotes/{remote}/main",
f"refs/remotes/{remote}/master",
):
base = _try_git(repo_dir, "merge-base", "HEAD", ref)
if base:
lines = run(
["log", "--format=%H %G?", f"{base}..HEAD"], cwd=repo_dir
).splitlines()
return base, dict(line.split() for line in lines)
raise PushRefusedError(
f"{repo_dir}: no {remote}/{branch}, no upstream and no {remote}/main or "
f"{remote}/master to sign against"
)
def sign_unpushed(repo_dir: str, *, preview: bool = False) -> None:
"""
Args:
repo_dir: checkout whose current branch is signed.
preview: print the fetch and the amend instead of running them.
"""
branch, remote = _push_target(repo_dir)
run(
["fetch", "--quiet", "--no-recurse-submodules", "--", remote],
cwd=repo_dir,
preview=preview,
)
base, signatures = _unpushed_signatures(repo_dir, remote, branch)
head = run(["rev-parse", "HEAD"], cwd=repo_dir)
older = sum(1 for sha, s in signatures.items() if sha != head and s != "G")
if older:
print(
f"[push] WARN {older} older unsigned commit(s) in {base[:10]}..HEAD~ of "
f"{repo_dir} stay unsigned: signing them would rewrite history"
)
if signatures.get(head, "G") == "G":
return
if run(
[
*SCAN_FLAGS,
"diff",
"--cached",
"--name-only",
"--no-ext-diff",
"--ignore-submodules=none",
"HEAD",
],
cwd=repo_dir,
):
raise PushRefusedError(f"{repo_dir}: staged changes, cannot sign HEAD")
cmd = [
"git",
*SCAN_FLAGS,
"commit",
"--amend",
"--no-edit",
"--no-verify",
"--allow-empty",
"-S",
]
print(f"[push] Signing HEAD {head[:10]} in {repo_dir}")
if preview:
print(f"[PREVIEW] Would run in {repo_dir!r}: {' '.join(cmd)}")
return
with _SIGN_LOCK:
result = subprocess.run(cmd, cwd=repo_dir, check=False)
if result.returncode != 0:
raise PushRefusedError(f"{repo_dir}: signing HEAD {head[:10]} failed")
def _lease(
repo_dir: str,
url: str,
branch_ref: str,
tracking: str | None,
keep: tuple[str, ...],
preview: bool,
) -> str:
if tracking is not None:
sha = _try_git(repo_dir, "rev-parse", "--verify", "--quiet", tracking) or ""
else:
listed = run(["ls-remote", "--", url, branch_ref], cwd=repo_dir).splitlines()
sha = next(
(ln.split()[0] for ln in listed if ln.split()[1:] == [branch_ref]), ""
)
if sha and not preview:
run(
[
"fetch",
"--quiet",
"--no-tags",
"--no-recurse-submodules",
"--",
url,
branch_ref,
],
cwd=repo_dir,
)
if not sha or preview:
return sha
lost: set[str] | None = None
for tip in keep:
plus = {
line[2:]
for line in run(["cherry", tip, sha], cwd=repo_dir).splitlines()
if line.startswith("+")
}
lost = plus if lost is None else lost & plus
if lost:
raise PushRefusedError(
f"{url}: {len(lost)} commit(s) exist only there, not overwritten: "
f"{' '.join(s[:10] for s in sorted(lost))}; "
"pass --rewrite for a deliberate history rewrite"
)
return sha
def push_branch(
repo_dir: str,
extra_args: list[str],
*,
sign: bool,
refs: tuple[str, ...] = (),
rewrite: bool = False,
preview: bool = False,
) -> None:
"""
Push the current branch to every pushurl of its push remote, each with its own lease.
The fetch URL is leased against the remote-tracking ref, every other
pushurl against its own branch tip; each only after `git cherry` shows it
holds no commit the new tip lacks. A mirror failure is reported after the
others were pushed; a canonical failure stops before any mirror.
Args:
repo_dir: checkout whose current branch is pushed.
extra_args: flags appended to every `git push`.
sign: refuse an unpushed, unsigned HEAD; False pushes it as it is.
refs: extra refs (e.g. a release tag) pushed to every URL.
rewrite: replace commits of the old canonical tip; the fetch URL skips
the cherry check, mirrors only refuse commits the old tip never had.
preview: print the pushes instead of running them.
"""
branch, remote = _push_target(repo_dir)
head = run(["rev-parse", "HEAD"], cwd=repo_dir)
if sign:
_base, signatures = _unpushed_signatures(repo_dir, remote, branch)
if not preview and signatures.get(head, "G") != "G":
raise PushRefusedError(
f"{repo_dir}: HEAD {head[:10]} is unsigned; run `pkgmgr push` to sign it"
)
fetch_url = run(["remote", "get-url", "--", remote], cwd=repo_dir)
push_urls = run(
["remote", "get-url", "--push", "--all", "--", remote], cwd=repo_dir
).splitlines()
branch_ref = f"refs/heads/{branch}"
tracking = f"refs/remotes/{remote}/{branch}"
old_tip = _try_git(repo_dir, "rev-parse", "--verify", "--quiet", tracking)
if fetch_url not in push_urls:
print(
f"[push] WARN {remote}: fetch URL is no pushurl; every URL is leased "
"as a mirror, tracking ref and upstream stay untouched"
)
failed: list[str] = []
for url in sorted(push_urls, key=lambda u: u != fetch_url):
if url == fetch_url:
keep: tuple[str, ...] = () if rewrite else (head,)
else:
keep = (head, old_tip) if rewrite and old_tip else (head,)
try:
expect = _lease(
repo_dir,
url,
branch_ref,
tracking if url == fetch_url else None,
keep,
preview,
)
run(
[
"push",
"--follow-tags",
"--no-recurse-submodules",
f"--force-with-lease={branch_ref}:{expect}",
*extra_args,
"--",
url,
branch_ref,
*refs,
],
cwd=repo_dir,
preview=preview,
)
except (GitRunError, PushRefusedError) as exc:
if url == fetch_url:
raise
print(f"[push] FAILED {url}\n{exc}")
failed.append(url)
continue
print(f"[push] {url} <- {branch} {head[:10]}")
if url == fetch_url and not preview:
run(["update-ref", tracking, head], cwd=repo_dir)
if not _try_git(repo_dir, "rev-parse", "--abbrev-ref", "@{u}"):
run(["branch", f"--set-upstream-to={remote}/{branch}"], cwd=repo_dir)
if failed:
raise PushRefusedError(f"{repo_dir}: push failed for {', '.join(failed)}")
def _push_one(
repo_dir: str, extra_args: list[str], sign: bool, rewrite: bool, preview: bool
) -> tuple[bool, str]:
try:
if sign:
sign_unpushed(repo_dir, preview=preview)
push_branch(repo_dir, extra_args, sign=sign, rewrite=rewrite, preview=preview)
return (True, "")
except GitPushArgsError as exc:
except (GitRunError, PushRefusedError) as exc:
return (False, str(exc))
def push_repo_dirs(
repos: list[RepoRef],
extra_args: list[str],
preview: bool,
sign: bool,
jobs: int = 1,
rewrite: bool = False,
) -> None:
"""
Sign and push each ``(identifier, repo_dir)``, optionally in parallel.
"""
run_on_repos(
repos,
lambda rd: _push_one(rd, extra_args, sign, rewrite, preview),
jobs=jobs,
op_name="push",
)
def push_in_parallel(
selected_repos: list[Repository],
repositories_base_dir: str,
all_repos: list[Repository],
extra_args: list[str],
preview: bool,
sign: bool,
jobs: int = 1,
rewrite: bool = False,
) -> None:
"""
Execute `git push` for each repository, optionally in parallel.
Sign and push each selected repository, optionally in parallel.
"""
repos = resolve_repos(selected_repos, repositories_base_dir, all_repos)
run_on_repos(
repos,
lambda rd: _push_one(rd, extra_args, preview),
push_repo_dirs(
resolve_repos(selected_repos, repositories_base_dir, all_repos),
extra_args,
preview,
sign,
jobs=jobs,
op_name="push",
rewrite=rewrite,
)
+7
View File
@@ -0,0 +1,7 @@
from __future__ import annotations
from .collect import collect_review
from .model import ReviewError
from .render import render_review, write_review
__all__ = ["ReviewError", "collect_review", "render_review", "write_review"]
+113
View File
@@ -0,0 +1,113 @@
from __future__ import annotations
import os
from pkgmgr.actions.branch.gc import GITLINK
from pkgmgr.core.git.errors import GitBaseError, GitRunError
from pkgmgr.core.git.queries import FILTER_CONFIG, local_config
from pkgmgr.core.git.run import run
from .model import FileView, ReviewError
from .repo import git_text
ATTRIBUTE_BINARY = (
"{} is shown as binary because {}, not because of its content: "
"run git diff --text to see it."
)
UNSET = "a gitattributes entry unsets its diff attribute"
DRIVER = "its gitattributes diff driver {0} sets diff.{0}.binary"
RUNS = "a command git would run on this host"
def hidden_entries(top: str) -> list[str]:
listing = git_text(["ls-files", "-v", "-z"], top)
return [
f"Index entry {entry[2:]} is marked "
f"{'skip-worktree' if entry[0] == 'S' else 'assume-unchanged'}: "
"its worktree changes are not shown."
for entry in listing.split("\0")
if entry[:1] == "S" or entry[:1].islower()
]
def _binary_drivers(top: str) -> set[str]:
try:
out = run(
["config", "-z", "--type=bool", "--get-regexp", r"^diff\..*\.binary$"],
cwd=top,
scan=True,
)
except GitRunError as exc:
if exc.returncode == 1:
return set()
raise ReviewError("git config failed reading diff.<driver>.binary") from None
last: dict[str, str] = {}
for record in filter(None, out.split("\0")):
key, _, value = record.partition("\n")
last[key.removeprefix("diff.").removesuffix(".binary")] = value
return {driver for driver, value in last.items() if value == "true"}
def attribute_binary(files: list[FileView], top: str) -> list[str]:
paths = list(dict.fromkeys(view.diff.path for view in files if view.diff.binary))
if not paths:
return []
fields = git_text(["check-attr", "-z", "diff", "--", *paths], top).split("\0")
reasons = {
**{driver: DRIVER.format(driver) for driver in _binary_drivers(top)},
"unset": UNSET,
}
return [
ATTRIBUTE_BINARY.format(path, reasons[value])
for path, value in zip(fields[0::3], fields[2::3])
if value in reasons
]
def _submodule_configs(top: str, prefix: str) -> list[str]:
"""
Args:
top: Real path of the top-level working tree.
prefix: Submodule path below ``top`` whose gitlinks are walked, or ""
for the top level itself.
Returns:
One reason per populated submodule, nested ones included, whose
config defines a filter or cannot be read. A gitlink behind a symbolic
link or without a ``.git`` is skipped: git runs nothing there either.
"""
listing = git_text(["ls-files", "--stage", "-z"], os.path.join(top, prefix))
gitlinks = dict.fromkeys(
path
for meta, _, path in (entry.partition("\t") for entry in listing.split("\0"))
if meta.startswith(f"{GITLINK} ")
)
reasons: list[str] = []
for path in gitlinks:
relative = os.path.join(prefix, path)
full = os.path.join(top, relative)
if os.path.realpath(full) != full or not os.path.lexists(
os.path.join(full, ".git")
):
continue
try:
keys = local_config(full, FILTER_CONFIG)
nested = _submodule_configs(top, relative)
except (GitBaseError, ReviewError):
reasons.append(
f"the config of submodule {relative} cannot be read, so its "
"filters cannot be ruled out"
)
continue
if keys:
reasons.append(
f"the config of submodule {relative} defines {', '.join(keys)}, {RUNS}"
)
reasons.extend(nested)
return reasons
def unsafe_configs(top: str) -> list[str]:
keys = local_config(top, FILTER_CONFIG)
own = [f"the repository config defines {', '.join(keys)}, {RUNS}"] if keys else []
return own + _submodule_configs(os.path.realpath(top), "")
+153
View File
@@ -0,0 +1,153 @@
from __future__ import annotations
import os
from pkgmgr.core.git.run import END_OF_OPTIONS
from .checks import attribute_binary, hidden_entries, unsafe_configs
from .diff import parse_patch
from .files import read_note, untracked_views
from .model import Commit, Review, Section
from .repo import base_ref, git_optional, git_text, top_level
from .views import file_views
DIFF_FLAGS = [
"--no-color",
"--no-ext-diff",
"--no-textconv",
"-M",
"--submodule=short",
"--ignore-submodules=none",
"--src-prefix=a/",
"--dst-prefix=b/",
]
def _commit(sha: str, anchor: str, top: str, limit: int) -> Commit:
meta = git_text(
[
"show",
"-s",
"--no-color",
"--no-show-signature",
"--format=%h%x00%an <%ae>%x00%s%x00%b",
END_OF_OPTIONS,
sha,
],
top,
)
short, author, subject, body = meta.split("\0", 3)
patch = git_text(
[
"show",
"--format=",
"--no-show-signature",
"--diff-merges=first-parent",
*DIFF_FLAGS,
END_OF_OPTIONS,
sha,
],
top,
)
return Commit(
anchor=anchor,
short=short,
author=author,
subject=subject,
body=body.rstrip("\n"),
files=file_views(parse_patch(patch), anchor, top, limit),
)
def collect_review(
cwd: str, max_file_lines: int, base: str | None, notes: list[str]
) -> Review:
"""
Args:
cwd: Directory inside the git working copy to review.
max_file_lines: Diff lines shown per file before the rest is summarised.
base: Branch, under refs/heads/ or refs/remotes/, whose ``REF..HEAD``
commits are listed, or None for no Commits section.
notes: Regular text files shown verbatim at the top, relative to
``cwd``; a symbolic link anywhere in the path is refused.
Returns:
The collected Review model.
"""
top = top_level(cwd)
base_name, base_sha = (None, None) if base is None else base_ref(base, top)
note_items = tuple(read_note(path, cwd) for path in notes)
unsafe = unsafe_configs(top)
skipped = (
f"Unstaged changes and index tree not computed: {'; '.join(unsafe)}."
if unsafe
else None
)
staged = Section(
key="staged",
title="Staged",
files=file_views(
parse_patch(git_text(["diff", "--cached", *DIFF_FLAGS], top)),
"staged",
top,
max_file_lines,
),
skipped=None,
)
unstaged = Section(
key="unstaged",
title="Unstaged",
files=()
if skipped
else file_views(
parse_patch(git_text(["diff", *DIFF_FLAGS], top)),
"unstaged",
top,
max_file_lines,
),
skipped=skipped,
)
untracked = Section(
key="untracked",
title="Untracked",
files=untracked_views(top, max_file_lines),
skipped=None,
)
commits: tuple[Commit, ...] = ()
if base_sha is not None:
shas = git_text(["rev-list", END_OF_OPTIONS, f"{base_sha}..HEAD"], top)
commits = tuple(
_commit(sha, f"commit-{index}", top, max_file_lines)
for index, sha in enumerate(shas.split(), 1)
)
tracked = [
*staged.files,
*unstaged.files,
*(view for commit in commits for view in commit.files),
]
return Review(
top_level=top,
name=os.path.basename(top),
branch=git_optional(
["symbolic-ref", "--short", "-q", END_OF_OPTIONS, "HEAD"], top
),
head=git_optional(
["rev-parse", "--verify", "--quiet", "--short", END_OF_OPTIONS, "HEAD"],
top,
),
index_tree="(not computed, see the warnings)"
if skipped
else git_optional(["write-tree"], top) or "(index has unmerged entries)",
base=base_name,
base_short=None
if base_sha is None
else git_text(["rev-parse", "--short", END_OF_OPTIONS, base_sha], top).strip(),
warnings=tuple(
([skipped] if skipped else [])
+ hidden_entries(top)
+ attribute_binary(tracked, top)
),
notes=note_items,
sections=(staged, unstaged, untracked),
commits=commits,
)
+154
View File
@@ -0,0 +1,154 @@
from __future__ import annotations
import re
from .model import FileDiff, Hunk, Line
HUNK_HEADER = re.compile(r"^@@ -(\d+)(?:,(\d+))? \+(\d+)(?:,(\d+))? @@")
GIT_HEADER = "diff --git "
ESCAPES = {
"a": 7,
"b": 8,
"t": 9,
"n": 10,
"v": 11,
"f": 12,
"r": 13,
'"': 34,
"\\": 92,
}
HEADER_FIELDS = (
"rename from ",
"rename to ",
"copy from ",
"copy to ",
"new file mode ",
"deleted file mode ",
"old mode ",
"new mode ",
"--- ",
"+++ ",
)
def unquote(text: str) -> str:
if len(text) < 2 or not (text.startswith('"') and text.endswith('"')):
return text
body = text[1:-1]
raw = bytearray()
index = 0
while index < len(body):
char = body[index]
escaped = body[index + 1 : index + 2]
octal = body[index + 1 : index + 4]
if char == "\\" and escaped in ESCAPES:
raw.append(ESCAPES[escaped])
index += 2
elif char == "\\" and len(octal) == 3 and set(octal) <= set("01234567"):
raw.append(int(octal, 8) & 0xFF)
index += 4
else:
raw.extend(char.encode("utf-8"))
index += 1
return raw.decode("utf-8", "replace")
def _side(value: str, prefix: str) -> str | None:
value = value.removesuffix("\t")
if value == "/dev/null":
return None
return unquote(value).removeprefix(prefix)
def _file(
git_line: str, headers: list[str], pending: list[tuple[str, int, list[Line]]]
) -> FileDiff:
fields: dict[str, str] = {}
for header in headers:
for key in HEADER_FIELDS:
if header.startswith(key):
fields[key.strip()] = header[len(key) :]
break
half = (len(git_line) - 1) // 2
old_path = _side(git_line[:half], "a/")
new_path = _side(git_line[half + 1 :], "b/")
if "---" in fields:
old_path = _side(fields["---"], "a/")
if "+++" in fields:
new_path = _side(fields["+++"], "b/")
status = "modified"
for kind, moved in (("rename", "renamed"), ("copy", "copied")):
if f"{kind} from" in fields and f"{kind} to" in fields:
old_path = unquote(fields[f"{kind} from"])
new_path = unquote(fields[f"{kind} to"])
status = moved
if "new file mode" in fields:
old_path, status = None, "added"
if "deleted file mode" in fields:
new_path, status = None, "deleted"
mode_change = None
if "old mode" in fields and "new mode" in fields:
mode_change = f"{fields['old mode']} → {fields['new mode']}"
return FileDiff(
old_path=old_path,
new_path=new_path,
status=status,
binary=any(header.startswith("Binary files ") for header in headers),
mode_change=mode_change,
hunks=tuple(
Hunk(header, start, tuple(lines)) for header, start, lines in pending
),
)
def parse_patch(text: str) -> list[FileDiff]:
"""
Args:
text: Output of ``git diff`` or ``git show`` produced with ``a/`` and
``b/`` prefixes and decoded without newline translation.
Returns:
One FileDiff per ``diff --git`` block, in git's order.
"""
files: list[FileDiff] = []
git_line: str | None = None
headers: list[str] = []
pending: list[tuple[str, int, list[Line]]] = []
old_no = new_no = old_left = new_left = 0
for raw in text.split("\n"):
if pending and (old_left > 0 or new_left > 0):
lines = pending[-1][2]
sign, body = raw[:1], raw[1:]
if sign == "+":
lines.append(Line("add", None, new_no, body))
new_no, new_left = new_no + 1, new_left - 1
elif sign == "-":
lines.append(Line("del", old_no, None, body))
old_no, old_left = old_no + 1, old_left - 1
elif sign == "\\":
lines.append(Line("meta", None, None, raw))
else:
lines.append(Line("ctx", old_no, new_no, body))
old_no, new_no = old_no + 1, new_no + 1
old_left, new_left = old_left - 1, new_left - 1
continue
if raw.startswith(GIT_HEADER):
if git_line is not None:
files.append(_file(git_line, headers, pending))
git_line, headers, pending = raw[len(GIT_HEADER) :], [], []
continue
if git_line is None:
continue
match = HUNK_HEADER.match(raw)
if match:
old_no, new_no = int(match[1]), int(match[3])
old_left = 1 if match[2] is None else int(match[2])
new_left = 1 if match[4] is None else int(match[4])
pending.append((raw, new_no, []))
elif pending and raw.startswith("\\"):
pending[-1][2].append(Line("meta", None, None, raw))
elif not pending:
headers.append(raw)
if git_line is not None:
files.append(_file(git_line, headers, pending))
return files
+115
View File
@@ -0,0 +1,115 @@
from __future__ import annotations
import errno
import os
import stat
from .model import FileDiff, FileView, Hunk, Line, Note, ReviewError
from .repo import git_text
from .views import BINARY_DETAIL, file_view
READ_MAX_BYTES = 1024 * 1024
BINARY_SNIFF_BYTES = 8000
def _listed(path: str, binary: bool) -> FileDiff:
return FileDiff(
old_path=None,
new_path=path,
status="untracked",
binary=binary,
mode_change=None,
hunks=(),
)
def _read_regular(path: str, limit: int) -> tuple[bytes, int] | None:
"""
Args:
path: Absolute path, opened one component at a time without following
a symbolic link anywhere along it.
limit: Most bytes to read.
Returns:
The bytes read and the file size, or None when a component is a
symbolic link or the target is not a regular file.
"""
names = [name for name in path.split(os.sep) if name not in ("", ".")]
fd = os.open(os.sep, os.O_RDONLY | os.O_DIRECTORY)
try:
for index, name in enumerate(names, 1):
kind = os.O_NONBLOCK if index == len(names) else os.O_DIRECTORY
child = os.open(name, os.O_RDONLY | os.O_NOFOLLOW | kind, dir_fd=fd)
os.close(fd)
fd = child
info = os.fstat(fd)
if not stat.S_ISREG(info.st_mode):
return None
with os.fdopen(os.dup(fd), "rb") as handle:
return handle.read(limit), info.st_size
except OSError as exc:
if exc.errno in (errno.ELOOP, errno.ENOTDIR):
return None
raise
finally:
os.close(fd)
def _untracked_file(path: str, top: str) -> tuple[FileDiff, str | None]:
try:
read = _read_regular(os.path.join(top, path), READ_MAX_BYTES + 1)
except OSError as exc:
return _listed(path, False), f"Unreadable: {exc.strerror}."
if read is None:
return _listed(path, False), "Not a regular file, contents not shown."
data, size = read
if len(data) > READ_MAX_BYTES:
return _listed(path, False), f"Large file ({size} bytes), contents not shown."
if b"\0" in data[:BINARY_SNIFF_BYTES]:
return _listed(path, True), BINARY_DETAIL
lines = data.decode("utf-8", "replace").split("\n")
if lines[-1] == "":
lines.pop()
added = tuple(
Line("add", None, number, text) for number, text in enumerate(lines, 1)
)
hunks = (Hunk(f"@@ -0,0 +1,{len(added)} @@", 1, added),) if added else ()
return (
FileDiff(
old_path=None,
new_path=path,
status="untracked",
binary=False,
mode_change=None,
hunks=hunks,
),
None,
)
def untracked_views(top: str, limit: int) -> tuple[FileView, ...]:
listing = git_text(["ls-files", "--others", "--exclude-standard", "-z"], top)
views = []
for index, path in enumerate(sorted(p for p in listing.split("\0") if p), 1):
diff, detail = _untracked_file(path, top)
views.append(file_view(diff, f"untracked-{index}", top, limit, detail))
return tuple(views)
def read_note(path: str, cwd: str) -> Note:
try:
read = _read_regular(
os.path.abspath(os.path.join(cwd, path)), READ_MAX_BYTES + 1
)
except OSError as exc:
raise ReviewError(f"cannot read note {path!r}: {exc.strerror}") from None
if read is None:
raise ReviewError(
f"cannot read note {path!r}: a symbolic link or not a regular file"
)
data, size = read
if len(data) > READ_MAX_BYTES:
raise ReviewError(
f"cannot read note {path!r}: {size} bytes, more than {READ_MAX_BYTES}"
)
return Note(title=os.path.basename(path), text=data.decode("utf-8", "replace"))
+136
View File
@@ -0,0 +1,136 @@
from __future__ import annotations
from dataclasses import dataclass
class ReviewError(RuntimeError):
pass
@dataclass(frozen=True)
class Line:
kind: str
old: int | None
new: int | None
text: str
@dataclass(frozen=True)
class Hunk:
header: str
new_start: int
lines: tuple[Line, ...]
@dataclass(frozen=True)
class FileDiff:
old_path: str | None
new_path: str | None
status: str
binary: bool
mode_change: str | None
hunks: tuple[Hunk, ...]
@property
def path(self) -> str:
return self.new_path if self.new_path is not None else str(self.old_path)
@property
def moved(self) -> bool:
return self.status in ("renamed", "copied")
@property
def added(self) -> int:
return sum(
1 for hunk in self.hunks for line in hunk.lines if line.kind == "add"
)
@property
def removed(self) -> int:
return sum(
1 for hunk in self.hunks for line in hunk.lines if line.kind == "del"
)
@dataclass(frozen=True)
class FileView:
anchor: str
diff: FileDiff
detail: str | None
link_base: str | None
link: str | None
hunks: tuple[Hunk, ...]
hidden: int
@property
def search(self) -> str:
if self.diff.moved:
return f"{self.diff.old_path} {self.diff.new_path}"
return self.diff.path
@dataclass(frozen=True)
class Section:
key: str
title: str
files: tuple[FileView, ...]
skipped: str | None
@property
def added(self) -> int:
return sum(view.diff.added for view in self.files)
@property
def removed(self) -> int:
return sum(view.diff.removed for view in self.files)
@dataclass(frozen=True)
class Commit:
anchor: str
short: str
author: str
subject: str
body: str
files: tuple[FileView, ...]
@property
def added(self) -> int:
return sum(view.diff.added for view in self.files)
@property
def removed(self) -> int:
return sum(view.diff.removed for view in self.files)
@dataclass(frozen=True)
class Note:
title: str
text: str
@dataclass(frozen=True)
class Review:
top_level: str
name: str
branch: str | None
head: str | None
index_tree: str
base: str | None
base_short: str | None
warnings: tuple[str, ...]
notes: tuple[Note, ...]
sections: tuple[Section, ...]
commits: tuple[Commit, ...]
@property
def commit_files(self) -> int:
return sum(len(commit.files) for commit in self.commits)
@property
def commit_added(self) -> int:
return sum(commit.added for commit in self.commits)
@property
def commit_removed(self) -> int:
return sum(commit.removed for commit in self.commits)
+86
View File
@@ -0,0 +1,86 @@
from __future__ import annotations
import base64
import hashlib
import os
import tempfile
from jinja2 import Environment, FileSystemLoader, StrictUndefined
from .collect import collect_review
from .model import Review, ReviewError
TEMPLATE = "review.html.j2"
INVISIBLE = {
code: f"<U+{code:04X}>"
for code in (
*range(0x200B, 0x2010),
*range(0x202A, 0x202F),
*range(0x2066, 0x206A),
0x061C,
0xFEFF,
)
}
def _csp_hash(text: str) -> str:
digest = hashlib.sha256(str(text).encode("utf-8")).digest()
return "sha256-" + base64.b64encode(digest).decode("ascii")
def _visible(value: object) -> object:
# not isinstance: Markup.translate() would splice the marks in unescaped,
# and macro output is Markup whose text was already marked when built
return value.translate(INVISIBLE) if type(value) is str else value
def render_review(review: Review) -> str:
environment = Environment(
loader=FileSystemLoader(os.path.dirname(os.path.abspath(__file__))),
autoescape=True,
undefined=StrictUndefined,
trim_blocks=True,
lstrip_blocks=True,
keep_trailing_newline=True,
finalize=_visible,
)
environment.filters["csp_hash"] = _csp_hash
return environment.get_template(TEMPLATE).render(review=review)
def write_review(
cwd: str, out: str, max_file_lines: int, base: str | None, notes: list[str]
) -> str:
"""
Args:
cwd: Directory inside the git working copy to review.
out: Output HTML path, relative to ``cwd`` unless absolute; written as
a new mode 0600 file renamed onto it, so a symbolic link there is
replaced, never followed.
max_file_lines: Diff lines shown per file before the rest is summarised.
base: Branch whose ``REF..HEAD`` commits are listed, or None.
notes: Text files shown verbatim at the top of the page.
Returns:
The absolute path of the written page.
"""
html = render_review(
collect_review(cwd=cwd, max_file_lines=max_file_lines, base=base, notes=notes)
)
path = os.path.abspath(os.path.join(cwd, out))
directory = os.path.dirname(path)
try:
os.makedirs(directory, exist_ok=True)
fd, temp = tempfile.mkstemp(prefix=".review-", suffix=".tmp", dir=directory)
try:
with os.fdopen(fd, "w", encoding="utf-8", newline="\n") as handle:
handle.write(html)
handle.flush()
os.fsync(handle.fileno())
os.replace(temp, path)
except OSError:
os.unlink(temp)
raise
except OSError as exc:
raise ReviewError(f"cannot write {path!r}: {exc.strerror}") from None
return path
+73
View File
@@ -0,0 +1,73 @@
from __future__ import annotations
import subprocess
from pkgmgr.core.git.errors import GitBaseError
from pkgmgr.core.git.run import END_OF_OPTIONS, SCAN_FLAGS, SCAN_TIMEOUT, _scan_env, run
from .model import ReviewError
BRANCH_NAMESPACES = ("refs/heads/", "refs/remotes/")
def git_text(args: list[str], cwd: str) -> str:
try:
result = subprocess.run(
["git", *SCAN_FLAGS, "-c", "core.quotepath=false", *args],
cwd=cwd,
env=_scan_env(),
stdin=subprocess.DEVNULL,
capture_output=True,
timeout=SCAN_TIMEOUT,
check=False,
)
except subprocess.TimeoutExpired:
raise ReviewError(f"git {args[0]} timed out") from None
if result.returncode != 0:
stderr = result.stderr.decode("utf-8", "replace").strip().splitlines()
reason = stderr[0] if stderr else f"exit code {result.returncode}"
raise ReviewError(f"git {args[0]} failed: {reason}")
return result.stdout.decode("utf-8", "replace")
def git_optional(args: list[str], cwd: str) -> str | None:
try:
return run(args, cwd=cwd, scan=True) or None
except GitBaseError:
return None
def top_level(cwd: str) -> str:
top = git_optional(["rev-parse", "--show-toplevel"], cwd)
if top is None:
raise ReviewError(f"not inside a git working copy: {cwd!r}")
return top
def base_ref(base: str, top: str) -> tuple[str, str]:
names = (
[base]
if base.startswith(BRANCH_NAMESPACES)
else [namespace + base for namespace in BRANCH_NAMESPACES]
)
refs: dict[str, str] = {}
for name in names:
sha = git_optional(
["show-ref", "--verify", "--hash", END_OF_OPTIONS, name], top
)
if sha is not None:
refs[name] = sha
if len(refs) > 1:
raise ReviewError(f"ambiguous base ref: {base!r} names {' and '.join(refs)}")
if not refs:
raise ReviewError(
f"unknown base ref: {base!r}, expected a branch under "
f"{' or '.join(BRANCH_NAMESPACES)}"
)
[(name, sha)] = refs.items()
commit = git_optional(
["rev-parse", "--verify", "--quiet", END_OF_OPTIONS, f"{sha}^{{commit}}"], top
)
if commit is None:
raise ReviewError(f"base ref {name!r} does not point to a commit")
return name, commit
+347
View File
@@ -0,0 +1,347 @@
{% set status_letters = {"added": "A", "deleted": "D", "modified": "M", "renamed": "R", "copied": "C", "untracked": "U"} %}
{% set signs = {"add": "+", "del": "-", "ctx": "", "meta": ""} %}
{% set style %}
:root {
color-scheme: light dark;
--bg: #f6f8fa;
--panel: #ffffff;
--subtle: #f6f8fa;
--border: #d1d9e0;
--border-muted: #e1e6eb;
--text: #1f2328;
--muted: #59636e;
--accent: #0969da;
--accent-bg: #ddf4ff;
--add-fg: #1a7f37;
--add-bg: #dafbe1;
--add-line: #e6ffec;
--add-num: #ccffd8;
--del-fg: #cf222e;
--del-bg: #ffebe9;
--del-line: #ffebe9;
--del-num: #ffd7d5;
--hunk-bg: #ddf4ff;
--hunk-fg: #59636e;
--warn-fg: #9a6700;
--warn-bg: #fff8c5;
--shadow: 0 1px 2px rgba(31, 35, 40, 0.06);
--mono: ui-monospace, SFMono-Regular, "SF Mono", Menlo, Consolas, "Liberation Mono", monospace;
--sans: system-ui, -apple-system, "Segoe UI", "Noto Sans", Helvetica, Arial, sans-serif;
}
@media (prefers-color-scheme: dark) {
:root {
--bg: #0d1117;
--panel: #151b23;
--subtle: #1b222c;
--border: #3d444d;
--border-muted: #2a313c;
--text: #e6edf3;
--muted: #9198a1;
--accent: #4493f8;
--accent-bg: rgba(56, 139, 253, 0.15);
--add-fg: #3fb950;
--add-bg: rgba(46, 160, 67, 0.2);
--add-line: rgba(46, 160, 67, 0.15);
--add-num: rgba(63, 185, 80, 0.3);
--del-fg: #f85149;
--del-bg: rgba(248, 81, 73, 0.2);
--del-line: rgba(248, 81, 73, 0.1);
--del-num: rgba(248, 81, 73, 0.3);
--hunk-bg: rgba(56, 139, 253, 0.1);
--hunk-fg: #9198a1;
--warn-fg: #d29922;
--warn-bg: rgba(187, 128, 9, 0.15);
--shadow: none;
}
}
*, *::before, *::after { box-sizing: border-box; }
[hidden] { display: none !important; }
html { scroll-padding-top: 64px; }
body { margin: 0; background: var(--bg); color: var(--text); font: 14px/1.5 var(--sans); }
a { color: var(--accent); text-decoration: none; }
a:hover { text-decoration: underline; }
pre, code { font-family: var(--mono); }
.muted { color: var(--muted); font-weight: 400; }
.masthead { background: var(--panel); border-bottom: 1px solid var(--border); }
.masthead-inner, .toolbar-inner, main { max-width: 1280px; margin: 0 auto; padding-left: 32px; padding-right: 32px; }
.masthead-inner { padding-top: 24px; padding-bottom: 20px; }
.eyebrow { margin: 0 0 4px; color: var(--muted); font-size: 12px; font-weight: 600; letter-spacing: 0.06em; text-transform: uppercase; }
h1 { display: flex; flex-wrap: wrap; align-items: center; gap: 10px; margin: 0; font-size: 22px; font-weight: 600; }
.pill { padding: 2px 10px; border-radius: 999px; background: var(--accent-bg); color: var(--accent); font: 500 12px/20px var(--mono); }
.facts { display: grid; grid-template-columns: max-content 1fr; gap: 4px 20px; margin: 16px 0 0; font-size: 13px; }
.facts dt { color: var(--muted); }
.facts dd { margin: 0; font-family: var(--mono); overflow-wrap: anywhere; }
.warnings, .skipped { border: 1px solid var(--warn-fg); border-radius: 8px; background: var(--warn-bg); color: var(--warn-fg); font-size: 13px; overflow-wrap: anywhere; }
.warnings { margin: 16px 0 0; padding: 10px 14px 10px 34px; }
.skipped { margin: 0; padding: 14px 16px; border-style: dashed; }
.totals { display: flex; flex-wrap: wrap; gap: 10px; margin-top: 18px; }
.total { display: flex; flex-direction: column; gap: 2px; min-width: 160px; padding: 10px 14px; border: 1px solid var(--border); border-radius: 8px; background: var(--subtle); color: var(--text); }
.total:hover { border-color: var(--accent); text-decoration: none; }
.total .label { color: var(--muted); font-size: 12px; font-weight: 600; letter-spacing: 0.04em; text-transform: uppercase; }
.total .files { font-size: 15px; font-weight: 600; }
.stat { display: inline-flex; gap: 6px; font: 600 12px var(--mono); white-space: nowrap; }
.stat .add { color: var(--add-fg); }
.stat .del { color: var(--del-fg); }
.toolbar { position: sticky; top: 0; z-index: 10; border-bottom: 1px solid var(--border); background: var(--panel); background: color-mix(in srgb, var(--panel) 86%, transparent); backdrop-filter: saturate(180%) blur(10px); box-shadow: var(--shadow); }
.toolbar-inner { display: flex; flex-wrap: wrap; align-items: center; gap: 8px; padding-top: 8px; padding-bottom: 8px; }
.jumps { display: flex; flex-wrap: wrap; gap: 2px; }
.jumps a { display: inline-flex; align-items: center; gap: 6px; padding: 4px 10px; border-radius: 6px; color: var(--text); font-weight: 500; }
.jumps a:hover { background: var(--accent-bg); text-decoration: none; }
.count { min-width: 20px; padding: 0 6px; border: 1px solid var(--border-muted); border-radius: 999px; background: var(--subtle); color: var(--muted); font-size: 12px; line-height: 18px; text-align: center; }
.spacer { flex: 1; }
input[type="search"] { width: 260px; max-width: 100%; padding: 5px 10px; border: 1px solid var(--border); border-radius: 6px; background: var(--bg); color: var(--text); font: 13px var(--mono); }
input[type="search"]:focus { outline: 2px solid var(--accent); outline-offset: -1px; }
button { padding: 5px 12px; border: 1px solid var(--border); border-radius: 6px; background: var(--subtle); color: var(--text); font: 500 13px var(--sans); cursor: pointer; }
button:hover { border-color: var(--muted); }
main { padding-top: 24px; padding-bottom: 80px; }
.group { margin-bottom: 36px; }
h2 { display: flex; flex-wrap: wrap; align-items: baseline; gap: 12px; margin: 0 0 12px; font-size: 17px; font-weight: 600; }
.note { margin: 0 0 14px; border: 1px solid var(--border); border-radius: 8px; background: var(--panel); overflow: hidden; }
.note figcaption { padding: 8px 14px; border-bottom: 1px solid var(--border); background: var(--subtle); font: 600 13px var(--mono); }
.note pre, .commit-body { margin: 0; padding: 12px 14px; font-size: 13px; line-height: 1.55; white-space: pre-wrap; overflow-wrap: anywhere; }
details.file, details.commit { margin-bottom: 12px; border: 1px solid var(--border); border-radius: 8px; background: var(--panel); box-shadow: var(--shadow); overflow: hidden; }
summary { display: flex; align-items: center; gap: 10px; padding: 8px 12px; cursor: pointer; list-style: none; }
summary::-webkit-details-marker { display: none; }
summary::before { content: ""; flex: none; width: 6px; height: 6px; margin: 0 4px 0 2px; border-right: 2px solid var(--muted); border-bottom: 2px solid var(--muted); transform: rotate(-45deg); transition: transform 0.12s ease; }
details[open] > summary::before { transform: rotate(45deg); }
details.file > summary { background: var(--subtle); font: 13px var(--mono); }
details.file[open] > summary, details.commit[open] > summary { border-bottom: 1px solid var(--border); }
summary:hover { background: var(--accent-bg); }
.badge { flex: none; width: 20px; height: 20px; border-radius: 5px; font: 700 11px/20px var(--sans); text-align: center; }
.badge.added, .badge.untracked { background: var(--add-bg); color: var(--add-fg); }
.badge.deleted { background: var(--del-bg); color: var(--del-fg); }
.badge.modified { background: var(--accent-bg); color: var(--accent); }
.badge.renamed, .badge.copied { background: var(--warn-bg); color: var(--warn-fg); }
.path { flex: 1; min-width: 0; font-weight: 600; overflow-wrap: anywhere; }
.path .old, .path .arrow { color: var(--muted); font-weight: 400; }
.tag { flex: none; padding: 0 8px; border: 1px solid var(--border); border-radius: 999px; color: var(--muted); font: 500 11px/18px var(--sans); }
.open { flex: none; font: 500 12px var(--sans); }
.placeholder, .empty { margin: 0; padding: 14px 16px; color: var(--muted); font-style: italic; }
.empty { border: 1px dashed var(--border); border-radius: 8px; }
table.diff { width: 100%; border-collapse: collapse; font: 12.5px/20px var(--mono); tab-size: 4; }
.diff td { height: 20px; padding: 0; vertical-align: top; }
td.num { width: 1%; min-width: 50px; padding: 0 10px; color: var(--muted); text-align: right; white-space: nowrap; user-select: none; }
td.num a { color: inherit; }
td.num a:hover { color: var(--accent); }
td.sign { width: 20px; padding-left: 10px; color: var(--muted); user-select: none; }
td.code { padding-right: 16px; white-space: pre-wrap; overflow-wrap: anywhere; }
tr.add td { background: var(--add-line); }
tr.add td.num { background: var(--add-num); }
tr.add td.sign { color: var(--add-fg); }
tr.del td { background: var(--del-line); }
tr.del td.num { background: var(--del-num); }
tr.del td.sign { color: var(--del-fg); }
tr.hunk td { padding-top: 4px; padding-bottom: 4px; background: var(--hunk-bg); color: var(--hunk-fg); }
tr.meta td.code { color: var(--muted); font-style: italic; }
tr.more td { padding: 8px 16px; border-top: 1px dashed var(--border); background: var(--subtle); color: var(--muted); font: italic 12px var(--sans); text-align: center; }
details.commit > summary { padding: 10px 14px; }
.sha { flex: none; padding: 1px 8px; border: 1px solid var(--border-muted); border-radius: 6px; background: var(--subtle); font-size: 12px; }
.subject { flex: 1; min-width: 0; font-weight: 600; overflow-wrap: anywhere; }
.commit-meta { padding: 10px 14px 0; color: var(--muted); font-size: 13px; }
.commit-files { padding: 12px 14px 2px; }
@media (max-width: 720px) {
.masthead-inner, .toolbar-inner, main { padding-left: 14px; padding-right: 14px; }
input[type="search"] { width: 100%; }
.spacer { display: none; }
}
{% endset %}
{% set script %}
(function () {
"use strict";
var files = Array.prototype.slice.call(document.querySelectorAll("details.file"));
var filter = document.getElementById("filter");
var counter = document.getElementById("filter-count");
function setOpen(open) {
document.querySelectorAll("details").forEach(function (item) {
item.open = open;
});
}
function applyFilter() {
var query = filter.value.trim().toLowerCase();
var shown = 0;
files.forEach(function (item) {
var match = query === "" || item.dataset.path.toLowerCase().indexOf(query) !== -1;
item.hidden = !match;
shown += match ? 1 : 0;
});
counter.textContent = query === "" ? "" : shown + " of " + files.length + " files";
}
document.getElementById("expand-all").addEventListener("click", function () {
setOpen(true);
});
document.getElementById("collapse-all").addEventListener("click", function () {
setOpen(false);
});
filter.addEventListener("input", applyFilter);
document.addEventListener("keydown", function (event) {
if (event.key === "/" && document.activeElement !== filter) {
event.preventDefault();
filter.focus();
} else if (event.key === "Escape" && document.activeElement === filter) {
filter.value = "";
applyFilter();
filter.blur();
}
});
applyFilter();
})();
{% endset %}
{% macro stat(added, removed) %}<span class="stat"><span class="add">+{{ added }}</span><span class="del">&minus;{{ removed }}</span></span>{% endmacro %}
{% macro plural(count, word) %}{{ count }} {{ word }}{{ "" if count == 1 else "s" }}{% endmacro %}
{% macro file_block(file, open) %}
<details class="file" id="{{ file.anchor }}" data-path="{{ file.search }}"{% if open %} open{% endif %}>
<summary>
<span class="badge {{ file.diff.status }}" title="{{ file.diff.status }}">{{ status_letters[file.diff.status] }}</span>
<span class="path">{% if file.diff.moved %}<span class="old">{{ file.diff.old_path }}</span><span class="arrow"> &rarr; </span>{% endif %}{{ file.diff.path }}</span>
{% if file.diff.mode_change %}
<span class="tag">mode {{ file.diff.mode_change }}</span>
{% endif %}
{% if file.diff.binary %}
<span class="tag">binary</span>
{% endif %}
{{ stat(file.diff.added, file.diff.removed) }}
{% if file.link %}
<a class="open" href="{{ file.link }}" title="Open in VS Code">Open &#8599;</a>
{% endif %}
</summary>
{% if file.detail %}
<p class="placeholder">{{ file.detail }}</p>
{% elif not file.hunks %}
<p class="placeholder">No content changes.</p>
{% else %}
<table class="diff">
{% for hunk in file.hunks %}
<tbody>
<tr class="hunk"><td class="num" colspan="2"></td><td class="code" colspan="2">{{ hunk.header }}</td></tr>
{% for line in hunk.lines %}
<tr class="{{ line.kind }}"><td class="num">{{ "" if line.old is none else line.old }}</td><td class="num">{% if line.new is not none and file.link_base %}<a href="{{ file.link_base }}:{{ line.new }}">{{ line.new }}</a>{% elif line.new is not none %}{{ line.new }}{% endif %}</td><td class="sign">{{ signs[line.kind] }}</td><td class="code">{{ line.text }}</td></tr>
{% endfor %}
</tbody>
{% endfor %}
{% if file.hidden %}
<tbody><tr class="more"><td colspan="4">&hellip; {{ plural(file.hidden, "more line") }} not shown</td></tr></tbody>
{% endif %}
</table>
{% endif %}
</details>
{% endmacro %}
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1">
<meta http-equiv="Content-Security-Policy" content="default-src 'none'; style-src '{{ style|csp_hash }}'; script-src '{{ script|csp_hash }}'; base-uri 'none'; form-action 'none'">
<meta name="referrer" content="no-referrer">
<title>Review: {{ review.name }}{% if review.branch %} ({{ review.branch }}){% endif %}</title>
<style>{{ style }}</style>
</head>
<body>
<header class="masthead">
<div class="masthead-inner">
<p class="eyebrow">Working copy review</p>
<h1>{{ review.name }} <span class="pill">{{ review.branch or "detached HEAD" }}</span></h1>
<dl class="facts">
<dt>Top level</dt><dd>{{ review.top_level }}</dd>
<dt>Branch</dt><dd>{{ review.branch or "(detached HEAD)" }}</dd>
<dt>HEAD</dt><dd>{{ review.head or "(no commits yet)" }}</dd>
<dt>Index tree</dt><dd>{{ review.index_tree }}</dd>
{% if review.base is not none %}
<dt>Base</dt><dd>{{ review.base }} @ {{ review.base_short }}</dd>
{% endif %}
</dl>
{% if review.warnings %}
<ul class="warnings">
{% for warning in review.warnings %}
<li>{{ warning }}</li>
{% endfor %}
</ul>
{% endif %}
<div class="totals">
{% for section in review.sections %}
<a class="total" href="#{{ section.key }}"><span class="label">{{ section.title }}</span><span class="files">{{ plural(section.files|length, "file") }}</span>{{ stat(section.added, section.removed) }}</a>
{% endfor %}
{% if review.base is not none %}
<a class="total" href="#commits"><span class="label">Commits</span><span class="files">{{ plural(review.commits|length, "commit") }} &middot; {{ plural(review.commit_files, "file") }}</span>{{ stat(review.commit_added, review.commit_removed) }}</a>
{% endif %}
</div>
</div>
</header>
<nav class="toolbar">
<div class="toolbar-inner">
<div class="jumps">
{% if review.notes %}
<a href="#notes">Notes <span class="count">{{ review.notes|length }}</span></a>
{% endif %}
{% for section in review.sections %}
<a href="#{{ section.key }}">{{ section.title }} <span class="count">{{ section.files|length }}</span></a>
{% endfor %}
{% if review.base is not none %}
<a href="#commits">Commits <span class="count">{{ review.commits|length }}</span></a>
{% endif %}
</div>
<div class="spacer"></div>
<input id="filter" type="search" placeholder="Filter files by path ( / )" autocomplete="off" spellcheck="false" aria-label="Filter files by path">
<span id="filter-count" class="muted"></span>
<button type="button" id="expand-all">Expand all</button>
<button type="button" id="collapse-all">Collapse all</button>
</div>
</nav>
<main>
{% if review.notes %}
<section class="group" id="notes">
<h2>Notes</h2>
{% for note in review.notes %}
<figure class="note">
<figcaption>{{ note.title }}</figcaption>
<pre>
{{ note.text }}</pre>
</figure>
{% endfor %}
</section>
{% endif %}
{% for section in review.sections %}
<section class="group" id="{{ section.key }}" data-files="{{ section.files|length }}" data-added="{{ section.added }}" data-removed="{{ section.removed }}">
<h2>{{ section.title }} <span class="muted">{{ plural(section.files|length, "file") }}</span>{{ stat(section.added, section.removed) }}</h2>
{% if section.skipped %}
<p class="skipped">{{ section.skipped }}</p>
{% else %}
{% for file in section.files %}
{{ file_block(file, true) }}
{% else %}
<p class="empty">No changes.</p>
{% endfor %}
{% endif %}
</section>
{% endfor %}
{% if review.base is not none %}
<section class="group" id="commits" data-commits="{{ review.commits|length }}" data-files="{{ review.commit_files }}" data-added="{{ review.commit_added }}" data-removed="{{ review.commit_removed }}">
<h2>Commits <span class="muted">{{ review.base }}..HEAD &middot; {{ plural(review.commits|length, "commit") }}</span>{{ stat(review.commit_added, review.commit_removed) }}</h2>
{% for commit in review.commits %}
<details class="commit" id="{{ commit.anchor }}" open>
<summary>
<code class="sha">{{ commit.short }}</code>
<span class="subject">{{ commit.subject }}</span>
{{ stat(commit.added, commit.removed) }}
</summary>
<div class="commit-meta">{{ commit.author }}</div>
{% if commit.body %}
<pre class="commit-body">
{{ commit.body }}</pre>
{% endif %}
<div class="commit-files">
{% for file in commit.files %}
{{ file_block(file, false) }}
{% else %}
<p class="placeholder">No file changes.</p>
{% endfor %}
</div>
</details>
{% else %}
<p class="empty">No commits between {{ review.base }} and HEAD.</p>
{% endfor %}
</section>
{% endif %}
</main>
<script>{{ script }}</script>
</body>
</html>
+67
View File
@@ -0,0 +1,67 @@
from __future__ import annotations
import os
from urllib.parse import quote
from .model import FileDiff, FileView, Hunk
BINARY_DETAIL = "Binary file, contents not shown."
def _first_changed_line(hunks: tuple[Hunk, ...]) -> int:
for hunk in hunks:
position = hunk.new_start
for line in hunk.lines:
if line.kind in ("add", "del"):
return max(position, 1)
if line.new is not None:
position = line.new + 1
return 1
def _truncate(hunks: tuple[Hunk, ...], limit: int) -> tuple[tuple[Hunk, ...], int]:
shown: list[Hunk] = []
budget = limit
for hunk in hunks:
if budget <= 0:
break
lines = hunk.lines[:budget]
shown.append(Hunk(hunk.header, hunk.new_start, lines))
budget -= len(lines)
total = sum(len(hunk.lines) for hunk in hunks)
return tuple(shown), total - (limit - budget)
def file_view(
diff: FileDiff, anchor: str, top: str, limit: int, detail: str | None
) -> FileView:
link_base = None
if diff.new_path is not None:
link_base = "vscode://file" + quote(os.path.join(top, diff.new_path))
hunks, hidden = _truncate(diff.hunks, limit)
return FileView(
anchor=anchor,
diff=diff,
detail=detail,
link_base=link_base,
link=None
if link_base is None
else f"{link_base}:{_first_changed_line(diff.hunks)}",
hunks=hunks,
hidden=hidden,
)
def file_views(
diffs: list[FileDiff], prefix: str, top: str, limit: int
) -> tuple[FileView, ...]:
return tuple(
file_view(
diff,
f"{prefix}-{index}",
top,
limit,
BINARY_DETAIL if diff.binary else None,
)
for index, diff in enumerate(diffs, 1)
)
Whitespace-only changes.
+237
View File
@@ -0,0 +1,237 @@
from __future__ import annotations
import os
import secrets
import stat
from contextlib import suppress
from dataclasses import dataclass, field
import yaml
from pkgmgr.actions.branch.nofollow import DIR_FLAGS
from pkgmgr.core.git.errors import GitBaseError
from pkgmgr.core.git.run import run
FLOW_DIR = "pkgmgr"
FLOW_FILE = "flow.yml"
RELATIVE = f"{FLOW_DIR}/{FLOW_FILE}"
VERSION = 1
MAX_BYTES = 64 * 1024
FOLDER_MODE = 0o700
FILE_MODE = 0o600
KEYS = ("version", "branch", "pull", "push")
REQUIRED = ("version", "pull", "push")
# O_NONBLOCK: opening a planted FIFO for reading would block until a writer comes
READ_FLAGS = os.O_RDONLY | os.O_NOFOLLOW | os.O_NONBLOCK
WRITE_FLAGS = os.O_WRONLY | os.O_CREAT | os.O_EXCL | os.O_NOFOLLOW
DETACHED = "a detached HEAD"
NONE_LISTED = "(none)"
class FlowError(ValueError):
pass
@dataclass
class Flow:
pull: list[str] = field(default_factory=list)
push: list[str] = field(default_factory=list)
branch: str | None = None
source: str = "none"
warnings: list[str] = field(default_factory=list)
def describe(exc: BaseException) -> str:
return (str(exc).strip().splitlines() or [type(exc).__name__])[0]
def _ref_format(name: str) -> bool:
# check-ref-format needs no repository, so / keeps any repository config
# out; --branch would expand @{-N} into another name inside one
try:
return (
run(["check-ref-format", "--branch", name], cwd=os.sep, scan=True) == name
)
except (GitBaseError, OSError, ValueError):
return False
def check_name(kind: str, name: object) -> str:
"""
Args:
kind: pull, push or branch, for the message.
name: an entry from the file or the command line.
Returns:
name, once it is a string git accepts as a branch name that does not
start with `-`; a remote-tracking name such as origin/main passes.
Raises:
FlowError: name is anything else.
"""
if not isinstance(name, str) or name.startswith("-") or not _ref_format(name):
raise FlowError(f"{kind} {name!r} is not a valid branch name")
return name
def check_names(kind: str, names: object) -> list[str]:
"""
Args:
kind: pull or push, for the message.
names: the list from the file or the command line.
Returns:
The names in order, each passed through check_name.
Raises:
FlowError: names is not a list or an entry is invalid.
"""
if not isinstance(names, list):
raise FlowError(f"{kind} must be a list of branch names")
return [check_name(kind, name) for name in names]
def _folder(git_dir: str, create: bool) -> int:
parent = os.open(git_dir, os.O_RDONLY | os.O_DIRECTORY)
try:
if create:
with suppress(FileExistsError):
os.mkdir(FLOW_DIR, FOLDER_MODE, dir_fd=parent)
return os.open(FLOW_DIR, DIR_FLAGS, dir_fd=parent)
finally:
os.close(parent)
def _read_text(git_dir: str) -> str | None:
try:
folder = _folder(git_dir, create=False)
except FileNotFoundError:
return None
try:
fd = os.open(FLOW_FILE, READ_FLAGS, dir_fd=folder)
except FileNotFoundError:
return None
finally:
os.close(folder)
# os.fdopen refuses a directory but leaves its fd open, so check first
if not stat.S_ISREG(os.fstat(fd).st_mode):
os.close(fd)
raise FlowError("not a regular file")
with os.fdopen(fd, "rb") as fh:
data = fh.read(MAX_BYTES + 1)
if len(data) > MAX_BYTES:
raise FlowError(f"larger than {MAX_BYTES} bytes")
return data.decode("utf-8")
def _parse(text: str) -> tuple[str | None, list[str], list[str]]:
data = yaml.safe_load(text)
if not isinstance(data, dict):
raise FlowError("not a mapping")
unknown = [repr(key) for key in data if key not in KEYS]
if unknown:
raise FlowError(f"unknown keys {', '.join(unknown)}")
missing = [key for key in REQUIRED if key not in data]
if missing:
raise FlowError(f"missing keys {', '.join(missing)}")
# bool is an int, and True == 1
if type(data["version"]) is not int or data["version"] != VERSION:
raise FlowError(f"version must be {VERSION}")
branch = data.get("branch")
if branch is not None:
check_name("branch", branch)
return branch, check_names("pull", data["pull"]), check_names("push", data["push"])
def read_flow(git_dir: str) -> Flow:
"""
Read `<git_dir>/pkgmgr/flow.yml`, which an agent may have written.
Args:
git_dir: the worktree's own git directory.
Returns:
The flow with source file; source none when there is no file; source
invalid, empty lists and a warning when the file is a symbolic link,
no regular file, larger than MAX_BYTES or malformed. Never raises.
"""
try:
text = _read_text(git_dir)
if text is None:
return Flow()
branch, pull, push = _parse(text)
# RecursionError: safe_load recurses once per nesting level
except (OSError, ValueError, yaml.YAMLError, RecursionError) as exc:
return Flow(source="invalid", warnings=[f"{RELATIVE} ignored: {describe(exc)}"])
return Flow(pull=pull, push=push, branch=branch, source="file")
def write_flow(
git_dir: str, branch: str | None, pull: list[str], push: list[str]
) -> None:
"""
Replace the flow atomically, never writing through a symbolic link.
Args:
git_dir: the worktree's own git directory.
branch: the worktree's branch now; None when detached.
pull: refs merged into the branch, in order, already through check_names.
push: branches the branch is merged into, in order, likewise checked.
Raises:
OSError: `pkgmgr` is a symbolic link or no directory, or a write failed.
"""
data = {"version": VERSION, "branch": branch, "pull": pull, "push": push}
text = yaml.safe_dump(data, sort_keys=False, default_flow_style=False)
folder = _folder(git_dir, create=True)
tmp = f".{FLOW_FILE}.{secrets.token_hex(8)}"
try:
fd = os.open(tmp, WRITE_FLAGS, FILE_MODE, dir_fd=folder)
with os.fdopen(fd, "w", encoding="utf-8") as fh:
fh.write(text)
fh.flush()
os.fsync(fh.fileno())
os.replace(tmp, FLOW_FILE, src_dir_fd=folder, dst_dir_fd=folder)
finally:
with suppress(FileNotFoundError):
os.unlink(tmp, dir_fd=folder)
os.close(folder)
def clear_flow(git_dir: str) -> None:
"""
Remove the flow file; a symbolic link in its place goes, never its target.
Args:
git_dir: the worktree's own git directory.
Raises:
OSError: `pkgmgr` is a symbolic link or no directory, or the unlink failed.
"""
try:
folder = _folder(git_dir, create=False)
except FileNotFoundError:
return
try:
with suppress(FileNotFoundError):
os.unlink(FLOW_FILE, dir_fd=folder)
finally:
os.close(folder)
def mismatch(flow: Flow, branch: str | None) -> list[str]:
"""
Args:
flow: a flow from read_flow.
branch: the worktree's current branch; None when detached.
Returns:
One warning when the file was written on another branch, else none.
"""
if flow.source != "file" or flow.branch == branch:
return []
warning = (
f"flow was set on {flow.branch or DETACHED}, "
f"the worktree is on {branch or DETACHED}"
)
return [warning]
+158
View File
@@ -0,0 +1,158 @@
from __future__ import annotations
import json
import sys
from pkgmgr.core.git.errors import GitBaseError
from pkgmgr.core.git.run import run
from .flow import (
NONE_LISTED,
Flow,
FlowError,
check_names,
clear_flow,
describe,
mismatch,
read_flow,
write_flow,
)
Located = tuple[str, str, str | None]
def locate(path: str) -> Located:
"""
Args:
path: any directory inside the worktree.
Returns:
The worktree root, its own git directory (`.git` of the main
worktree, `<common>/worktrees/<id>` of a linked one) and its branch,
None when detached.
Raises:
GitBaseError: path is not inside a worktree, e.g. a bare repository.
OSError: path is no directory.
ValueError: a path git printed spans lines.
"""
out = run(
["rev-parse", "--absolute-git-dir", "--show-toplevel"], cwd=path, scan=True
)
git_dir, top = out.split("\n")
try:
ref = run(["symbolic-ref", "--quiet", "HEAD"], cwd=top, scan=True)
except GitBaseError:
ref = ""
branch = ref.removeprefix("refs/heads/") if ref.startswith("refs/heads/") else None
return top, git_dir, branch
def _located(path: str) -> Located | None:
try:
return locate(path)
except (GitBaseError, OSError, ValueError) as exc:
print(
f"Refusing: {path} is not a git worktree: {describe(exc)}",
file=sys.stderr,
)
return None
def _report(located: Located, flow: Flow, as_json: bool) -> None:
top, git_dir, branch = located
result = {
"worktree": top,
"git_dir": git_dir,
"branch": branch,
"flow_branch": flow.branch,
"pull": flow.pull,
"push": flow.push,
"source": flow.source,
"warnings": flow.warnings + mismatch(flow, branch),
}
if as_json:
print(json.dumps(result, indent=2))
return
for warning in result["warnings"]:
print(f"WARN {warning}")
for key in ("pull", "push"):
print(f"{key}: {', '.join(result[key]) or NONE_LISTED}")
def worktree_flow_show(path: str, *, as_json: bool) -> int:
"""
Print the flow of the worktree at path.
Args:
path: any directory inside the worktree.
as_json: print the result as JSON instead of text.
Returns:
0, also without a flow or with an invalid file; 2 when path is not a
git worktree.
"""
located = _located(path)
if located is None:
return 2
_report(located, read_flow(located[1]), as_json)
return 0
def worktree_flow_set(
path: str, *, pull: list[str], push: list[str], as_json: bool
) -> int:
"""
Replace both lists of the worktree's flow and print the result.
Args:
path: any directory inside the worktree.
pull: refs merged into the branch before a gate, deploy or merge, in
order; may be empty.
push: branches the branch is merged into when finished, in order;
may be empty.
as_json: print the result as JSON instead of text.
Returns:
0 when written, 1 when the write failed, 2 on an invalid name or when
path is not a git worktree; nothing is written unless 0.
"""
try:
pull, push = check_names("pull", pull), check_names("push", push)
except FlowError as exc:
print(f"Refusing: {exc}", file=sys.stderr)
return 2
located = _located(path)
if located is None:
return 2
try:
write_flow(located[1], located[2], pull, push)
except OSError as exc:
print(f"Cannot write the flow of {located[0]}: {exc}", file=sys.stderr)
return 1
_report(located, read_flow(located[1]), as_json)
return 0
def worktree_flow_clear(path: str, *, as_json: bool) -> int:
"""
Remove the worktree's flow and print the empty result.
Args:
path: any directory inside the worktree.
as_json: print the result as JSON instead of text.
Returns:
0 when no flow is left, 1 when the removal failed, 2 when path is not
a git worktree.
"""
located = _located(path)
if located is None:
return 2
try:
clear_flow(located[1])
except OSError as exc:
print(f"Cannot clear the flow of {located[0]}: {exc}", file=sys.stderr)
return 1
_report(located, read_flow(located[1]), as_json)
return 0
+144
View File
@@ -0,0 +1,144 @@
from __future__ import annotations
import os
import re
import shutil
import signal
import subprocess
import time
from .trust import root_resolved, trusted_path
INIT_HOOK = "worktree-init"
FINI_HOOK = "worktree-fini"
MAKEFILES = ("GNUmakefile", "makefile", "Makefile")
MAKEFILE_LIMIT = 1 << 20
HOOK_TIMEOUT = 1800
HOOK_GRACE = 5
PLACEHOLDER = re.compile(r"\{(path|slot)\}")
def defined_in(path: str, target: str) -> str | None:
"""
Name the makefile in path that defines target, read as text and never run:
make expands $(shell) and include lines even under -n.
Args:
path: directory that may hold a makefile.
target: make target name.
Returns:
The first makefile, in make's lookup order, with a `<target>:` rule;
None when none has one.
"""
rule = re.compile(rf"^{re.escape(target)}\s*:", re.MULTILINE)
for name in MAKEFILES:
file = os.path.join(path, name)
if os.path.islink(file) or not os.path.isfile(file):
continue
with open(file, encoding="utf-8", errors="replace") as fh:
if rule.search(fh.read(MAKEFILE_LIMIT)):
return name
return None
def hook_command(name: str, argv: list[str], path: str, slot: int | None) -> list[str]:
"""
Fill a pinned hook command in one pass and resolve its program.
Args:
name: the hook, for error messages.
argv: the command pinned in the `worktree_hooks` config.
path: the worktree, replacing `{path}`.
slot: the worktree's slot, replacing `{slot}`; None leaves it as is.
Returns:
The argv to run; its program is an absolute path, for a bare name the
resolved file that was checked, so a multi-call binary sees that
file's name; pin the link's absolute path to keep the link's name.
Raises:
RuntimeError: the program is neither absolute nor a bare name found,
as a root-controlled file, on a root-owned PATH entry no group or
other user can write; anything relative resolves in the worktree.
"""
def fill(match: re.Match[str]) -> str:
if match.group(1) == "path":
return path
return match.group(0) if slot is None else str(slot)
filled = [PLACEHOLDER.sub(fill, arg) for arg in argv]
program = filled[0] if filled else ""
if not os.path.isabs(program):
found = (
shutil.which(program, path=trusted_path())
if program and "/" not in program
else None
)
resolved = root_resolved(found) if found else None
if resolved is None:
raise RuntimeError(
f"{name} pin {program!r} is neither an absolute path nor a "
"root-owned command on a root-owned PATH entry"
)
program = resolved
return [program, *filled[1:]]
def run_hook(
name: str, command: list[str], path: str, *, check: bool, **kwargs
) -> subprocess.CompletedProcess:
"""
Run a resolved hook command in the worktree.
Args:
name: the hook, for error messages.
command: the argv from hook_command.
path: the worktree to run in.
check: raise CalledProcessError on a non-zero exit.
kwargs: further subprocess.Popen arguments, such as output handling.
Returns:
The completed process; stdin was closed, PATH held only its
root-controlled entries and HOOK_TIMEOUT applied to its whole process
group.
Raises:
RuntimeError: the hook ran longer than HOOK_TIMEOUT, or no PATH entry
is root-controlled.
"""
with subprocess.Popen(
command,
cwd=path,
stdin=subprocess.DEVNULL,
env={**os.environ, "PATH": trusted_path()},
start_new_session=True,
**kwargs,
) as process:
try:
stdout, stderr = process.communicate(timeout=HOOK_TIMEOUT)
except subprocess.TimeoutExpired as exc:
_stop_group(process)
raise RuntimeError(f"{name} timed out after {HOOK_TIMEOUT} s") from exc
if check and process.returncode:
raise subprocess.CalledProcessError(
process.returncode, command, output=stdout, stderr=stderr
)
return subprocess.CompletedProcess(command, process.returncode, stdout, stderr)
def _stop_group(process: subprocess.Popen) -> None:
for sig in (signal.SIGTERM, signal.SIGKILL):
try:
os.killpg(process.pid, sig)
except OSError:
return
deadline = time.monotonic() + HOOK_GRACE
while time.monotonic() < deadline:
process.poll()
try:
os.killpg(process.pid, 0)
except OSError:
return
time.sleep(0.05)
+193
View File
@@ -0,0 +1,193 @@
from __future__ import annotations
import json
import time
from dataclasses import asdict
from pkgmgr.actions.branch.gc import Item, human_size, prepare
DAY = 86400
ORDER = ("merged-clean", "merged-dirty", "stale-open", "stale", "active")
PROPOSALS = {
"merged-clean": "pkgmgr gc",
"merged-dirty": "ask",
"stale-open": "ask",
"stale": "git worktree prune",
"active": "keep",
}
GUARDS = ("process:", "lock:")
HEADER = (
"Class",
"Branch",
"Worktree",
"Dirty",
"Unpushed",
"Last activity",
"Proposal",
)
def classify(item: Item, now: float, limit: float) -> str:
"""
Sort one scanned item into an orphan class.
Args:
item: item from a scan with activity recorded.
now: current epoch seconds.
limit: seconds of inactivity after which unfinished work is stale.
Returns:
One of ORDER.
"""
if item.state == "stale":
return "stale"
if (
item.state == "protected"
or (item.branch or "").startswith("archive/")
or item.skip == "checked out in main worktree"
):
return "active"
if item.merged:
return "merged-dirty" if item.state == "dirty" else "merged-clean"
pending = item.state == "dirty" or bool(item.unpushed) or not item.remotes
idle = item.last_activity is not None and now - item.last_activity > limit
guarded = bool(item.skip) and item.skip.startswith(GUARDS)
return "stale-open" if pending and idle and not guarded else "active"
def proposal(item: Item, cls: str) -> str:
if cls == "active":
return "keep"
if item.skip and item.skip.startswith(GUARDS):
return f"keep ({item.skip})"
return PROPOSALS[cls]
def _cells(row: dict, now: float, sizes: bool) -> list[str]:
activity = row["last_activity"]
when = (
f"{time.strftime('%Y-%m-%d', time.localtime(activity))} "
f"({int((now - activity) // DAY)} d)"
if activity
else "-"
)
values = [
row["class"],
row["branch"] or "(detached)",
row["worktree"] or "-",
*(
[human_size(row["size_bytes"]) if row["size_bytes"] else "-"]
if sizes
else []
),
str(row["dirty"]),
"-" if row["unpushed"] is None else str(row["unpushed"]),
when,
row["proposal"],
]
return [value.replace("|", "\\|") for value in values]
def _table(rows: list[dict], now: float, sizes: bool) -> list[str]:
header = [*HEADER[:3], *(["Size"] if sizes else []), *HEADER[3:]]
lines = [
"| " + " | ".join(header) + " |",
"| " + " | ".join("---" for _ in header) + " |",
]
return lines + ["| " + " | ".join(_cells(r, now, sizes)) + " |" for r in rows]
def worktree_orphans(
repo_roots: list[str],
*,
stale_days: int,
top: int,
as_json: bool,
include: list[str],
exclude: list[str],
merged_into: list[str],
sizes: bool,
) -> int:
"""
List merged and abandoned branches and worktrees with a proposal each.
Read-only apart from the fetch the scan does. Merged covers ancestry, a
merged pull request, an integration line and patch equivalence.
Args:
repo_roots: main worktree roots of the repositories to scan.
stale_days: unfinished work idle for longer is stale-open.
top: rows per class in the text table; 0 shows the counts only.
as_json: print every row, active ones included, as JSON.
include: only items whose branch or worktree path matches one of these.
exclude: drop items whose branch or worktree path matches one of these.
merged_into: integration lines (refs) that also count as merged.
sizes: measure each worktree's disk usage and order by it.
Returns:
0 on success, 2 when refused.
"""
options = {
"merged_into": merged_into,
"sizes": sizes,
"min_age": None,
"scope": "all",
"activity": True,
"hooks": {},
}
prepared = prepare(repo_roots, include, exclude, options)
if prepared is None:
return 2
repos, host_ns, chosen = prepared
now = time.time()
limit = stale_days * DAY
report = []
for repo in repos:
rows = []
for item in repo.items:
if not chosen(item):
continue
cls = classify(item, now, limit)
rows.append({**asdict(item), "class": cls, "proposal": proposal(item, cls)})
rows.sort(
key=lambda r: (
ORDER.index(r["class"]),
-(r["size_bytes"] or 0) if sizes else 0,
r["last_activity"] or 0,
)
)
report.append({"repo": repo.repo, "warnings": repo.warnings, "rows": rows})
if as_json:
print(
json.dumps(
{
"host_pid_namespace": host_ns,
"stale_days": stale_days,
"repos": report,
},
indent=2,
)
)
return 0
for entry in report:
print(f"== {entry['repo']} (stale after {stale_days} days)")
for warning in entry["warnings"]:
print(f"WARN {warning}")
for cls in ORDER:
count = sum(r["class"] == cls for r in entry["rows"])
print(f"- {cls}: {count} ({PROPOSALS[cls]})")
shown = [
row
for cls in ORDER[:-1]
for row in [r for r in entry["rows"] if r["class"] == cls][:top]
]
if shown:
print()
print("\n".join(_table(shown, now, sizes)))
if not host_ns:
print(
"WARN busy and lock checks unavailable: not in the host PID namespace"
)
return 0
+53
View File
@@ -0,0 +1,53 @@
from __future__ import annotations
import os
SLOT_FILE = "pkgmgr-slot"
def used_slots(common_dir: str) -> set[int]:
base = os.path.join(common_dir, "worktrees")
slots: set[int] = set()
for entry in os.listdir(base) if os.path.isdir(base) else []:
try:
with open(os.path.join(base, entry, SLOT_FILE), encoding="utf-8") as fh:
slots.add(int(fh.read().strip()))
except (OSError, ValueError):
continue
return slots
def next_slot(used: set[int]) -> int:
slot = 1
while slot in used:
slot += 1
return slot
def new_worktree_meta(path: str) -> str:
"""
Args:
path: a worktree this process just added, whose `.git` file names
its metadata.
Returns:
The worktree's own git directory, `<common>/worktrees/<id>`.
"""
with open(os.path.join(path, ".git"), encoding="utf-8") as fh:
return os.path.normpath(
os.path.join(path, fh.read().strip().removeprefix("gitdir: "))
)
def write_slot(path: str, slot: int) -> None:
"""
Record a slot in the metadata of a worktree this process just added.
Args:
path: the new worktree, whose `.git` file names its metadata.
slot: the slot to record.
"""
with open(
os.path.join(new_worktree_meta(path), SLOT_FILE), "w", encoding="utf-8"
) as fh:
fh.write(f"{slot}\n")
+93
View File
@@ -0,0 +1,93 @@
from __future__ import annotations
import os
import stat
ROOT_UID = 0
UID_MAP = "/proc/self/uid_map"
INITIAL_MAP = [[0, 0, 4294967295]]
OVERFLOW_UID = "/proc/sys/kernel/overflowuid"
SHARED_WRITE = stat.S_IWGRP | stat.S_IWOTH
def root_uids() -> frozenset[int]:
"""
Returns:
The owner uids that stand for root. In the initial user namespace,
whose uid_map is the identity over every uid, that is 0. In any other
namespace a mapped uid 0 may be any user, so only the overflow uid
counts: the real root's files show up under it there, as do those of
every other unmapped owner. Empty when a file it needs is unreadable
or this process's own uid is unmapped, since its own files then show
up as the overflow uid too.
"""
try:
with open(UID_MAP, encoding="ascii") as fh:
ranges = [[int(n) for n in line.split()] for line in fh if line.strip()]
if ranges == INITIAL_MAP:
return frozenset({ROOT_UID})
with open(OVERFLOW_UID, encoding="ascii") as fh:
overflow = int(fh.read())
except (OSError, ValueError):
return frozenset()
if os.geteuid() == overflow:
return frozenset()
return frozenset({overflow})
def root_resolved(path: str) -> str | None:
"""
Args:
path: an absolute file or directory.
Returns:
The path, resolved once, when it and every directory above it are
owned by root and writable by neither group nor others, so no other
user can replace what it names; None otherwise. Callers keep and use
the returned path: resolving path again can follow a link swapped in
after this check. In the initial user namespace a component on a
nosuid mount also yields None: a desktop user can mount an image of
root-owned files through udisks2 or FUSE, always nosuid, while the
system's own directories never are; every bubblewrap mount is nosuid,
so the rule cannot apply inside one.
"""
owners = root_uids()
initial = owners == {ROOT_UID}
resolved = os.path.realpath(path)
current = resolved
while True:
try:
info = os.stat(current)
nosuid = initial and os.statvfs(current).f_flag & os.ST_NOSUID
except OSError:
return None
if info.st_uid not in owners or info.st_mode & SHARED_WRITE or nosuid:
return None
parent = os.path.dirname(current)
if parent == current:
return resolved
current = parent
def trusted_path() -> str:
"""
Returns:
The PATH entries that resolve to root-controlled directories, else
those of os.defpath, each in its resolved form and joined by
os.pathsep, so a symbolic link another user can retarget never stays
on it.
Raises:
RuntimeError: no entry of either qualifies; an empty PATH would make
the hook search its working directory, the worktree.
"""
for source in (os.get_exec_path(), os.defpath.split(os.pathsep)):
resolved = (root_resolved(entry) for entry in source if os.path.isabs(entry))
entries = dict.fromkeys(
entry for entry in resolved if entry and os.path.isdir(entry)
)
if entries:
return os.pathsep.join(entries)
raise RuntimeError(
"no PATH entry is a root-owned directory closed to other writers"
)
+286
View File
@@ -0,0 +1,286 @@
from __future__ import annotations
import json
import os
import re
import sys
from pkgmgr.core.git.errors import GitRunError
from pkgmgr.core.git.queries import FILTER_CONFIG, local_config
from pkgmgr.core.git.run import END_OF_OPTIONS, run
from .flow import NONE_LISTED, FlowError, check_name, check_names, write_flow
from .hooks import INIT_HOOK, defined_in, hook_command, run_hook
from .slots import new_worktree_meta, next_slot, used_slots, write_slot
PORT_PREFIX = re.compile(r"^\d+/")
HOST_AND_PATH = re.compile(r"([^:/]+)[:/](.*)")
NETWORK_CONFIG = (
r"^(core\.(sshcommand|gitproxy)|remote\..*\.(uploadpack|receivepack)"
r"|credential\..*helper|url\..*\.insteadof|protocol\..*allow)$"
)
def worktree_location(url: str) -> str:
"""
Map a remote URL to `<host>/<owner>/<repo>`.
Args:
url: remote URL in https, ssh or scp form.
Returns:
The host followed by the repository path, without `.git` or a port;
a local path loses its leading slash, so it always stays relative.
"""
rest = url.removesuffix(".git").split("://", 1)[-1].split("@", 1)[-1]
match = HOST_AND_PATH.match(rest)
if not match:
return rest.strip("/")
host, path = match.groups()
return f"{host}/{PORT_PREFIX.sub('', path).strip('/')}"
def _common_dir(cwd: str) -> str:
return run(
["rev-parse", "--path-format=absolute", "--git-common-dir"], cwd=cwd, scan=True
)
def _inside(path: str, root: str) -> bool:
real, base = os.path.realpath(path), os.path.realpath(root)
return os.path.commonpath([real, base]) == base
def _ref_exists(repo: str, ref: str) -> bool:
try:
run(
["rev-parse", "--verify", "--quiet", f"{ref}^{{commit}}"],
cwd=repo,
scan=True,
)
except GitRunError:
return False
return True
def _commit_of(repo: str, ref: str) -> str:
# worktree add reads a start point that begins with "-" as HEAD even after
# "--" or --end-of-options, so it only ever gets the resolved commit id
return run(
["rev-parse", "--verify", "--quiet", END_OF_OPTIONS, f"{ref}^{{commit}}"],
cwd=repo,
scan=True,
)
def _default_base(repo: str) -> str:
try:
return run(
["symbolic-ref", "--short", "refs/remotes/origin/HEAD"], cwd=repo, scan=True
)
except GitRunError:
return "origin/main" if _ref_exists(repo, "origin/main") else "origin/master"
def _local_branch(repo: str, ref: str) -> str:
try:
remotes = run(["remote"], cwd=repo, scan=True).split()
except GitRunError:
return ref
owner = next((r for r in remotes if ref.startswith(f"{r}/")), None)
return ref.removeprefix(f"{owner}/") if owner else ref
def _invalid_flow(pull: list[str] | None, push: list[str] | None) -> str | None:
try:
for kind, names in (("pull", pull), ("push", push)):
if names is not None:
check_names(kind, names)
except FlowError as exc:
return str(exc)
return None
def _record_flow(
root: str,
path: str,
branch: str,
lists: tuple[list[str] | None, list[str] | None],
base: str | None,
warnings: list[str],
) -> dict[str, list[str]] | None:
pull, push = lists
if pull is None or push is None:
default = base or _default_base(root)
try:
check_name("base", default)
except FlowError as exc:
warnings.append(f"flow not set: {exc}")
return None
pull = [default] if pull is None else pull
push = [_local_branch(root, default)] if push is None else push
try:
write_flow(new_worktree_meta(path), branch, pull, push)
except OSError as exc:
warnings.append(f"flow not written: {exc}")
return None
return {"pull": pull, "push": push}
def worktree_up(
repo_dir: str,
branch: str,
*,
base: str | None,
worktrees_dir: str,
path: str | None,
hooks: dict[str, dict[str, list[str]]],
as_json: bool,
pull: list[str] | None,
push: list[str] | None,
) -> int:
"""
Check a branch out into its own worktree under the standard path.
Args:
repo_dir: any directory inside the repository.
branch: branch to check out; created from base when it exists nowhere.
base: start point of a new branch; None takes origin's default branch.
worktrees_dir: root of the standard path
`<worktrees_dir>/<host>/<owner>/<repo>/<branch>`.
path: exact checkout path instead of the standard one, e.g. a
directory an agent sandbox releases; None keeps the standard path.
hooks: pinned hooks by managed repository root realpath, from the
CLI; the worktree-init hook runs only when the repository root is
pinned there, never from the Makefile or by origin's URL.
as_json: print the result as JSON instead of text.
pull: refs merged into the branch before a gate, deploy or merge, in
order, for the worktree's flow; None takes the base.
push: branches the branch is merged into when finished, in order;
None takes the base.
Returns:
0 when the worktree is ready, 1 when the worktree-init hook failed
(the worktree stays), 2 when refused.
"""
try:
common = _common_dir(repo_dir)
root = os.path.dirname(common)
url = run(["remote", "get-url", "--", "origin"], cwd=root, scan=True)
run(["check-ref-format", "--branch", branch], cwd=root, scan=True)
filters = local_config(root, FILTER_CONFIG)
except GitRunError as exc:
print(f"Refusing: {str(exc).splitlines()[0]}", file=sys.stderr)
return 2
if filters:
print(
f"Refusing: {', '.join(filters)} in the repository config would run "
"on the checkout",
file=sys.stderr,
)
return 2
invalid = _invalid_flow(pull, push)
if invalid:
print(f"Refusing: {invalid}", file=sys.stderr)
return 2
location = worktree_location(url)
if path:
path = os.path.abspath(os.path.expanduser(path))
else:
standard = os.path.expanduser(worktrees_dir)
path = os.path.join(standard, location, branch)
if not _inside(path, standard):
print(f"Refusing: {path} lies outside {standard}", file=sys.stderr)
return 2
if os.path.lexists(path):
print(f"Refusing: {path} already exists", file=sys.stderr)
return 2
slot = next_slot(used_slots(common))
pinned = hooks.get(os.path.realpath(root), {}).get(INIT_HOOK)
try:
command = hook_command(INIT_HOOK, pinned, path, slot) if pinned else None
except RuntimeError as exc:
print(f"Refusing: {exc}", file=sys.stderr)
return 2
warnings: list[str] = []
network = local_config(root, NETWORK_CONFIG)
if network:
warnings.append(f"fetch skipped: {', '.join(network)} in the repository config")
else:
try:
run(
["fetch", "--quiet", "--no-auto-maintenance", "origin"],
cwd=root,
scan=True,
)
except GitRunError as exc:
warnings.append(f"fetch failed: {str(exc).splitlines()[0]}")
os.makedirs(os.path.dirname(path), exist_ok=True)
start: str | None = None
if _ref_exists(root, f"refs/heads/{branch}"):
origin, add = "existing", ["worktree", "add", "--", path, branch]
elif _ref_exists(root, f"refs/remotes/origin/{branch}"):
origin = f"tracking origin/{branch}"
add = [
"worktree",
"add",
"--track",
"-b",
branch,
"--",
path,
f"origin/{branch}",
]
else:
start = base or _default_base(root)
origin = f"new from {start}"
add = ["worktree", "add", "--no-track", "-b", branch, "--", path]
try:
if start is not None:
add.append(_commit_of(root, start))
run(add, cwd=root, scan=True)
except GitRunError as exc:
print(f"Refusing: {str(exc).splitlines()[0]}", file=sys.stderr)
return 2
write_slot(path, slot)
flow = _record_flow(root, path, branch, (pull, push), start or base, warnings)
hook = "none"
code = 0
if command:
try:
ran = run_hook(INIT_HOOK, command, path, check=False, stdout=sys.stderr)
failure = None if ran.returncode == 0 else str(ran.returncode)
except (RuntimeError, OSError) as exc:
failure = str(exc)
hook = INIT_HOOK if failure is None else f"{INIT_HOOK} failed ({failure})"
code = 0 if failure is None else 1
else:
makefile = defined_in(path, INIT_HOOK)
if makefile:
hook = f"not pinned: {INIT_HOOK} in {makefile}"
result = {
"path": path,
"branch": branch,
"origin": origin,
"slot": slot,
"hook": hook,
"flow": flow,
"warnings": warnings,
}
if as_json:
print(json.dumps(result, indent=2))
else:
for warning in warnings:
print(f"WARN {warning}")
for key in ("path", "branch", "origin", "slot", "hook"):
print(f"{key:<8} {result[key]}")
for key, names in (flow or {}).items():
print(f"{key:<8} {', '.join(names) or NONE_LISTED}")
return code
+5
View File
@@ -8,8 +8,10 @@ from .mirror import handle_mirror_command
from .publish import handle_publish
from .release import handle_release
from .repos import handle_repos_command
from .review import handle_review
from .tools import handle_tools_command
from .version import handle_version
from .worktree import handle_gc, handle_worktree
__all__ = [
"handle_archive",
@@ -17,11 +19,14 @@ __all__ = [
"handle_changelog",
"handle_code_scanning",
"handle_config",
"handle_gc",
"handle_make",
"handle_mirror_command",
"handle_publish",
"handle_release",
"handle_repos_command",
"handle_review",
"handle_tools_command",
"handle_version",
"handle_worktree",
]
+129 -4
View File
@@ -1,9 +1,102 @@
from __future__ import annotations
import os
import sys
from pkgmgr.actions.branch import close_branch, drop_branch, open_branch
from pkgmgr.actions.branch import (
branch_purge,
cleanup,
close_branch,
drop_branch,
open_branch,
tree,
)
from pkgmgr.actions.branch.gc import main_repo_root
from pkgmgr.cli.context import CLIContext
from pkgmgr.core.git.errors import GitBaseError
from pkgmgr.core.repository.dir import get_repo_dir
def _repo_dir(ctx: CLIContext, repo: dict) -> str:
return (
os.path.expanduser(str(repo["directory"]))
if repo.get("directory")
else get_repo_dir(ctx.repositories_base_dir, repo)
)
def _known_dirs(ctx: CLIContext) -> list[str]:
return [_repo_dir(ctx, r) for r in ctx.all_repositories]
def pinned_hooks(ctx: CLIContext) -> dict[str, dict[str, list[str]]]:
"""
Bind the `worktree_hooks` config to managed repositories, never to a
remote URL, which the repository's own config sets.
Args:
ctx: CLI context; each `worktree_hooks` key names the
`<provider>/<account>/<repository>` of a `repositories` entry.
Returns:
Realpath of each pinned managed repository's directory -> hook name
-> argv; keys that match no managed repository are dropped.
"""
pins = ctx.config_merged["worktree_hooks"]
return {
os.path.realpath(_repo_dir(ctx, repo)): pins[key]
for repo in ctx.config_merged["repositories"]
if all(repo.get(part) for part in ("provider", "account", "repository"))
and (key := f"{repo['provider']}/{repo['account']}/{repo['repository']}")
in pins
}
def refresh_tree(roots: list[str]) -> None:
for root in roots:
try:
tree.refresh_tree_entry(root)
except (OSError, ValueError, KeyError, GitBaseError) as exc:
print(f"WARN tree file not rewritten for {root}: {exc}", file=sys.stderr)
def scan_repo_roots(args, ctx: CLIContext) -> list[str]:
"""
Resolve the repositories a scan covers: the current one, or every
pkgmgr-managed one with --all. Exits with 2 outside a repository.
"""
if not args.all:
try:
return [main_repo_root(".")]
except GitBaseError as exc:
print(f"Not inside a git repository: {exc}", file=sys.stderr)
sys.exit(2)
return [d for d in _known_dirs(ctx) if os.path.exists(os.path.join(d, ".git"))]
def scan_options(args) -> dict:
return {
"include": args.branch,
"exclude": args.exclude,
"merged_into": args.merged_into,
"sizes": args.sizes,
}
def run_cleanup(args, ctx: CLIContext, scope: str) -> None:
roots = scan_repo_roots(args, ctx)
code = cleanup(
roots,
scope=scope,
apply=args.apply,
as_json=args.json,
min_age=args.min_age,
hooks=pinned_hooks(ctx),
**scan_options(args),
)
if args.apply:
refresh_tree(roots)
sys.exit(code)
def handle_branch(args, ctx: CLIContext) -> None:
@@ -11,16 +104,46 @@ def handle_branch(args, ctx: CLIContext) -> None:
Handle `pkgmgr branch` subcommands.
Currently supported:
- pkgmgr branch open [<name>] [--base <branch>]
- pkgmgr branch close [<name>] [--base <branch>] [--force|-f]
- pkgmgr branch drop [<name>] [--base <branch>] [--force|-f]
- pkgmgr branch open [<name>] [--base <branch>]
- pkgmgr branch close [<name>] [--base <branch>] [--force|-f]
- pkgmgr branch drop [<name>] [--base <branch>] [--force|-f]
- pkgmgr branch cleanup [--apply] [--json] [--all] [--branch N]... [--exclude N]...
[--merged-into REF]... [--sizes] [--min-age HOURS]
- pkgmgr branch purge [--all] [--branch N]... [--exclude N]...
[--merged-into REF]... [--sizes] [--min-age HOURS]
- pkgmgr branch tree [--format json|yaml] [--output FILE]
"""
if args.subcommand == "tree":
sys.exit(
tree.branch_tree(
_known_dirs(ctx),
fmt=args.format,
output=args.output,
worktrees_root=tree.WORKTREES_ROOT,
)
)
if args.subcommand == "cleanup":
run_cleanup(args, ctx, "branch")
if args.subcommand == "purge":
roots = scan_repo_roots(args, ctx)
code = branch_purge(
roots,
min_age=args.min_age,
hooks=pinned_hooks(ctx),
**scan_options(args),
)
refresh_tree(roots)
sys.exit(code)
if args.subcommand == "open":
open_branch(
name=getattr(args, "name", None),
base_branch=getattr(args, "base", "main"),
cwd=".",
)
refresh_tree([main_repo_root(".")])
return
if args.subcommand == "close":
@@ -30,6 +153,7 @@ def handle_branch(args, ctx: CLIContext) -> None:
cwd=".",
force=getattr(args, "force", False),
)
refresh_tree([main_repo_root(".")])
return
if args.subcommand == "drop":
@@ -39,6 +163,7 @@ def handle_branch(args, ctx: CLIContext) -> None:
cwd=".",
force=getattr(args, "force", False),
)
refresh_tree([main_repo_root(".")])
return
print(f"Unknown branch subcommand: {args.subcommand}")
+23
View File
@@ -0,0 +1,23 @@
from __future__ import annotations
import os
import sys
from pathlib import Path
from pkgmgr.actions.review import ReviewError, write_review
from pkgmgr.cli.context import CLIContext
def handle_review(args, ctx: CLIContext) -> None:
try:
path = write_review(
cwd=os.getcwd(),
out=args.out,
max_file_lines=args.max_file_lines,
base=args.base,
notes=args.note,
)
except ReviewError as exc:
print(f"[ERROR] {exc}", file=sys.stderr)
sys.exit(2)
print(Path(path).as_uri())
+136
View File
@@ -0,0 +1,136 @@
from __future__ import annotations
import json
import os
import sys
from pkgmgr.actions.branch.gc import main_repo_root
from pkgmgr.actions.gc.report import disk_report, render_text
from pkgmgr.actions.worktree.flow_actions import (
worktree_flow_clear,
worktree_flow_set,
worktree_flow_show,
)
from pkgmgr.actions.worktree.orphans import worktree_orphans
from pkgmgr.actions.worktree.up import worktree_up
from pkgmgr.cli.context import CLIContext
from .branch import (
pinned_hooks,
refresh_tree,
run_cleanup,
scan_options,
scan_repo_roots,
)
MEBIBYTE = 1 << 20
FLOW_ACTIONS = ("set", "clear")
def _flow_usage(action: str, args) -> str | None:
if action == "show" and args.path is not None:
return f"unknown flow action {args.target!r}; use set or clear"
if action != "set":
flags = {
"--pull": args.pull,
"--push": args.push,
"--no-pull": args.no_pull,
"--no-push": args.no_push,
}
used = [flag for flag, value in flags.items() if value]
return f"{', '.join(used)} only with `flow set`" if used else None
for kind in ("pull", "push"):
if bool(getattr(args, kind)) == getattr(args, f"no_{kind}"):
return f"flow set takes either --{kind} or --no-{kind}"
return None
def _flow(args) -> int:
action, path = (
(args.target, args.path)
if args.target in FLOW_ACTIONS
else ("show", args.target)
)
problem = _flow_usage(action, args)
if problem:
print(f"Refusing: {problem}", file=sys.stderr)
return 2
target = path or "."
if action == "show":
return worktree_flow_show(target, as_json=args.json)
if action == "set":
code = worktree_flow_set(
target, pull=args.pull or [], push=args.push or [], as_json=args.json
)
else:
code = worktree_flow_clear(target, as_json=args.json)
if code == 0:
refresh_tree([main_repo_root(target)])
return code
def handle_worktree(args, ctx: CLIContext) -> None:
"""
Handle `pkgmgr worktree` subcommands.
- pkgmgr worktree up <branch> [--base REF] [--path DIR] [--pull REF ...]
[--push BRANCH ...] [--json]
- pkgmgr worktree flow [PATH] [--json]
- pkgmgr worktree flow set [PATH] (--pull REF ... | --no-pull)
(--push BRANCH ... | --no-push) [--json]
- pkgmgr worktree flow clear [PATH] [--json]
- pkgmgr worktree cleanup [--apply] [--json] [scan options] [--min-age HOURS]
- pkgmgr worktree orphans --stale-days N --top N [--json] [scan options]
"""
if args.subcommand == "up":
code = worktree_up(
".",
args.name,
base=args.base,
worktrees_dir=ctx.config_merged["directories"]["worktrees"],
path=args.path,
hooks=pinned_hooks(ctx),
as_json=args.json,
pull=args.pull,
push=args.push,
)
refresh_tree([main_repo_root(".")])
sys.exit(code)
if args.subcommand == "flow":
sys.exit(_flow(args))
if args.subcommand == "cleanup":
run_cleanup(args, ctx, "worktree")
if args.subcommand == "orphans":
sys.exit(
worktree_orphans(
scan_repo_roots(args, ctx),
stale_days=args.stale_days,
top=args.top,
as_json=args.json,
**scan_options(args),
)
)
print(f"Unknown worktree subcommand: {args.subcommand}")
sys.exit(2)
def handle_gc(args, ctx: CLIContext) -> None:
if not args.report:
run_cleanup(args, ctx, "all")
if args.apply:
print(
"--report only reads; it cannot be combined with --apply", file=sys.stderr
)
sys.exit(2)
report = disk_report(
[os.path.abspath(os.path.expanduser(root)) for root in args.root],
scan_repo_roots(args, ctx),
args.threshold_mb * MEBIBYTE,
args.docker,
)
print(json.dumps(report, indent=2) if args.json else render_text(report))
sys.exit(0)
+15
View File
@@ -10,13 +10,16 @@ from pkgmgr.cli.commands import (
handle_changelog,
handle_code_scanning,
handle_config,
handle_gc,
handle_make,
handle_mirror_command,
handle_publish,
handle_release,
handle_repos_command,
handle_review,
handle_tools_command,
handle_version,
handle_worktree,
)
from pkgmgr.cli.context import CLIContext
from pkgmgr.cli.proxy import maybe_handle_proxy
@@ -69,6 +72,10 @@ def dispatch_command(args, ctx: CLIContext) -> None:
handle_code_scanning(args, ctx)
return
if args.command == "review":
handle_review(args, ctx)
return
commands_with_selection = {
"install",
"update",
@@ -163,6 +170,14 @@ def dispatch_command(args, ctx: CLIContext) -> None:
handle_branch(args, ctx)
return
if args.command == "worktree":
handle_worktree(args, ctx)
return
if args.command == "gc":
handle_gc(args, ctx)
return
if args.command == "mirror":
handle_mirror_command(ctx, args, selected)
return
+6
View File
@@ -10,6 +10,7 @@ from .changelog_cmd import add_changelog_subparser
from .code_scanning_cmd import add_code_scanning_subparser
from .common import SortedSubParsersAction
from .config_cmd import add_config_subparsers
from .gc_cmd import add_gc_subparser
from .install_update import add_install_update_subparsers
from .list_cmd import add_list_subparser
from .make_cmd import add_make_subparsers
@@ -17,7 +18,9 @@ from .mirror_cmd import add_mirror_subparsers
from .navigation_cmd import add_navigation_subparsers
from .publish_cmd import add_publish_subparser
from .release_cmd import add_release_subparser
from .review_cmd import add_review_subparser
from .version_cmd import add_version_subparser
from .worktree_cmd import add_worktree_subparsers
def create_parser(description_text: str) -> argparse.ArgumentParser:
@@ -58,6 +61,9 @@ def create_parser(description_text: str) -> argparse.ArgumentParser:
add_navigation_subparsers(subparsers)
add_branch_subparsers(subparsers)
add_review_subparser(subparsers)
add_worktree_subparsers(subparsers)
add_gc_subparser(subparsers)
add_release_subparser(subparsers)
add_publish_subparser(subparsers)
+97
View File
@@ -96,3 +96,100 @@ def add_branch_subparsers(
action="store_true",
help="Skip confirmation prompt and drop the branch directly",
)
branch_cleanup = branch_subparsers.add_parser(
"cleanup",
help=(
"Remove merged branches that no worktree holds, locally and on the "
"remotes (dry-run unless --apply)"
),
)
add_apply_options(branch_cleanup)
branch_purge = branch_subparsers.add_parser(
"purge",
help="Delete unmerged or dirty branches after a confirmation per item",
)
for sub in (branch_cleanup, branch_purge):
add_scan_options(sub)
add_min_age_option(sub)
branch_tree = branch_subparsers.add_parser(
"tree",
help=(
"Print the branches and worktrees of every pkgmgr-managed repository "
"with their cleanup state, without fetching"
),
)
branch_tree.add_argument(
"--format",
choices=("json", "yaml"),
default="json",
help="Output format (default: json)",
)
branch_tree.add_argument(
"--output",
metavar="FILE",
help="Write atomically to FILE instead of stdout",
)
def add_apply_options(sub: argparse.ArgumentParser) -> None:
sub.add_argument(
"--apply",
action="store_true",
help="Execute the plan instead of only printing it",
)
sub.add_argument(
"--json",
action="store_true",
help="Print the plan as JSON",
)
def add_scan_options(sub: argparse.ArgumentParser) -> None:
sub.add_argument(
"--all",
action="store_true",
help="Scan every pkgmgr-managed repository instead of the current one",
)
sub.add_argument(
"--branch",
action="append",
default=[],
metavar="NAME",
help="Only this branch (or worktree path); repeatable",
)
sub.add_argument(
"--exclude",
action="append",
default=[],
metavar="NAME",
help="Skip this branch (or worktree path); repeatable",
)
sub.add_argument(
"--merged-into",
action="append",
default=[],
metavar="REF",
help=(
"Also count an item as merged when its commits are in REF (an "
"integration line); REF is protected and a checkout at its tip "
"counts as fresh; repeatable"
),
)
sub.add_argument(
"--sizes",
action="store_true",
help="Measure each worktree's disk usage (du) into the plan",
)
def add_min_age_option(sub: argparse.ArgumentParser) -> None:
sub.add_argument(
"--min-age",
type=float,
metavar="HOURS",
help="Skip items whose newest commit, worktree index or changed file is younger",
)
+46
View File
@@ -0,0 +1,46 @@
from __future__ import annotations
import argparse
from .branch_cmd import add_apply_options, add_min_age_option, add_scan_options
def add_gc_subparser(subparsers: argparse._SubParsersAction) -> None:
gc = subparsers.add_parser(
"gc",
help=(
"Garbage-collect the current repository, or every one with --all: "
"worktree cleanup and branch cleanup in one pass (dry-run unless --apply)"
),
)
add_apply_options(gc)
add_scan_options(gc)
add_min_age_option(gc)
gc.add_argument(
"--report",
action="store_true",
help=(
"Print a read-only disk inventory instead of the cleanup plan: "
"worktrees, preserved changes, --root children, core dumps, "
"docker usage and free space per filesystem"
),
)
gc.add_argument(
"--root",
action="append",
default=[],
metavar="DIR",
help="With --report: also measure the direct children of DIR; repeatable",
)
gc.add_argument(
"--threshold-mb",
type=int,
default=0,
metavar="MB",
help="With --report: list only --root children larger than MB (0: all)",
)
gc.add_argument(
"--docker",
action="store_true",
help="With --report: also read `docker system df`",
)
+47
View File
@@ -0,0 +1,47 @@
from __future__ import annotations
import argparse
def _positive_int(value: str) -> int:
number = int(value)
if number < 1:
raise argparse.ArgumentTypeError(f"must be a positive integer, got {value}")
return number
def add_review_subparser(subparsers: argparse._SubParsersAction) -> None:
parser = subparsers.add_parser(
"review",
help="Render the working copy's changes as a local HTML review page.",
description=(
"Write a self-contained HTML page showing the staged, unstaged and "
"untracked changes of the current git working copy (plus the commits "
"of REF..HEAD with --base) and print its file:// URL."
),
)
parser.add_argument(
"--out",
required=True,
metavar="FILE",
help="Write the HTML page to FILE; parent directories are created.",
)
parser.add_argument(
"--max-file-lines",
required=True,
type=_positive_int,
metavar="N",
help="Show at most N diff lines per file and summarise the rest.",
)
parser.add_argument(
"--base",
metavar="REF",
help="Add a Commits section for REF..HEAD; omitted means no Commits section.",
)
parser.add_argument(
"--note",
action="append",
default=[],
metavar="FILE",
help="Show FILE verbatim in a panel at the top; repeatable.",
)
+132
View File
@@ -0,0 +1,132 @@
from __future__ import annotations
import argparse
from .branch_cmd import add_apply_options, add_min_age_option, add_scan_options
def _add_flow_options(parser: argparse.ArgumentParser, default: str) -> None:
parser.add_argument(
"--pull",
action="append",
metavar="REF",
help=(
"Ref merged into the branch before a gate, deploy or merge; "
f"repeatable, kept in order ({default})"
),
)
parser.add_argument(
"--push",
action="append",
metavar="BRANCH",
help=(
"Branch the branch is merged into when finished, then pushed by "
f"the operator; repeatable, kept in order ({default})"
),
)
def _add_flow_parser(worktree_subparsers: argparse._SubParsersAction) -> None:
flow = worktree_subparsers.add_parser(
"flow",
help=(
"Show, set or clear the worktree's merge flow in "
"<git-dir>/pkgmgr/flow.yml: pull refs merged in, push branches "
"merged into"
),
)
flow.add_argument(
"target",
nargs="?",
metavar="set|clear|PATH",
help="`set` or `clear`, else the worktree to show (default: .)",
)
flow.add_argument(
"path",
nargs="?",
metavar="PATH",
help="Worktree of set or clear (default: .)",
)
_add_flow_options(flow, "set only, replaces the list")
flow.add_argument(
"--no-pull", action="store_true", help="set only: an empty pull list"
)
flow.add_argument(
"--no-push", action="store_true", help="set only: an empty push list"
)
flow.add_argument("--json", action="store_true", help="Print the flow as JSON")
def add_worktree_subparsers(subparsers: argparse._SubParsersAction) -> None:
worktree_parser = subparsers.add_parser(
"worktree",
help=(
"Worktree utilities: create, keep a merge flow, clean up and find "
"abandoned worktrees"
),
)
worktree_subparsers = worktree_parser.add_subparsers(
dest="subcommand",
help="Worktree subcommands",
required=True,
)
up = worktree_subparsers.add_parser(
"up",
help=(
"Check a branch out under <directories.worktrees>/<host>/<owner>/"
"<repo>/<branch>, assign a slot and run `make worktree-init SLOT=<n>` "
"when the project defines it"
),
)
up.add_argument("name", help="Branch to check out; created when it exists nowhere")
up.add_argument(
"--base",
metavar="REF",
help="Start point of a new branch (origin's default branch when omitted)",
)
up.add_argument(
"--path",
metavar="DIR",
help=(
"Exact checkout path instead of the standard one, e.g. a directory "
"an agent sandbox releases"
),
)
_add_flow_options(up, "default: the base, for push its local branch")
up.add_argument("--json", action="store_true", help="Print the result as JSON")
_add_flow_parser(worktree_subparsers)
cleanup = worktree_subparsers.add_parser(
"cleanup",
help=(
"Remove clean worktrees whose commits are merged or pushed: "
"`make worktree-fini`, compose stack, checkout; the branch stays "
"(dry-run unless --apply)"
),
)
add_apply_options(cleanup)
add_scan_options(cleanup)
add_min_age_option(cleanup)
orphans = worktree_subparsers.add_parser(
"orphans",
help="List merged and abandoned branches and worktrees with a proposal each",
)
add_scan_options(orphans)
orphans.add_argument(
"--stale-days",
type=int,
required=True,
metavar="DAYS",
help="Unfinished work idle for longer counts as stale-open",
)
orphans.add_argument(
"--top",
type=int,
required=True,
metavar="N",
help="Rows per class in the text table; 0 prints the counts only",
)
orphans.add_argument("--json", action="store_true", help="Print every row as JSON")
+39 -1
View File
@@ -8,8 +8,9 @@ from typing import Any
from pkgmgr.actions.proxy import exec_proxy_command
from pkgmgr.actions.repository.clone import clone_repos
from pkgmgr.actions.repository.pull import pull_with_verification
from pkgmgr.actions.repository.push import push_in_parallel
from pkgmgr.actions.repository.push import push_in_parallel, push_repo_dirs
from pkgmgr.cli.context import CLIContext
from pkgmgr.core.git import GitRunError, run
from pkgmgr.core.repository.dir import get_repo_dir
from pkgmgr.core.repository.selected import get_selected_repos
@@ -185,6 +186,25 @@ def register_proxy_commands(
"(default: min(cpu_count, 8)). Use 1 for sequential."
),
)
if subcommand == "push":
parser.add_argument(
"--rewrite",
action="store_true",
default=False,
help=(
"Deliberate history rewrite: replace commits the remote "
"already has (amend, squash, drop); leases still apply."
),
)
parser.add_argument(
"--no-sign",
action="store_true",
default=False,
help=(
"Push unsigned commits as they are instead of re-signing "
"them; leases still apply."
),
)
if subcommand == "clone":
parser.add_argument(
"--clone-mode",
@@ -209,6 +229,22 @@ def maybe_handle_proxy(args: argparse.Namespace, ctx: CLIContext) -> bool:
if args.command not in all_proxy_subcommands:
return False
if args.command == "push" and not _proxy_has_explicit_selection(args):
try:
toplevel = run(["rev-parse", "--show-toplevel"])
except GitRunError:
print("[ERROR] Current directory is not inside a git checkout.")
sys.exit(1)
push_repo_dirs(
[(toplevel, toplevel)],
args.extra_args,
args.preview,
not args.no_sign,
jobs=args.jobs,
rewrite=args.rewrite,
)
sys.exit(0)
# Default semantics: without explicit selection → repo of current folder.
if _proxy_has_explicit_selection(args):
selected = get_selected_repos(args, ctx.all_repositories)
@@ -251,7 +287,9 @@ def maybe_handle_proxy(args: argparse.Namespace, ctx: CLIContext) -> bool:
ctx.all_repositories,
args.extra_args,
args.preview,
not args.no_sign,
jobs=args.jobs,
rewrite=args.rewrite,
)
else:
exec_proxy_command(
+7
View File
@@ -4,6 +4,13 @@ directories:
repositories: ~/Repositories/
workspaces: ~/Workspaces/
binaries: ~/.local/bin/
worktrees: ~/.local/share/worktrees/
# worktree_hooks: <provider>/<account>/<repository> of an entry under
# repositories -> worktree-init / worktree-fini -> argv, with {path} and {slot}
# replaced; it runs only for that entry's directory, never selected by origin's
# URL, and without an entry no hook runs, whatever the repository's Makefile
# defines
worktree_hooks: {}
repositories:
- account: kevinveenbirkenbach
description: A configurable Python package manager that automates repository tasks—including cloning, installation, updates, and status reporting—based on a YAML configuration file for streamlined software management which gives you access to the Kevin Veen-Birkenbach Code Universe.
+13 -15
View File
@@ -144,6 +144,7 @@ def _load_layer_dir(
{
"directories": {...},
"repositories": [...],
<other key>: dicts deep-merged, other values from the last file,
}
"""
defaults: dict[str, Any] = {"directories": {}, "repositories": []}
@@ -165,21 +166,16 @@ def _load_layer_dir(
for path in yaml_files:
data = _load_yaml_file(path)
category_name = path.stem
dirs = data.get("directories")
if isinstance(dirs, dict):
defaults.setdefault("directories", {})
_deep_merge(defaults["directories"], dirs)
repos = data.get("repositories")
repos = data.pop("repositories", None)
if isinstance(repos, list):
defaults.setdefault("repositories", [])
_merge_repo_lists(
defaults["repositories"],
repos,
category_name=category_name,
category_name=path.stem,
)
if not isinstance(data.get("directories", {}), dict):
del data["directories"]
_deep_merge(defaults, data)
return defaults
@@ -229,7 +225,8 @@ def load_config(user_config_path: str) -> dict[str, Any]:
Steps:
1. Determine ~/.config/pkgmgr/ (or dir of user_config_path).
2. Load all *.yml/*.yaml in that dir (except the user config file) as defaults.
3. If nothing found, fall back to package defaults.
3. If nothing found, fall back to package defaults. Package keys other
than directories and repositories always lie under the layers.
4. Load the user config file (if present).
5. Merge:
- directories: deep-merge (defaults <- user)
@@ -247,13 +244,14 @@ def load_config(user_config_path: str) -> dict[str, Any]:
# 1+2) Defaults from user directory
defaults = _load_layer_dir(config_dir, skip_filename=user_cfg_name)
package = _load_defaults_from_package_or_project()
# 3) Fallback to package defaults
if not defaults["directories"] and not defaults["repositories"]:
defaults = _load_defaults_from_package_or_project()
defaults.setdefault("directories", {})
defaults.setdefault("repositories", [])
defaults["directories"] = package["directories"]
defaults["repositories"] = package["repositories"]
del package["directories"], package["repositories"]
defaults = _deep_merge(package, defaults)
# 4) User config
user_cfg: dict[str, Any] = {}
-3
View File
@@ -19,7 +19,6 @@ from .pull import GitPullError, pull
from .pull_args import GitPullArgsError, pull_args
from .pull_ff_only import GitPullFfOnlyError, pull_ff_only
from .push import GitPushError, push
from .push_args import GitPushArgsError, push_args
from .push_upstream import GitPushUpstreamError, push_upstream
from .set_remote_url import GitSetRemoteUrlError, set_remote_url
from .tag_annotated import GitTagAnnotatedError, tag_annotated
@@ -43,7 +42,6 @@ __all__ = [
"GitPullArgsError",
"GitPullError",
"GitPullFfOnlyError",
"GitPushArgsError",
"GitPushError",
"GitPushUpstreamError",
"GitSetRemoteUrlError",
@@ -67,7 +65,6 @@ __all__ = [
"pull_args",
"pull_ff_only",
"push",
"push_args",
"push_upstream",
"set_remote_url",
"tag_annotated",
@@ -11,7 +11,7 @@ class GitDeleteLocalBranchError(GitCommandError):
def delete_local_branch(branch: str, cwd: str = ".", force: bool = False) -> None:
flag = "-D" if force else "-d"
try:
run(["branch", flag, branch], cwd=cwd)
run(["branch", flag, "--", branch], cwd=cwd)
except GitRunError as exc:
raise GitDeleteLocalBranchError(
f"Failed to delete local branch {branch!r} (flag {flag}).",
@@ -10,7 +10,7 @@ class GitDeleteRemoteBranchError(GitCommandError):
def delete_remote_branch(remote: str, branch: str, cwd: str = ".") -> None:
try:
run(["push", remote, "--delete", branch], cwd=cwd)
run(["push", "--delete", "--", remote, branch], cwd=cwd)
except GitRunError as exc:
raise GitDeleteRemoteBranchError(
f"Failed to delete remote branch {branch!r} on {remote!r}.",
+2 -4
View File
@@ -20,11 +20,9 @@ def push(
Push a ref to a remote, optionally forced.
Equivalent to:
git push <remote> <ref> [--force]
git push [--force] -- <remote> <ref>
"""
args = ["push", remote, ref]
if force:
args.append("--force")
args = ["push", *(["--force"] if force else []), "--", remote, ref]
try:
run(args, cwd=cwd, preview=preview)
-37
View File
@@ -1,37 +0,0 @@
from __future__ import annotations
from ..errors import GitCommandError, GitRunError
from ..run import run
class GitPushArgsError(GitCommandError):
"""Raised when `git push` with arbitrary args fails."""
def push_args(
args: list[str] | None = None,
*,
cwd: str = ".",
preview: bool = False,
) -> None:
"""
Execute `git push` with caller-provided arguments.
Examples:
[] -> git push
["--force"] -> git push --force
["origin", "main"] -> git push origin main
["-u", "origin", "feature"] -> git push -u origin feature
"""
extra = args or []
try:
run(["push", *extra], cwd=cwd, preview=preview)
except GitRunError as exc:
details = getattr(exc, "output", None) or getattr(exc, "stderr", None) or ""
raise GitPushArgsError(
(
f"Failed to run `git push` with args={extra!r} "
f"in cwd={cwd!r}.\n{details}"
).rstrip(),
cwd=cwd,
) from exc
@@ -19,10 +19,10 @@ def push_upstream(
Push a branch and set upstream tracking.
Equivalent to:
git push -u <remote> <branch>
git push -u -- <remote> <branch>
"""
try:
run(["push", "-u", remote, branch], cwd=cwd, preview=preview)
run(["push", "-u", "--", remote, branch], cwd=cwd, preview=preview)
except GitRunError as exc:
raise GitPushUpstreamError(
f"Failed to push branch {branch!r} to {remote!r} with upstream tracking.",
+4
View File
@@ -20,6 +20,7 @@ from .get_tags_at_ref import GitTagsAtRefQueryError, get_tags_at_ref
from .get_upstream_ref import get_upstream_ref
from .list_remotes import list_remotes
from .list_tags import list_tags
from .local_config import FILTER_CONFIG, UNSAFE_CONFIG, local_config
from .probe_remote_reachable import (
probe_remote_reachable,
probe_remote_reachable_detail,
@@ -27,6 +28,8 @@ from .probe_remote_reachable import (
from .resolve_base_branch import GitBaseBranchNotFoundError, resolve_base_branch
__all__ = [
"FILTER_CONFIG",
"UNSAFE_CONFIG",
"GitBaseBranchNotFoundError",
"GitChangelogQueryError",
"GitLatestSigningKeyQueryError",
@@ -46,6 +49,7 @@ __all__ = [
"get_upstream_ref",
"list_remotes",
"list_tags",
"local_config",
"probe_remote_reachable",
"probe_remote_reachable_detail",
"resolve_base_branch",
+3 -3
View File
@@ -3,13 +3,13 @@ from __future__ import annotations
from ..run import run
def list_remotes(cwd: str = ".") -> list[str]:
def list_remotes(cwd: str = ".", scan: bool = False) -> list[str]:
"""
Return a list of configured git remotes (e.g. ['origin', 'upstream']).
Raises GitBaseError if the command fails.
scan=True runs git hardened. Raises GitBaseError if the command fails.
"""
output = run(["remote"], cwd=cwd)
output = run(["remote"], cwd=cwd, scan=scan)
if not output:
return []
return [line.strip() for line in output.splitlines() if line.strip()]
@@ -0,0 +1,43 @@
from __future__ import annotations
from ..errors import GitRunError
from ..run import run
FILTER_CONFIG = r"^filter\."
UNSAFE_CONFIG = (
r"^(core\.(sshcommand|gitproxy|askpass|fsmonitor|alternaterefscommand)$"
r"|credential\.(.+\.)?helper$|gpg\.(.+\.)?program$|gpg\.ssh\.defaultkeycommand$"
r"|protocol\.|hook\.|gc\.recentobjectshook$"
r"|http\.(.+\.)?(proxy|sslverify|sslcainfo|sslcapath|cookiefile|savecookies)$"
r"|remote\..+\.(receivepack|uploadpack|vcs|proxy)$"
r"|url\..+\.(insteadof|pushinsteadof)$)"
)
LOCAL_SCOPES = ("local", "worktree")
def local_config(path: str, pattern: str) -> list[str]:
"""
List the config keys matching pattern that the repository itself sets.
Args:
path: any directory inside the repository.
pattern: `git config --get-regexp` pattern over the canonical key names.
Returns:
Each matching key set in the local or worktree scope, in config order;
global values never count, system config is not read.
"""
try:
out = run(
["config", "--show-scope", "--get-regexp", pattern], cwd=path, scan=True
)
except GitRunError as exc:
if exc.returncode == 1:
return []
raise
keys: list[str] = []
for line in out.splitlines():
scope, _, rest = line.partition("\t")
if scope in LOCAL_SCOPES:
keys.append(rest.split(" ", 1)[0])
return keys
@@ -37,12 +37,13 @@ def resolve_base_branch(
preferred: str = "main",
fallback: str = "master",
cwd: str = ".",
scan: bool = False,
) -> str:
"""
Resolve the base branch to use.
Try `preferred` first (default: main),
fall back to `fallback` (default: master).
fall back to `fallback` (default: master); scan=True runs git hardened.
Raises GitBaseBranchNotFoundError if neither exists.
Raises GitRunError for other git failures (e.g., not a git repository).
@@ -51,7 +52,7 @@ def resolve_base_branch(
for candidate in (preferred, fallback):
try:
run(["rev-parse", "--verify", candidate], cwd=cwd)
run(["rev-parse", "--verify", candidate], cwd=cwd, scan=scan)
return candidate
except GitRunError as exc:
if _is_branch_missing_error(exc):
+85 -3
View File
@@ -1,35 +1,111 @@
from __future__ import annotations
import os
import subprocess
from .errors import GitNotRepositoryError, GitRunError
SCAN_FLAGS = (
"--no-pager",
"-c",
"core.fsmonitor=false",
"-c",
"core.hooksPath=/dev/null",
# hook.<name>.command hooks from the config ignore core.hooksPath
"-c",
"hook.reference-transaction.enabled=false",
"-c",
"hook.prepare-commit-msg.enabled=false",
"-c",
"hook.post-commit.enabled=false",
"-c",
"hook.post-rewrite.enabled=false",
"-c",
"hook.post-index-change.enabled=false",
"-c",
"hook.pre-push.enabled=false",
"-c",
"hook.post-checkout.enabled=false",
"-c",
"core.pager=cat",
"-c",
"gc.auto=0",
"-c",
"maintenance.auto=false",
"-c",
"log.showSignature=false",
)
SCAN_ENV = {
"GIT_CONFIG_NOSYSTEM": "1",
"GIT_NO_LAZY_FETCH": "1",
"GIT_NO_REPLACE_OBJECTS": "1",
"GIT_OPTIONAL_LOCKS": "0",
"GIT_TERMINAL_PROMPT": "0",
"GIT_ASKPASS": "/bin/false",
"LC_ALL": "C",
}
SCAN_UNSET = (
"GIT_DIR",
"GIT_WORK_TREE",
"GIT_INDEX_FILE",
"GIT_EXTERNAL_DIFF",
"GIT_PAGER",
"PAGER",
)
SCAN_TIMEOUT = 60
# ref and remote names come from repositories an agent may write: one named
# like --output=<path> would otherwise make rev-list create that file
END_OF_OPTIONS = "--end-of-options"
def _is_not_repo_error(stderr: str) -> bool:
msg = (stderr or "").lower()
return "not a git repository" in msg
def _scan_env() -> dict[str, str]:
env = {key: value for key, value in os.environ.items() if key not in SCAN_UNSET}
return {**env, **SCAN_ENV}
def run(
args: list[str],
*,
cwd: str = ".",
preview: bool = False,
scan: bool = False,
input_text: str | None = None,
) -> str:
"""
Run a Git command and return its stdout as a stripped string.
If preview=True, the command is printed but NOT executed.
Args:
args: the git arguments.
cwd: directory to run in.
preview: print the command instead of running it.
scan: read a repository an agent may have written: no fsmonitor, hook,
pager, lazy fetch, replace object, system config or prompt, stdin
closed, a timeout.
input_text: text fed to stdin, e.g. for `cat-file --batch-check`;
replaces the closed stdin of scan mode.
Raises GitRunError (or a subclass) if execution fails.
Raises GitRunError (or a subclass) if execution fails or times out.
"""
cmd = ["git"] + args
cmd = ["git", *SCAN_FLAGS, *args] if scan else ["git"] + args
cmd_str = " ".join(cmd)
if preview:
print(f"[PREVIEW] Would run in {cwd!r}: {cmd_str}")
return ""
hardened: dict = (
{"env": _scan_env(), "stdin": subprocess.DEVNULL, "timeout": SCAN_TIMEOUT}
if scan
else {}
)
if input_text is not None:
hardened.pop("stdin", None)
hardened["input"] = input_text
try:
result = subprocess.run(
cmd,
@@ -37,7 +113,13 @@ def run(
check=True,
capture_output=True,
text=True,
**hardened,
)
except subprocess.TimeoutExpired as exc:
err = GitRunError(f"Git command timed out in {cwd!r}: {cmd_str}")
err.cwd, err.cmd, err.cmd_str, err.returncode = cwd, cmd, cmd_str, -1
err.stdout, err.stderr = "", ""
raise err from exc
except subprocess.CalledProcessError as exc:
stderr = exc.stderr or ""
stdout = exc.stdout or ""
+37
View File
@@ -232,3 +232,40 @@ def read_ansible_galaxy_version(repo_dir: str) -> str | None:
return None
return None
def read_cargo_version(repo_dir: str) -> str | None:
"""
Read [package].version from Cargo.toml in repo_dir, if present.
"""
path = os.path.join(repo_dir, "Cargo.toml")
if not os.path.isfile(path):
return None
try:
import tomllib
except ImportError:
import tomli as tomllib # type: ignore
try:
with open(path, "rb") as f:
version = (tomllib.load(f).get("package") or {}).get("version")
except (OSError, UnicodeDecodeError, tomllib.TOMLDecodeError):
return None
return version.strip() if isinstance(version, str) and version.strip() else None
def read_changelog_version(repo_dir: str) -> str | None:
"""
Read the newest release from CHANGELOG.md ("## [X.Y.Z] - date").
"""
path = os.path.join(repo_dir, "CHANGELOG.md")
if not os.path.isfile(path):
return None
try:
with open(path, encoding="utf-8") as f:
match = re.search(r"(?m)^##\s+\[(v?\d+\.\d+\.\d+)\]", f.read())
except (OSError, UnicodeDecodeError):
return None
return match.group(1).strip() if match else None
+589
View File
@@ -0,0 +1,589 @@
from __future__ import annotations
import io
import json
import os
import subprocess
import tempfile
import unittest
from contextlib import redirect_stdout
from types import SimpleNamespace
from unittest.mock import patch
from pkgmgr.actions.branch import gc
from pkgmgr.actions.branch.gc_forge import ForgeFacts
from pkgmgr.cli.commands.branch import handle_branch
from pkgmgr.cli.parser import create_parser
GIT_ENV = {
**os.environ,
"GIT_AUTHOR_NAME": "t",
"GIT_AUTHOR_EMAIL": "t@example.org",
"GIT_COMMITTER_NAME": "t",
"GIT_COMMITTER_EMAIL": "t@example.org",
}
def git(cwd: str, *args: str) -> str:
return subprocess.run(
["git", "-c", "commit.gpgsign=false", *args],
cwd=cwd,
env=GIT_ENV,
check=True,
capture_output=True,
text=True,
).stdout.strip()
def docker_available() -> bool:
try:
subprocess.run(["docker", "info"], check=True, capture_output=True)
except (FileNotFoundError, subprocess.CalledProcessError):
return False
return True
class GcFixture:
def setUp(self) -> None:
self._tmp = tempfile.TemporaryDirectory()
self.root = os.path.realpath(self._tmp.name)
self.origin = os.path.join(self.root, "origin.git")
self.repo = os.path.join(self.root, "repo")
git(self.root, "init", "--bare", "-b", "main", self.origin)
git(self.root, "init", "-b", "main", self.repo)
git(self.repo, "remote", "add", "origin", self.origin)
self._commit("init")
git(self.repo, "push", "-u", "origin", "main")
self._branch_with_commit("merged")
git(self.repo, "merge", "--no-ff", "-m", "merge merged", "merged")
git(self.repo, "push", "origin", "main")
git(self.repo, "branch", "fresh")
git(self.repo, "push", "origin", "fresh")
self._branch_with_commit("wip")
self._branch_with_commit("squashed")
self._branch_with_commit("locked")
git(self.repo, "merge", "--no-ff", "-m", "merge locked", "locked")
git(self.repo, "push", "origin", "main")
git(self.repo, "remote", "set-head", "origin", "main")
self.merged_wt = os.path.join(self.root, "wt-merged")
git(self.repo, "worktree", "add", self.merged_wt, "merged")
self.dirty_wt = os.path.join(self.root, "wt-dirty")
git(self.repo, "worktree", "add", "-b", "dirty", self.dirty_wt, "merged")
with open(os.path.join(self.dirty_wt, "scratch.txt"), "w") as fh:
fh.write("x")
self.locked_wt = os.path.join(self.root, "wt-locked")
git(self.repo, "worktree", "add", self.locked_wt, "locked")
self.squashed_sha = git(self.repo, "rev-parse", "squashed")
self.forge = ForgeFacts(
merged_heads={self.squashed_sha: 7}, open_heads={"wip": 3}
)
def tearDown(self) -> None:
self._tmp.cleanup()
def _commit(self, name: str) -> None:
with open(os.path.join(self.repo, f"{name}.txt"), "w") as fh:
fh.write(name)
git(self.repo, "add", f"{name}.txt")
git(self.repo, "commit", "-m", name)
def _branch_with_commit(self, name: str) -> None:
git(self.repo, "checkout", "-b", name, "main")
self._commit(name)
git(self.repo, "push", "origin", name)
git(self.repo, "checkout", "main")
def _lock(self, reason: str) -> None:
git(self.repo, "worktree", "lock", "--reason", reason, self.locked_wt)
def _plan(
self,
host_ns: bool,
cwds: dict[int, str],
merged_into: list[str] | None = None,
sizes: bool = False,
min_age: float | None = None,
scope: str = "all",
activity: bool = False,
hooks: dict | None = None,
containers: list[gc.Container] | None = None,
) -> gc.Repo:
with (
patch.object(gc, "fetch_forge_facts", return_value=self.forge),
patch.object(gc, "_process_cwds", return_value=cwds),
):
return gc.plan_repo(
self.repo,
containers or [],
host_ns,
merged_into=merged_into or [],
sizes=sizes,
min_age=min_age,
scope=scope,
activity=activity,
hooks=hooks or {},
)
def _integration_line(self) -> str:
git(self.repo, "checkout", "-b", "line", "main")
self._commit("line-base")
git(self.repo, "checkout", "-b", "feat", "line")
self._commit("feat")
git(self.repo, "checkout", "line")
git(self.repo, "merge", "--no-ff", "-m", "merge feat", "feat")
git(self.repo, "push", "origin", "line", "feat")
git(self.repo, "branch", "start", "line")
git(self.repo, "checkout", "main")
feat_wt = os.path.join(self.root, "wt-feat")
git(self.repo, "worktree", "add", feat_wt, "feat")
return feat_wt
def _items(self, plan: gc.Repo) -> dict[str, gc.Item]:
return {item.branch: item for item in plan.items if item.branch}
class TestBranchGc(GcFixture, unittest.TestCase):
def test_classifies_every_branch(self) -> None:
items = self._items(self._plan(host_ns=True, cwds={}))
self.assertEqual(items["main"].state, "protected")
self.assertEqual(items["merged"].state, "merged-clean")
self.assertEqual(items["merged"].merged_by, "ancestor")
self.assertEqual(
items["merged"].actions,
["worktree", "local-branch", "remote-branch:origin"],
)
self.assertTrue(items["merged"].eligible)
self.assertEqual(items["squashed"].merged_by, "pr#7")
self.assertTrue(items["squashed"].eligible)
self.assertEqual(items["fresh"].state, "unmerged")
self.assertEqual(items["fresh"].reason, "no own commits")
self.assertEqual(items["wip"].state, "unmerged")
self.assertEqual(items["wip"].skip, "open-pr #3")
self.assertEqual(items["dirty"].state, "dirty")
def test_merged_into_counts_commits_in_an_integration_line(self) -> None:
self._integration_line()
before = self._items(self._plan(host_ns=True, cwds={}))
self.assertEqual(before["feat"].state, "unmerged")
self.assertIsNone(before["feat"].merged_into)
for ref in ("line", "origin/line"):
with self.subTest(ref=ref):
items = self._items(
self._plan(host_ns=True, cwds={}, merged_into=[ref])
)
self.assertEqual(items["feat"].state, "merged-clean")
self.assertEqual(items["feat"].merged_by, f"line:{ref}")
self.assertEqual(items["feat"].merged_into, ref)
self.assertTrue(items["feat"].eligible)
self.assertEqual(items["line"].state, "protected")
self.assertEqual(items["start"].state, "unmerged")
self.assertEqual(items["merged"].merged_into, "origin/main")
def test_unknown_merged_into_ref_is_a_warning(self) -> None:
plan = self._plan(host_ns=True, cwds={}, merged_into=["no-such-line"])
self.assertIn("merged-into ref no-such-line not found", plan.warnings)
self.assertEqual(self._items(plan)["merged"].state, "merged-clean")
def test_sizes_and_min_age(self) -> None:
items = self._items(self._plan(host_ns=True, cwds={}, sizes=True))
self.assertGreater(items["merged"].size_bytes, 0)
self.assertIsNone(items["squashed"].size_bytes)
self.assertIn("size ", gc._line(items["merged"]))
young = self._items(self._plan(host_ns=True, cwds={}, min_age=1000))
self.assertEqual(young["merged"].skip, "younger than 1000 h")
self.assertFalse(young["merged"].eligible)
old = self._items(self._plan(host_ns=True, cwds={}, min_age=0))
self.assertIsNone(old["merged"].skip)
def test_busy_worktree_is_skipped(self) -> None:
cwds = {4242: os.path.join(self.merged_wt, "sub")}
items = self._items(self._plan(host_ns=True, cwds=cwds))
self.assertTrue(items["merged"].skip.startswith("process: pid 4242"))
self.assertFalse(items["merged"].eligible)
def test_lock_with_live_pid_is_skipped(self) -> None:
self._lock(f"claude agent x (pid {os.getpid()} on host)")
items = self._items(self._plan(host_ns=True, cwds={}))
self.assertEqual(items["locked"].skip, f"lock: pid {os.getpid()} alive")
def test_lock_with_dead_pid_is_overridden(self) -> None:
self._lock("claude agent x (pid 999999999 on host)")
items = self._items(self._plan(host_ns=True, cwds={}))
self.assertIsNone(items["locked"].skip)
self.assertTrue(items["locked"].eligible)
def test_lock_outside_host_namespace_is_unknown(self) -> None:
self._lock("claude agent x (pid 999999999 on host)")
items = self._items(self._plan(host_ns=False, cwds={}))
self.assertEqual(items["locked"].skip, "lock: liveness unknown")
def test_lock_without_pid_is_respected(self) -> None:
self._lock("manual hold")
items = self._items(self._plan(host_ns=True, cwds={}))
self.assertEqual(items["locked"].skip, "lock: manual hold")
def test_forge_unavailable_skips_everything(self) -> None:
self.forge = ForgeFacts(unavailable="no GitHub token")
items = self._items(self._plan(host_ns=True, cwds={}))
self.assertEqual(items["merged"].skip, "forge unavailable")
self.assertIsNone(items["squashed"].merged_by)
def test_stale_metadata_is_listed_not_pruned(self) -> None:
stub = os.path.join(self.repo, ".git", "worktrees", "stub")
os.makedirs(stub)
open(os.path.join(stub, "commondir"), "w").close()
plan = self._plan(host_ns=True, cwds={})
stale = [i for i in plan.items if i.state == "stale"]
self.assertEqual([i.worktree for i in stale], [stub])
self.assertTrue(os.path.isdir(stub))
def test_execute_removes_worktree_and_branches(self) -> None:
plan = self._plan(host_ns=True, cwds={})
item = self._items(plan)["merged"]
meta = os.path.join(self.repo, ".git", "worktrees", "wt-merged")
with patch.object(gc, "_process_cwds", return_value={}):
gc.execute(plan, item)
self.assertEqual(item.result, "done")
self.assertFalse(os.path.exists(self.merged_wt))
self.assertFalse(os.path.exists(meta))
self.assertEqual(git(self.repo, "branch", "--list", "merged"), "")
self.assertEqual(git(self.origin, "branch", "--list", "merged"), "")
@unittest.skipUnless(docker_available(), "docker is not available")
def test_execute_removes_foreign_owned_worktree_via_container(self) -> None:
with open(os.path.join(self.repo, ".git", "info", "exclude"), "a") as fh:
fh.write("node_modules/\n")
subprocess.run(
[
"docker",
"run",
"--rm",
"-v",
f"{self.merged_wt}:/w",
gc.CLEANUP_IMAGE,
"sh",
"-c",
(
"mkdir -p /w/node_modules/pkg && touch /w/node_modules/pkg/index.js"
" && chown -R 65534:65534 /w/node_modules"
),
],
check=True,
capture_output=True,
)
self.assertTrue(gc._foreign_owned(self.merged_wt))
plan = self._plan(host_ns=True, cwds={})
item = self._items(plan)["merged"]
meta = os.path.join(self.repo, ".git", "worktrees", "wt-merged")
with patch.object(gc, "_process_cwds", return_value={}):
gc.execute(plan, item)
self.assertEqual(item.result, "done")
self.assertFalse(os.path.lexists(self.merged_wt))
self.assertFalse(os.path.exists(meta))
self.assertEqual(git(self.repo, "branch", "--list", "merged"), "")
def test_metadata_kept_when_checkout_survives_removal(self) -> None:
plan = self._plan(host_ns=True, cwds={})
item = self._items(plan)["merged"]
meta = os.path.join(self.repo, ".git", "worktrees", "wt-merged")
with (
patch.object(gc, "_process_cwds", return_value={}),
patch.object(gc, "_foreign_owned", return_value=True),
patch.object(gc, "_docker", return_value="") as docker,
):
gc.execute(plan, item)
docker.assert_called_once_with(
[
"run",
"--rm",
"-v",
f"{self.root}:/w",
gc.CLEANUP_IMAGE,
"rm",
"-rf",
"/w/wt-merged",
]
)
self.assertTrue(item.result.startswith("failed: worktree"))
self.assertTrue(os.path.isdir(self.merged_wt))
self.assertTrue(os.path.isdir(meta))
self.assertNotEqual(git(self.repo, "branch", "--list", "merged"), "")
def test_execute_skips_item_changed_since_plan(self) -> None:
plan = self._plan(host_ns=True, cwds={})
item = self._items(plan)["merged"]
with open(os.path.join(self.merged_wt, "late.txt"), "w") as fh:
fh.write("late")
with patch.object(gc, "_process_cwds", return_value={}):
gc.execute(plan, item)
self.assertTrue(item.result.startswith("skipped: changed since plan"))
self.assertTrue(os.path.isdir(self.merged_wt))
def test_worktree_path_with_newlines_cannot_forge_a_record(self) -> None:
forged = os.path.join(self.root, "a\n\nworktree -evil")
git(self.repo, "worktree", "add", "-b", "forged", forged, "main")
paths = {tree.path: tree.branch for tree in gc._worktrees(self.repo)}
self.assertEqual(paths[forged], "forged")
self.assertNotIn("-evil", paths)
def test_ref_named_like_an_option_writes_no_file(self) -> None:
target = os.path.join(self.root, "pwn")
os.makedirs(target)
remote = f"--output={target}"
main = git(self.repo, "rev-parse", "main")
git(self.repo, "remote", "remove", "origin")
git(self.repo, "config", f"remote.{remote}.url", self.origin)
git(self.repo, "update-ref", f"refs/remotes/{remote}/main", main)
git(
self.repo,
"symbolic-ref",
f"refs/remotes/{remote}/HEAD",
f"refs/remotes/{remote}/main",
)
plan = self._plan(host_ns=True, cwds={})
self.assertEqual(plan.default, f"{remote}/main")
self.assertEqual(os.listdir(target), [])
def test_execute_deletes_branches_named_like_options(self) -> None:
sha = git(self.repo, "rev-parse", "merged")
other = os.path.join(self.root, "other.git")
git(self.root, "init", "--bare", other)
git(self.repo, "push", other, "merged:refs/heads/-evil")
git(self.repo, "config", "remote.-r.url", other)
git(self.repo, "update-ref", "refs/heads/-evil", sha)
git(self.repo, "update-ref", "refs/remotes/-r/-evil", sha)
plan = self._plan(host_ns=True, cwds={})
item = self._items(plan)["-evil"]
self.assertEqual(item.remotes, {"-r": sha})
with patch.object(gc, "_process_cwds", return_value={}):
gc.execute(plan, item)
self.assertEqual(item.result, "done")
self.assertEqual(git(self.repo, "for-each-ref", "refs/heads/-evil"), "")
self.assertEqual(git(other, "for-each-ref", "refs/heads/-evil"), "")
def _fetches(self) -> int:
with patch.object(gc, "run", wraps=gc.run) as spy:
self.plan = self._plan(host_ns=True, cwds={})
return sum("fetch" in call.args[0] for call in spy.call_args_list)
def test_unsafe_repository_config_skips_fetch_and_remote_deletes(self) -> None:
self.assertEqual(self._fetches(), 1)
git(self.repo, "config", "core.sshCommand", "false")
self.assertEqual(self._fetches(), 0)
plan = self.plan
item = self._items(plan)["merged"]
self.assertIn(
"fetch and remote branch deletes skipped: the repository config sets "
"core.sshcommand",
plan.warnings,
)
self.assertIsNone(item.skip)
self.assertEqual(item.actions, ["worktree", "local-branch"])
with patch.object(gc, "_process_cwds", return_value={}):
gc.execute(plan, item)
self.assertEqual(item.result, "done")
self.assertFalse(os.path.exists(self.merged_wt))
self.assertEqual(git(self.origin, "branch", "--list", "merged"), "merged")
def test_config_set_after_plan_stops_execute(self) -> None:
plan = self._plan(host_ns=True, cwds={})
item = self._items(plan)["merged"]
git(self.repo, "config", "credential.helper", "false")
with patch.object(gc, "_process_cwds", return_value={}):
gc.execute(plan, item)
self.assertEqual(
item.result, "skipped: repository config sets credential.helper"
)
self.assertTrue(os.path.isdir(self.merged_wt))
self.assertEqual(git(self.origin, "branch", "--list", "merged"), "merged")
def test_compose_project_with_containers_elsewhere_is_skipped(self) -> None:
inside = gc.Container("proj", os.path.join(self.merged_wt, "app"), "img")
elsewhere = gc.Container("proj", os.path.join(self.root, "other"), "img")
alone = self._items(self._plan(host_ns=True, cwds={}, containers=[inside]))
shared = self._items(
self._plan(host_ns=True, cwds={}, containers=[inside, elsewhere])
)
self.assertIsNone(alone["merged"].skip)
self.assertIn("compose-down:proj", alone["merged"].actions)
self.assertEqual(
shared["merged"].skip,
"compose project proj also has containers outside the worktree",
)
self.assertFalse(shared["merged"].eligible)
def test_execute_runs_no_repository_hook(self) -> None:
marker = os.path.join(self.root, "hooked")
hook = os.path.join(self.repo, ".git", "hooks", "reference-transaction")
with open(hook, "w") as fh:
fh.write(f"#!/bin/sh\ntouch {marker}\n")
os.chmod(hook, 0o755)
plan = self._plan(host_ns=True, cwds={})
item = self._items(plan)["merged"]
with patch.object(gc, "_process_cwds", return_value={}):
gc.execute(plan, item)
self.assertEqual(item.result, "done")
self.assertEqual(git(self.origin, "branch", "--list", "merged"), "")
self.assertFalse(os.path.exists(marker))
def test_local_branch_delete_runs_no_config_hook(self) -> None:
marker = os.path.join(self.root, "hooked")
git(self.repo, "branch", "local-only", "merged")
git(self.repo, "config", "hook.probe.command", f"touch {marker}")
git(self.repo, "config", "hook.probe.event", "reference-transaction")
plan = self._plan(host_ns=True, cwds={})
item = self._items(plan)["local-only"]
self.assertEqual(item.actions, ["local-branch"])
with patch.object(gc, "_process_cwds", return_value={}):
gc.execute(plan, item)
self.assertEqual(item.result, "done")
self.assertEqual(git(self.repo, "branch", "--list", "local-only"), "")
self.assertFalse(os.path.exists(marker))
def test_cleanup_json_and_refusal(self) -> None:
out = io.StringIO()
with (
patch.object(gc, "host_pid_namespace", return_value=True),
patch.object(gc, "list_compose_containers", return_value=[]),
patch.object(gc, "fetch_forge_facts", return_value=self.forge),
patch.object(gc, "_process_cwds", return_value={}),
redirect_stdout(out),
):
code = gc.cleanup(
[self.repo],
scope="all",
apply=False,
as_json=True,
include=["merged"],
exclude=[],
merged_into=["origin/main"],
sizes=True,
min_age=None,
hooks={},
)
data = json.loads(out.getvalue())
self.assertEqual(code, 0)
self.assertEqual(data["repos"][0]["merged_into"], ["origin/main"])
items = data["repos"][0]["items"]
self.assertEqual([i["branch"] for i in items], ["merged"])
self.assertEqual(items[0]["merged_into"], "origin/main")
self.assertGreater(items[0]["size_bytes"], 0)
with patch.object(gc, "host_pid_namespace", return_value=False):
self.assertEqual(
gc.cleanup(
[self.repo],
scope="all",
apply=True,
as_json=False,
include=[],
exclude=[],
merged_into=[],
sizes=False,
min_age=None,
hooks={},
),
2,
)
def test_cli_forwards_cleanup_flags(self) -> None:
args = create_parser("test").parse_args(
[
"branch",
"cleanup",
"--apply",
"--json",
"--branch",
"a",
"--branch",
"b",
"--exclude",
"c",
"--merged-into",
"origin/line",
"--merged-into",
"fork/line",
"--sizes",
"--min-age",
"24",
]
)
pins = {"worktree-fini": ["true"]}
managed = {
"provider": "host",
"account": "o",
"repository": "r",
"directory": self.repo,
}
ctx = SimpleNamespace(
config_merged={
"worktree_hooks": {"host/o/r": pins},
"repositories": [managed],
}
)
with (
patch("pkgmgr.cli.commands.branch.main_repo_root", return_value=self.repo),
patch("pkgmgr.cli.commands.branch.cleanup", return_value=1) as run_gc,
patch("pkgmgr.cli.commands.branch.refresh_tree") as refresh,
self.assertRaises(SystemExit) as exit_ctx,
):
handle_branch(args, ctx=ctx)
self.assertEqual(exit_ctx.exception.code, 1)
run_gc.assert_called_once_with(
[self.repo],
scope="branch",
apply=True,
as_json=True,
include=["a", "b"],
exclude=["c"],
merged_into=["origin/line", "fork/line"],
sizes=True,
min_age=24.0,
hooks={os.path.realpath(self.repo): pins},
)
refresh.assert_called_once_with([self.repo])
if __name__ == "__main__":
unittest.main()
+442
View File
@@ -0,0 +1,442 @@
from __future__ import annotations
import io
import json
import os
import shutil
import stat
import subprocess
import tempfile
import time
import unittest
from contextlib import redirect_stdout
from unittest.mock import patch
import yaml
from pkgmgr.actions.branch import gc, tree
from pkgmgr.actions.branch.gc_forge import ForgeFacts
from pkgmgr.cli.commands.branch import handle_branch
from pkgmgr.cli.context import CLIContext
from pkgmgr.cli.parser import create_parser
from pkgmgr.core.git.run import run
GIT_ENV = {
**os.environ,
"GIT_AUTHOR_NAME": "t",
"GIT_AUTHOR_EMAIL": "t@example.org",
"GIT_COMMITTER_NAME": "t",
"GIT_COMMITTER_EMAIL": "t@example.org",
}
VOLATILE = ("generated_at", "scanned_at")
def git(cwd: str, *args: str) -> str:
return subprocess.run(
["git", "-c", "commit.gpgsign=false", *args],
cwd=cwd,
env=GIT_ENV,
check=True,
capture_output=True,
text=True,
).stdout.strip()
def stable(data):
if isinstance(data, dict):
return {
key: stable(value) for key, value in data.items() if key not in VOLATILE
}
if isinstance(data, list):
return [stable(value) for value in data]
return data
class TestBranchTree(unittest.TestCase):
def setUp(self) -> None:
self._tmp = tempfile.TemporaryDirectory()
self.addCleanup(self._tmp.cleanup)
self.root = os.path.realpath(self._tmp.name)
self.origin = os.path.join(self.root, "origin.git")
self.repo = os.path.join(self.root, "repos", "repo")
self.shared = os.path.join(self.root, "shared-worktrees")
os.makedirs(self.shared)
git(self.root, "init", "--bare", "-b", "main", self.origin)
git(self.root, "init", "-b", "main", self.repo)
git(self.repo, "remote", "add", "origin", self.origin)
self._commit("init")
git(self.repo, "push", "-u", "origin", "main")
git(self.repo, "remote", "set-head", "origin", "main")
self._write(os.path.join(self.repo, ".git", "info"), "exclude", ".worktrees/\n")
self._pushed("merged")
git(self.repo, "merge", "--no-ff", "-m", "merge merged", "merged")
git(self.repo, "push", "origin", "main")
self._pushed("ahead")
git(self.repo, "checkout", "ahead")
self._commit("ahead-local")
git(self.repo, "checkout", "main")
self._pushed("behind", commits=2)
git(self.repo, "branch", "-f", "behind", "behind~1")
self._pushed("gone")
git(self.repo, "push", "origin", "--delete", "gone")
self.changed = os.path.join(self.root, "repos", "repo", ".worktrees", "changed")
git(self.repo, "worktree", "add", "-b", "changed", self.changed, "main")
self._write(self.changed, "staged.txt", "s")
git(self.changed, "add", "staged.txt")
self._write(self.changed, "init.txt", "edited")
self._write(self.changed, "loose.txt", "u")
self.detached = os.path.join(self.shared, "repo", "detached")
git(self.repo, "worktree", "add", "--detach", self.detached, "main")
self.missing = os.path.join(self.root, "wt-missing")
git(self.repo, "worktree", "add", "-b", "missing", self.missing, "main")
shutil.rmtree(self.missing)
self.facts = ForgeFacts(merged_heads={}, open_heads={})
self.tree_file = os.path.join(self.root, "state", "tree", "branch-tree.json")
self.env = patch.dict(
os.environ,
{
"PKGMGR_TREE_FILE": self.tree_file,
"XDG_CACHE_HOME": os.path.join(self.root, "cache"),
},
)
self.env.start()
self.addCleanup(self.env.stop)
self.patches = [
patch.object(gc, "host_pid_namespace", return_value=True),
patch.object(gc, "list_compose_containers", return_value=[]),
patch.object(gc, "_process_cwds", return_value={}),
patch.object(
tree,
"secret_service_token",
return_value=(None, "Secret Service locked"),
),
]
for patcher in self.patches:
patcher.start()
self.addCleanup(patcher.stop)
def _write(self, cwd: str, name: str, text: str) -> None:
with open(os.path.join(cwd, name), "w", encoding="utf-8") as fh:
fh.write(text)
def _commit(self, name: str, cwd: str | None = None) -> None:
self._write(cwd or self.repo, f"{name}.txt", name)
git(cwd or self.repo, "add", f"{name}.txt")
git(cwd or self.repo, "commit", "-m", name)
def _pushed(self, name: str, commits: int = 1) -> None:
git(self.repo, "checkout", "-b", name, "main")
for number in range(commits):
self._commit(f"{name}-{number}")
git(self.repo, "push", "-u", "origin", name)
git(self.repo, "checkout", "main")
def _tree(self, fmt: str = "json", repos: list[str] | None = None) -> str:
out = io.StringIO()
with redirect_stdout(out):
code = tree.branch_tree(
repos or [self.repo], fmt=fmt, output=None, worktrees_root=self.shared
)
self.assertEqual(code, 0)
return out.getvalue()
def _entry(self) -> dict:
data = json.loads(self._tree())
return next(repo for repo in data["repos"] if repo["path"] == self.repo)
def _by_branch(self, entry: dict) -> dict[str, dict]:
return {item["branch"]: item for item in entry["items"] if item["branch"]}
def test_json_carries_repository_branch_and_worktree_fields(self) -> None:
data = json.loads(self._tree())
entry = next(repo for repo in data["repos"] if repo["path"] == self.repo)
branches = self._by_branch(entry)
self.assertIn("generated_at", data)
self.assertEqual(
(entry["remote"], entry["default"], entry["missing"]),
(self.origin, "main", False),
)
self.assertEqual(
(
branches["ahead"]["upstream"],
branches["ahead"]["ahead"],
branches["ahead"]["behind"],
),
("origin/ahead", 1, 0),
)
self.assertEqual(
(branches["behind"]["ahead"], branches["behind"]["behind"]), (0, 1)
)
self.assertTrue(branches["gone"]["gone"])
self.assertFalse(branches["merged"]["gone"])
changed = branches["changed"]
self.assertEqual(
(changed["worktree"], changed["worktree_state"]), (self.changed, "changed")
)
self.assertEqual(
(changed["staged"], changed["unstaged"], changed["untracked"]), (1, 1, 1)
)
self.assertEqual(changed["head"], git(self.repo, "rev-parse", "main"))
self.assertEqual(changed["state"], "dirty")
self.assertEqual(branches["missing"]["worktree_state"], "missing")
self.assertEqual(branches["main"]["worktree_state"], "clean")
detached = [item for item in entry["items"] if item["detached"]]
self.assertEqual(
[(item["worktree"], item["worktree_state"]) for item in detached],
[(self.detached, "clean")],
)
self.assertEqual(branches["merged"]["merged_by"], "ancestor")
self.assertEqual(entry["forge"], "none")
self.assertEqual(
(data["delete_ready"], data["delete_reason"]),
(False, "Secret Service locked"),
)
def test_yaml_prints_the_same_data(self) -> None:
self.assertEqual(
stable(yaml.safe_load(self._tree("yaml"))),
stable(json.loads(self._tree("json"))),
)
def test_tree_and_gc_share_every_state(self) -> None:
out = io.StringIO()
with (
patch.object(gc, "fetch_forge_facts", return_value=ForgeFacts()),
redirect_stdout(out),
):
gc.cleanup(
[self.repo],
scope="all",
apply=False,
as_json=True,
include=[],
exclude=[],
merged_into=[],
sizes=False,
min_age=None,
hooks={},
)
planned = {
(item["branch"], item["worktree"]): (
item["state"],
item["merged_by"],
item["skip"],
)
for item in json.loads(out.getvalue())["repos"][0]["items"]
}
listed = {
(item["branch"], item["worktree"]): (
item["state"],
item["merged_by"],
item["skip"],
)
for item in self._entry()["items"]
}
self.assertEqual(listed, planned)
def test_the_scan_runs_no_hook_fsmonitor_or_local_filter(self) -> None:
marker = os.path.join(self.root, "ran")
script = os.path.join(self.root, "plant.sh")
self._write(self.root, "plant.sh", f"#!/bin/sh\ntouch {marker}\n")
os.chmod(script, stat.S_IRWXU)
hooks = os.path.join(self.root, "hooks")
os.makedirs(hooks)
for hook in ("post-index-change", "reference-transaction", "post-checkout"):
shutil.copy(script, os.path.join(hooks, hook))
git(self.repo, "config", "core.fsmonitor", script)
git(self.repo, "config", "core.hooksPath", hooks)
git(self.repo, "config", "core.pager", script)
git(self.repo, "config", "filter.planted.clean", script)
self._write(self.changed, ".gitattributes", "* filter=planted\n")
with patch("subprocess.run", wraps=subprocess.run) as spy:
changed = self._by_branch(self._entry())["changed"]
self.assertFalse(os.path.exists(marker))
self.assertEqual(
(changed["worktree_state"], changed["staged"]), ("changed", None)
)
self.assertIn("filter.planted.clean", changed["reason"])
scans = [
call
for call in spy.call_args_list
if call.args and call.args[0][:1] == ["git"]
]
self.assertTrue(scans)
for call in scans:
with self.subTest(argv=call.args[0][:6]):
self.assertNotIn("fetch", call.args[0])
self.assertEqual(call.kwargs["env"]["GIT_NO_LAZY_FETCH"], "1")
self.assertIn("core.fsmonitor=false", call.args[0])
def test_a_worktree_with_submodules_is_not_read(self) -> None:
head = git(self.changed, "rev-parse", "HEAD")
git(self.changed, "update-index", "--add", "--cacheinfo", f"160000,{head},sub")
changed = self._by_branch(self._entry())["changed"]
self.assertEqual((changed["state"], changed["staged"]), ("dirty", None))
self.assertIn("submodules", changed["reason"])
def test_the_scan_never_runs_a_planted_gpg_program(self) -> None:
marker = os.path.join(self.root, "verified")
self._write(self.root, "gpg.sh", f"#!/bin/sh\ntouch {marker}\nexit 1\n")
os.chmod(os.path.join(self.root, "gpg.sh"), stat.S_IRWXU)
git(self.repo, "config", "log.showSignature", "true")
git(self.repo, "config", "gpg.program", os.path.join(self.root, "gpg.sh"))
body = git(self.repo, "cat-file", "commit", "HEAD").replace(
"\n\n",
"\ngpgsig -----BEGIN PGP SIGNATURE-----\n \n -----END PGP SIGNATURE-----\n\n",
1,
)
signed = subprocess.run(
["git", "hash-object", "-t", "commit", "-w", "--stdin"],
cwd=self.repo,
input=body,
capture_output=True,
text=True,
check=True,
).stdout.strip()
git(self.repo, "log", "-1", signed)
self.assertTrue(os.path.exists(marker))
os.unlink(marker)
run(["log", "-1", "--format=%ct", signed], cwd=self.repo, scan=True)
self.assertFalse(os.path.exists(marker))
def test_output_is_written_atomically(self) -> None:
target = os.path.join(self.root, "out", "branch-tree.json")
code = tree.branch_tree(
[self.repo], fmt="json", output=target, worktrees_root=self.shared
)
self.assertEqual(code, 0)
with open(target, encoding="utf-8") as fh:
self.assertEqual(json.load(fh)["repos"][0]["path"], self.repo)
self.assertEqual(
sorted(os.listdir(os.path.dirname(target))),
[".branch-tree.json.lock", "branch-tree.json"],
)
def test_lists_every_known_repository_also_plain_and_missing(self) -> None:
plain = os.path.join(self.root, "repos", "plain")
git(self.root, "init", "-b", "main", plain)
self._commit("plain", cwd=plain)
absent = os.path.join(self.root, "repos", "absent")
ctx = CLIContext(
config_merged={},
repositories_base_dir=os.path.join(self.root, "base"),
all_repositories=[
{
"provider": "x",
"account": "y",
"repository": "repo",
"directory": self.repo,
},
{
"provider": "x",
"account": "y",
"repository": "plain",
"directory": plain,
},
{
"provider": "x",
"account": "y",
"repository": "absent",
"directory": absent,
},
],
binaries_dir="",
user_config_path="",
)
args = create_parser("test").parse_args(["branch", "tree", "--format", "json"])
out = io.StringIO()
with (
patch.object(tree, "WORKTREES_ROOT", self.shared),
redirect_stdout(out),
self.assertRaises(SystemExit) as done,
):
handle_branch(args, ctx)
self.assertEqual(done.exception.code, 0)
repos = {repo["path"]: repo for repo in json.loads(out.getvalue())["repos"]}
self.assertEqual(set(repos), {self.repo, plain, absent})
self.assertEqual([item["branch"] for item in repos[plain]["items"]], ["main"])
self.assertEqual((repos[absent]["missing"], repos[absent]["items"]), (True, []))
def test_a_shared_worktree_of_an_unknown_repository_adds_it(self) -> None:
other = os.path.join(self.root, "elsewhere", "other")
git(self.root, "init", "-b", "main", other)
self._commit("other", cwd=other)
orphan = os.path.join(self.shared, "other", "feature")
git(other, "worktree", "add", "-b", "feature", orphan, "main")
repos = {repo["path"]: repo for repo in json.loads(self._tree())["repos"]}
self.assertEqual(
(repos[self.repo]["known"], repos[other]["known"]), (True, False)
)
self.assertIn(orphan, [item["worktree"] for item in repos[other]["items"]])
def test_forge_facts_come_from_the_secret_service_at_most_every_ten_minutes(
self,
) -> None:
git(self.repo, "remote", "set-url", "origin", "https://github.com/me/repo.git")
locked = self._entry()
self.assertEqual(locked["forge"], "unavailable: Secret Service locked")
self.assertEqual(self._by_branch(locked)["merged"]["merged_by"], "ancestor")
self.assertEqual(self._by_branch(locked)["ahead"]["skip"], "forge unavailable")
facts = ForgeFacts(
merged_heads={
git(self.repo, "rev-parse", ref): 9 for ref in ("ahead", "origin/ahead")
},
open_heads={},
)
with (
patch.object(tree, "secret_service_token", return_value=("t0ken", None)),
patch.object(tree, "facts_with_token", return_value=facts) as live,
):
first = self._by_branch(self._entry())
second = self._by_branch(self._entry())
self.assertEqual(live.call_count, 1)
with patch.object(tree, "now", return_value=time.time() + 601):
self._entry()
self.assertEqual(live.call_count, 2)
self.assertEqual(
(first["ahead"]["merged_by"], second["ahead"]["merged_by"]),
("pr#9", "pr#9"),
)
data = json.loads(self._tree())
self.assertEqual(
(data["delete_ready"], data["delete_reason"]),
(False, "Secret Service locked"),
)
self.assertEqual(data["repos"][0]["forge"], "cached")
with patch.object(tree, "secret_service_token", return_value=("t0ken", None)):
self.assertTrue(json.loads(self._tree())["delete_ready"])
def test_an_unwritable_cache_keeps_the_scan_and_names_it(self) -> None:
git(self.repo, "remote", "set-url", "origin", "https://github.com/me/repo.git")
blocker = os.path.join(self.root, "cache-is-a-file")
self._write(self.root, "cache-is-a-file", "")
with (
patch.dict(os.environ, {"XDG_CACHE_HOME": blocker}),
patch.object(tree, "secret_service_token", return_value=("t0ken", None)),
patch.object(tree, "facts_with_token", return_value=ForgeFacts()),
):
entry = self._entry()
self.assertTrue(entry["forge"].startswith("live: cache not written"))
self.assertIn("main", self._by_branch(entry))
def test_branch_actions_rewrite_the_tree_file_when_it_exists(self) -> None:
tree.branch_tree(
[self.repo], fmt="json", output=self.tree_file, worktrees_root=self.shared
)
args = create_parser("test").parse_args(["branch", "open", "fresh"])
cwd = os.getcwd()
os.chdir(self.repo)
self.addCleanup(os.chdir, cwd)
handle_branch(args, ctx=None)
with open(self.tree_file, encoding="utf-8") as fh:
entry = json.load(fh)["repos"][0]
self.assertIn("fresh", [item["branch"] for item in entry["items"]])
if __name__ == "__main__":
unittest.main()
+105
View File
@@ -0,0 +1,105 @@
from __future__ import annotations
import io
import json
import os
import unittest
from contextlib import redirect_stdout
from unittest.mock import patch
from pkgmgr.actions.branch import gc, tree
from pkgmgr.actions.worktree import flow
from tests.integration.test_branch_gc import GcFixture
class TestBranchTreeFlow(GcFixture, unittest.TestCase):
def setUp(self) -> None:
super().setUp()
self.shared = os.path.join(self.root, "shared-worktrees")
os.makedirs(self.shared)
self.metas = os.path.join(self.repo, ".git", "worktrees")
for patcher in (
patch.object(gc, "host_pid_namespace", return_value=True),
patch.object(gc, "list_compose_containers", return_value=[]),
patch.object(gc, "_process_cwds", return_value={}),
patch.object(tree, "secret_service_token", return_value=(None, "locked")),
):
patcher.start()
self.addCleanup(patcher.stop)
def _entry(self) -> dict:
out = io.StringIO()
with redirect_stdout(out):
code = tree.branch_tree(
[self.repo], fmt="json", output=None, worktrees_root=self.shared
)
self.assertEqual(code, 0)
return json.loads(out.getvalue())["repos"][0]
def _flows(self, entry: dict) -> dict[str, dict | None]:
return {
item["branch"]: item["flow"] for item in entry["items"] if item["branch"]
}
def _flow_warnings(self, entry: dict) -> list[str]:
return [w for w in entry["warnings"] if w.startswith("flow of ")]
def test_items_carry_their_flow_or_null(self) -> None:
flow.write_flow(
os.path.join(self.metas, "wt-merged"), "merged", ["origin/main"], ["prod"]
)
flow.write_flow(os.path.join(self.repo, ".git"), "main", [], ["origin/main"])
entry = self._entry()
flows = self._flows(entry)
self.assertEqual(
flows["merged"],
{"pull": ["origin/main"], "push": ["prod"], "source": "file"},
)
self.assertEqual(
flows["main"], {"pull": [], "push": ["origin/main"], "source": "file"}
)
self.assertIsNone(flows["dirty"])
self.assertIsNone(flows["wip"])
self.assertTrue(all("flow" in item for item in entry["items"]))
self.assertEqual(self._flow_warnings(entry), [])
def test_invalid_and_mismatched_flows_warn_without_failing(self) -> None:
folder = os.path.join(self.metas, "wt-dirty", flow.FLOW_DIR)
os.makedirs(folder)
with open(os.path.join(folder, flow.FLOW_FILE), "w", encoding="utf-8") as fh:
fh.write("version: 1\npull: [a..b]\npush: []\n")
flow.write_flow(os.path.join(self.metas, "wt-merged"), "gone", ["main"], [])
entry = self._entry()
flows = self._flows(entry)
self.assertEqual(flows["dirty"], {"pull": [], "push": [], "source": "invalid"})
self.assertEqual(flows["merged"]["source"], "file")
warnings = self._flow_warnings(entry)
self.assertEqual(len(warnings), 2)
self.assertTrue(
any(
w.startswith(f"flow of {self.dirty_wt}: pkgmgr/flow.yml")
for w in warnings
)
)
self.assertIn(
f"flow of {self.merged_wt}: flow was set on gone, the worktree is on merged",
warnings,
)
def test_a_forged_registration_reads_no_flow(self) -> None:
flow.write_flow(os.path.join(self.metas, "wt-merged"), "merged", ["main"], [])
with open(os.path.join(self.metas, "wt-locked", "gitdir"), "w") as fh:
fh.write(os.path.join(self.merged_wt, ".git") + "\n")
items = {item["branch"]: item for item in self._entry()["items"]}
for branch in ("merged", "locked"):
with self.subTest(branch=branch):
self.assertEqual(items[branch]["worktree"], self.merged_wt)
self.assertIsNone(items[branch]["flow"])
if __name__ == "__main__":
unittest.main()
@@ -0,0 +1,321 @@
from __future__ import annotations
import os
import subprocess
import tempfile
import unittest
from pathlib import Path
REPO_ROOT = Path(__file__).resolve().parents[2]
BASE = "--build-arg BASE_IMAGE=ghcr.io/kevinveenbirkenbach/base-{distro}:latest ."
STAMP = "1767225600"
DIGEST = "{{json .Manifest.Digest}}"
FAKE_DOCKER = """#!/usr/bin/env bash
echo "$*" >> "$FAKE_DOCKER_LOG"
case "$*" in
"run "*" ghcr.io/"*)
for tag in ${FAKE_PUBLISHED_TAGS}; do
if [[ "$*" == *" ${tag} -lc "* ]]; then echo "${FAKE_PUBLISHED_SET}"; exit 0; fi
done
exit 1 ;;
"run "*) echo "${FAKE_BUILT_SET}" ;;
"buildx imagetools create --dry-run "*) echo "${FAKE_PLANNED_INDEX}" ;;
"buildx imagetools inspect --raw "*)
if [[ " ${FAKE_CURRENT_TAGS} " == *" ${!#} "* ]]; then
echo "${FAKE_PLANNED_INDEX}"
else
echo stale
fi ;;
"buildx imagetools inspect --format "*)
if [[ "${!#}" == "${FAKE_DRIFTED_TAG}" ]]; then echo old; else echo new; fi ;;
esac
"""
def refs(image: str, *tags: str) -> list[str]:
return [f"ghcr.io/o/{image}:{tag}" for tag in tags]
def sources(distro: str, version: str) -> str:
return " ".join(refs(f"pkgmgr-{distro}", f"{version}-amd64", f"{version}-arm64"))
def tag_args(tags: list[str]) -> str:
return " ".join(f"-t {tag}" for tag in tags)
class TestBuildImagePublish(unittest.TestCase):
def setUp(self) -> None:
self._tmp = tempfile.TemporaryDirectory()
tmp = Path(self._tmp.name)
self.log = tmp / "docker.log"
fake = tmp / "docker"
fake.write_text(FAKE_DOCKER)
fake.chmod(0o755)
date = tmp / "date"
date.write_text(f"#!/usr/bin/env bash\necho {STAMP}\n")
date.chmod(0o755)
self.env = {
k: v for k, v in os.environ.items() if not k.startswith("BASE_IMAGES_")
}
self.env["PATH"] = f"{tmp}{os.pathsep}{os.environ['PATH']}"
self.env["FAKE_DOCKER_LOG"] = str(self.log)
self.env["FAKE_BUILT_SET"] = "built"
self.env["FAKE_PUBLISHED_SET"] = "published"
self.env["FAKE_PUBLISHED_TAGS"] = ""
self.env["FAKE_PLANNED_INDEX"] = "planned"
self.env["FAKE_CURRENT_TAGS"] = ""
self.env["FAKE_DRIFTED_TAG"] = ""
def tearDown(self) -> None:
self._tmp.cleanup()
def _run(self, script: str, distro: str, *args: str, **env: str):
result = subprocess.run(
["bash", f"scripts/build/{script}", *args],
check=False,
cwd=REPO_ROOT,
env={**self.env, "PKGMGR_DISTRO": distro, **env},
capture_output=True,
text=True,
)
calls = self.log.read_text().splitlines() if self.log.exists() else []
return result, calls
def _publish(self, distro: str, version: str, *args: str, **env: str):
return self._run(
"image.sh",
distro,
"--publish",
"--registry",
"ghcr.io",
"--owner",
"o",
"--version",
version,
*args,
**env,
)
def test_platform_pushes_only_the_arch_tag(self) -> None:
result, calls = self._publish("debian", "2.1.2", "--platform", "linux/arm64")
self.assertEqual(result.returncode, 0, result.stderr)
self.assertEqual(
calls,
[
("buildx build --load --platform linux/arm64 -t pkgmgr-debian:probe ")
+ BASE.format(distro="debian"),
(
"run --rm --platform linux/arm64 --entrypoint bash "
"pkgmgr-debian:probe -lc dpkg-query -W"
),
(
"run --rm --pull always --platform linux/arm64 --entrypoint bash "
"ghcr.io/o/pkgmgr-debian:2.1.2 -lc dpkg-query -W"
),
(
"buildx build --push -t ghcr.io/o/pkgmgr-debian:2.1.2-arm64 "
"--platform linux/arm64 "
)
+ BASE.format(distro="debian"),
],
)
def test_platform_unchanged_since_its_own_published_version_pushes_nothing(
self,
) -> None:
result, calls = self._publish(
"debian",
"2.3.0",
"--platform",
"linux/arm64",
FAKE_BUILT_SET="same",
FAKE_PUBLISHED_SET="same",
FAKE_PUBLISHED_TAGS="ghcr.io/o/pkgmgr-debian:2.3.0",
)
self.assertEqual(result.returncode, 0, result.stderr)
self.assertIn("nothing to push", result.stdout)
self.assertFalse([c for c in calls if c.startswith("buildx build --push")])
def test_platform_pushes_when_its_packages_changed(self) -> None:
result, calls = self._publish(
"debian",
"2.3.0",
"--platform",
"linux/arm64",
FAKE_PUBLISHED_TAGS="ghcr.io/o/pkgmgr-debian:2.3.0",
)
self.assertEqual(result.returncode, 0, result.stderr)
self.assertTrue(calls[-1].startswith("buildx build --push"), calls)
def test_platform_pushes_while_its_version_is_not_out(self) -> None:
result, calls = self._publish(
"debian",
"2.4.0",
"--platform",
"linux/arm64",
FAKE_BUILT_SET="same",
FAKE_PUBLISHED_SET="same",
FAKE_PUBLISHED_TAGS="ghcr.io/o/pkgmgr-debian:latest",
)
self.assertEqual(result.returncode, 0, result.stderr)
self.assertIn(
(
"run --rm --pull always --platform linux/arm64 --entrypoint bash "
"ghcr.io/o/pkgmgr-debian:2.4.0 -lc dpkg-query -W"
),
calls,
)
self.assertTrue(calls[-1].startswith("buildx build --push"), calls)
def test_merge_of_the_newest_release_tags_latest_and_its_prefixes(self) -> None:
result, calls = self._publish(
"arch", "2.4.0", "--aliases", "latest 2 2.4", "--merge"
)
self.assertEqual(result.returncode, 0, result.stderr)
moving = refs("pkgmgr-arch", "2.4.0", "latest", "2", "2.4") + refs(
"pkgmgr", "2.4.0", "latest", "2", "2.4"
)
stamps = refs("pkgmgr-arch", f"2.4.0.{STAMP}") + refs(
"pkgmgr", f"2.4.0.{STAMP}"
)
self.assertEqual(
calls,
[
(
"buildx imagetools create --dry-run "
f"-t ghcr.io/o/pkgmgr-arch:2.4.0 {sources('arch', '2.4.0')}"
),
"buildx imagetools inspect --raw ghcr.io/o/pkgmgr-arch:2.4.0",
(
f"buildx imagetools create {tag_args(moving + stamps)} "
f"{sources('arch', '2.4.0')}"
),
f"buildx imagetools inspect --format {DIGEST} ghcr.io/o/pkgmgr-arch:2.4.0",
]
+ [f"buildx imagetools inspect --format {DIGEST} {t}" for t in moving],
)
def test_merge_of_an_older_stable_release_tags_stable_and_only_its_own_prefixes(
self,
) -> None:
result, calls = self._publish(
"debian", "2.3.0", "--aliases", "2.3 stable", "--merge"
)
self.assertEqual(result.returncode, 0, result.stderr)
tags = refs("pkgmgr-debian", "2.3.0", "2.3", "stable", f"2.3.0.{STAMP}")
self.assertIn(
f"buildx imagetools create {tag_args(tags)} {sources('debian', '2.3.0')}",
calls,
)
def test_merge_moves_stable_to_an_unchanged_build_without_a_new_stamp(
self,
) -> None:
result, calls = self._publish(
"debian",
"2.4.0",
"--aliases",
"latest 2 2.4 stable",
"--merge",
FAKE_CURRENT_TAGS=" ".join(
refs("pkgmgr-debian", "2.4.0", "latest", "2", "2.4")
),
)
self.assertEqual(result.returncode, 0, result.stderr)
self.assertEqual(
[c for c in calls if c.startswith("buildx imagetools create -t")],
[
"buildx imagetools create -t ghcr.io/o/pkgmgr-debian:stable "
+ sources("debian", "2.4.0")
],
)
def test_merge_pushes_nothing_when_every_tag_shows_the_build(self) -> None:
result, calls = self._publish(
"arch",
"2.4.0",
"--aliases",
"latest 2 2.4 stable",
"--merge",
FAKE_CURRENT_TAGS=" ".join(
refs("pkgmgr-arch", "2.4.0", "latest", "2", "2.4", "stable")
+ refs("pkgmgr", "2.4.0", "latest", "2", "2.4", "stable")
),
)
self.assertEqual(result.returncode, 0, result.stderr)
self.assertIn("nothing to push", result.stdout)
self.assertEqual(
[c for c in calls if c.startswith("buildx imagetools create -t")], []
)
def test_merge_fails_when_a_tag_drifts_from_its_version(self) -> None:
result, _ = self._publish(
"debian",
"2.1.2",
"--aliases",
"2.1",
"--merge",
FAKE_DRIFTED_TAG="ghcr.io/o/pkgmgr-debian:2.1",
)
self.assertEqual(result.returncode, 1)
self.assertIn(
":2.1.2 and ghcr.io/o/pkgmgr-debian:2.1 point at different images",
result.stderr,
)
def test_unknown_platform_merge_without_publish_or_aliases_are_refused(
self,
) -> None:
unknown, _ = self._publish("debian", "2.1.2", "--platform", "linux/s390x")
no_publish, _ = self._run("image.sh", "debian", "--merge")
no_aliases, calls = self._publish("debian", "2.1.2", "--merge")
self.assertEqual(unknown.returncode, 2)
self.assertEqual(no_publish.returncode, 2)
self.assertEqual(no_aliases.returncode, 2)
self.assertEqual(calls, [])
def test_publish_builds_each_platform_then_merges(self) -> None:
result, calls = self._run(
"publish.sh",
"fedora",
OWNER="o",
VERSION="2.1.2",
ALIASES="2.1 stable",
DISTROS="fedora",
)
self.assertEqual(result.returncode, 0, result.stderr)
pushes = [
c
for c in calls
if c.startswith(("buildx build --push", "buildx imagetools create"))
]
self.assertEqual(
[c.split(" -t ")[0] for c in pushes],
[
"buildx build --push",
"buildx build --push",
"buildx imagetools create --dry-run",
"buildx imagetools create",
],
)
self.assertIn("--platform linux/amd64", pushes[0])
self.assertIn("--platform linux/arm64", pushes[1])
self.assertIn("-t ghcr.io/o/pkgmgr-fedora:stable ", pushes[3])
self.assertNotIn(":latest", pushes[3])
if __name__ == "__main__":
unittest.main()
+215
View File
@@ -0,0 +1,215 @@
from __future__ import annotations
import io
import json
import os
import subprocess
import tempfile
import unittest
from contextlib import redirect_stdout
from unittest.mock import patch
from pkgmgr.actions.gc.report import disk_report, render_text
from pkgmgr.actions.gc.report import docker_usage as docker_module
from pkgmgr.actions.gc.report import walk as walk_module
from pkgmgr.cli.commands.worktree import handle_gc
from pkgmgr.cli.parser import create_parser
GIT_ENV = {
**os.environ,
"GIT_CONFIG_GLOBAL": os.devnull,
"GIT_CONFIG_NOSYSTEM": "1",
"GIT_AUTHOR_NAME": "t",
"GIT_AUTHOR_EMAIL": "t@example.org",
"GIT_COMMITTER_NAME": "t",
"GIT_COMMITTER_EMAIL": "t@example.org",
}
THRESHOLD = 32 * 1024
ELF_CORE = b"\x7fELF\x02\x01\x01" + bytes(9) + (4).to_bytes(2, "little")
ELF_EXEC = b"\x7fELF\x02\x01\x01" + bytes(9) + (2).to_bytes(2, "little")
DOCKER_DF = (
'{"Active":"1","Reclaimable":"0B","Size":"1GB","TotalCount":"3","Type":"Images"}\n'
)
def git(cwd: str, *args: str) -> str:
return subprocess.run(
["git", *args], cwd=cwd, env=GIT_ENV, check=True, capture_output=True, text=True
).stdout
def write(path: str, data: bytes | str, size: int | None = None) -> str:
os.makedirs(os.path.dirname(path), exist_ok=True)
with open(path, "wb") as fh:
fh.write(data.encode() if isinstance(data, str) else data)
if size is not None:
os.truncate(path, size)
return path
class TestDiskReport(unittest.TestCase):
def setUp(self) -> None:
self._tmp = tempfile.TemporaryDirectory()
self.tmp = os.path.realpath(self._tmp.name)
self.repo = os.path.join(self.tmp, "repo")
git(self.tmp, "init", "-q", "-b", "main", self.repo)
write(os.path.join(self.repo, "a.txt"), "base")
git(self.repo, "add", "a.txt")
git(self.repo, "commit", "-q", "-m", "base")
git(self.repo, "checkout", "-q", "-b", "other")
write(os.path.join(self.repo, "a.txt"), "kept")
write(os.path.join(self.repo, "b.txt"), "kept-b")
git(self.repo, "add", "a.txt", "b.txt")
git(self.repo, "commit", "-q", "-m", "other")
git(self.repo, "checkout", "-q", "main")
self.clean = self._worktree("wt-clean", "-b", "clean")
self.kept = self._worktree("wt-kept", "-b", "kept")
write(os.path.join(self.kept, "a.txt"), "kept")
git(self.kept, "add", "a.txt")
write(os.path.join(self.kept, "b.txt"), "kept-b")
self.lost = self._worktree("wt-lost", "-b", "lost")
write(os.path.join(self.lost, "new.txt"), "only here")
self.detached = self._worktree("wt-detached", "--detach")
write(os.path.join(self.detached, "c.txt"), "c")
git(self.detached, "add", "c.txt")
git(self.detached, "commit", "-q", "-m", "unreachable")
write(os.path.join(self.detached, "a.txt"), "kept")
agents = os.path.join(self.repo, ".claude", "worktrees")
self.agent = self._worktree(os.path.join(agents, "agent-y"), "-b", "agent-y")
self.orphan = os.path.join(agents, "agent-x")
write(os.path.join(self.orphan, "notes.txt"), "left behind")
self.root = os.path.join(self.tmp, "root")
self.dump = write(
os.path.join(self.root, "core"), ELF_CORE + b"c" * 64 * 1024, 1 << 30
)
self.binary = write(
os.path.join(self.root, "core.12"), ELF_EXEC + b"e" * 64 * 1024
)
write(os.path.join(self.root, "bigdir", "blob"), b"b" * 128 * 1024)
write(os.path.join(self.root, "small"), "s")
def tearDown(self) -> None:
self._tmp.cleanup()
def _worktree(self, name: str, *args: str) -> str:
path = os.path.join(self.tmp, name)
git(self.repo, "worktree", "add", "-q", *args, path, "main")
return path
def _report(self) -> dict:
with patch.object(docker_module, "_docker", return_value=DOCKER_DF):
return disk_report([self.root], [self.repo], THRESHOLD, docker=True)
def _cli(self, *argv: str) -> tuple[int, str]:
args = create_parser("test").parse_args(["gc", *argv])
out = io.StringIO()
with (
patch("pkgmgr.cli.commands.branch.main_repo_root", return_value=self.repo),
patch(
"pkgmgr.cli.commands.branch.cleanup",
side_effect=AssertionError("the report must not run a cleanup"),
),
patch.object(docker_module, "_docker", return_value=DOCKER_DF),
redirect_stdout(out),
self.assertRaises(SystemExit) as stop,
):
handle_gc(args, None)
return stop.exception.code, out.getvalue()
def test_cli_report_prints_the_json_inventory(self) -> None:
code, out = self._cli("--report", "--json", "--root", self.root, "--docker")
report = json.loads(out)
self.assertEqual(code, 0)
self.assertIn(self.dump, [dump["path"] for dump in report["core_dumps"]])
self.assertIn(self.lost, [tree["path"] for tree in report["worktrees"]])
self.assertIsNotNone(report["docker"])
def test_cli_report_refuses_apply(self) -> None:
self.assertEqual(self._cli("--report", "--apply"), (2, ""))
def test_worktrees_and_orphaned_agent_dir(self) -> None:
trees = {t["path"]: t for t in self._report()["worktrees"]}
expected = {
self.clean,
self.kept,
self.lost,
self.detached,
self.agent,
self.orphan,
}
self.assertEqual(set(trees), expected)
self.assertTrue(trees[self.orphan]["orphaned"])
self.assertFalse(trees[self.agent]["orphaned"])
self.assertEqual(trees[self.kept]["branch"], "kept")
self.assertIsNone(trees[self.detached]["branch"])
for tree in trees.values():
self.assertGreater(tree["size_bytes"], 0)
self.assertIsNotNone(tree["last_activity"])
self.assertEqual(tree["foreign_entries"], 0)
def test_foreign_entries_through_patched_owner_check(self) -> None:
with patch.object(
walk_module,
"is_foreign",
side_effect=lambda path, st: os.path.basename(path) == "new.txt",
):
trees = {t["path"]: t for t in self._report()["worktrees"]}
self.assertEqual(trees[self.lost]["foreign_entries"], 1)
self.assertEqual(trees[self.clean]["foreign_entries"], 0)
def test_preserved_verdicts_and_nothing_written(self) -> None:
before = git(self.kept, "status", "--porcelain=v2", "--untracked-files=all")
verdicts = {p["path"]: p for p in self._report()["preserved"]}
self.assertEqual(set(verdicts), {self.kept, self.lost, self.detached})
self.assertTrue(verdicts[self.kept]["preserved"])
self.assertEqual(verdicts[self.kept]["paths"], 2)
self.assertFalse(verdicts[self.lost]["preserved"])
self.assertEqual(verdicts[self.lost]["reasons"], ["new.txt: content in no ref"])
self.assertFalse(verdicts[self.detached]["preserved"])
self.assertEqual(
verdicts[self.detached]["reasons"], ["detached HEAD reachable from no ref"]
)
after = git(self.kept, "status", "--porcelain=v2", "--untracked-files=all")
self.assertEqual(before, after)
def test_roots_core_dumps_docker_and_balance(self) -> None:
report = self._report()
children = report["roots"][0]["children"]
self.assertEqual(
[(os.path.basename(c["path"]), c["kind"]) for c in children],
[("bigdir", "dir"), ("core", "core-dump"), ("core.12", "file")],
)
self.assertEqual([d["path"] for d in report["core_dumps"]], [self.dump])
dump = report["core_dumps"][0]
self.assertEqual(dump["apparent_bytes"], 1 << 30)
self.assertLess(dump["size_bytes"], dump["apparent_bytes"])
self.assertEqual(report["docker"][0]["total"], 3)
self.assertEqual(len(report["balance"]), 1)
listed = sum(c["size_bytes"] for c in children)
listed += sum(t["size_bytes"] for t in report["worktrees"])
self.assertEqual(report["balance"][0]["attributed_bytes"], listed)
self.assertEqual(report["warnings"], [])
json.dumps(report)
text = render_text(report)
self.assertIn("orphaned", text)
self.assertIn("core-dump", text)
def test_unknown_repo_and_root_are_warnings(self) -> None:
missing = os.path.join(self.tmp, "absent")
report = disk_report([missing], [self.root], THRESHOLD, docker=False)
self.assertEqual(report["worktrees"], [])
self.assertIsNone(report["docker"])
self.assertEqual(len(report["warnings"]), 2)
if __name__ == "__main__":
unittest.main()
@@ -0,0 +1,72 @@
from __future__ import annotations
import os
import subprocess
import tempfile
import unittest
from pkgmgr.core.git.commands import delete_remote_branch, push, push_upstream
REMOTE = "-r"
def git(cwd: str, *args: str) -> str:
return subprocess.run(
[
"git",
"-c",
"user.name=t",
"-c",
"user.email=t@example.org",
"-c",
"commit.gpgsign=false",
*args,
],
cwd=cwd,
check=True,
capture_output=True,
text=True,
).stdout.strip()
class TestPushCommandsEndOptions(unittest.TestCase):
def setUp(self) -> None:
self._tmp = tempfile.TemporaryDirectory()
self.addCleanup(self._tmp.cleanup)
root = os.path.realpath(self._tmp.name)
self.origin = os.path.join(root, "origin.git")
self.repo = os.path.join(root, "repo")
git(root, "init", "-q", "--bare", "-b", "main", self.origin)
git(root, "init", "-q", "-b", "main", self.repo)
git(self.repo, "config", f"remote.{REMOTE}.url", self.origin)
git(
self.repo,
"config",
f"remote.{REMOTE}.fetch",
f"+refs/heads/*:refs/remotes/{REMOTE}/*",
)
git(self.repo, "commit", "-q", "--allow-empty", "-m", "init")
def _branches(self) -> str:
return git(self.origin, "for-each-ref", "--format=%(refname:short)")
def test_remote_named_like_an_option(self) -> None:
push_upstream(REMOTE, "main", cwd=self.repo)
self.assertEqual(self._branches(), "main")
self.assertEqual(
git(self.repo, "rev-parse", "--abbrev-ref", "@{u}"), f"{REMOTE}/main"
)
git(self.repo, "commit", "-q", "--amend", "--allow-empty", "-m", "rewritten")
push(REMOTE, "main:feature", cwd=self.repo)
push(REMOTE, "main", force=True, cwd=self.repo)
self.assertEqual(
git(self.origin, "rev-parse", "main"), git(self.repo, "rev-parse", "HEAD")
)
delete_remote_branch(REMOTE, "feature", cwd=self.repo)
self.assertEqual(self._branches(), "main")
if __name__ == "__main__":
unittest.main()
@@ -0,0 +1,239 @@
from __future__ import annotations
import json
import os
import subprocess
import tempfile
import unittest
from pathlib import Path
import yaml
REPO_ROOT = Path(__file__).resolve().parents[2]
RESOLVER = (
REPO_ROOT
/ "scripts"
/ "github"
/ "publish-containers"
/ "compute-publish-container-info.sh"
)
WORKFLOW = REPO_ROOT / ".github" / "workflows" / "publish-containers.yml"
GIT_ENV = {
**os.environ,
"GIT_CONFIG_GLOBAL": os.devnull,
"GIT_CONFIG_NOSYSTEM": "1",
"GIT_AUTHOR_NAME": "test",
"GIT_AUTHOR_EMAIL": "test@example.invalid",
"GIT_COMMITTER_NAME": "test",
"GIT_COMMITTER_EMAIL": "test@example.invalid",
}
class TestComputePublishContainerInfo(unittest.TestCase):
def setUp(self) -> None:
self._tmp = tempfile.TemporaryDirectory()
tmp = Path(self._tmp.name)
self.repo = tmp / "repo"
self.repo.mkdir()
self.output = tmp / "github_output"
self.git("init", "-q", "-b", "main")
def tearDown(self) -> None:
self._tmp.cleanup()
def git(self, *args: str) -> str:
return subprocess.run(
["git", *args],
cwd=self.repo,
env=GIT_ENV,
capture_output=True,
text=True,
check=True,
).stdout.strip()
def commit(self, *tags: str) -> str:
self.git("commit", "-q", "--allow-empty", "-m", " ".join(tags) or "untagged")
for tag in tags:
self.git("tag", tag)
return self.git("rev-parse", "HEAD")
def push_main(self) -> None:
self.git("update-ref", "refs/remotes/origin/main", "HEAD")
def resolve(self, workflow_run_sha: str):
self.output.write_text("")
result = subprocess.run(
["bash", str(RESOLVER)],
cwd=self.repo,
env={
**GIT_ENV,
"GITHUB_OUTPUT": str(self.output),
"WORKFLOW_RUN_SHA": workflow_run_sha,
},
capture_output=True,
text=True,
check=False,
)
outputs = dict(
line.split("=", 1) for line in self.output.read_text().splitlines()
)
return result, outputs
def releases(self, workflow_run_sha: str) -> list[dict[str, str]]:
result, outputs = self.resolve(workflow_run_sha)
self.assertEqual(result.returncode, 0, result.stderr)
return json.loads(outputs["releases"])
def test_a_rebuild_with_stable_on_the_newest_release_publishes_it_once(
self,
) -> None:
self.commit("v2.3.0")
newest = self.commit("v2.4.0", "stable")
self.push_main()
self.assertEqual(
self.releases(""),
[{"sha": newest, "version": "2.4.0", "aliases": "latest 2 2.4 stable"}],
)
def test_a_rebuild_with_stable_behind_publishes_the_newest_and_the_stable_release(
self,
) -> None:
self.commit("v2.2.0")
stable = self.commit("v2.3.0", "stable")
newest = self.commit("v2.4.0")
self.commit("v2.5.0-rc1")
self.push_main()
self.assertEqual(
self.releases(""),
[
{"sha": newest, "version": "2.4.0", "aliases": "latest 2 2.4"},
{"sha": stable, "version": "2.3.0", "aliases": "2.3 stable"},
],
)
def test_the_stable_release_takes_no_prefix_a_newer_release_leads(self) -> None:
stable = self.commit("v2.3.0", "stable")
self.commit("v2.3.1")
newest = self.commit("v2.4.0")
self.push_main()
self.assertEqual(
self.releases(""),
[
{"sha": newest, "version": "2.4.0", "aliases": "latest 2 2.4"},
{"sha": stable, "version": "2.3.0", "aliases": "stable"},
],
)
def test_a_rebuild_publishes_the_newest_release_on_main_not_its_tip(
self,
) -> None:
self.commit("v1.9.0")
newest = self.commit("v1.10.0")
self.commit()
self.push_main()
self.git("checkout", "-q", "-b", "side")
self.commit("v3.0.0")
result, outputs = self.resolve("")
self.assertEqual(result.returncode, 0, result.stderr)
self.assertEqual(
json.loads(outputs["releases"]),
[{"sha": newest, "version": "1.10.0", "aliases": "latest 1 1.10"}],
)
self.assertIn("the stable image stays where it is", result.stdout)
def test_a_workflow_run_publishes_the_commit_it_tagged(self) -> None:
self.commit("v2.3.0")
tagged = self.commit("v2.4.0", "stable")
self.push_main()
self.assertEqual(
self.releases(tagged),
[{"sha": tagged, "version": "2.4.0", "aliases": "latest 2 2.4 stable"}],
)
def test_a_workflow_run_of_a_backport_moves_only_the_prefixes_it_leads(
self,
) -> None:
self.commit("v2.3.0")
self.commit("v2.4.0", "stable")
backport = self.commit("v2.3.1")
self.push_main()
self.assertEqual(
self.releases(backport),
[{"sha": backport, "version": "2.3.1", "aliases": "2.3"}],
)
def test_a_workflow_run_outside_main_publishes_nothing(self) -> None:
self.commit("v2.3.0")
self.push_main()
self.git("checkout", "-q", "-b", "side")
outside = self.commit("v9.0.0")
self.assertEqual(self.releases(outside), [])
def test_a_workflow_run_without_an_x_y_z_tag_publishes_nothing(self) -> None:
self.commit("v2.3.0")
candidate = self.commit("v2.4.0-rc1")
self.push_main()
self.assertEqual(self.releases(candidate), [])
def test_a_rebuild_without_a_release_on_main_fails(self) -> None:
self.commit("v1.0.0-rc1", "stable")
self.push_main()
result, outputs = self.resolve("")
self.assertEqual(result.returncode, 1)
self.assertEqual(outputs, {})
self.assertIn("No vX.Y.Z tag is reachable from origin/main", result.stderr)
class TestPublishContainersWorkflow(unittest.TestCase):
def setUp(self) -> None:
self.workflow = yaml.safe_load(WORKFLOW.read_text())
def test_the_releases_are_rebuilt_daily_at_03_utc_and_on_demand(self) -> None:
triggers = self.workflow[True]
steps = {
step.get("id"): step for step in self.workflow["jobs"]["prepare"]["steps"]
}
self.assertEqual(triggers["schedule"], [{"cron": "0 3 * * *"}])
self.assertIn("workflow_dispatch", triggers)
self.assertEqual(
steps["info"]["run"],
"bash scripts/github/publish-containers/compute-publish-container-info.sh",
)
self.assertEqual(
steps["info"]["env"]["WORKFLOW_RUN_SHA"],
"${{ github.event.workflow_run.head_sha }}",
)
def test_every_resolved_release_is_built_and_merged_with_its_own_tags(
self,
) -> None:
jobs = self.workflow["jobs"]
merge_env = jobs["merge"]["steps"][-1]["env"]
for job in ("build", "merge"):
self.assertEqual(
jobs[job]["strategy"]["matrix"]["release"],
"${{ fromJSON(needs.prepare.outputs.releases) }}",
)
self.assertEqual(
jobs[job]["steps"][0]["with"]["ref"], "${{ matrix.release.sha }}"
)
self.assertEqual(merge_env["VERSION"], "${{ matrix.release.version }}")
self.assertEqual(merge_env["ALIASES"], "${{ matrix.release.aliases }}")
if __name__ == "__main__":
unittest.main()
@@ -0,0 +1,407 @@
from __future__ import annotations
import io
import os
import shutil
import subprocess
import tempfile
import unittest
from contextlib import redirect_stdout
from pathlib import Path
from unittest.mock import patch
from pkgmgr.actions.release.retry import retry_release
from pkgmgr.actions.release.workflow import _release_impl
from pkgmgr.actions.repository.push import (
PushRefusedError,
push_branch,
sign_unpushed,
)
from pkgmgr.cli.context import CLIContext
from pkgmgr.cli.parser import create_parser
from pkgmgr.cli.proxy import maybe_handle_proxy
def git(cwd: str, *args: str) -> str:
return subprocess.run(
["git", *args], cwd=cwd, check=True, capture_output=True, text=True
).stdout.strip()
@unittest.skipUnless(shutil.which("ssh-keygen"), "ssh-keygen signs the commits")
class TestPushMirrorLeases(unittest.TestCase):
def setUp(self) -> None:
self._tmp = tempfile.TemporaryDirectory()
self.addCleanup(self._tmp.cleanup)
root = Path(os.path.realpath(self._tmp.name))
self.root = str(root)
key = root / "key"
subprocess.run(
["ssh-keygen", "-q", "-t", "ed25519", "-N", "", "-f", str(key)],
check=True,
)
(root / "allowed").write_text(f"t@example.org {(root / 'key.pub').read_text()}")
(root / "gitconfig").write_text(
"[user]\n\tname = t\n\temail = t@example.org\n"
f"\tsigningkey = {key}\n"
"[gpg]\n\tformat = ssh\n"
f'[gpg "ssh"]\n\tallowedSignersFile = {root / "allowed"}\n'
"[commit]\n\tgpgsign = false\n"
"[init]\n\tdefaultBranch = main\n"
)
env = patch.dict(
os.environ,
{"GIT_CONFIG_GLOBAL": str(root / "gitconfig"), "GIT_CONFIG_NOSYSTEM": "1"},
)
env.start()
self.addCleanup(env.stop)
self.canonical = str(root / "canonical.git")
self.mirror = str(root / "mirror.git")
self.repo = str(root / "repo")
git(self.root, "init", "-q", "--bare", self.canonical)
git(self.root, "init", "-q", "--bare", self.mirror)
git(self.root, "init", "-q", self.repo)
git(self.repo, "remote", "add", "origin", self.canonical)
git(self.repo, "remote", "set-url", "--add", "--push", "origin", self.canonical)
git(self.repo, "remote", "set-url", "--add", "--push", "origin", self.mirror)
self.init = self._commit("init")
git(self.repo, "push", "-q", self.canonical, "main")
git(self.repo, "push", "-q", self.mirror, "main")
git(self.repo, "fetch", "-q", "origin")
git(self.repo, "branch", "-q", "--set-upstream-to=origin/main")
def _commit(self, name: str, cwd: str | None = None, *flags: str) -> str:
cwd = cwd or self.repo
Path(cwd, f"{name}.txt").write_text(name)
git(cwd, "add", f"{name}.txt")
git(cwd, "commit", "-q", *flags, "-m", name)
return git(cwd, "rev-parse", "HEAD")
def _push(self, rewrite: bool = False) -> str:
with redirect_stdout(io.StringIO()) as out:
sign_unpushed(self.repo)
push_branch(self.repo, [], sign=True, rewrite=rewrite)
return out.getvalue()
def _cli_push(self, cwd: str, *argv: str) -> None:
ctx = CLIContext(
config_merged={},
repositories_base_dir=os.path.join(self.root, "base"),
all_repositories=[],
binaries_dir=os.path.join(self.root, "bin"),
user_config_path=os.path.join(self.root, "config.yaml"),
)
args = create_parser("pkgmgr").parse_args(["push", "-j", "1", *argv])
prev = os.getcwd()
os.chdir(cwd)
self.addCleanup(os.chdir, prev)
with redirect_stdout(io.StringIO()), self.assertRaises(SystemExit) as cm:
maybe_handle_proxy(args, ctx)
self.assertEqual(cm.exception.code, 0)
def _tip(self, bare: str, ref: str = "refs/heads/main") -> str:
return subprocess.run(
["git", "rev-parse", "--verify", "--quiet", ref],
cwd=bare,
check=False,
capture_output=True,
text=True,
).stdout.strip()
def _assert_shipped_signed(self) -> None:
head = git(self.repo, "rev-parse", "HEAD")
self.assertEqual(self._tip(self.canonical), head)
self.assertEqual(self._tip(self.mirror), head)
signatures = git(self.repo, "log", "--format=%G?", f"{self.init}..HEAD")
self.assertEqual(set(signatures.split()), {"G"})
def test_mirror_equal_gets_the_signed_commit(self) -> None:
self._commit("feature")
self._push()
self._assert_shipped_signed()
def test_mirror_behind_is_fast_forwarded(self) -> None:
self._commit("first", None, "-S")
git(self.repo, "push", "-q", self.canonical, "main")
git(self.repo, "fetch", "-q", "origin")
self._commit("second")
self._push()
self._assert_shipped_signed()
def test_mirror_on_older_rewrite_is_replaced_after_resign(self) -> None:
unsigned = self._commit("feature")
git(self.repo, "push", "-q", self.mirror, "main")
self._push()
self._assert_shipped_signed()
self.assertNotEqual(self._tip(self.mirror), unsigned)
def test_mirror_only_commit_is_refused_and_kept(self) -> None:
other = os.path.join(self.root, "other")
git(self.root, "clone", "-q", self.mirror, other)
foreign = self._commit("foreign", other)
git(other, "push", "-q", "origin", "main")
self._commit("feature")
with self.assertRaises(PushRefusedError) as cm:
self._push()
with self.assertRaises(PushRefusedError):
self._push(rewrite=True)
self.assertIn(self.mirror, str(cm.exception))
self.assertEqual(self._tip(self.mirror), foreign)
self.assertEqual(self._tip(self.canonical), git(self.repo, "rev-parse", "HEAD"))
def test_content_rewrite_needs_the_rewrite_flag(self) -> None:
self._commit("feature")
self._push()
shipped = git(self.repo, "rev-parse", "HEAD")
Path(self.repo, "feature.txt").write_text("changed")
git(self.repo, "commit", "-q", "-a", "--amend", "--no-edit")
with self.assertRaises(PushRefusedError) as cm:
self._push()
self.assertIn(self.canonical, str(cm.exception))
self.assertEqual(self._tip(self.canonical), shipped)
self.assertEqual(self._tip(self.mirror), shipped)
self._push(rewrite=True)
self._assert_shipped_signed()
self.assertNotEqual(self._tip(self.mirror), shipped)
def test_worktree_outside_base_dir_is_pushed(self) -> None:
worktree = os.path.join(self.root, "wt")
git(self.repo, "worktree", "add", "-q", "-b", "feature", worktree)
head_before = self._commit("feature", worktree)
self._cli_push(worktree)
head = git(worktree, "rev-parse", "HEAD")
self.assertNotEqual(head, head_before)
self.assertEqual(self._tip(self.canonical, "refs/heads/feature"), head)
self.assertEqual(self._tip(self.mirror, "refs/heads/feature"), head)
self.assertEqual(
git(worktree, "rev-parse", "--abbrev-ref", "@{u}"), "origin/feature"
)
def test_no_sign_pushes_unsigned_commits_to_every_url(self) -> None:
unsigned = self._commit("feature")
self._cli_push(self.repo, "--no-sign")
self.assertEqual(git(self.repo, "rev-parse", "HEAD"), unsigned)
self.assertEqual(self._tip(self.canonical), unsigned)
self.assertEqual(self._tip(self.mirror), unsigned)
self.assertEqual(git(self.repo, "log", "-1", "--format=%G?"), "N")
def test_only_head_is_signed_and_older_unsigned_commits_stay(self) -> None:
older = self._commit("older")
self._commit("head")
out = self._push()
head = git(self.repo, "rev-parse", "HEAD")
self.assertEqual(git(self.repo, "rev-parse", "HEAD^"), older)
self.assertEqual(git(self.repo, "log", "-1", "--format=%G?", head), "G")
self.assertEqual(git(self.repo, "log", "-1", "--format=%G?", older), "N")
self.assertIn("WARN 1 older unsigned commit(s)", out)
self.assertEqual(self._tip(self.canonical), head)
self.assertEqual(self._tip(self.mirror), head)
def test_pushed_remote_branch_is_the_base_without_upstream(self) -> None:
git(self.repo, "checkout", "-q", "-b", "feature")
with patch.dict(os.environ, {"GIT_COMMITTER_DATE": "2001-01-01T00:00:00+0000"}):
pushed = self._commit("pushed", None, "-S")
git(self.repo, "push", "-q", "origin", "feature")
git(self.repo, "fetch", "-q", "origin")
self._commit("new")
upstream = subprocess.run(
["git", "rev-parse", "--abbrev-ref", "@{u}"],
cwd=self.repo,
check=False,
capture_output=True,
)
self.assertNotEqual(upstream.returncode, 0)
out = self._push()
head = git(self.repo, "rev-parse", "HEAD")
self.assertEqual(git(self.repo, "rev-parse", "HEAD^"), pushed)
self.assertEqual(git(self.repo, "log", "-1", "--format=%G?", head), "G")
self.assertNotIn("WARN", out)
self.assertEqual(self._tip(self.canonical, "refs/heads/feature"), head)
def test_repository_config_that_runs_a_program_is_refused(self) -> None:
self._commit("feature")
for key in (
"core.sshCommand",
"core.gitProxy",
"core.askPass",
"core.fsmonitor",
"core.alternateRefsCommand",
"credential.helper",
"credential.https://example.org.helper",
"gpg.program",
"gpg.ssh.program",
"gpg.ssh.defaultKeyCommand",
"protocol.ext.allow",
"hook.probe.command",
"gc.recentObjectsHook",
"http.proxy",
"http.sslVerify",
"http.sslCAInfo",
"http.sslCAPath",
"http.cookieFile",
"http.saveCookies",
"remote.origin.receivepack",
"remote.origin.uploadpack",
"remote.origin.vcs",
"remote.origin.proxy",
"url.x.insteadOf",
"url.x.pushInsteadOf",
):
for push in (self._push, lambda: push_branch(self.repo, [], sign=False)):
with self.subTest(key=key, push=push):
git(self.repo, "config", key, "false")
with self.assertRaises(PushRefusedError) as cm:
push()
git(self.repo, "config", "--unset", key)
self.assertIn(key.lower(), str(cm.exception))
self.assertEqual(self._tip(self.canonical), self.init)
self.assertEqual(self._tip(self.mirror), self.init)
def test_ordinary_repository_config_is_pushed(self) -> None:
for key, value in (
("core.hooksPath", ".husky/_"),
("filter.lfs.process", "git-lfs filter-process"),
("gpg.format", "ssh"),
("credential.useHttpPath", "true"),
("include.path", "../shared.gitconfig"),
):
git(self.repo, "config", key, value)
self._commit("feature")
self._push()
self._assert_shipped_signed()
def test_staged_gitlink_is_not_folded_into_the_signed_head(self) -> None:
self._commit("feature")
head = git(self.repo, "rev-parse", "HEAD")
git(self.repo, "config", "diff.ignoreSubmodules", "all")
git(self.repo, "update-index", "--add", "--cacheinfo", f"160000,{head},sub")
with self.assertRaises(PushRefusedError) as cm:
self._push()
self.assertIn("staged changes", str(cm.exception))
self.assertEqual(git(self.repo, "rev-parse", "HEAD"), head)
def test_unstaged_edit_stays_out_of_the_signed_head(self) -> None:
self._commit("feature")
Path(self.repo, "feature.txt").write_text("unstaged")
self._push()
self._assert_shipped_signed()
self.assertEqual(git(self.repo, "status", "--porcelain"), "M feature.txt")
def test_signing_amend_runs_no_repository_hook(self) -> None:
marker = Path(self.root, "hooked")
hook = Path(self.repo, ".git", "hooks", "post-commit")
hook.write_text(f"#!/bin/sh\ntouch {marker}\n")
hook.chmod(0o755)
self._commit("feature")
marker.unlink()
self._push()
self._assert_shipped_signed()
self.assertFalse(marker.exists())
def test_signing_amend_runs_no_config_hook(self) -> None:
marker = Path(self.root, "hooked")
git(self.repo, "config", "hook.probe.command", f"touch {marker}")
git(self.repo, "config", "hook.probe.event", "post-commit")
self._commit("feature")
hooks_fire = marker.exists()
marker.unlink(missing_ok=True)
try:
self._push()
except PushRefusedError:
pass
self.assertFalse(marker.exists(), f"config hooks fire on commit: {hooks_fire}")
def test_mirror_lease_runs_no_textconv(self) -> None:
marker = Path(self.root, "textconv")
Path(self.repo, ".gitattributes").write_text("* diff=probe\n")
git(self.repo, "add", ".gitattributes")
git(self.repo, "commit", "-q", "-S", "-m", "attributes")
git(self.repo, "push", "-q", self.mirror, "main")
git(self.repo, "config", "diff.probe.textconv", f"touch {marker};cat")
self._commit("feature")
git(self.repo, "diff", "HEAD~1", "HEAD")
self.assertTrue(marker.exists())
marker.unlink()
self._push()
self._assert_shipped_signed()
self.assertFalse(marker.exists())
def test_remote_named_like_an_option_is_pushed(self) -> None:
git(self.repo, "config", "remote.-r.url", self.canonical)
git(self.repo, "config", "remote.-r.fetch", "+refs/heads/*:refs/remotes/-r/*")
git(self.repo, "config", "branch.main.pushRemote", "-r")
git(self.repo, "fetch", "-q", "--", "-r")
self._commit("feature")
self._push()
self.assertEqual(self._tip(self.canonical), git(self.repo, "rev-parse", "HEAD"))
def test_release_refuses_unsafe_config_before_any_change(self) -> None:
head = git(self.repo, "rev-parse", "HEAD")
git(self.repo, "config", "core.sshCommand", "false")
cwd = os.getcwd()
os.chdir(self.repo)
self.addCleanup(os.chdir, cwd)
with redirect_stdout(io.StringIO()), self.assertRaises(PushRefusedError):
_release_impl(
pyproject_path=os.path.join(self.repo, "pyproject.toml"), force=True
)
self.assertEqual(git(self.repo, "rev-parse", "HEAD"), head)
self.assertEqual(git(self.repo, "status", "--porcelain"), "")
def test_release_retry_refuses_unsigned_range(self) -> None:
self._commit("release")
git(self.repo, "tag", "-a", "v1.0.0", "-m", "v1.0.0")
cwd = os.getcwd()
os.chdir(self.repo)
self.addCleanup(os.chdir, cwd)
with redirect_stdout(io.StringIO()), self.assertRaises(PushRefusedError):
retry_release(pyproject_path=os.path.join(self.repo, "pyproject.toml"))
self.assertEqual(self._tip(self.canonical), git(self.repo, "rev-parse", "@{u}"))
self.assertEqual(self._tip(self.canonical, "refs/tags/v1.0.0"), "")
if __name__ == "__main__":
unittest.main()
+237
View File
@@ -0,0 +1,237 @@
from __future__ import annotations
import io
import os
import subprocess
import tempfile
import unittest
from contextlib import redirect_stderr, redirect_stdout
from pathlib import Path
from unittest.mock import patch
from urllib.parse import quote
from pkgmgr.actions.review import ReviewError, write_review
from pkgmgr.cli.commands.review import handle_review
from pkgmgr.cli.parser import create_parser
GIT_ENV = {
**os.environ,
"GIT_AUTHOR_NAME": "t",
"GIT_AUTHOR_EMAIL": "t@example.org",
"GIT_COMMITTER_NAME": "t",
"GIT_COMMITTER_EMAIL": "t@example.org",
}
COMMIT_MESSAGE = (
'feat: <script>alert("subject")</script>\n\nBody <script>alert("body")</script>'
)
def git(cwd: str, *args: str) -> str:
return subprocess.run(
["git", "-c", "commit.gpgsign=false", *args],
cwd=cwd,
env=GIT_ENV,
check=True,
capture_output=True,
text=True,
).stdout.strip()
def write(path: str, content: bytes) -> None:
os.makedirs(os.path.dirname(path), exist_ok=True)
with open(path, "wb") as handle:
handle.write(content)
class TestReview(unittest.TestCase):
def setUp(self) -> None:
self._tmp = tempfile.TemporaryDirectory()
self.root = os.path.realpath(self._tmp.name)
self.repo = os.path.join(self.root, "my repo")
git(self.root, "init", "-b", "main", self.repo)
app = "".join(f"line {n}\n" for n in range(1, 21))
write(self._path("app.py"), app.encode())
write(self._path("notes.md"), b"# Notes\n")
git(self.repo, "add", "-A")
git(self.repo, "commit", "-m", "init")
git(self.repo, "checkout", "-b", "feature")
write(self._path("notes.md"), b"# Notes\nfeature\n")
git(self.repo, "commit", "-am", COMMIT_MESSAGE)
write(self._path("app.py"), app.replace("line 5\n", "line five\n").encode())
big = "".join(f"big {n}\n" for n in range(1, 31))
write(self._path("dir with space", "big.txt"), big.encode())
write(self._path("image.png"), b"\x89PNG\x00\x01\x02")
git(self.repo, "mv", "notes.md", "renamed.md")
git(self.repo, "add", "-A")
unstaged = app.replace("line 5\n", "line five\n")
write(
self._path("app.py"), unstaged.replace("line 15\n", "line 15!\n").encode()
)
write(
self._path("todo.txt"),
b"first\n<script>alert('file')</script>\nlast\n",
)
write(self._path("blob.bin"), b"\x00\xffdata")
self.note = os.path.join(self.root, "score.txt")
write(self.note, b"<b>score</b> 90/5\n")
def tearDown(self) -> None:
self._tmp.cleanup()
def _path(self, *parts: str) -> str:
return os.path.join(self.repo, *parts)
def _render(self, out: str, base: str | None, notes: list[str]) -> str:
path = write_review(
cwd=self.repo,
out=os.path.join(self.root, "out", out),
max_file_lines=10,
base=base,
notes=notes,
)
return Path(path).read_text(encoding="utf-8")
def _link(self, rel: str, line: int) -> str:
return f"vscode://file{quote(os.path.join(self.repo, rel))}:{line}"
def test_sections_and_counts(self) -> None:
html = self._render("page.html", None, [])
self.assertIn(
'id="staged" data-files="4" data-added="31" data-removed="1"', html
)
self.assertIn(
'id="unstaged" data-files="1" data-added="1" data-removed="1"', html
)
self.assertIn(
'id="untracked" data-files="2" data-added="3" data-removed="0"', html
)
self.assertNotIn('id="commits"', html)
self.assertIn(
'notes.md</span><span class="arrow"> &rarr; </span>renamed.md', html
)
self.assertIn('<span class="tag">binary</span>', html)
self.assertIn('data-path="blob.bin"', html)
self.assertIn("Binary file, contents not shown.", html)
self.assertIn(git(self.repo, "write-tree"), html)
self.assertIn(git(self.repo, "rev-parse", "--short", "HEAD"), html)
def test_base_adds_commits_section(self) -> None:
html = self._render("page.html", "main", [])
self.assertIn(
'id="commits" data-commits="1" data-files="1" data-added="1" '
'data-removed="0"',
html,
)
self.assertIn("t &lt;t@example.org&gt;", html)
self.assertIn(git(self.repo, "rev-parse", "--short", "feature"), html)
def test_untrusted_text_is_escaped(self) -> None:
html = self._render("page.html", "main", [self.note])
self.assertEqual(html.count("<script"), 1)
self.assertNotIn("<script>alert", html)
self.assertIn("&lt;script&gt;alert(&#39;file&#39;)&lt;/script&gt;", html)
self.assertIn("feat: &lt;script&gt;alert(&#34;subject&#34;)", html)
self.assertIn("Body &lt;script&gt;alert(&#34;body&#34;)", html)
self.assertIn("<figcaption>score.txt</figcaption>", html)
self.assertIn("&lt;b&gt;score&lt;/b&gt; 90/5", html)
def test_rendering_is_deterministic(self) -> None:
first = self._render("one.html", "main", [self.note])
second = self._render("two.html", "main", [self.note])
self.assertEqual(first.encode("utf-8"), second.encode("utf-8"))
def test_vscode_links_and_truncation(self) -> None:
html = self._render("page.html", None, [])
self.assertIn(f'class="open" href="{self._link("app.py", 5)}"', html)
self.assertIn(f'class="open" href="{self._link("app.py", 15)}"', html)
self.assertIn(f'<a href="{self._link("app.py", 5)}">5</a>', html)
self.assertIn("my%20repo/dir%20with%20space/big.txt:1", html)
self.assertIn("big 10</td>", html)
self.assertNotIn("big 11</td>", html)
self.assertIn("20 more lines not shown", html)
def test_errors_exit_two(self) -> None:
plain = os.path.join(self.root, "plain")
os.makedirs(plain)
with (
patch.dict(os.environ, {"GIT_CEILING_DIRECTORIES": self.root}),
self.assertRaisesRegex(ReviewError, "not inside a git working copy"),
):
write_review(cwd=plain, out="x.html", max_file_lines=5, base=None, notes=[])
with self.assertRaisesRegex(ReviewError, "unknown base ref"):
write_review(
cwd=self.repo,
out="x.html",
max_file_lines=5,
base="no-such-ref",
notes=[],
)
args = create_parser("test").parse_args(
["review", "--out", "x.html", "--max-file-lines", "5", "--base", "nope"]
)
stderr = io.StringIO()
with (
patch("pkgmgr.cli.commands.review.os.getcwd", return_value=self.repo),
redirect_stderr(stderr),
self.assertRaises(SystemExit) as exit_ctx,
):
handle_review(args, ctx=None)
self.assertEqual(exit_ctx.exception.code, 2)
self.assertEqual(len(stderr.getvalue().splitlines()), 1)
self.assertFalse(os.path.exists(self._path("x.html")))
def test_cli_forwards_every_option(self) -> None:
args = create_parser("test").parse_args(
[
"review",
"--out",
"page.html",
"--max-file-lines",
"50",
"--base",
"origin/main",
"--note",
"score.txt",
"--note",
"message.txt",
]
)
target = os.path.join(self.root, "page.html")
stdout = io.StringIO()
with (
patch("pkgmgr.cli.commands.review.os.getcwd", return_value=self.repo),
patch(
"pkgmgr.cli.commands.review.write_review", return_value=target
) as run_review,
redirect_stdout(stdout),
):
handle_review(args, ctx=None)
run_review.assert_called_once_with(
cwd=self.repo,
out="page.html",
max_file_lines=50,
base="origin/main",
notes=["score.txt", "message.txt"],
)
self.assertEqual(stdout.getvalue(), f"file://{target}\n")
bare = create_parser("test").parse_args(
["review", "--out", "p.html", "--max-file-lines", "3"]
)
self.assertIsNone(bare.base)
self.assertEqual(bare.note, [])
if __name__ == "__main__":
unittest.main()
@@ -0,0 +1,148 @@
from __future__ import annotations
import os
import unittest
from pkgmgr.actions.review import ReviewError
from tests.integration.test_review import git, write
from tests.integration.test_review_hardening import MODULE, ReviewRepoCase
INVISIBLE = "".join(
map(
chr,
(
*range(0x202A, 0x202F),
*range(0x2066, 0x206A),
*range(0x200B, 0x2010),
0x061C,
0xFEFF,
),
)
)
RLO, LRI, ZWSP, BOM = map(chr, (0x202E, 0x2066, 0x200B, 0xFEFF))
HIDDEN = "<li>Index entry {} is marked {}: its worktree changes are not shown.</li>"
ATTRIBUTE_BINARY = (
"{} is shown as binary because {}, not because of its content: "
"run git diff --text to see it."
)
UNSET = "a gitattributes entry unsets its diff attribute"
DRIVER = "its gitattributes diff driver {0} sets diff.{0}.binary"
def marks(text: str) -> str:
return "".join(f"&lt;U+{ord(char):04X}&gt;" for char in text)
class TestReviewConcealment(ReviewRepoCase):
def test_replace_objects_cannot_mask_the_index(self) -> None:
evil = self._object("blob", "print('evil')\n")
benign = self._object("blob", "print('staged')\n")
git(self.repo, "update-index", "--cacheinfo", f"100644,{evil},app.py")
git(self.repo, "replace", evil, benign)
git(self.repo, "update-index", "--skip-worktree", "app.py")
git(self.repo, "update-index", "--assume-unchanged", "lib.py")
write(self._path("lib.py"), b"print('hidden')\n")
html = self._render(None, [])
self.assertIn("print(&#39;evil&#39;)", html)
self.assertNotIn("print(&#39;staged&#39;)", html)
self.assertIn(HIDDEN.format("app.py", "skip-worktree"), html)
self.assertIn(HIDDEN.format("lib.py", "assume-unchanged"), html)
def test_ignored_submodule_change_is_shown(self) -> None:
sub = self._submodule(self.repo, "sub", "\tignore = all\n")
git(sub, "commit", "--allow-empty", "-m", "two")
new = git(sub, "rev-parse", "HEAD")
git(self.repo, "update-index", "--cacheinfo", f"160000,{new},sub")
gitmodules = self._render(None, [])
write(self._path(".gitmodules"), MODULE.format("sub", "").encode())
git(self.repo, "config", "diff.ignoreSubmodules", "all")
config = self._render(None, [])
for source, html in (("gitmodules", gitmodules), ("config", config)):
with self.subTest(source=source):
self.assertIn(f"Subproject commit {new}", html)
def test_attribute_binary_is_warned(self) -> None:
write(self._path(".gitattributes"), b"*.txt -diff\n")
write(self._path(".git", "info", "attributes"), b"*.md binary\n")
write(self._path("plain.txt"), b"plain text\n")
write(self._path("readme.md"), b"# readme\n")
write(self._path("image.bin"), b"\x00\x01binary\n")
git(self.repo, "add", ".gitattributes", "plain.txt", "readme.md", "image.bin")
html = self._render(None, [])
self.assertIn('data-path="image.bin"', html)
for path in ("plain.txt", "readme.md"):
self.assertIn(f"<li>{ATTRIBUTE_BINARY.format(path, UNSET)}</li>", html)
for path in ("image.bin", "new.txt"):
self.assertNotIn(f"{path} is shown as binary", html)
def test_binary_diff_driver_is_warned(self) -> None:
write(self._path(".gitattributes"), b"*.dat diff=opaque\n*.bin diff=plain\n")
git(self.repo, "config", "diff.opaque.binary", "yes")
write(self._path("data.dat"), b"text data\n")
write(self._path("image.bin"), b"\x00\x01binary\n")
git(self.repo, "add", ".gitattributes", "data.dat", "image.bin")
html = self._render(None, [])
self.assertIn('data-path="image.bin"', html)
warning = ATTRIBUTE_BINARY.format("data.dat", DRIVER.format("opaque"))
self.assertIn(f"<li>{warning}</li>", html)
self.assertNotIn("image.bin is shown as binary", html)
def test_base_resolves_the_branch_not_a_shadow(self) -> None:
head = git(self.repo, "rev-parse", "HEAD")
git(self.repo, "update-ref", "refs/tags/main", head)
tag_shadow = self._render("main", [])
write(self._path(".git", "main"), f"{head}\n".encode())
dir_shadow = self._render("main", [])
for html in (tag_shadow, dir_shadow):
self.assertIn('id="commits" data-commits="1"', html)
self.assertIn("<dt>Base</dt><dd>refs/heads/main @ ", html)
def test_base_refuses_ambiguous_and_other_refs(self) -> None:
head = git(self.repo, "rev-parse", "HEAD")
git(self.repo, "update-ref", "refs/remotes/main", head)
git(self.repo, "update-ref", "refs/tags/v1", head)
cases = {
"main": "ambiguous base ref",
"refs/tags/v1": "unknown base ref",
"HEAD~1": "unknown base ref",
head: "unknown base ref",
}
for base, message in cases.items():
with (
self.subTest(base=base),
self.assertRaisesRegex(ReviewError, message),
):
self._render(base, [])
def test_invisible_characters_are_marked(self) -> None:
tree = git(self.repo, "rev-parse", "HEAD^{tree}")
message = f"fix{RLO}: subject\n\nbody{LRI}text"
commit = git(self.repo, "commit-tree", tree, "-p", "HEAD", "-m", message)
git(self.repo, "update-ref", "HEAD", commit)
write(self._path(f"evil{RLO}txt.py"), f"code{ZWSP}line{BOM}\n".encode())
note = os.path.join(self.root, "note.txt")
write(note, f"note{INVISIBLE}\n".encode())
html = self._render("main", [note])
for char in INVISIBLE:
self.assertNotIn(char, html, f"U+{ord(char):04X}")
self.assertIn(f"fix{marks(RLO)}: subject", html)
self.assertIn(f"body{marks(LRI)}text", html)
self.assertIn(f"evil{marks(RLO)}txt.py", html)
self.assertIn(f"code{marks(ZWSP)}line{marks(BOM)}", html)
self.assertIn(f"note{marks(INVISIBLE)}", html)
if __name__ == "__main__":
unittest.main()
+229
View File
@@ -0,0 +1,229 @@
from __future__ import annotations
import os
import stat
import subprocess
import tempfile
import unittest
from pathlib import Path
from unittest.mock import patch
from pkgmgr.actions.review import ReviewError, write_review
from tests.integration.test_review import GIT_ENV, git, write
SECRET = b"SECRET-KEY-MATERIAL\n"
SIGNATURE = (
"gpgsig -----BEGIN PGP SIGNATURE-----\n \n iQEzBAABCAAdFiEE\n"
" -----END PGP SIGNATURE-----"
)
MODULE = '[submodule "{0}"]\n\tpath = {0}\n\turl = ./{0}\n{1}'
class ReviewRepoCase(unittest.TestCase):
def setUp(self) -> None:
self._tmp = tempfile.TemporaryDirectory()
self.root = os.path.realpath(self._tmp.name)
self.repo = os.path.join(self.root, "repo")
self.marker = os.path.join(self.root, "marker")
self.secret = os.path.join(self.root, "secret", "id_ed25519")
write(self.secret, SECRET)
git(self.root, "init", "-b", "main", self.repo)
write(self._path("app.py"), b"print('base')\n")
write(self._path("lib.py"), b"print('lib')\n")
git(self.repo, "add", "-A")
git(self.repo, "commit", "-m", "init")
git(self.repo, "checkout", "-b", "feature")
write(self._path("app.py"), b"print('feature')\n")
git(self.repo, "commit", "-am", "feature")
write(self._path("app.py"), b"print('staged')\n")
git(self.repo, "add", "app.py")
write(self._path("app.py"), b"print('unstaged')\n")
write(self._path("new.txt"), b"untracked\n")
def tearDown(self) -> None:
self._tmp.cleanup()
def _path(self, *parts: str) -> str:
return os.path.join(self.repo, *parts)
def _render(self, base: str | None, notes: list[str]) -> str:
path = write_review(
cwd=self.repo,
out=os.path.join(self.root, "out", "page.html"),
max_file_lines=50,
base=base,
notes=notes,
)
return Path(path).read_text(encoding="utf-8")
def _trap(self, path: str, tail: str) -> str:
write(path, f"#!/bin/sh\ntouch '{self.marker}'\n{tail}\n".encode())
os.chmod(path, 0o755)
return path
def _assert_fires(self, *args: str) -> None:
subprocess.run(
["git", *args],
cwd=self.repo,
env=GIT_ENV,
stdin=subprocess.DEVNULL,
capture_output=True,
check=False,
)
self.assertTrue(os.path.exists(self.marker), f"plain git {args[0]} fires")
os.remove(self.marker)
def _submodule(self, parent: str, name: str, extra: str = "") -> str:
path = os.path.join(parent, name)
git(parent, "init", "-b", "main", path)
write(os.path.join(path, "f.txt"), b"one\n")
git(path, "add", "-A")
git(path, "commit", "-m", "one")
head = git(path, "rev-parse", "HEAD")
write(os.path.join(parent, ".gitmodules"), MODULE.format(name, extra).encode())
git(parent, "update-index", "--add", "--cacheinfo", f"160000,{head},{name}")
git(parent, "add", ".gitmodules")
git(parent, "commit", "-m", name)
return path
def _object(self, kind: str, content: str) -> str:
return subprocess.run(
["git", "hash-object", "-t", kind, "-w", "--stdin"],
cwd=self.repo,
env=GIT_ENV,
input=content,
capture_output=True,
text=True,
check=True,
).stdout.strip()
class TestReviewHardening(ReviewRepoCase):
def test_fsmonitor_never_runs(self) -> None:
monitor = self._trap(os.path.join(self.root, "fsmonitor"), "exit 1")
git(self.repo, "config", "core.fsmonitor", monitor)
self._assert_fires("diff")
self._render("main", [])
self.assertFalse(os.path.exists(self.marker))
def test_repository_filter_never_runs(self) -> None:
write(self._path(".git", "info", "attributes"), b"* filter=trap\n")
clean = self._trap(os.path.join(self.root, "clean"), "exec cat")
git(self.repo, "config", "filter.trap.clean", clean)
self._assert_fires("diff")
html = self._render("main", [])
self.assertFalse(os.path.exists(self.marker))
self.assertIn('id="unstaged" data-files="0"', html)
self.assertIn("filter.trap.clean", html)
self.assertNotIn("print(&#39;unstaged&#39;)", html)
self.assertIn("<dd>(not computed, see the warnings)</dd>", html)
def test_submodule_filter_never_runs(self) -> None:
sub = self._submodule(self.repo, "sub")
inner = self._submodule(sub, "inner")
clean = self._trap(os.path.join(self.root, "clean"), "exec cat")
for path in (sub, inner):
write(os.path.join(path, ".git", "info", "attributes"), b"* filter=x\n")
write(os.path.join(path, "f.txt"), b"two\n")
git(path, "config", "filter.x.clean", clean)
self._assert_fires("diff")
html = self._render(None, [])
self.assertFalse(os.path.exists(self.marker))
self.assertIn('id="unstaged" data-files="0"', html)
self.assertIn("the config of submodule sub defines filter.x.clean", html)
self.assertIn("the config of submodule sub/inner defines filter.x.clean", html)
self.assertIn("<dd>(not computed, see the warnings)</dd>", html)
def test_signature_program_never_runs(self) -> None:
header, message = git(self.repo, "cat-file", "commit", "HEAD").split("\n\n", 1)
signed = self._object("commit", f"{header}\n{SIGNATURE}\n\n{message}\n")
git(self.repo, "update-ref", "HEAD", signed)
program = self._trap(os.path.join(self.root, "gpg"), "exit 1")
git(self.repo, "config", "gpg.program", program)
self._assert_fires("log", "-1", "--format=%G?")
html = self._render("main", [])
self.assertFalse(os.path.exists(self.marker))
self.assertIn('id="commits" data-commits="1"', html)
self.assertNotIn('<span class="sig', html)
def test_hooks_never_run(self) -> None:
hooks = os.path.join(self.root, "hooks")
self._trap(os.path.join(hooks, "post-index-change"), "exit 0")
git(self.repo, "config", "core.hooksPath", hooks)
self._assert_fires("update-index", "--force-write-index")
self._render("main", [])
self.assertFalse(os.path.exists(self.marker))
def test_note_symlinks_are_refused(self) -> None:
link = os.path.join(self.root, "score.txt")
os.symlink(self.secret, link)
parent = os.path.join(self.root, "notes")
os.symlink(os.path.dirname(self.secret), parent)
for note in (link, os.path.join(parent, "id_ed25519")):
with (
self.subTest(note=note),
self.assertRaisesRegex(ReviewError, "cannot read note"),
):
self._render(None, [note])
def test_untracked_parent_swap_is_not_followed(self) -> None:
write(self._path("sub", "id_ed25519"), b"harmless\n")
real_run = subprocess.run
swapped: list[list[str]] = []
def swap(cmd, *args, **kwargs):
result = real_run(cmd, *args, **kwargs)
if "--others" in cmd and not swapped:
os.rename(self._path("sub"), os.path.join(self.root, "sub-real"))
os.symlink(os.path.dirname(self.secret), self._path("sub"))
swapped.append(cmd)
return result
with patch("subprocess.run", swap):
html = self._render(None, [])
self.assertTrue(swapped)
self.assertNotIn(SECRET.decode().strip(), html)
self.assertIn("Not a regular file, contents not shown.", html)
def test_out_symlink_is_replaced_not_followed(self) -> None:
keys = b"ssh-ed25519 AAAA operator\n"
victim = os.path.join(self.root, "authorized_keys")
write(victim, keys)
missing = os.path.join(self.root, "missing_keys")
out = os.path.join(self.root, "out")
os.makedirs(out)
os.symlink(victim, os.path.join(out, "page.html"))
os.symlink(missing, os.path.join(out, "dangling.html"))
for name in ("page.html", "dangling.html"):
with self.subTest(out=name):
path = write_review(
cwd=self.repo,
out=os.path.join(out, name),
max_file_lines=50,
base=None,
notes=[],
)
info = os.lstat(path)
self.assertTrue(stat.S_ISREG(info.st_mode))
self.assertEqual(stat.S_IMODE(info.st_mode), 0o600)
self.assertEqual(Path(victim).read_bytes(), keys)
self.assertFalse(os.path.lexists(missing))
self.assertEqual(sorted(os.listdir(out)), ["dangling.html", "page.html"])
if __name__ == "__main__":
unittest.main()
+775
View File
@@ -0,0 +1,775 @@
from __future__ import annotations
import io
import json
import os
import shlex
import shutil
import sys
import time
import unittest
from contextlib import redirect_stderr, redirect_stdout
from types import SimpleNamespace
from unittest.mock import patch
import yaml
from pkgmgr.actions.branch import gc
from pkgmgr.actions.worktree import hooks, orphans, up
from pkgmgr.cli.commands.worktree import handle_gc, handle_worktree
from pkgmgr.cli.context import CLIContext
from pkgmgr.cli.dispatch import dispatch_command
from pkgmgr.cli.parser import create_parser
from pkgmgr.core.config.load import load_config
from tests.integration.test_branch_gc import GcFixture, git
class WorktreeFixture(GcFixture):
def _pushed_worktree(self, name: str, makefile: str | None) -> str:
git(self.repo, "checkout", "-b", name, "main")
if makefile is None:
self._commit(name)
else:
with open(os.path.join(self.repo, "Makefile"), "w") as fh:
fh.write(makefile)
git(self.repo, "add", "Makefile")
git(self.repo, "commit", "-m", f"{name} makefile")
git(self.repo, "push", "origin", name)
git(self.repo, "checkout", "main")
path = os.path.join(self.root, f"wt-{name}")
git(self.repo, "worktree", "add", path, name)
return path
def _execute(self, plan: gc.Repo, item: gc.Item) -> None:
with patch.object(gc, "_process_cwds", return_value={}):
gc.execute(plan, item)
def _pinned(self, hook: str, argv: list[str]) -> dict:
return {os.path.realpath(self.repo): {hook: argv}}
class TestCleanupScopes(WorktreeFixture, unittest.TestCase):
def test_worktree_scope_removes_pushed_worktree_and_keeps_branch(self) -> None:
path = self._pushed_worktree("pushed", None)
plan = self._plan(host_ns=True, cwds={}, scope="worktree")
items = self._items(plan)
self.assertNotIn("squashed", items)
item = items["pushed"]
self.assertEqual(item.state, "unmerged")
self.assertEqual(item.unpushed, 0)
self.assertEqual(item.actions, ["worktree"])
self.assertEqual(gc._label(item), "REMOVE")
self._execute(plan, item)
self.assertEqual(item.result, "done")
self.assertFalse(os.path.exists(path))
self.assertNotEqual(git(self.repo, "branch", "--list", "pushed"), "")
self.assertNotEqual(git(self.origin, "branch", "--list", "pushed"), "")
def test_worktree_scope_keeps_unpushed_worktree(self) -> None:
path = self._pushed_worktree("ahead", None)
with open(os.path.join(path, "more.txt"), "w") as fh:
fh.write("more")
git(path, "add", "more.txt")
git(path, "commit", "-m", "more")
item = self._items(self._plan(host_ns=True, cwds={}, scope="worktree"))["ahead"]
self.assertEqual(item.unpushed, 1)
self.assertEqual(item.actions, [])
self.assertFalse(item.eligible)
def test_branch_scope_skips_checked_out_and_takes_free_branches(self) -> None:
items = self._items(self._plan(host_ns=True, cwds={}, scope="branch"))
self.assertTrue(items["merged"].skip.startswith("checked out in worktree"))
self.assertEqual(items["merged"].actions, [])
self.assertEqual(gc._label(items["merged"]), "SKIP")
self.assertEqual(
items["squashed"].actions, ["local-branch", "remote-branch:origin"]
)
self.assertTrue(items["squashed"].eligible)
def test_patch_equivalent_branch_counts_as_merged(self) -> None:
git(self.repo, "checkout", "-b", "picked", "main")
self._commit("picked")
git(self.repo, "push", "origin", "picked")
sha = git(self.repo, "rev-parse", "HEAD")
git(self.repo, "checkout", "main")
self._commit("main-moved")
git(self.repo, "cherry-pick", sha)
git(self.repo, "push", "origin", "main")
item = self._items(self._plan(host_ns=True, cwds={}))["picked"]
self.assertEqual(item.merged_by, "patch")
self.assertEqual(item.state, "merged-clean")
def test_dotdot_behind_a_symlink_never_reaches_another_checkout(self) -> None:
victim = os.path.join(self.root, "victim")
os.makedirs(victim)
git(victim, "init", "--quiet", "-b", "main")
git(victim, "commit", "--quiet", "--allow-empty", "-m", "unpushed work")
git(self.repo, "branch", "evil", "main")
hop = os.path.join(self.repo, "A")
os.makedirs(os.path.join(hop, "victim"))
os.symlink(victim, os.path.join(hop, "L"))
meta = os.path.join(self.repo, ".git", "worktrees", "evil")
os.makedirs(meta)
files = {
os.path.join(hop, "victim", ".git"): f"gitdir: {meta}\n",
os.path.join(meta, "gitdir"): os.path.join(
hop, "L", "..", "victim", ".git"
),
os.path.join(meta, "HEAD"): "ref: refs/heads/evil\n",
os.path.join(meta, "commondir"): "../..\n",
}
for name, text in files.items():
with open(name, "w") as fh:
fh.write(text + ("" if text.endswith("\n") else "\n"))
plan = self._plan(host_ns=True, cwds={}, scope="worktree")
item = self._items(plan)["evil"]
self._execute(plan, item)
self.assertNotIn("worktree", item.actions)
self.assertTrue(os.path.isdir(os.path.join(victim, ".git")))
def test_pinned_fini_hook_runs_before_removal(self) -> None:
marker = os.path.join(self.root, "fini-ran")
path = self._pushed_worktree("hooked", None)
pins = self._fini_pins(["touch", marker])
plan = self._plan(host_ns=True, cwds={}, scope="worktree", hooks=pins)
item = self._items(plan)["hooked"]
self.assertEqual(item.actions, ["hook:worktree-fini", "worktree"])
self._execute(plan, item)
self.assertEqual(item.result, "done")
self.assertTrue(os.path.exists(marker))
self.assertFalse(os.path.exists(path))
def _fini_pins(self, argv: list[str]) -> dict:
return self._pinned("worktree-fini", argv)
def _cleanup_merged_forged_to(self, target: str) -> dict:
meta = os.path.join(self.repo, ".git", "worktrees", "wt-merged")
with open(os.path.join(meta, "gitdir"), "w") as fh:
fh.write(os.path.join(target, ".git") + "\n")
pins = self._fini_pins(["touch", "{path}/fini-ran"])
out = io.StringIO()
with (
patch.object(gc, "host_pid_namespace", return_value=True),
patch.object(gc, "list_compose_containers", return_value=[]),
patch.object(gc, "fetch_forge_facts", return_value=self.forge),
patch.object(gc, "_process_cwds", return_value={}),
redirect_stdout(out),
redirect_stderr(io.StringIO()),
):
gc.cleanup(
[self.repo],
scope="worktree",
apply=True,
as_json=True,
include=["merged"],
exclude=[],
merged_into=[],
sizes=False,
min_age=None,
hooks=pins,
)
return json.loads(out.getvalue())["repos"][0]["items"][0]
def test_forged_gitdir_runs_nothing_in_another_checkout(self) -> None:
other = os.path.join(self.root, "other")
git(self.root, "clone", "--quiet", self.repo, other)
item = self._cleanup_merged_forged_to(other)
self.assertFalse(os.path.exists(os.path.join(other, "fini-ran")))
self.assertEqual(item["worktree"], other)
self.assertEqual(item["skip"], gc.NOT_LINKED)
self.assertEqual(item["actions"], [])
self.assertTrue(os.path.isdir(os.path.join(other, ".git")))
def test_gitdir_forged_to_a_sibling_worktree_runs_nothing_there(self) -> None:
sibling = self._pushed_worktree("sibling", None)
item = self._cleanup_merged_forged_to(sibling)
self.assertFalse(os.path.exists(os.path.join(sibling, "fini-ran")))
self.assertEqual(item["worktree"], sibling)
self.assertEqual(item["skip"], gc.NOT_LINKED)
self.assertEqual(item["actions"], [])
self.assertTrue(os.path.isdir(sibling))
def test_fini_hook_commit_keeps_worktree_and_branch(self) -> None:
commit = [
"git",
"-c",
"user.name=t",
"-c",
"user.email=t@example.org",
"-c",
"commit.gpgsign=false",
"commit",
"--quiet",
"--allow-empty",
"-m",
"hook",
]
plan = self._plan(host_ns=True, cwds={}, hooks=self._fini_pins(commit))
item = self._items(plan)["merged"]
self.assertIn("local-branch", item.actions)
self._execute(plan, item)
self.assertEqual(
item.result, "skipped: changed during worktree-fini (HEAD moved)"
)
self.assertTrue(os.path.isdir(self.merged_wt))
self.assertNotEqual(git(self.repo, "rev-parse", "merged"), item.local_sha)
def test_fini_hook_timeout_kills_its_process_group(self) -> None:
self._pushed_worktree("spawner", None)
marker = os.path.join(self.root, "grandchild-ran")
script = f"(sleep 1; touch {shlex.quote(marker)}) & sleep 30"
pins = self._fini_pins(["sh", "-c", script])
plan = self._plan(host_ns=True, cwds={}, scope="worktree", hooks=pins)
item = self._items(plan)["spawner"]
with patch.object(hooks, "HOOK_TIMEOUT", 0.3):
self._execute(plan, item)
time.sleep(1.5)
self.assertEqual(item.result, "failed: worktree-fini timed out after 0.3 s")
self.assertFalse(os.path.exists(marker))
def test_fini_hook_timeout_fails_without_removal(self) -> None:
path = self._pushed_worktree("slow", None)
sleep = [sys.executable, "-c", "import time; time.sleep(30)"]
pins = self._fini_pins(sleep)
plan = self._plan(host_ns=True, cwds={}, scope="worktree", hooks=pins)
item = self._items(plan)["slow"]
with patch.object(hooks, "HOOK_TIMEOUT", 0.5):
self._execute(plan, item)
self.assertEqual(item.result, "failed: worktree-fini timed out after 0.5 s")
self.assertTrue(os.path.isdir(path))
def test_fini_hook_output_keeps_the_worktree(self) -> None:
path = self._pushed_worktree("dumps", None)
pins = self._fini_pins(["touch", "{path}/dump.sql"])
plan = self._plan(host_ns=True, cwds={}, scope="worktree", hooks=pins)
item = self._items(plan)["dumps"]
self._execute(plan, item)
self.assertEqual(
item.result, "skipped: changed during worktree-fini (dirty: 1 paths)"
)
self.assertTrue(os.path.isfile(os.path.join(path, "dump.sql")))
@unittest.skipUnless(shutil.which("make"), "make is not available")
def test_makefile_hooks_never_run_unpinned(self) -> None:
marker = os.path.join(self.root, "fini-ran")
planted = f"worktree-fini:\n\ttouch {marker}\n"
path = self._pushed_worktree("planted", makefile=planted)
with open(os.path.join(self.repo, ".git", "info", "exclude"), "a") as fh:
fh.write("GNUmakefile\n")
with open(os.path.join(path, "GNUmakefile"), "w") as fh:
fh.write(planted)
plan = self._plan(host_ns=True, cwds={}, scope="worktree")
item = self._items(plan)["planted"]
self.assertEqual(item.actions, ["worktree"])
self._execute(plan, item)
self.assertEqual(item.result, "done")
self.assertFalse(os.path.exists(marker))
def test_activity_is_recorded_on_request(self) -> None:
quiet = self._items(self._plan(host_ns=True, cwds={}))
loud = self._items(self._plan(host_ns=True, cwds={}, activity=True))
self.assertIsNone(quiet["merged"].last_activity)
self.assertLessEqual(loud["merged"].last_activity, time.time())
class TestWorktreeUp(WorktreeFixture, unittest.TestCase):
def setUp(self) -> None:
super().setUp()
self.worktrees = os.path.join(self.root, "wts")
def _up(
self, branch: str, path: str | None, hooks: dict | None = None
) -> tuple[int, dict]:
out = io.StringIO()
with redirect_stdout(out):
code = up.worktree_up(
self.repo,
branch,
base=None,
worktrees_dir=self.worktrees,
path=path,
hooks=hooks or {},
as_json=True,
pull=None,
push=None,
)
return code, json.loads(out.getvalue()) if code != 2 else {}
def _commit_file(self, name: str, text: str) -> None:
with open(os.path.join(self.repo, name), "w") as fh:
fh.write(text)
git(self.repo, "add", name)
git(self.repo, "commit", "-m", name)
git(self.repo, "push", "origin", "main")
def _commit_script(self, name: str) -> str:
marker = os.path.join(self.root, f"{name}-ran")
script = os.path.join(self.repo, name)
with open(script, "w") as fh:
fh.write(f"#!/bin/sh\ntouch {marker}\n")
os.chmod(script, 0o755)
git(self.repo, "add", name)
git(self.repo, "commit", "-m", name)
git(self.repo, "push", "origin", "main")
return marker
def _pins(self, argv: list[str]) -> dict:
return self._pinned("worktree-init", argv)
def test_slot_in_branch_name_stays_literal(self) -> None:
write_path = "import sys; open('path.txt', 'w').write(sys.argv[1])"
pins = self._pins([sys.executable, "-c", write_path, "{path}"])
code, result = self._up("feature/x{slot}", None, pins)
self.assertEqual(code, 0)
self.assertTrue(result["path"].endswith("/feature/x{slot}"))
with open(os.path.join(result["path"], "path.txt")) as fh:
self.assertEqual(fh.read(), result["path"])
@unittest.skipUnless(shutil.which("make"), "make is not available")
def test_empty_path_entry_never_runs_a_planted_command(self) -> None:
marker = self._commit_script("make")
search = os.pathsep + os.environ["PATH"]
with patch.dict(os.environ, {"PATH": search}):
code, result = self._up("feature/p", None, self._pins(["make", "-v"]))
self.assertEqual(code, 0)
self.assertEqual(result["hook"], "worktree-init")
self.assertFalse(os.path.exists(marker))
def test_bare_pin_skips_a_path_entry_another_user_can_write(self) -> None:
planted = os.path.join(self.root, "planted-bin")
os.makedirs(planted)
marker = os.path.join(self.root, "planted-ran")
fake = os.path.join(planted, "true")
with open(fake, "w") as fh:
fh.write(f"#!/bin/sh\ntouch {marker}\n")
os.chmod(fake, 0o755)
if os.getuid() == 0:
os.chown(planted, 65534, 65534)
os.chown(fake, 65534, 65534)
search = planted + os.pathsep + os.environ["PATH"]
with patch.dict(os.environ, {"PATH": search}):
code, result = self._up("feature/t", None, self._pins(["true"]))
self.assertEqual((code, result["hook"]), (0, "worktree-init"))
self.assertFalse(os.path.exists(marker))
def test_relative_pin_is_refused_before_checkout(self) -> None:
marker = self._commit_script("init.sh")
code, _ = self._up("feature/r", None, self._pins(["./init.sh"]))
self.assertEqual(code, 2)
self.assertFalse(os.path.exists(marker))
self.assertFalse(os.path.exists(self.worktrees))
def test_init_hook_timeout_fails_and_keeps_worktree(self) -> None:
pins = self._pins([sys.executable, "-c", "import time; time.sleep(30)"])
with patch.object(hooks, "HOOK_TIMEOUT", 0.5):
code, result = self._up("feature/t", None, pins)
self.assertEqual(code, 1)
self.assertEqual(
result["hook"],
"worktree-init failed (worktree-init timed out after 0.5 s)",
)
self.assertTrue(os.path.isdir(result["path"]))
def test_location_of_remote_urls(self) -> None:
cases = {
"git@github.com:owner/repo.git": "github.com/owner/repo",
"https://github.com/owner/repo": "github.com/owner/repo",
"ssh://git@host:2201/owner/repo.git": "host/owner/repo",
"/tmp/x/origin.git": "tmp/x/origin",
}
for url, expected in cases.items():
with self.subTest(url=url):
self.assertEqual(up.worktree_location(url), expected)
def test_new_branches_get_path_slots_and_pinned_init_hook(self) -> None:
write_slot = "import sys; open('slot.txt', 'w').write(sys.argv[1])"
hooks = self._pins([sys.executable, "-c", write_slot, "{slot}"])
code, first = self._up("feature/x", None, hooks)
expected = os.path.join(
self.worktrees, up.worktree_location(self.origin), "feature/x"
)
self.assertEqual(code, 0)
self.assertEqual(first["path"], expected)
self.assertEqual(first["slot"], 1)
self.assertEqual(first["hook"], "worktree-init")
self.assertTrue(first["origin"].startswith("new from origin/main"))
with open(os.path.join(expected, "slot.txt")) as fh:
self.assertEqual(fh.read().strip(), "1")
_, second = self._up("feature/y", None, hooks)
self.assertEqual(second["slot"], 2)
self.assertEqual(self._up("feature/x", None, hooks)[0], 2)
@unittest.skipUnless(shutil.which("make"), "make is not available")
def test_unpinned_makefile_is_never_run_or_probed(self) -> None:
parsed = os.path.join(self.root, "parsed")
ran = os.path.join(self.root, "ran")
self._commit_file(
"Makefile", f"X := $(shell touch {parsed})\nworktree-init:\n\ttouch {ran}\n"
)
code, result = self._up("feature/m", None)
self.assertEqual(code, 0)
self.assertEqual(result["hook"], "not pinned: worktree-init in Makefile")
self.assertFalse(os.path.exists(parsed))
self.assertFalse(os.path.exists(ran))
def test_checkout_runs_no_config_hook(self) -> None:
marker = os.path.join(self.root, "hooked")
git(self.repo, "config", "hook.probe.command", f"touch {marker}")
git(self.repo, "config", "hook.probe.event", "post-checkout")
code, _ = self._up("feature/h", None)
self.assertEqual(code, 0)
self.assertFalse(os.path.exists(marker))
def test_origin_url_cannot_escape_the_worktrees_dir(self) -> None:
git(self.repo, "remote", "set-url", "origin", "https://host/../../escape")
self.assertEqual(self._up("feature/e", None)[0], 2)
self.assertFalse(os.path.exists(os.path.join(self.root, "escape")))
def test_default_base_named_like_an_option_is_a_ref(self) -> None:
git(self.repo, "checkout", "-b", "side", "main")
self._commit("side")
sha = git(self.repo, "rev-parse", "HEAD")
git(self.repo, "checkout", "main")
git(self.repo, "update-ref", "refs/heads/-f", sha)
git(self.repo, "symbolic-ref", "refs/remotes/origin/HEAD", "refs/heads/-f")
code, result = self._up("feature/o", None)
self.assertEqual(code, 0)
self.assertEqual(git(result["path"], "rev-parse", "HEAD"), sha)
def test_local_filter_config_blocks_the_checkout(self) -> None:
marker = os.path.join(self.root, "smudged")
self._commit_file(".gitattributes", "* filter=evil\n")
git(self.repo, "config", "filter.evil.smudge", f"touch {marker}; cat")
self.assertEqual(self._up("feature/s", None)[0], 2)
self.assertFalse(os.path.exists(marker))
def test_local_ssh_command_skips_the_fetch(self) -> None:
marker = os.path.join(self.root, "ssh-ran")
git(self.repo, "config", "core.sshCommand", f"touch {marker}; false")
git(self.repo, "config", "remote.origin.pushurl", self.origin)
git(self.repo, "remote", "set-url", "origin", "ssh://example.invalid/r.git")
code, result = self._up("feature/f", None)
self.assertEqual(code, 0)
self.assertFalse(os.path.exists(marker))
self.assertIn("core.sshcommand", " ".join(result["warnings"]))
def test_remote_branch_is_tracked(self) -> None:
git(self.repo, "checkout", "-b", "remote-only", "main")
self._commit("remote-only")
git(self.repo, "push", "origin", "remote-only")
git(self.repo, "checkout", "main")
git(self.repo, "branch", "-D", "remote-only")
code, result = self._up("remote-only", None)
self.assertEqual(code, 0)
self.assertEqual(result["origin"], "tracking origin/remote-only")
self.assertEqual(result["hook"], "none")
self.assertEqual(
git(result["path"], "rev-parse", "--abbrev-ref", "@{u}"),
"origin/remote-only",
)
def test_explicit_path_replaces_standard_path(self) -> None:
released = os.path.join(self.root, "released", "feature-z")
code, result = self._up("feature/z", released)
self.assertEqual(code, 0)
self.assertEqual(result["path"], released)
self.assertEqual(
git(released, "rev-parse", "--abbrev-ref", "HEAD"), "feature/z"
)
self.assertFalse(os.path.exists(self.worktrees))
def test_invalid_branch_is_refused(self) -> None:
self.assertEqual(self._up("../evil", None)[0], 2)
self.assertFalse(os.path.exists(self.worktrees))
class TestWorktreeOrphans(WorktreeFixture, unittest.TestCase):
def _report(self, cwds: dict[int, str]) -> dict[str, dict]:
out = io.StringIO()
with (
patch.object(gc, "host_pid_namespace", return_value=True),
patch.object(gc, "list_compose_containers", return_value=[]),
patch.object(gc, "fetch_forge_facts", return_value=self.forge),
patch.object(gc, "_process_cwds", return_value=cwds),
redirect_stdout(out),
):
code = orphans.worktree_orphans(
[self.repo],
stale_days=0,
top=5,
as_json=True,
include=[],
exclude=[],
merged_into=[],
sizes=False,
)
self.assertEqual(code, 0)
rows = json.loads(out.getvalue())["repos"][0]["rows"]
return {row["branch"]: row for row in rows if row["branch"]}
def test_classes_and_proposals(self) -> None:
git(self.repo, "checkout", "-b", "lonely", "main")
self._commit("lonely")
git(self.repo, "checkout", "main")
time.sleep(1.1)
rows = self._report({})
self.assertEqual(rows["main"]["class"], "active")
self.assertEqual(rows["merged"]["class"], "merged-clean")
self.assertEqual(rows["merged"]["proposal"], "pkgmgr gc")
self.assertEqual(rows["dirty"]["class"], "merged-dirty")
self.assertEqual(rows["lonely"]["class"], "stale-open")
self.assertEqual(rows["lonely"]["unpushed"], 1)
self.assertEqual(rows["wip"]["class"], "active")
def test_busy_worktree_is_kept(self) -> None:
rows = self._report({4242: self.merged_wt})
self.assertEqual(rows["merged"]["class"], "merged-clean")
self.assertTrue(
rows["merged"]["proposal"].startswith("keep (process: pid 4242")
)
class TestLoadedConfigCli(WorktreeFixture, unittest.TestCase):
HOOKS = ("worktree-init", "worktree-fini")
def setUp(self) -> None:
super().setUp()
self.addCleanup(os.chdir, os.getcwd())
self.markers = {h: os.path.join(self.root, f"{h}-ran") for h in self.HOOKS}
def _ctx(self, directory: str, pinned: bool) -> CLIContext:
location = up.worktree_location(self.origin)
provider, _, rest = location.partition("/")
account, _, repository = rest.rpartition("/")
config: dict = {
"directories": {"worktrees": os.path.join(self.root, "wts")},
"repositories": [
{
"provider": provider,
"account": account,
"repository": repository,
"directory": directory,
}
],
}
if pinned:
config["worktree_hooks"] = {
location: {h: ["touch", m] for h, m in self.markers.items()}
}
config_dir = os.path.join(self.root, "config")
os.makedirs(config_dir, exist_ok=True)
user_config = os.path.join(config_dir, "config.yaml")
with open(user_config, "w") as fh:
yaml.safe_dump(config, fh)
merged = load_config(user_config)
return CLIContext(
config_merged=merged,
repositories_base_dir=os.path.join(self.root, "repos"),
all_repositories=merged["repositories"],
binaries_dir=os.path.join(self.root, "bin"),
user_config_path=user_config,
)
def _run(self, argv: list[str], ctx: CLIContext) -> tuple[int, str]:
os.chdir(self.repo)
out = io.StringIO()
with (
patch.object(gc, "host_pid_namespace", return_value=True),
patch.object(gc, "list_compose_containers", return_value=[]),
patch.object(gc, "fetch_forge_facts", return_value=self.forge),
patch.object(gc, "_process_cwds", return_value={}),
patch.object(gc, "_docker", return_value=""),
patch("pkgmgr.cli.commands.branch.refresh_tree"),
patch("pkgmgr.cli.commands.worktree.refresh_tree"),
redirect_stdout(out),
self.assertRaises(SystemExit) as exit_ctx,
):
dispatch_command(create_parser("test").parse_args(argv), ctx)
return exit_ctx.exception.code, out.getvalue()
def test_gc_dry_run_with_config_lacking_worktree_hooks(self) -> None:
code, out = self._run(["gc", "--json"], self._ctx(self.repo, pinned=False))
self.assertEqual(code, 0)
self.assertEqual(json.loads(out)["repos"][0]["repo"], self.repo)
def test_origin_url_of_a_managed_repository_selects_no_pin(self) -> None:
managed = os.path.join(self.root, "managed")
git(self.root, "clone", "--quiet", self.repo, managed)
ctx = self._ctx(managed, pinned=True)
code, out = self._run(["worktree", "up", "feature/a", "--json"], ctx)
self.assertEqual((code, json.loads(out)["hook"]), (0, "none"))
self.assertEqual(self._run(["gc", "--apply", "--branch", "merged"], ctx)[0], 0)
self.assertFalse(os.path.exists(self.markers["worktree-init"]))
self.assertFalse(os.path.exists(self.markers["worktree-fini"]))
self.assertFalse(os.path.exists(self.merged_wt))
def test_pinned_directory_borrowing_another_git_dir_runs_no_pin(self) -> None:
borrower = os.path.join(self.root, "borrower")
git(self.root, "init", "--quiet", "-b", "main", borrower)
with open(os.path.join(borrower, ".git", "commondir"), "w") as fh:
fh.write(os.path.join(self.repo, ".git") + "\n")
ctx = self._ctx(borrower, pinned=True)
code, out = self._run(
["gc", "--all", "--apply", "--json", "--branch", "merged"], ctx
)
warnings = json.loads(out)["repos"][0]["warnings"]
self.assertEqual(code, 0)
self.assertTrue(any("pinned hooks skipped" in w for w in warnings))
self.assertFalse(os.path.exists(self.markers["worktree-fini"]))
def test_managed_repository_runs_its_pins(self) -> None:
ctx = self._ctx(self.repo, pinned=True)
code, out = self._run(["worktree", "up", "feature/b", "--json"], ctx)
self.assertEqual((code, json.loads(out)["hook"]), (0, "worktree-init"))
self.assertEqual(self._run(["gc", "--apply", "--branch", "merged"], ctx)[0], 0)
self.assertTrue(os.path.exists(self.markers["worktree-init"]))
self.assertTrue(os.path.exists(self.markers["worktree-fini"]))
self.assertFalse(os.path.exists(self.merged_wt))
class TestWorktreeCli(unittest.TestCase):
def test_branch_gc_is_gone(self) -> None:
with self.assertRaises(SystemExit):
create_parser("test").parse_args(["branch", "gc"])
def test_up_forwards_to_worktree_up(self) -> None:
args = create_parser("test").parse_args(
[
"worktree",
"up",
"feature/x",
"--base",
"origin/main",
"--path",
"/srv/wt/x",
"--json",
]
)
pins = {"worktree-init": ["true"]}
ctx = SimpleNamespace(
repositories_base_dir="/srv",
config_merged={
"directories": {"worktrees": "~/wt"},
"repositories": [
{"provider": "host", "account": "o", "repository": "r"}
],
"worktree_hooks": {"host/o/r": pins, "host/o/unmanaged": pins},
},
)
with (
patch("pkgmgr.cli.commands.worktree.worktree_up", return_value=0) as run_up,
patch("pkgmgr.cli.commands.worktree.refresh_tree") as refresh,
self.assertRaises(SystemExit) as exit_ctx,
):
handle_worktree(args, ctx)
self.assertEqual(exit_ctx.exception.code, 0)
run_up.assert_called_once_with(
".",
"feature/x",
base="origin/main",
worktrees_dir="~/wt",
path="/srv/wt/x",
hooks={os.path.realpath("/srv/host/o/r"): pins},
as_json=True,
pull=None,
push=None,
)
refresh.assert_called_once()
def test_cleanup_and_gc_pick_their_scope(self) -> None:
for argv, handler, scope in (
(["worktree", "cleanup", "--apply"], handle_worktree, "worktree"),
(["gc", "--apply"], handle_gc, "all"),
):
with self.subTest(scope=scope):
args = create_parser("test").parse_args(argv)
ctx = SimpleNamespace(
config_merged={"worktree_hooks": {}, "repositories": []}
)
with (
patch(
"pkgmgr.cli.commands.branch.main_repo_root", return_value="/r"
),
patch(
"pkgmgr.cli.commands.branch.cleanup", return_value=0
) as run_cleanup,
patch("pkgmgr.cli.commands.branch.refresh_tree") as refresh,
self.assertRaises(SystemExit),
):
handler(args, ctx)
self.assertEqual(run_cleanup.call_args.kwargs["scope"], scope)
self.assertTrue(run_cleanup.call_args.kwargs["apply"])
self.assertEqual(run_cleanup.call_args.kwargs["hooks"], {})
refresh.assert_called_once_with(["/r"])
def test_orphans_requires_explicit_limits(self) -> None:
with self.assertRaises(SystemExit):
create_parser("test").parse_args(["worktree", "orphans"])
args = create_parser("test").parse_args(
["worktree", "orphans", "--stale-days", "7", "--top", "5"]
)
self.assertEqual((args.stale_days, args.top), (7, 5))
if __name__ == "__main__":
unittest.main()
+247
View File
@@ -0,0 +1,247 @@
from __future__ import annotations
import io
import json
import os
import unittest
from contextlib import redirect_stderr, redirect_stdout
from types import SimpleNamespace
from unittest.mock import patch
import yaml
from pkgmgr.actions.worktree import up
from pkgmgr.cli.commands.worktree import handle_worktree
from pkgmgr.cli.parser import create_parser
from tests.integration.test_branch_gc import GcFixture, git
PULL = ["origin/main", "store-i18n"]
PUSH = ["feature/iam-profile-rbac", "production"]
class FlowFixture(GcFixture):
def setUp(self) -> None:
super().setUp()
self.linked_git_dir = os.path.join(self.repo, ".git", "worktrees", "wt-merged")
self.linked_file = os.path.join(self.linked_git_dir, "pkgmgr", "flow.yml")
self.addCleanup(os.chdir, os.getcwd())
def _cli(self, *argv: str) -> tuple[int, str, str]:
out, err = io.StringIO(), io.StringIO()
args = create_parser("test").parse_args(["worktree", *argv])
with (
patch("pkgmgr.cli.commands.worktree.refresh_tree") as self.refresh,
redirect_stdout(out),
redirect_stderr(err),
self.assertRaises(SystemExit) as done,
):
handle_worktree(args, SimpleNamespace())
return done.exception.code, out.getvalue(), err.getvalue()
def _json(self, *argv: str) -> dict:
code, out, err = self._cli(*argv, "--json")
self.assertEqual(code, 0, err)
return json.loads(out)
class TestFlowCli(FlowFixture, unittest.TestCase):
def test_set_show_clear_json_on_a_linked_worktree(self) -> None:
argv = [f"--pull={ref}" for ref in PULL] + [f"--push={b}" for b in PUSH]
expected = {
"worktree": self.merged_wt,
"git_dir": self.linked_git_dir,
"branch": "merged",
"flow_branch": "merged",
"pull": PULL,
"push": PUSH,
"source": "file",
"warnings": [],
}
self.assertEqual(self._json("flow", "set", self.merged_wt, *argv), expected)
self.refresh.assert_called_once_with([self.repo])
self.assertEqual(self._json("flow", self.merged_wt), expected)
self.assertTrue(os.path.isfile(self.linked_file))
self.assertEqual(git(self.merged_wt, "status", "--porcelain"), "")
cleared = self._json("flow", "clear", self.merged_wt)
self.assertEqual(
{**expected, "flow_branch": None, "pull": [], "push": [], "source": "none"},
cleared,
)
self.assertFalse(os.path.exists(self.linked_file))
def test_set_show_clear_text_on_the_main_worktree(self) -> None:
os.chdir(self.repo)
code, out, _ = self._cli("flow", "set", "--pull", "origin/main", "--no-push")
self.assertEqual((code, out), (0, "pull: origin/main\npush: (none)\n"))
self.assertEqual(
self._cli("flow")[:2], (0, "pull: origin/main\npush: (none)\n")
)
shown = self._json("flow")
self.assertEqual(
(shown["worktree"], shown["git_dir"], shown["branch"]),
(self.repo, os.path.join(self.repo, ".git"), "main"),
)
self.assertTrue(
os.path.isfile(os.path.join(self.repo, ".git", "pkgmgr", "flow.yml"))
)
self.assertEqual(
self._cli("flow", "clear")[:2], (0, "pull: (none)\npush: (none)\n")
)
def test_show_without_a_flow_succeeds_empty(self) -> None:
shown = self._json("flow", self.merged_wt)
self.assertEqual(
(shown["source"], shown["flow_branch"], shown["pull"], shown["push"]),
("none", None, [], []),
)
self.assertEqual(shown["warnings"], [])
def test_invalid_names_exit_2_and_write_nothing(self) -> None:
escape = os.path.join(self.root, "escape")
for bad in ("-x", "a..b", f"--output={escape}"):
for flag, other in (("--pull", "--push"), ("--push", "--pull")):
with self.subTest(bad=bad, flag=flag):
code, out, err = self._cli(
"flow", "set", self.merged_wt, f"{flag}={bad}", f"{other}=main"
)
self.assertEqual((code, out), (2, ""))
self.assertIn("is not a valid branch name", err)
self.assertFalse(os.path.exists(os.path.join(self.linked_git_dir, "pkgmgr")))
self.assertFalse(os.path.exists(escape))
def test_usage_errors_exit_2_and_write_nothing(self) -> None:
cases = (
("flow", "set", self.merged_wt, "--pull", "main"),
("flow", "set", self.merged_wt, "--pull", "main", "--no-pull", "--no-push"),
("flow", self.merged_wt, "--push", "main"),
("flow", "clear", self.merged_wt, "--no-pull"),
("flow", "show", self.merged_wt),
)
for argv in cases:
with self.subTest(argv=argv[1:]):
self.assertEqual(self._cli(*argv)[:2], (2, ""))
self.assertFalse(os.path.exists(os.path.join(self.linked_git_dir, "pkgmgr")))
def test_a_path_that_is_no_worktree_exits_2(self) -> None:
absent = os.path.join(self.root, "absent")
for path in (self.root, self.origin, absent):
for argv in (
("flow", path),
("flow", "set", path, "--no-pull", "--no-push"),
):
with self.subTest(argv=argv):
code, out, err = self._cli(*argv)
self.assertEqual((code, out), (2, ""))
self.assertIn("is not a git worktree", err)
self.assertFalse(os.path.exists(os.path.join(self.origin, "pkgmgr")))
def test_another_branch_warns_of_the_mismatch(self) -> None:
self._json("flow", "set", self.merged_wt, "--pull", "main", "--push", "main")
git(self.merged_wt, "checkout", "--quiet", "-b", "renamed")
shown = self._json("flow", self.merged_wt)
self.assertEqual(
(shown["branch"], shown["flow_branch"], shown["source"], shown["pull"]),
("renamed", "merged", "file", ["main"]),
)
warning = "flow was set on merged, the worktree is on renamed"
self.assertEqual(shown["warnings"], [warning])
self.assertEqual(
self._cli("flow", self.merged_wt)[1],
f"WARN {warning}\npull: main\npush: main\n",
)
def test_a_planted_symlink_reads_invalid_and_set_replaces_it(self) -> None:
target = os.path.join(self.root, "planted.yml")
with open(target, "w", encoding="utf-8") as fh:
yaml.safe_dump({"version": 1, "pull": ["planted"], "push": []}, fh)
os.makedirs(os.path.dirname(self.linked_file))
os.symlink(target, self.linked_file)
shown = self._json("flow", self.merged_wt)
self.assertEqual(
(shown["source"], shown["pull"], shown["push"]), ("invalid", [], [])
)
self.assertTrue(shown["warnings"][0].startswith("pkgmgr/flow.yml ignored: "))
self._json("flow", "set", self.merged_wt, "--pull", "main", "--no-push")
self.assertFalse(os.path.islink(self.linked_file))
with open(target, encoding="utf-8") as fh:
self.assertEqual(yaml.safe_load(fh)["pull"], ["planted"])
class TestUpFlow(FlowFixture, unittest.TestCase):
def _up(
self,
branch: str,
pull: list[str] | None,
push: list[str] | None,
base: str | None = None,
) -> tuple[int, dict]:
out = io.StringIO()
with redirect_stdout(out), redirect_stderr(io.StringIO()):
code = up.worktree_up(
self.repo,
branch,
base=base,
worktrees_dir=os.path.join(self.root, "wts"),
path=None,
hooks={},
as_json=True,
pull=pull,
push=push,
)
return code, json.loads(out.getvalue()) if code != 2 else {}
def _shown(self, path: str) -> tuple:
shown = self._json("flow", path)
return shown["source"], shown["flow_branch"], shown["pull"], shown["push"]
def test_without_lists_the_flow_is_the_base(self) -> None:
code, result = self._up("feature/d", None, None)
self.assertEqual(code, 0)
self.assertEqual(result["flow"], {"pull": ["origin/main"], "push": ["main"]})
self.assertEqual(
self._shown(result["path"]),
("file", "feature/d", ["origin/main"], ["main"]),
)
def test_given_lists_replace_the_base_each_on_its_own(self) -> None:
_, both = self._up("feature/b", PULL, PUSH, base="origin/main")
_, one = self._up("feature/o", None, ["production"])
self.assertEqual(self._shown(both["path"]), ("file", "feature/b", PULL, PUSH))
self.assertEqual(one["flow"], {"pull": ["origin/main"], "push": ["production"]})
def test_a_base_that_is_no_branch_name_sets_no_flow(self) -> None:
code, result = self._up("feature/c", None, None, base="main~0")
self.assertEqual((code, result["flow"]), (0, None))
self.assertIn("flow not set: base 'main~0'", " ".join(result["warnings"]))
self.assertEqual(self._shown(result["path"])[0], "none")
def test_invalid_names_refuse_before_the_checkout(self) -> None:
escape = os.path.join(self.root, "escape")
for pull, push in (
(["-x"], None),
(None, ["a..b"]),
([f"--output={escape}"], None),
):
with self.subTest(pull=pull, push=push):
self.assertEqual(self._up("feature/i", pull, push)[0], 2)
self.assertFalse(os.path.exists(os.path.join(self.root, "wts")))
self.assertFalse(os.path.exists(escape))
self.assertEqual(git(self.repo, "branch", "--list", "feature/i"), "")
def test_up_parses_repeatable_lists(self) -> None:
args = create_parser("test").parse_args(
["worktree", "up", "feature/x", "--pull", "a", "--pull", "b", "--push", "c"]
)
self.assertEqual((args.pull, args.push), (["a", "b"], ["c"]))
if __name__ == "__main__":
unittest.main()
@@ -0,0 +1,64 @@
from __future__ import annotations
import sys
import types
import unittest
from unittest.mock import MagicMock, patch
from pkgmgr.actions.branch import tree
class SecretStorageError(Exception):
pass
def fake_secretstorage(items: list[MagicMock]) -> types.ModuleType:
module = types.ModuleType("secretstorage")
module.exceptions = types.SimpleNamespace(SecretStorageException=SecretStorageError)
module.connection = MagicMock()
module.dbus_init = MagicMock(return_value=module.connection)
module.search_items = MagicMock(return_value=iter(items))
return module
def item(locked: bool, secret: bytes = b"t0ken") -> MagicMock:
found = MagicMock()
found.is_locked.return_value = locked
found.get_secret.return_value = secret
return found
class TestSecretServiceToken(unittest.TestCase):
def _token(self, module: types.ModuleType) -> tuple[str | None, str | None]:
with patch.dict(sys.modules, {"secretstorage": module}):
return tree.secret_service_token()
def test_a_locked_item_is_never_unlocked(self) -> None:
locked = item(True)
module = fake_secretstorage([locked])
self.assertEqual(self._token(module), (None, "Secret Service locked"))
locked.get_secret.assert_not_called()
locked.unlock.assert_not_called()
module.connection.close.assert_called_once()
def test_an_unlocked_item_returns_its_token(self) -> None:
module = fake_secretstorage([item(True), item(False, b"live")])
self.assertEqual(self._token(module), ("live", None))
module.search_items.assert_called_once_with(
module.connection, {"service": "pkgmgr:github", "username": "github.com|-"}
)
def test_no_item_and_no_bus_name_their_reason(self) -> None:
self.assertEqual(
self._token(fake_secretstorage([])),
(None, "no GitHub token in the Secret Service"),
)
module = fake_secretstorage([])
module.dbus_init.side_effect = SecretStorageError("no bus")
self.assertEqual(
self._token(module), (None, "Secret Service unavailable: no bus")
)
if __name__ == "__main__":
unittest.main()
Whitespace-only changes.
Whitespace-only changes.
@@ -0,0 +1,47 @@
from __future__ import annotations
import os
import tempfile
import unittest
from pkgmgr.actions.gc.report.balance import balance
class TestBalance(unittest.TestCase):
def setUp(self) -> None:
self._tmp = tempfile.TemporaryDirectory()
self.root = os.path.realpath(self._tmp.name)
for rel in ("a/b", "c", "a-x"):
os.makedirs(os.path.join(self.root, rel))
def tearDown(self) -> None:
self._tmp.cleanup()
def _path(self, rel: str) -> str:
return os.path.join(self.root, rel)
def test_nested_and_repeated_items_count_once(self) -> None:
items = [
(self._path("a/b"), 50),
(self._path("a"), 100),
(self._path("a-x"), 7),
(self._path("c"), 10),
(self._path("c"), 10),
(self._path("gone"), 1000),
]
rows = balance([self.root, self.root], items)
self.assertEqual(len(rows), 1)
row = rows[0]
self.assertEqual(row["attributed_bytes"], 117)
self.assertTrue(os.path.ismount(row["filesystem"]))
self.assertGreater(row["total_bytes"], 0)
self.assertLessEqual(row["free_bytes"], row["total_bytes"])
def test_unreadable_paths_get_no_row(self) -> None:
self.assertEqual(balance([self._path("gone")], []), [])
if __name__ == "__main__":
unittest.main()
@@ -0,0 +1,79 @@
from __future__ import annotations
import os
import tempfile
import unittest
from unittest.mock import ANY, patch
from pkgmgr.actions.gc.report.core_dumps import (
ELF_HEADER,
ET_CORE,
core_dumps,
is_core_dump,
)
ET_EXEC = 2
def elf(e_type: int, order: str = "little") -> bytes:
data = 1 if order == "little" else 2
return b"\x7fELF" + bytes([2, data, 1]) + bytes(9) + e_type.to_bytes(2, order)
class TestCoreDumps(unittest.TestCase):
def setUp(self) -> None:
self._tmp = tempfile.TemporaryDirectory()
self.root = os.path.realpath(self._tmp.name)
def tearDown(self) -> None:
self._tmp.cleanup()
def _file(self, name: str, data: bytes, size: int | None = None) -> str:
path = os.path.join(self.root, name)
with open(path, "wb") as fh:
fh.write(data + bytes(46))
if size is not None:
os.truncate(path, size)
return path
def test_et_core_is_detected_in_both_byte_orders(self) -> None:
self.assertTrue(is_core_dump(self._file("core", elf(ET_CORE))))
self.assertTrue(is_core_dump(self._file("core.1", elf(ET_CORE, "big"))))
def test_other_elf_types_and_non_elf_are_ignored(self) -> None:
self.assertFalse(is_core_dump(self._file("core", elf(ET_EXEC))))
self.assertFalse(is_core_dump(self._file("core.2", b"#!/bin/sh\n")))
short = os.path.join(self.root, "core.3")
with open(short, "wb") as fh:
fh.write(b"\x7fELF")
self.assertFalse(is_core_dump(short))
def test_symlink_is_not_followed(self) -> None:
target = self._file("real", elf(ET_CORE))
link = os.path.join(self.root, "core")
os.symlink(target, link)
self.assertFalse(is_core_dump(link))
def test_reads_no_more_than_the_elf_header(self) -> None:
path = self._file("core", elf(ET_CORE), size=1 << 20)
with patch("os.read", wraps=os.read) as read:
is_core_dump(path)
read.assert_called_once_with(ANY, ELF_HEADER)
def test_sparse_dump_is_sized_by_blocks_and_others_dropped(self) -> None:
dump = self._file("core", elf(ET_CORE), size=1 << 30)
binary = self._file("core.7", elf(ET_EXEC))
dumps = core_dumps([dump, binary, dump])
self.assertEqual([d["path"] for d in dumps], [dump])
self.assertEqual(dumps[0]["apparent_bytes"], 1 << 30)
self.assertLess(dumps[0]["size_bytes"], dumps[0]["apparent_bytes"])
self.assertEqual(dumps[0]["mtime"], os.lstat(dump).st_mtime)
if __name__ == "__main__":
unittest.main()
Loaded 100 of 116 files, more files were not shown because too many files have changed in this diff. Show more