Add Git repository mirroring plugin (tarball approach)

[slauger]

Overview

Add support for mirroring complete Git repositories. This enables:

1. Repository backup: Secure copy with full history in case of force-push or deletion
2. Air-gapped development: Mirror source repositories for offline access
3. Compliance: Archival of dependencies with full Git history

Use Case

Problem 1: Force-push protection

• Upstream maintainers can force-push and rewrite history
• Critical dependencies can disappear (left-pad incident)
• Need immutable backup of repository state

Problem 2: Air-gapped source access

• Development environments without internet access
• Need full Git repos including all branches/tags/history
• Not just release tarballs - need actual Git repository

Examples:

• Kubernetes source: https://github.com/kubernetes/kubernetes
• Helm source: https://github.com/helm/helm
• Internal dependencies built from Git repos

Proposed Implementation

Plugin Type: git-mirror

Uses tarball approach to fit into content-addressed storage model.

Architecture

Key Concept: Git repositories are archived as tarballs and stored in the pool.

Sync:
git clone --mirror → tar.gz → SHA256 → pool

Publishing:
Extract tarball → /var/www/repos/repo-name/latest/

No hardlinks possible (directories, not files), but deduplication still works via tarball checksums.

Example Configuration

repositories:

• id: kubernetes-source
  type: git-mirror
  feed: https://github.com/kubernetes/kubernetes
  enabled: true
  git:
  sync_mode: mirror # All branches and tags

• id: helm-source
  type: git-mirror
  feed: https://github.com/helm/helm
  git:
  sync_mode: selective
  branches: [main, release-3.14, release-3.13]
  tags_pattern: "^v3\.(1[34])\." # v3.13.x, v3.14.x

Sync Workflow

1. Clone or update mirror:

   • git clone --mirror (initial)
   • git fetch --all --prune (update)

2. Create tarball:

   • tar czf repo-mirror.tar.gz -C /tmp repo.git/

3. Calculate SHA256

4. Store in pool if not exists:

   • /var/lib/chantal/pool/ab/cd/abcd1234...tar.gz

5. Create ContentItem in database

Deduplication

Works automatically via tarball SHA256.

Scenario: No new commits since last sync

• git fetch --all --prune (no changes)
• tar czf creates identical tarball
• Same SHA256 → already in pool, skip storage

Deduplication confirmed!

Note: Tar archives are deterministic if using --sort=name --mtime flags.

Publishing

Cannot use hardlinks (tarball contains directory structure).

Publishing workflow:

1. Extract tarball to staging directory
2. Atomic swap via symlink
3. Run git update-server-info for HTTP access

Published structure:
/var/www/repos/kubernetes-source/
├── latest/kubernetes.git/
└── snapshots/2025-01/kubernetes.git/

Git Access Methods

HTTP (dumb protocol) - simplest:

• Requires git update-server-info
• Works with standard Apache/NGINX
• Client: git clone http://mirror/repos/kubernetes-source/latest/kubernetes.git

HTTP (smart protocol):

• Requires git-http-backend CGI setup
• Faster for large repos

SSH:

• Requires SSH server with git-shell
• Most secure option

Database Schema

GitMirror table:

• repository_id, last_sync, head_commit, branch_count, tag_count

GitMirrorSnapshot table:

• mirror_id, content_item_id, name, created_at, head_commit
• refs_snapshot (JSON): refs/heads/main, refs/tags/v1.28.0, etc.

Snapshots

Snapshot = Tarball reference + metadata

Benefits:

• Points to tarball in pool (no duplication)
• Stores Git metadata (commits, refs)
• Can be published to separate directory

Storage Optimization

Problem: Large repositories (Linux Kernel: 4 GB bare repo)

Mitigations:

1. Selective mirroring (only specific branches/tags)
2. Shallow clones (optional, not a true mirror)
3. Compression testing (tar vs tar.gz for Git objects)

Benefits

• Deduplication works (unchanged repo → same tarball SHA256)
• Snapshots work (tarball references + metadata)
• Fits into content-addressed storage (tarballs are files)
• Force-push protection (immutable snapshots)
• Simple implementation (git clone + tar + extract)

Limitations

• No hardlinks (must extract tarball for publishing)
• Large repos are large (Linux Kernel: 4 GB)
• Publishing not instant (extraction takes time)
• Deterministic tarballs need --sort=name --mtime flags

Alternative Considered

Native Git pool (store repos directly without tarball):

• Rejected: Does not fit content-addressed model
• No SHA256-based deduplication
• Snapshots would require full repo copies
• Tarball approach is simpler

Implementation Steps

1. Create src/chantal/plugins/git_mirror_sync.py
2. Implement git clone --mirror wrapper
3. Implement tarball creation (deterministic tar)
4. Store tarball in content-addressed pool
5. Add database models (GitMirror, GitMirrorSnapshot)
6. Implement publishing (extract tarball atomically)
7. Add git update-server-info for HTTP access
8. Add chantal repo sync --type git-mirror
9. Add chantal content list --type git-mirror
10. Documentation (HTTP/SSH access setup)
11. Example configurations in examples/git/

Example Repositories to Mirror

Kubernetes Ecosystem:

• kubernetes/kubernetes
• helm/helm
• etcd-io/etcd

Build Dependencies:

• golang/go
• nodejs/node

Internal Dependencies:

• GitLab/GitHub Enterprise repos

Future Enhancements

• Support for Git LFS (large file storage)
• Support for submodules
• Incremental tarball updates (only changed objects)
• Git GC optimization before tarball creation
• Support for Gitea, Gogs, Bitbucket

Related

• Issue Add GitHub/GitLab Release Asset mirroring plugin #19: GitHub/GitLab Release Assets (different use case - binaries, not repos)
• Complements release assets: This mirrors source code, Add GitHub/GitLab Release Asset mirroring plugin #19 mirrors binaries