Add GitHub/GitLab Release Asset mirroring plugin

[slauger]

Overview

Add support for mirroring binary release assets from GitHub and GitLab releases. This enables mirroring of CLI tools and binaries for air-gapped environments.

Use Case

Many Kubernetes/DevOps tools distribute binaries via GitHub/GitLab releases:

• k9s: k9s_linux_amd64.tar.gz, k9s_Darwin_arm64.tar.gz
• ArgoCD: argocd-linux-amd64, argocd-darwin-arm64
• helmfile: Platform-specific binaries
• kustomize, kind, stern, etc.

Problem: Air-gapped environments cannot access GitHub/GitLab directly.

Solution: Mirror release assets to local repository using Chantal.

Proposed Implementation

Plugin Type: github-release and gitlab-release

Fits perfectly into existing content-addressed storage model (single files).

Example Configuration

repositories:
  # k9s CLI binaries
  - id: k9s-binaries
    type: github-release
    feed: https://github.com/derailed/k9s
    enabled: true
    filters:
      patterns:
        include:
          - "^k9s_linux_amd64\\.tar\\.gz$"
          - "^k9s_Darwin_.*\\.tar\\.gz$"
        exclude:
          - "\\.sbom\\.json$"
          - "^checksums\\.sha256$"
      post_processing:
        only_latest_version: false  # Keep all versions
  
  # ArgoCD CLI binaries
  - id: argocd-binaries
    type: github-release
    feed: https://github.com/argoproj/argo-cd
    enabled: true
    filters:
      patterns:
        include:
          - "^argocd-linux-amd64$"
          - "^argocd-darwin-.*$"
      post_processing:
        only_latest_version: true  # Only latest version

Sync Workflow

1. Fetch releases via API:

   GitHub: GET /repos/{owner}/{repo}/releases
   GitLab: GET /api/v4/projects/{id}/releases
   

2. Extract assets from each release:

   • Release tag (e.g., v0.31.14)
   • Release date
   • Asset list with download URLs

3. Filter assets based on patterns (same as RPM/Helm filtering)

4. Download assets:

   • Calculate SHA256
   • Store in content-addressed pool
   • Create ContentItem in database

5. Track metadata:

   • Release tag
   • Release date
   • Asset name
   • Download URL (for reference)

Database Schema

class GithubRelease(Base):
    """GitHub/GitLab release metadata."""
    id = Column(Integer, primary_key=True)
    repository_id = Column(Integer, ForeignKey("repositories.id"))
    release_tag = Column(String)      # v0.31.14
    release_name = Column(String)     # k9s v0.31.14
    release_date = Column(DateTime)
    release_url = Column(String)
    
class GithubReleaseAsset(Base):
    """Individual release assets."""
    id = Column(Integer, primary_key=True)
    release_id = Column(Integer, ForeignKey("github_releases.id"))
    content_item_id = Column(Integer, ForeignKey("content_items.id"))
    asset_name = Column(String)       # k9s_linux_amd64.tar.gz
    download_url = Column(String)

Publishing

Standard publishing works as-is (hardlinks):

/var/www/repos/k9s-binaries/
  ├── latest/
  │   ├── k9s_linux_amd64.tar.gz        # Hardlink to pool
  │   └── k9s_Darwin_arm64.tar.gz
  └── snapshots/
      └── 2025-01/
          ├── k9s_linux_amd64.tar.gz
          └── k9s_Darwin_arm64.tar.gz

Content Deduplication

Works automatically via SHA256:

• Same binary across different releases → stored once
• Different binaries → separate pool entries

API Rate Limiting

GitHub API:

• Unauthenticated: 60 requests/hour
• Authenticated: 5,000 requests/hour

Solution:

• Use GitHub token for authentication
• Cache release metadata
• Use conditional requests (ETag/If-Modified-Since)

Configuration Options

repositories:
  - id: k9s-binaries
    type: github-release
    feed: https://github.com/derailed/k9s
    github:
      token: ${GITHUB_TOKEN}  # Optional, increases rate limit
      include_prereleases: false
      include_drafts: false
    filters:
      patterns:
        include: ["^k9s_linux_amd64\\.tar\\.gz$"]
      post_processing:
        only_latest_version: false

Benefits

• ✅ Fits perfectly into existing content-addressed storage
• ✅ Deduplication works automatically (SHA256)
• ✅ Snapshots work out-of-the-box
• ✅ Publishing uses hardlinks (instant, zero-copy)
• ✅ Same filtering mechanism as RPM/Helm
• ✅ Clear use case for air-gapped environments

Example Tools to Mirror

Kubernetes Ecosystem:

• k9s, kustomize, kind, stern, kubectx, kubens
• ArgoCD, Flux CLI
• helmfile, helm-diff

HashiCorp (also have GitHub releases):

• terraform, vault, consul (in addition to RPM repos)

Monitoring:

• Prometheus, Grafana binaries

Implementation Steps

1. Create src/chantal/plugins/github_release_sync.py
2. Implement GitHub API client
3. Add release/asset discovery
4. Integrate with existing ContentItem model
5. Add database models for releases/assets
6. Implement filtering (reuse existing patterns)
7. Add chantal repo sync support
8. Add chantal content list --type github-release support
9. Documentation
10. Example configurations in examples/binaries/

Related

• Similar to Helm plugin (index.yaml parsing)
• Uses same content-addressed storage as RPM/APK
• GitLab variant should be similar (different API, same concept)

Future Enhancements

• Support for release notes/changelog
• Automatic checksum verification (many releases include .sha256 files)
• Support for other git hosting platforms (Gitea, Gogs)