展示 HN：Repogen – 一个用于软件包仓库的静态站点生成器

原文

⚠️ Alpha Software Warning

Repogen is alpha software and has not been extensively battle-tested in production environments. While it includes comprehensive test coverage and has been validated with package managers, use it with caution for critical infrastructure. Always verify generated repositories work correctly with your package manager before deploying to production.

Repogen is a CLI tool that generates static repository structures for multiple package managers. It scans directories for packages, generates appropriate metadata files, and signs repositories with GPG/RSA keys.

Debian/APT (.deb packages)
Yum/RPM (.rpm packages)
Alpine/APK (.apk packages)
Arch Linux/Pacman (.pkg.tar.zst, .pkg.tar.xz, .pkg.tar.gz)
Homebrew (bottle files)

Automatic Package Detection: Scans directories and auto-detects package types using magic bytes
Metadata Generation: Creates all necessary index and metadata files for each repository type
Repository Signing: Signs repositories with GPG (Debian/RPM/Pacman) or RSA (Alpine) keys
Unsigned Repository Support:
- Always generates InRelease files (required by Debian Trixie)
- InRelease contains Release content without signature for unsigned repos
- Compatible with [trusted=yes] apt option
Static Output: Generates static file structures that can be served by any web server
Simple Component Structure: Uses single component/pool structure for simplicity

# Clone the repository
git clone https://github.com/ralt/repogen
cd repogen

# Build
go build -o repogen ./cmd/repogen

# Optional: Install to PATH
sudo cp repogen /usr/local/bin/

# Scan current directory and generate repositories
repogen generate

# Scan specific directory
repogen generate --input-dir /path/to/packages --output-dir /path/to/repo

# Enable verbose logging
repogen generate -v

Incremental mode allows you to add new packages to an existing repository without regenerating everything from scratch. This is useful when:

You have a large repository and only want to add new package versions
You're syncing from S3 and don't want to download all package files locally
You want faster repository updates

How It Works:

Reads existing metadata files (Packages, trust.db, repomd.xml, etc.) from the output directory
Adds only new packages without removing existing ones
Errors if a package with the same name+version already exists (conflict detection)
Regenerates metadata files with both existing and new packages
Re-signs metadata if signing is enabled

Basic Incremental Usage:

# Add new packages to existing repository
repogen generate \
  --input-dir ./new-packages \
  --output-dir ./repo \
  --incremental

S3 Workflow Examples:

The incremental mode is particularly powerful when combined with S3. You can sync only the metadata files (not the packages themselves), add new packages, and regenerate.

# Sync only metadata from S3 (not package files)
aws s3 sync s3://my-bucket/repo/dists ./repo/dists --delete

# Add new packages with repogen
repogen generate --input-dir ./new-packages --output-dir ./repo --incremental

# Sync everything back to S3 (without --delete to preserve existing packages)
aws s3 sync ./repo s3://my-bucket/repo

# Sync only metadata
aws s3 sync s3://my-bucket/repo/40/x86_64/repodata ./repo/40/x86_64/repodata --delete

# Add new packages
repogen generate \
  --input-dir ./new-packages \
  --output-dir ./repo \
  --incremental \
  --version 40

# Sync back (without --delete to preserve existing packages)
aws s3 sync ./repo s3://my-bucket/repo

# Sync only database files (exclude actual package files)
aws s3 sync s3://my-bucket/repo/x86_64 ./repo/x86_64 \
  --exclude "*.pkg.tar.zst" \
  --exclude "*.pkg.tar.zst.sig"

# Add new packages
repogen generate \
  --input-dir ./new-packages \
  --output-dir ./repo \
  --repo-name myrepo \
  --incremental

# Sync back (without --delete to preserve existing packages)
aws s3 sync ./repo s3://my-bucket/repo

# Sync metadata only
aws s3 cp s3://my-bucket/repo/x86_64/APKINDEX.tar.gz ./repo/x86_64/APKINDEX.tar.gz

# Add new packages
repogen generate --input-dir ./new-packages --output-dir ./repo --incremental

# Sync back (without --delete to preserve existing packages)
aws s3 sync ./repo s3://my-bucket/repo

Important Notes:

Incremental mode will error if a package with the same name+version already exists (conflict detection)
If metadata files don't exist, it falls back to normal mode automatically
Package files from existing metadata don't need to be present locally
You can use incremental mode with or without signing

Debian/RPM/Pacman (GPG Signing)

# Generate signed Debian/RPM repositories
repogen generate \
  --input-dir ./packages \
  --output-dir ./repo \
  --gpg-key /path/to/private.key \
  --gpg-passphrase "your-passphrase"

# Generate signed Pacman repository (requires --repo-name)
repogen generate \
  --input-dir ./packages \
  --output-dir ./repo \
  --repo-name "myrepo" \
  --gpg-key /path/to/private.key \
  --gpg-passphrase "your-passphrase"

# Generate signed Alpine repository
repogen generate \
  --input-dir ./packages \
  --output-dir ./repo \
  --rsa-key /path/to/rsa-private.pem \
  --rsa-passphrase "your-passphrase" \
  --key-name "mykey"

repogen generate [flags]

Flags:
  # Input/Output
  -i, --input-dir string        Input directory to scan (default ".")
  -o, --output-dir string       Output directory (default "./repo")
  -v, --verbose                 Enable verbose logging

  # Incremental Mode
      --incremental             Add new packages to existing repository without removing existing ones

  # GPG Signing (Debian/RPM)
  -k, --gpg-key string          Path to GPG private key
  -p, --gpg-passphrase string   GPG key passphrase

  # RSA Signing (Alpine)
      --rsa-key string          Path to RSA private key
      --rsa-passphrase string   RSA key passphrase
      --key-name string         Key name for Alpine signatures (default "repogen")

  # Repository Metadata
      --origin string           Repository origin name
      --label string            Repository label
      --repo-name string        Repository name (required for Pacman)
      --codename string         Codename for Debian repos (default "stable")
      --suite string            Suite for Debian repos (defaults to codename)
      --components strings      Components for Debian repos (default [main])
      --arch strings            Architectures to support (default [amd64])

  # Homebrew
      --base-url string         Base URL for Homebrew bottles

Generated Repository Structures

repo/
├── dists/
│   └── stable/
│       ├── InRelease              # Cleartext signed Release (or unsigned copy for unsigned repos)
│       ├── Release                # Main metadata
│       ├── Release.gpg            # Detached GPG signature (only for signed repos)
│       └── main/
│           └── binary-amd64/
│               ├── Packages        # Package metadata
│               ├── Packages.gz     # Compressed
│               └── Release
└── pool/
    └── main/
        └── {letter}/              # First letter of package name
            └── {package-name}/
                └── package.deb

Using the Repository:

# Add repository (unsigned)
echo "deb [trusted=yes] http://your-server.com/repo stable main" | sudo tee /etc/apt/sources.list.d/repo.list

# Add repository (signed)
# First, import the public key
wget -qO - http://your-server.com/repo/public.key | sudo apt-key add -
echo "deb http://your-server.com/repo stable main" | sudo tee /etc/apt/sources.list.d/repo.list

# Update and install
sudo apt update
sudo apt install package-name

repo/
├── repodata/
│   ├── repomd.xml              # Main metadata index
│   ├── repomd.xml.asc          # GPG signature
│   └── {hash}-primary.xml.gz   # Package metadata
└── Packages/
    └── *.rpm

Using the Repository:

# Create repo file
sudo tee /etc/yum.repos.d/repo.repo <<EOF
[myrepo]
name=My Repository
baseurl=http://your-server.com/repo
enabled=1
gpgcheck=0
EOF

# With GPG checking
sudo rpm --import http://your-server.com/repo/public.key
sudo tee /etc/yum.repos.d/repo.repo <<EOF
[myrepo]
name=My Repository
baseurl=http://your-server.com/repo
enabled=1
gpgcheck=1
gpgkey=http://your-server.com/repo/public.key
EOF

# Install packages
sudo yum install package-name

repo/
└── x86_64/
    ├── APKINDEX.tar.gz         # Package index
    ├── APKINDEX.tar.gz.SIGN.RSA.repogen.pub  # RSA signature
    └── package-1.0.0-r0.apk

Using the Repository:

# Add repository
echo "http://your-server.com/repo" | sudo tee -a /etc/apk/repositories

# With signing (copy public key first)
sudo cp repogen.pub /etc/apk/keys/
echo "http://your-server.com/repo" | sudo tee -a /etc/apk/repositories

# Update and install
sudo apk update
sudo apk add package-name

Arch Linux/Pacman Repository

repo/
└── x86_64/
    ├── myrepo.db.tar.zst       # Package database
    ├── myrepo.db               # Symlink/copy of .db.tar.zst
    ├── myrepo.db.tar.zst.sig   # GPG signature (if signed)
    ├── myrepo.db.sig           # Symlink/copy of signature
    ├── package-1.0.0-1-x86_64.pkg.tar.zst
    └── package-1.0.0-1-x86_64.pkg.tar.zst.sig  # Package signature (if signed)

Using the Repository:

# Add repository to /etc/pacman.conf
sudo tee -a /etc/pacman.conf <<EOF
[myrepo]
Server = http://your-server.com/repo/\$arch
SigLevel = Optional TrustAll
EOF

# With GPG signing (import public key first)
sudo pacman-key --add public.key
sudo pacman-key --lsign-key KEY_ID
# Update SigLevel in /etc/pacman.conf:
# SigLevel = Required DatabaseOptional

# Update and install
sudo pacman -Sy
sudo pacman -S package-name

repo/
├── Formula/
│   └── package-name.rb         # Ruby formula
└── bottles/
    └── package--1.0.0.monterey.bottle.tar.gz

Using the Repository:

# Add tap (assuming repo is in GitHub)
brew tap username/repo https://github.com/username/repo

# Install package
brew install package-name

Generate GPG Key for Signing

# Generate key
gpg --full-generate-key

# Export private key
gpg --export-secret-keys YOUR_KEY_ID > private.key

# Export public key (for distribution)
gpg --export --armor YOUR_KEY_ID > public.key

Generate RSA Key for Alpine

# Generate RSA private key
openssl genrsa -out private.pem 2048

# Extract public key
openssl rsa -in private.pem -pubout -out public.pem

# With passphrase
openssl genrsa -aes256 -out private.pem 2048

Repository Structure Details

Repogen generates Debian repositories following the standard format:

InRelease: Cleartext signed Release file (preferred by modern apt). For unsigned repositories, contains the same content as Release file without signature wrapper.
Release: Contains metadata and checksums of all index files
Release.gpg: Detached signature of Release file (only for signed repositories)
Packages: RFC 822-style package metadata
pool/: Organized by first letter of package name

Key fields in Packages file:

Package, Version, Architecture
Filename (relative to repo root)
Size, MD5sum, SHA1, SHA256, SHA512
Description, Depends, Maintainer

Repogen generates RPM repositories compatible with yum/dnf:

repomd.xml: Master index with checksums of metadata files
primary.xml.gz: Core package information and dependencies
Minimal metadata (primary only) for simplicity

The generated repositories can be consumed by:

yum (RHEL/CentOS 7 and earlier)
dnf (RHEL/CentOS 8+, Fedora)
zypper (openSUSE)

Repogen generates Alpine repositories in the apk v2 format:

APKINDEX.tar.gz: Contains DESCRIPTION and APKINDEX files
APKINDEX: Letter:value format package metadata
- C: Checksum (Q1 prefix + base64 SHA1)
- P: Package name
- V: Version
- A: Architecture
- S: Size
- T: Description
- L: License
- D: Dependencies (space-separated)

Repogen generates Pacman (Arch Linux) repositories:

Database file (e.g., myrepo.db.tar.zst): Tarball containing package metadata
desc files: Package information in Pacman format within the database
Package files: .pkg.tar.zst, .pkg.tar.xz, or .pkg.tar.gz
Signatures: Binary GPG signatures (.sig files) for database and packages
Database structure: Each package has a directory with desc file containing:
- %FILENAME%, %NAME%, %VERSION%, %DESC%
- %CSIZE%, %ISIZE% (compressed and installed size)
- %MD5SUM%, %SHA256SUM%
- %ARCH%, %BUILDDATE%, %PACKAGER%, %URL%, %LICENSE%
- %DEPENDS%, %CONFLICTS%, %GROUPS%

Repogen generates Homebrew taps with:

Formula/: Ruby formula files auto-generated from bottles
bottles/: Binary packages
Multi-architecture support (arm64, x86_64)
Platform detection from filename patterns

Bottle filename format: {package}--{version}.{platform}.bottle.tar.gz

Example 1: Simple Debian Repository

# Organize packages
mkdir -p packages
cp *.deb packages/

# Generate repository
repogen generate --input-dir packages --output-dir /var/www/repo

# Serve with nginx
sudo ln -s /var/www/repo /usr/share/nginx/html/repo

Example 2: Multi-Architecture Debian Repository

repogen generate \
  --input-dir packages \
  --output-dir /var/www/repo \
  --arch amd64,arm64,i386 \
  --codename bookworm \
  --origin "My Company" \
  --label "Production Packages"

Example 3: Signed RPM Repository

# Generate repository with GPG signing
repogen generate \
  --input-dir rpms \
  --output-dir /var/www/repo \
  --gpg-key ~/.gnupg/secring.gpg \
  --gpg-passphrase "secret"

# Export public key for users
gpg --export --armor YOUR_KEY_ID > /var/www/repo/RPM-GPG-KEY

Example 4: Homebrew Tap with Multiple Bottles

# Organize bottles
mkdir bottles
cp *.bottle.tar.gz bottles/

# Generate tap
repogen generate \
  --input-dir bottles \
  --output-dir homebrew-tap \
  --base-url "https://github.com/username/homebrew-tap/releases/download/v1.0"

Example 5: Pacman Repository with Signing

# Organize packages
mkdir packages
cp *.pkg.tar.zst packages/

# Generate signed repository
repogen generate \
  --input-dir packages \
  --output-dir /var/www/repo \
  --repo-name "myrepo" \
  --arch x86_64,aarch64 \
  --gpg-key ~/.gnupg/secring.gpg \
  --gpg-passphrase "secret"

# Export public key for users
gpg --export --armor YOUR_KEY_ID > /var/www/repo/myrepo.key

Repogen includes a comprehensive test suite with Docker-based integration tests that verify each repository type works correctly in its native environment.

# Build the binary
make build

# Build test packages
make test-packages

# Run all tests (unit + integration)
make test

# Run only integration tests
make test-integration

Test packages are minimal dummy packages used to verify repository functionality:

# Build test packages natively (requires dpkg-deb, rpmbuild)
make test-packages

# Build test packages using Docker (recommended if tools not available)
make test-packages-docker

This creates:

test/fixtures/debs/repogen-test_1.0.0_amd64.deb
test/fixtures/rpms/repogen-test-1.0.0-1.x86_64.rpm
test/fixtures/apks/repogen-test-1.0.0-r0.apk
test/fixtures/pacman/repogen-test-1.0.0-1-x86_64.pkg.tar.zst
test/fixtures/bottles/repogen-test--1.0.0.x86_64_linux.bottle.tar.gz

Integration tests use Docker to:

Generate repositories with test packages
Spin up distribution-specific containers
Configure package managers to use test repositories
Install test packages
Verify successful installation

Tested Distributions:

Debian: Debian Bookworm and Trixie containers
RPM: Fedora latest container
Alpine: Alpine latest container
Pacman: Arch Linux latest container
Homebrew: Formula validation (local)

Running Integration Tests:

# Requires Docker
make test-integration

# Or run directly with Go
go test -v -timeout 15m ./test

# Skip integration tests if Docker not available
go test -v -short ./...

Integration tests verify:

✓ Repository structure (all expected files present)
✓ Metadata files (Release, Packages, APKINDEX, repomd.xml)
✓ Package manager can read repository metadata
✓ Package manager can install packages
✓ Installed binaries execute successfully

Example output:

=== RUN   TestIntegration
=== RUN   TestIntegration/Debian
    Generating Debian repository...
    Testing repository in Debian container...
    ✓ Debian repository test passed
=== RUN   TestIntegration/RPM
    Generating RPM repository...
    Testing repository in Fedora container...
    ✓ RPM repository test passed
=== RUN   TestIntegration/Alpine
    Generating Alpine repository...
    Testing repository in Alpine container...
    ✓ Alpine repository test passed
=== RUN   TestIntegration/Pacman
    Generating Pacman repository...
    Testing repository in Arch Linux container...
    ✓ Pacman repository test passed
=== RUN   TestIntegration/Homebrew
    Generating Homebrew repository...
    ✓ Homebrew repository test passed

Example GitHub Actions workflow:

name: Test
on: [push, pull_request]

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - uses: actions/setup-go@v4
        with:
          go-version: '1.23'

      - name: Build test packages
        run: make test-packages-docker

      - name: Run tests
        run: make test

You can manually test repositories:

# Generate a test repository
./repogen generate --input-dir test/fixtures/debs --output-dir /tmp/test-repo

# Serve with Python
cd /tmp/test-repo
python3 -m http.server 8000

# In another terminal, test with Docker
docker run -it --rm debian:bookworm bash
# Inside container:
echo "deb [trusted=yes] http://host.docker.internal:8000 stable main" > /etc/apt/sources.list.d/test.list
apt update
apt install repogen-test

Check that package files have correct extensions (.deb, .rpm, .apk, .pkg.tar.zst/.pkg.tar.xz/.pkg.tar.gz, .bottle.tar.gz)
Verify magic bytes in files (packages may be corrupted)
Use --verbose flag to see detailed scanning output

Verify GPG key is not encrypted or provide correct passphrase
Check that private key file is readable
Ensure go-crypto library supports your key type

Verify all metadata files were generated in output directory
Check file permissions (should be readable by web server)
Test with unsigned repository first ([trusted=yes] for apt)
Review web server logs for 404s

Debian Trixie requires InRelease files

Even with [trusted=yes], Debian Trixie expects InRelease files to exist. Repogen now automatically generates InRelease files for all repositories:

Signed repositories: InRelease contains cleartext signature
Unsigned repositories: InRelease contains Release content (no signature)

This ensures compatibility with both old (Bookworm) and new (Trixie) Debian releases.

Ensure Docker is installed and running: docker version
Build test packages first: make test-packages
Check Docker can pull images: docker pull debian:bookworm
Increase timeout for slow systems: go test -timeout 30m ./test

Contributions are welcome! Please feel free to submit pull requests or open issues for bugs and feature requests.

# Clone and build
git clone https://github.com/ralt/repogen
cd repogen
make build

# Make changes and test
make fmt              # Format code
make lint             # Run linter (requires golangci-lint)
make test             # Run all tests

# Before committing
make test-packages    # Ensure test packages build
make test             # Ensure all tests pass

MIT License.

See RELEASING.md for detailed instructions on:

Preparing and creating releases
What happens during the automated release workflow
Deploying the generated repository archive to S3 or web servers
Troubleshooting common issues

Quick start:

# Run tests
make test

# Create and push a tag
git tag -a v1.0.0 -m "Release version 1.0.0"
git push origin v1.0.0

# GitHub Actions will automatically:
# - Build binaries for 4 platforms
# - Create native packages (deb, rpm, apk, bottle)
# - Generate a repository using repogen itself
# - Create release with all artifacts + repository archive

Test Workflow (.github/workflows/test.yml): Runs on PRs and pushes to main
Release Workflow (.github/workflows/release.yml): Runs on version tags (v*..)

The release workflow generates a repogen-repository-VERSION.zip archive containing a complete repository that you can extract and deploy to S3, GitHub Pages, or any web server.