FireSync

Infrastructure as Code for Google Cloud Firestore

FireSync is a lightweight Python tool that brings version control and deployment automation to your Firestore database schema. Manage composite indexes, single-field indexes, and TTL policies using a familiar pull-plan-apply workflow inspired by Terraform.

Features

🔄 Version Control - Track Firestore schema changes in git
🔍 Plan Before Apply - Preview changes before deploying
🛡️ Safety First - Idempotent operations, no accidental deletions
🔑 Flexible Auth - Use any GCP service account key
🐍 Zero Dependencies - Pure Python stdlib, just needs gcloud CLI
🪟 Cross-Platform - Works on Linux, macOS, and Windows

Quick Start

Prerequisites

Python 3.8+
Google Cloud SDK (gcloud CLI)
GCP service account with Firestore permissions:
- datastore.indexes.list
- datastore.indexes.create
- datastore.indexes.update
- firebase.projects.get

Installation

Install from PyPI:

pip install firestore-schema-migration

This installs the firesync command globally.

Alternative: Install from source

git clone https://github.com/PavelRavvich/firesync.git
cd firesync
pip install -e .

All Commands

Get help on any command:

firesync --help
firesync pull --help
firesync plan --help
firesync apply --help
firesync env --help

Available Commands:

firesync init - Initialize workspace with config.yaml
firesync env add <name> - Add environment to workspace
firesync env remove <name> - Remove environment from workspace
firesync env list - List all environments
firesync env show <name> - Show environment details
firesync pull - Export Firestore schema to local files
firesync plan - Compare local vs remote schemas
firesync apply - Deploy local schema to Firestore

Quick Setup

Initialize workspace:

firesync init

Add your environments:

# Using direct path to key file
firesync env add dev --key-path=./secrets/gcp-key-dev.json --description="Development"

# Using environment variable (auto-detects JSON content or file path)
firesync env add prod --key-env=GCP_PROD_KEY --description="Production"

Pull schemas from all environments:

firesync pull --all

Commands Overview

firesync init

Initialize a new FireSync workspace. Creates config.yaml to manage multiple environments.

# Initialize in current directory
firesync init

# Initialize in custom directory
firesync init --path ./my-project

Options:

--path PATH - Target directory for workspace (default: current directory)

Creates:

config.yaml - Workspace configuration
firestore_schema/ - Default schema directory

firesync env

Manage environments in your workspace.

Authentication Options:

FireSync supports two ways to provide GCP service account credentials:

Option	Description	Use Case
`--key-path`	Direct path to key file	Local development
`--key-env`	Environment variable name	CI/CD and shared environments

The --key-env option is smart: it auto-detects whether the environment variable contains:

JSON content (the key itself) - creates a temp file for gcloud
File path (path to key file) - reads the key from that path

Add environment:

# Option 1: Direct path to key file
firesync env add dev --key-path=./secrets/gcp-key-dev.json

# Option 2: Environment variable (auto-detects JSON content or file path)
firesync env add prod --key-env=GCP_PROD_KEY --description="Production environment"

Example environment variable usage:

# Store JSON content in env var (for CI/CD secrets)
export GCP_KEY='{"type":"service_account","project_id":"my-project",...}'

# Or store path to key file in env var
export GCP_KEY='/path/to/service-account.json'

# Both work with the same command:
firesync env add prod --key-env=GCP_KEY

List environments:

firesync env list
# Shows environment names with key paths (including absolute paths) or key env variables
# Example output:
#   dev
#     key_path: secrets/gcp-key-dev.json - Development environment (/absolute/path/to/secrets/gcp-key-dev.json)
#   prod
#     key_env: GCP_PROD_KEY - Production environment

Show environment details:

firesync env show dev

Remove environment:

firesync env remove dev

firesync pull

Export Firestore schema from GCP to local JSON files.

Pull all environments:

firesync pull --all

Pull single environment:

firesync pull --env=dev

Creates three JSON files in schema directory:

composite-indexes.json - Composite indexes
field-indexes.json - Single-field indexes
ttl-policies.json - TTL policies

firesync plan

Compare local schema against remote Firestore and preview changes.

Compare local vs remote:

firesync plan --env=dev

Migration mode - compare two environments (local vs local):

firesync plan --env-from=dev --env-to=staging

With custom schema directory:

firesync plan --env=dev --schema-dir=custom_schemas

Output shows:

[+] WILL CREATE - Resource exists locally but not remotely
[-] WILL DELETE - Resource exists remotely but not locally
[~] WILL UPDATE - Resource differs between local and remote
[~] No changes - Schemas are in sync

firesync apply

Deploy local schema to Firestore.

Apply to environment:

firesync apply --env=dev

Migration mode - apply source schema to target environment:

firesync apply --env-from=dev --env-to=staging

With custom schema directory:

firesync apply --env=prod --schema-dir=custom_schemas

Note: Apply operations are idempotent and skip existing resources. Delete operations are not implemented for safety.

Workspace Configuration

FireSync uses config.yaml to manage multiple environments with separate schemas.

Workspace Structure

your-project/
├── config.yaml                    # Workspace configuration
├── firestore_schema/              # Schema directories per environment
│   ├── dev/
│   │   ├── composite-indexes.json
│   │   ├── field-indexes.json
│   │   └── ttl-policies.json
│   ├── staging/
│   │   ├── composite-indexes.json
│   │   ├── field-indexes.json
│   │   └── ttl-policies.json
│   └── prod/
│       ├── composite-indexes.json
│       ├── field-indexes.json
│       └── ttl-policies.json
└── secrets/                       # GCP service account keys (gitignored)
    ├── gcp-key-dev.json
    ├── gcp-key-staging.json
    └── gcp-key-prod.json

config.yaml Format

version: 1
schema_base_dir: firestore_schema

environments:
  dev:
    key_path: secrets/gcp-key-dev.json        # Direct path to key file
    description: "Development environment"
  prod:
    key_env: GCP_PROD_KEY                     # Env var (JSON content OR file path)
    description: "Production environment"

Fields:

version - Config format version (always 1)
schema_base_dir - Base directory for all schema files
environments - Map of environment configurations (choose ONE per environment):
- key_path - Direct path to GCP service account key file (relative to config.yaml)
- key_env - Environment variable name (auto-detects JSON content or file path)
- description - Optional description of the environment

Usage Examples

Basic Workflow

# 1. Initialize workspace
firesync init

# 2. Add environment
firesync env add dev --key-path=./secrets/gcp-key-dev.json --description="Development environment"

# 3. Pull current schema
firesync pull --env=dev

# 4. Edit schema files (add/modify indexes)
vim firestore_schema/dev/composite-indexes.json

# 5. Preview changes
firesync plan --env=dev

# 6. Apply changes
firesync apply --env=dev

# 7. Commit to git
git add config.yaml firestore_schema/
git commit -m "Add user query index"

Multi-Environment Management

Manage dev, staging, and production environments separately:

# Set up all environments
firesync init
firesync env add dev --key-path=./secrets/gcp-key-dev.json --description="Development environment"
firesync env add staging --key-path=./secrets/gcp-key-staging.json --description="Staging environment"
firesync env add prod --key-env=GCP_PROD_KEY --description="Production environment"

# Pull schemas from all environments
firesync pull --all

# Work on dev environment
vim firestore_schema/dev/composite-indexes.json
firesync plan --env=dev
firesync apply --env=dev

# Promote dev schema to staging
# --env-from: source environment (reads LOCAL schema files from firestore_schema/dev/)
# --env-to: target environment (applies to REMOTE Firestore in staging project)
firesync plan --env-from=dev --env-to=staging
firesync apply --env-from=dev --env-to=staging

# IMPORTANT: Local files for staging are NOT updated automatically!
# Pull staging schema to update local files after migration:
firesync pull --env=staging

# After testing, promote to production
firesync plan --env-from=dev --env-to=prod
firesync apply --env-from=dev --env-to=prod

# Update production local files
firesync pull --env=prod

Environment Migration

Compare and migrate schemas between environments:

# Compare dev and staging schemas (local vs local)
firesync plan --env-from=dev --env-to=staging

# Apply dev schema to staging environment
firesync apply --env-from=dev --env-to=staging

# Verify changes
firesync pull --env=staging
git diff firestore_schema/staging/

Configuration

Custom Schema Directory

Override the schema directory for any command:

firesync plan --env=dev --schema-dir=custom_schemas
firesync apply --env=staging --schema-dir=backups/schemas_2024

Schema Files

FireSync manages three types of Firestore configurations:

composite-indexes.json

Composite indexes for complex queries spanning multiple fields. Each index includes metadata like name, state, and density from Firestore.

Example:

[
  {
    "density": "SPARSE_ALL",
    "fields": [
      {
        "fieldPath": "userId",
        "order": "ASCENDING"
      },
      {
        "fieldPath": "status",
        "order": "ASCENDING"
      },
      {
        "fieldPath": "createdAt",
        "order": "DESCENDING"
      },
      {
        "fieldPath": "__name__",
        "order": "DESCENDING"
      }
    ],
    "name": "projects/your-project-id/databases/(default)/collectionGroups/orders/indexes/CICAgICAgICA",
    "queryScope": "COLLECTION",
    "state": "READY"
  }
]

Note: When creating new indexes manually, you only need to specify fields and queryScope. The name, state, and density fields are populated by Firestore.

field-indexes.json

Single-field index configurations including default index settings and exemptions.

Example:

[
  {
    "indexConfig": {
      "indexes": [
        {
          "fields": [
            {
              "fieldPath": "*",
              "order": "ASCENDING"
            }
          ],
          "queryScope": "COLLECTION",
          "state": "READY"
        },
        {
          "fields": [
            {
              "fieldPath": "*",
              "order": "DESCENDING"
            }
          ],
          "queryScope": "COLLECTION",
          "state": "READY"
        },
        {
          "fields": [
            {
              "arrayConfig": "CONTAINS",
              "fieldPath": "*"
            }
          ],
          "queryScope": "COLLECTION",
          "state": "READY"
        }
      ]
    },
    "name": "projects/your-project-id/databases/(default)/collectionGroups/__default__/fields/*"
  }
]

Note: The __default__/fields/* entry controls default indexing behavior for all collections.

ttl-policies.json

Time-to-live policies for automatic document deletion. Documents are deleted when the TTL field value expires.

Example:

[
  {
    "indexConfig": {
      "ancestorField": "projects/your-project-id/databases/(default)/collectionGroups/__default__/fields/*",
      "indexes": [
        {
          "fields": [
            {
              "fieldPath": "expiresAt",
              "order": "ASCENDING"
            }
          ],
          "queryScope": "COLLECTION",
          "state": "READY"
        },
        {
          "fields": [
            {
              "fieldPath": "expiresAt",
              "order": "DESCENDING"
            }
          ],
          "queryScope": "COLLECTION",
          "state": "READY"
        },
        {
          "fields": [
            {
              "arrayConfig": "CONTAINS",
              "fieldPath": "expiresAt"
            }
          ],
          "queryScope": "COLLECTION",
          "state": "READY"
        }
      ],
      "usesAncestorConfig": true
    },
    "name": "projects/your-project-id/databases/(default)/collectionGroups/sessions/fields/expiresAt",
    "ttlConfig": {
      "state": "ACTIVE"
    }
  }
]

Note: TTL policies automatically create necessary indexes for the TTL field (ASCENDING, DESCENDING, and CONTAINS). The ttlConfig.state can be ACTIVE or INACTIVE.

Project Structure

Your workspace (after firesync init):

your-project/
├── config.yaml                # Workspace configuration
├── firestore_schema/          # Schema directories (per environment)
│   ├── dev/
│   │   ├── composite-indexes.json
│   │   ├── field-indexes.json
│   │   └── ttl-policies.json
│   ├── staging/
│   │   └── ...
│   └── prod/
│       └── ...
└── secrets/                   # GCP service account keys (gitignored)
    ├── gcp-key-dev.json
    ├── gcp-key-staging.json
    └── gcp-key-prod.json

Source code structure (for contributors):

firesync/
├── src/firesync/              # Main Python package
│   ├── __init__.py
│   ├── __main__.py            # python -m firesync support
│   ├── main.py                # CLI entry point
│   ├── cli.py                 # CLI argument parsing
│   ├── config.py              # Configuration management
│   ├── gcloud.py              # GCloud CLI wrapper
│   ├── normalizers.py         # Data normalization
│   ├── operations.py          # Resource operations
│   ├── schema.py              # Schema file handling
│   ├── workspace.py           # Workspace configuration
│   └── commands/              # Command implementations
│       ├── __init__.py
│       ├── init.py
│       ├── env.py
│       ├── pull.py
│       ├── plan.py
│       └── apply.py
├── tests/                     # Unit tests
│   ├── test_cli.py
│   ├── test_config.py
│   ├── test_gcloud.py
│   ├── test_normalizers.py
│   ├── test_operations.py
│   ├── test_schema.py
│   └── test_workspace.py
├── pyproject.toml             # Package configuration
├── LICENSE
└── README.md

CI/CD Integration

GitHub Actions - Single Environment

name: Deploy Firestore Schema

on:
  push:
    branches: [main]

jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3

      - name: Set up Python
        uses: actions/setup-python@v4
        with:
          python-version: '3.11'

      - name: Install firesync
        run: pip install firestore-schema-migration

      - name: Install gcloud CLI
        uses: google-github-actions/setup-gcloud@v1

      - name: Create config.yaml
        run: |
          echo "version: 1" > config.yaml
          echo "schema_base_dir: firestore_schema" >> config.yaml
          echo "environments:" >> config.yaml
          echo "  prod:" >> config.yaml
          echo "    key_env: GCP_KEY" >> config.yaml
          echo "    schema_dir: prod" >> config.yaml

      - name: Plan schema changes
        env:
          GCP_KEY: ${{ secrets.GCP_SERVICE_ACCOUNT_KEY }}
        run: firesync plan --env=prod

      - name: Apply schema changes
        env:
          GCP_KEY: ${{ secrets.GCP_SERVICE_ACCOUNT_KEY }}
        run: firesync apply --env=prod

GitHub Actions - Multi-Environment Pipeline

name: Deploy to All Environments

on:
  push:
    branches: [main]

jobs:
  deploy-dev:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - uses: actions/setup-python@v4
        with:
          python-version: '3.11'
      - run: pip install firestore-schema-migration
      - uses: google-github-actions/setup-gcloud@v1

      - name: Deploy to dev
        env:
          GCP_DEV_KEY: ${{ secrets.GCP_DEV_KEY }}
        run: |
          firesync plan --env=dev
          firesync apply --env=dev

  deploy-staging:
    needs: deploy-dev
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - uses: actions/setup-python@v4
        with:
          python-version: '3.11'
      - run: pip install firestore-schema-migration
      - uses: google-github-actions/setup-gcloud@v1

      - name: Deploy to staging
        env:
          GCP_STAGING_KEY: ${{ secrets.GCP_STAGING_KEY }}
        run: |
          firesync plan --env=staging
          firesync apply --env=staging

  deploy-prod:
    needs: deploy-staging
    runs-on: ubuntu-latest
    environment: production
    steps:
      - uses: actions/checkout@v3
      - uses: actions/setup-python@v4
        with:
          python-version: '3.11'
      - run: pip install firestore-schema-migration
      - uses: google-github-actions/setup-gcloud@v1

      - name: Deploy to production
        env:
          GCP_PROD_KEY: ${{ secrets.GCP_PROD_KEY }}
        run: |
          firesync plan --env=prod
          firesync apply --env=prod

Limitations

Delete operations: Plan shows deletions but Apply doesn't implement them (safety feature)
Manual schema editing: Schema files must be edited manually or pulled from existing Firestore
Single region: Assumes single Firestore region per project

Troubleshooting

Authentication Issues

# Verify gcloud is installed
gcloud --version

# Check active account
gcloud auth list

# Re-authenticate if needed
gcloud auth login

Permission Denied

Ensure your service account has these IAM roles:

roles/datastore.indexAdmin or
roles/datastore.owner

Schema Files Not Found

# Pull schema first if firestore_schema/ is empty
firesync pull --env=dev

Contributing

Contributions are welcome! Please feel free to submit issues or pull requests.

License

MIT License - see LICENSE for details.

Author

Pavel Ravvich

Note: Always test schema changes in development/staging environments before applying to production.

Name		Name	Last commit message	Last commit date
Latest commit History 34 Commits
.githooks		.githooks
.github/workflows		.github/workflows
src/firesync		src/firesync
tests		tests
.gitignore		.gitignore
CHANGELOG.md		CHANGELOG.md
CLAUDE.MD		CLAUDE.MD
CONTRIBUTING.md		CONTRIBUTING.md
LICENSE		LICENSE
README.md		README.md
pyproject.toml		pyproject.toml

License

PavelRavvich/firesync

Folders and files

Latest commit

History

Repository files navigation

FireSync

Table of Contents

Features

Quick Start

Prerequisites

Installation

All Commands

Quick Setup

Commands Overview

firesync init

firesync env

firesync pull

firesync plan

firesync apply

Workspace Configuration

Workspace Structure

config.yaml Format

Usage Examples

Basic Workflow

Multi-Environment Management

Environment Migration

Configuration

Custom Schema Directory

Schema Files

composite-indexes.json

field-indexes.json

ttl-policies.json

Project Structure

CI/CD Integration

GitHub Actions - Single Environment

GitHub Actions - Multi-Environment Pipeline

Limitations

Troubleshooting

Authentication Issues

Permission Denied

Schema Files Not Found

Contributing

License

Author

About

Topics

Resources

License

Contributing

Uh oh!

Stars

Watchers

Forks

Releases

Packages 0

Contributors 2

Languages

Packages