Skip to content

DEPLOYMENT.md

Latest commit

 

History

History
536 lines (386 loc) · 10.5 KB

DEPLOYMENT.md

File metadata and controls

536 lines (386 loc) · 10.5 KB
Raw

Deployment Guide

This guide covers deploying FIML in various environments from development to production.

Minimum Required API Keys

FIML is designed to work with minimal configuration. Here are the API key requirements:

No API Keys Required (Free Tier)

FIML works out-of-the-box with these free providers:

  • Yahoo Finance - Real-time and historical stock data (no API key)
  • CoinGecko - Cryptocurrency data (no API key for basic tier)
  • Mock Provider - For testing and development

Optional API Keys (Enhanced Features)

For additional data sources and features, you can configure these optional API keys:

Provider Environment Variable Purpose Free Tier
Alpha Vantage ALPHA_VANTAGE_API_KEY Stock fundamentals, forex Yes (5 calls/min)
FMP FMP_API_KEY Financial modeling data Yes (250 calls/day)
Polygon POLYGON_API_KEY Real-time market data Yes (limited)
Finnhub FINNHUB_API_KEY Stock & crypto data Yes (60 calls/min)
NewsAPI NEWSAPI_API_KEY Financial news Yes (100 calls/day)
TwelveData TWELVEDATA_API_KEY Time series data Yes (800 calls/day)
Tiingo TIINGO_API_KEY Stock & crypto prices Yes (limited)
CoinMarketCap COINMARKETCAP_API_KEY Cryptocurrency data Yes (333 calls/day)

AI Features (Optional)

For narrative generation and AI-powered insights:

Service Environment Variables Purpose
Azure OpenAI AZURE_OPENAI_ENDPOINT, AZURE_OPENAI_API_KEY, AZURE_OPENAI_DEPLOYMENT_NAME AI narrative generation

Alert Notifications (Optional)

Service Environment Variables Purpose
Email (SMTP) SMTP_HOST, SMTP_PORT, SMTP_USER, SMTP_PASSWORD Email alerts
Telegram TELEGRAM_BOT_TOKEN Telegram notifications

Development Deployment

Quick Start

See Installation Guide for development setup.

git clone https://github.com/kiarashplusplus/FIML.git
cd FIML
./quickstart.sh

Production Deployment

Docker Compose (Small to Medium Scale)

Best for: Single server deployments, staging environments

1. Prepare Environment

# Clone repository
git clone https://github.com/kiarashplusplus/FIML.git
cd FIML

# Create production environment file
cp .env.example .env.production

Edit .env.production with production settings:

ENVIRONMENT=production
LOG_LEVEL=WARNING
SERVER_WORKERS=8

# Use strong passwords
POSTGRES_PASSWORD=your_strong_password
REDIS_PASSWORD=your_redis_password

# Add monitoring
SENTRY_DSN=your_sentry_dsn

2. Build and Deploy

# Build production images
docker-compose -f docker-compose.yml build

# Start services
docker-compose -f docker-compose.yml up -d

# Verify deployment
curl http://localhost:8000/health

3. Configure Reverse Proxy

Use Nginx or Caddy for SSL termination and load balancing.

Nginx Example:

server {
    listen 80;
    server_name fiml.yourdomain.com;
    return 301 https://$server_name$request_uri;
}

server {
    listen 443 ssl http2;
    server_name fiml.yourdomain.com;

    ssl_certificate /path/to/cert.pem;
    ssl_certificate_key /path/to/key.pem;

    location / {
        proxy_pass http://localhost:8000;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }

    # WebSocket support
    location /ws {
        proxy_pass http://localhost:8000;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
    }
}

Kubernetes (Large Scale)

Best for: Production clusters, high availability, auto-scaling

Prerequisites

  • Kubernetes cluster (v1.25+)
  • kubectl configured
  • Helm 3.x

1. Create Namespace

kubectl create namespace fiml

2. Configure Secrets

kubectl create secret generic fiml-secrets \
  --from-literal=postgres-password=your_password \
  --from-literal=redis-password=your_password \
  --from-literal=alpha-vantage-key=your_key \
  --from-literal=fmp-key=your_key \
  -n fiml

3. Deploy with Helm

# Add FIML Helm repository (if available)
helm repo add fiml https://charts.fiml.ai
helm repo update

# Install
helm install fiml fiml/fiml \
  --namespace fiml \
  --values values-production.yaml

Or deploy using kubectl:

kubectl apply -f k8s/

4. Verify Deployment

kubectl get pods -n fiml
kubectl logs -f deployment/fiml-api -n fiml

Cloud Provider Deployments

AWS

ECS Deployment

Use the provided CloudFormation template:

aws cloudformation create-stack \
  --stack-name fiml-production \
  --template-body file://cloudformation/ecs-deployment.yaml \
  --parameters file://cloudformation/parameters.json

EKS Deployment

Deploy to Amazon EKS:

# Create EKS cluster
eksctl create cluster -f eks-cluster.yaml

# Deploy application
kubectl apply -f k8s/

Google Cloud Platform

GKE Deployment

# Create GKE cluster
gcloud container clusters create fiml-cluster \
  --num-nodes=3 \
  --machine-type=n1-standard-2

# Deploy
kubectl apply -f k8s/

Azure

AKS Deployment

# Create AKS cluster
az aks create \
  --resource-group fiml-rg \
  --name fiml-cluster \
  --node-count 3

# Deploy
kubectl apply -f k8s/

Database Setup

PostgreSQL

Production Configuration

# In .env.production
POSTGRES_HOST=your-postgres-host
POSTGRES_PORT=5432
POSTGRES_DB=fiml
POSTGRES_USER=fiml
POSTGRES_PASSWORD=strong_password

# Connection pooling
DATABASE_POOL_SIZE=20
DATABASE_MAX_OVERFLOW=10

Managed Database Services

  • AWS RDS: Use PostgreSQL 14+ with TimescaleDB extension
  • Google Cloud SQL: PostgreSQL with high availability
  • Azure Database: PostgreSQL flexible server

Redis

Production Configuration

# In .env.production
REDIS_HOST=your-redis-host
REDIS_PORT=6379
REDIS_PASSWORD=strong_password
REDIS_MAX_CONNECTIONS=100

Managed Redis Services

  • AWS ElastiCache: Redis cluster mode
  • Google Memorystore: Redis with HA
  • Azure Cache: Redis premium tier

Monitoring Setup

Prometheus

Already configured in docker-compose.yml:

prometheus:
  image: prom/prometheus
  ports:
    - "9091:9090"
  volumes:
    - ./config/prometheus.yml:/etc/prometheus/prometheus.yml

Grafana

Access dashboards at http://localhost:3000:

Default credentials: admin/admin

Import provided dashboards:

  • FIML API Metrics
  • Cache Performance
  • Provider Health

Sentry

Add to .env.production:

SENTRY_DSN=your_sentry_dsn
SENTRY_ENVIRONMENT=production
SENTRY_TRACES_SAMPLE_RATE=0.1

Scaling Considerations

Horizontal Scaling

Scale API instances:

# Docker Compose
docker-compose up -d --scale api=4

# Kubernetes
kubectl scale deployment fiml-api --replicas=4 -n fiml

Vertical Scaling

Adjust resource limits:

# docker-compose.yml
services:
  api:
    deploy:
      resources:
        limits:
          cpus: '4.0'
          memory: 8G

Cache Scaling

  • Redis: Use Redis Cluster for large datasets
  • PostgreSQL: Enable read replicas for L2 cache

Backup & Recovery

Database Backups

# PostgreSQL backup
docker-compose exec postgres pg_dump -U fiml fiml > backup.sql

# Restore
docker-compose exec -T postgres psql -U fiml fiml < backup.sql

Automated Backups

Configure in cloud provider:

  • AWS RDS: Automated snapshots
  • Google Cloud SQL: Automated backups
  • Azure: Automated backup policy

Security Checklist

  • Use strong passwords for all services
  • Enable SSL/TLS for all connections
  • Configure firewall rules
  • Enable API authentication
  • Set up rate limiting
  • Enable audit logging
  • Regular security updates
  • Secrets management (Vault, AWS Secrets Manager)
  • Network isolation
  • Enable CORS properly

Health Checks

FIML provides comprehensive health check endpoints for monitoring system status.

API Health

Basic application health check:

curl http://localhost:8000/health

Response:

{
  "status": "healthy",
  "version": "0.2.2",
  "environment": "production"
}

Database Health

PostgreSQL/TimescaleDB health check:

curl http://localhost:8000/health/db

Response:

{
  "status": "healthy",
  "service": "postgresql",
  "host": "localhost",
  "port": 5432,
  "database": "fiml"
}

Cache Health

Redis (L1 cache) health check:

curl http://localhost:8000/health/cache

Response:

{
  "status": "healthy",
  "service": "redis",
  "host": "localhost",
  "port": 6379
}

Provider Health

Health status of all data providers:

curl http://localhost:8000/health/providers

Response:

{
  "status": "healthy",
  "total_providers": 5,
  "healthy_providers": 5,
  "providers": {
    "yahoo_finance": {
      "is_healthy": true,
      "uptime_percent": 99.9,
      "avg_latency_ms": 150.5,
      "success_rate": 0.98
    }
  }
}

Individual Provider Health

Health status of a specific provider:

curl http://localhost:8000/health/providers/yahoo_finance

Prometheus Metrics

Prometheus-compatible metrics endpoint:

curl http://localhost:8000/metrics

Available metrics include:

  • fiml_requests_total - Total API requests by method, endpoint, status
  • fiml_request_duration_seconds - Request latency histogram
  • fiml_provider_requests_total - Provider API requests by provider/operation
  • fiml_provider_latency_seconds - Provider latency histogram
  • fiml_active_requests - Current active requests gauge
  • fiml_slow_queries_total - Count of slow queries (>1s)

Troubleshooting

Container Logs

# Docker Compose
docker-compose logs -f api

# Kubernetes
kubectl logs -f deployment/fiml-api -n fiml

Database Connection Issues

Check connection string and credentials:

docker-compose exec api python -c "from fiml.db import test_connection; test_connection()"

Memory Issues

Monitor memory usage:

docker stats

Adjust limits if needed in docker-compose.yml.

Next Steps