Skip to content

deployment.md

Latest commit

 

History

History
176 lines (143 loc) · 10.4 KB

deployment.md

File metadata and controls

176 lines (143 loc) · 10.4 KB
Raw

⬅️ Back to README

Deployment Configuration

LLM-Shield-Proxy is configured entirely via environment variables, following 12-factor app principles. Configuration governs upstream routing, cryptographic keys, performance thresholds, and security policies.

Minimum configuration

The proxy answers every request with a 401 until it has both of these:

  1. A client key. VALID_VIRTUAL_KEYS lists the keys your clients send as their bearer token. Generate one with python -c "import secrets; print(secrets.token_urlsafe(32))".
  2. A provider key. The proxy replaces the client key with the provider key for the request's destination: OPENAI_API_KEY for api.openai.com, ANTHROPIC_API_KEY, GEMINI_API_KEY and DEEPSEEK_API_KEY for theirs, and UPSTREAM_API_KEY for any other host or when the provider's own key is unset. With no provider key the proxy answers 500.
export VALID_VIRTUAL_KEYS=<client key>
export OPENAI_API_KEY=<provider key>
llm-shield-proxy --host 127.0.0.1 --port 8000

At startup the proxy warns if no client could authenticate. Every other setting below is optional.

Complete Configuration Reference

Core Proxy Settings

Environment Variable Default Description
HOST 0.0.0.0 IP address to bind the server.
PORT 8000 Port to bind the server.
WORKERS 1 Number of Uvicorn worker processes.
LOG_LEVEL INFO Application log verbosity (DEBUG, INFO, WARNING, ERROR).
UPSTREAM_BASE_URL https://api.openai.com The base URL of the target LLM provider.

HOST, PORT, WORKERS, LOG_LEVEL and the TLS_* settings are read by the llm-shield-proxy command, which is also the container image's entrypoint. Starting the app with uvicorn directly ignores them.

API Keys

Provide the central API keys the proxy will use to authenticate with upstream providers.

Environment Variable Description
UPSTREAM_API_KEY Key for any upstream host not listed below, and the fallback when a listed provider's key is unset.
OPENAI_API_KEY OpenAI API key.
ANTHROPIC_API_KEY Anthropic API key.
GEMINI_API_KEY Google Gemini API key.
DEEPSEEK_API_KEY DeepSeek API key.

Security & Authentication

Environment Variable Default Description
VALID_VIRTUAL_KEYS "" Comma-separated list of authorized client tokens. Required: with it empty, and neither setting below on, every request gets a 401.
ENABLE_OPEN_BYOK_PASSTHROUGH False Allow unrecognized provider-formatted keys (e.g. sk-proj-...) to pass through to the upstream provider.
OVERRIDE_CLIENT_AUTH False Strip the client's API key and replace it with the proxy's configured upstream API key.
CORS_ALLOWED_ORIGINS "" Comma-separated list of allowed browser origins for CORS. Leave empty for strict isolation.
SHIELD_FAILURE_MODE FAIL_CLOSED Security posture on unexpected errors (FAIL_CLOSED terminates, FAIL_OPEN bypasses).

PII Redaction Engines

Environment Variable Default Description
ENABLE_TIER2_ENTROPY True Enable Shannon entropy scanning for unstructured secrets.
SHANNON_ENTROPY_THRESHOLD 4.5 Minimum entropy bits/char to trigger detection.
ENABLE_TIER3_ONNX_NER False Enable local ONNX BERT-NER processing.
ONNX_MODEL_PATH None Path to the ONNX model weights.
CUSTOM_REGEX_PATH None Path to custom_regex.yaml for Bring-Your-Own-Regex rules.
ENABLE_SYNTHETIC_SWAPPING True Replace detected entities with format-preserving synthetic data.
SHIELD_DEFAULT_MASKING_MODE SYNTHETIC The default masking strategy (SYNTHETIC, SCRUB, STATELESS_CRYPTO).
ENABLE_RESPONSE_PII_REDACTION False Redact personal data the model produced, values that were never in the request and so cannot be restored, before the reply reaches the client. Off by default because a value the model rephrases rather than quotes is redacted and not restored. The request path is redacted either way. The benchmark's response-split profile passes only with this on.
RELAY_UPSTREAM_ERROR_MESSAGES True When the provider answers 4xx or 5xx, the client gets the provider's reason (Upstream provider answered HTTP 401: Incorrect API key provided: [REDACTED] ...) with every credential the proxy sent removed, the text scrubbed one-way by the PII engine and cut at 512 characters, plus the provider's type and code under error.upstream. Off, the client gets only the status and Failed to communicate with upstream provider.
ENABLE_DEEP_PAYLOAD_REDACTION True Walk fields outside the known request shapes: metadata, user, response_format, and any field this proxy does not know by name. Turning it off lets those fields reach the provider unredacted. Tool definitions are redacted either way.
PAYLOAD_PROTECTED_KEYS "" Extra JSON keys deep redaction must never rewrite, comma separated, added to the built-in structural set (model, type, enum, $ref, and similar). Per-role equivalent: payload_skip_keys.
PAYLOAD_MAX_REDACT_STRING_LENGTH 8192 Strings longer than this, and any data: URI, are forwarded without inspection.
UNMAPPED_BLOB_POLICY warn What happens when a string past that ceiling appears in a field no policy claims. skip forwards it silently, warn forwards it and writes a signed audit record naming the JSON path, block rejects the request with HTTP 413.

Rolling out deep redaction

Deep redaction is on by default because a passthrough proxy forwards every field, so any field left unwalked is egressed. Two settings control what it costs.

Start on UNMAPPED_BLOB_POLICY=warn. Every blob in an unclaimed field is forwarded and audited as UNMAPPED_BLOB_FORWARDED with its JSON path. Collect those paths, add them to the owning role's payload_skip_keys, then move to block once the audit trail is quiet.

Both UNMAPPED_BLOB_POLICY and PAYLOAD_MAX_REDACT_STRING_LENGTH can be set per role in policies.yaml, so one tenant can run on block while another is still onboarding.

Raising PAYLOAD_MAX_REDACT_STRING_LENGTH past your largest attachment is the one change here that visibly slows the proxy, because the walk then scans base64 that no text detector can match. Measure it on your own payload shapes:

python benchmarks/payload_walk_latency.py --turns 200 --image-mb 4

The blob cost column is what raising the ceiling would cost you. These are local microbenchmarks; they exclude network, TLS and model time, so use them to compare settings against each other rather than as an end-to-end latency figure.

Cryptography & Auditing

Environment Variable Default Description
SHIELD_ENCRYPTION_KEY None 256-bit AES-GCM key used for STATELESS_CRYPTO masking and stream digest receipts. Without it no stream digest receipt is written (one warning at the first stream).
AUDIT_DURABILITY best_effort Durability mode (best_effort, durable, required).
AUDIT_DURABLE_PATH None File path for append-only JSONL logs (required for durable modes).
AUDIT_SIGNING_KEY_FILE None Path to Ed25519 private key for signing audit logs.

Distributed State & Resiliency

Environment Variable Default Description
REDIS_URL None Connection string for Redis (used for distributed state and rate limiting).
SESSION_TTL_SECONDS 3600 Expiration time (in seconds) for stateful PII mappings.
ENABLE_RATE_LIMITING False Enable Redis-backed token bucket rate limiting.
RATE_LIMIT_RPM 6000 Maximum requests per minute per virtual key.
MAX_RETRIES 3 Maximum automatic retries for transient upstream errors.
FALLBACK_BASE_URL None Secondary provider URL for automatic failover.

Advanced Integrations

Environment Variable Default Description
POLICIES_FILE_PATH policies.yaml Path to the YAML file containing RBAC roles and tool policies.
ENABLE_EXT_PROC True Enable the Envoy gRPC External Processing listener. If the listener cannot start (for example, no write access to the socket directory outside the container image), the proxy logs a warning and serves HTTP; set this explicitly to make that a startup error.
EXT_PROC_SOCK_PATH /var/run/llm-shield/ext_proc.sock Unix Domain Socket path for Envoy ext_proc.
TELEMETRY_ENABLED False Enable OpenTelemetry (OTLP) tracing exports.
ENABLE_VAULT_SECRETS False Enable HashiCorp Vault integration for dynamic secrets.
ENABLE_K8S_WEBHOOK False Serve the Kubernetes mutating admission webhook at /v1/k8s/mutate.

TLS/HTTPS Deployment

In production, run the proxy behind TLS. You can terminate TLS directly at the proxy using standard certificate files:

llm-shield-proxy --tls-cert-file /etc/ssl/certs/server.crt --tls-key-file /etc/ssl/private/server.key

Docker Compose Example

services:
  llm-shield:
    image: ghcr.io/ninadphalak/llm-shield-proxy:latest
    ports:
      - "8443:8000"
    environment:
      - UPSTREAM_BASE_URL=https://api.openai.com
      - VALID_VIRTUAL_KEYS=${SHIELD_CLIENT_KEYS}
      - OPENAI_API_KEY=${OPENAI_API_KEY}
      - TLS_CERT_FILE=/certs/server.crt
      - TLS_KEY_FILE=/certs/server.key
    volumes:
      - ./certs:/certs:ro

Kubernetes (Helm)

The chart in deploy/helm/llm-shield-proxy takes the keys from a Secret:

kubectl create secret generic llm-shield-keys \
  --from-literal=VALID_VIRTUAL_KEYS=<client key> \
  --from-literal=OPENAI_API_KEY=<provider key>

helm install llm-shield deploy/helm/llm-shield-proxy \
  --set "envFrom[0].secretRef.name=llm-shield-keys"

Any other setting goes in extraEnv as a name/value list. webhook.enabled=true also serves the mutating webhook; set webhook.sidecarSecretName to the Secret each injected sidecar should load its keys from. The chart runs as the image's non-root user, adds a PodDisruptionBudget, and requests 128Mi of memory with a 256Mi limit: the proxy holds about 75-100 MiB idle.

This is the only chart. The second one that lived in charts/llm-shield-proxy was removed in 1.6.8; its values map to this one as env.UPSTREAM_API_KEY and config.VALID_VIRTUAL_KEYS to keys in the Secret above.