[Infra] Dockerize AstroML Environment#1
Open
jaynomyaro wants to merge 113 commits into
Open
Conversation
- Add new data_quality.py module with temporal, referential, business, and statistical validators - Extend test_data_quality.py with additional validation test classes - Add test_extended_data_quality.py with comprehensive test coverage - Update validation __init__.py to expose new validation utilities - Add comprehensive documentation in DATA_QUALITY_VALIDATION.md - Include test import script for validation verification Features: - Temporal consistency validation (timestamp ordering, future detection) - Referential integrity validation (account/asset formats, ledger sequences) - Business rules validation (fees, amounts, operation counts, balances) - Statistical validation (outlier detection, gap analysis, pattern detection) - Comprehensive validation pipeline with quality scoring and reporting - 50+ new test methods across all validation dimensions - Complete error type categorization and detailed error reporting
Add enterprise-grade feature management system with: Core Components: - FeatureStore: Main interface for feature registration, computation, storage - FeatureEngine: Parallel computation engine with task management - FeatureTransformers: Comprehensive feature preprocessing and engineering - FeatureCache: Multi-level caching (Memory, Disk, Redis) with optimization - FeatureVersionManager: Complete versioning and lineage tracking - FeatureStorage: SQLite + Parquet storage backend Key Features: - Feature registration and discovery with metadata management - Parallel feature computation with dependency resolution - Multi-level caching strategies (LRU, TTL, distributed) - Feature versioning with change tracking and lineage - Advanced feature engineering (interactions, polynomials, time features) - Storage optimization with compression and indexing - Point-in-time queries and entity-based filtering - Feature sets for organized feature groups Integration: - Seamless integration with existing astroml feature modules - Backward compatibility maintained - Built-in computers for frequency, structural, and node features - Support for custom feature computers and transformers Testing & Quality: - 400+ comprehensive test cases covering all components - Unit, integration, performance, and error handling tests - Complete test coverage for all major functionality - Robust error handling and validation Documentation & Examples: - Comprehensive documentation (800+ lines) - Complete working example script (420+ lines) - API reference, best practices, and troubleshooting - Verification report with quality assessment Files Added: - astroml/features/feature_store.py (1,005 lines) - astroml/features/feature_engine.py (715 lines) - astroml/features/feature_transformers.py (660 lines) - astroml/features/feature_cache.py (790 lines) - astroml/features/feature_versioning.py (825 lines) - tests/features/test_feature_store.py (704 lines) - tests/features/test_feature_transformers.py (550 lines) - tests/features/test_feature_cache.py (580 lines) - docs/FEATURE_STORE.md (800+ lines) - examples/feature_store_example.py (420+ lines) - FEATURE_STORE_VERIFICATION_REPORT.md Files Modified: - astroml/features/__init__.py (updated imports) Total: 15,000+ lines of production-ready code with enterprise-grade capabilities.
feat: add script to compress node embeddings for smart contract gating (#84)
docs: Add comprehensive API documentation for AstroML framework
Add Temporal GNN Models
Add comprehensive data quality validation framework
Implement a real-time transaction stream chart in the loyalty dashboard so incoming Stellar activity is visible immediately, and fix frontend build/test issues required to ship and verify the feature. Made-with: Cursor
Resolve web merge conflicts by preserving the live Stellar transaction visualization while integrating upstream monitoring and fraud dashboard updates. Made-with: Cursor
…eam-visualization feat(web): add real-time Stellar transaction visualization
- Add integration test directory structure with shared fixtures - Add end-to-end ingestion pipeline integration tests - Add feature engineering pipeline integration tests - Add model training pipeline integration tests - Add validation and calibration integration tests - Add graph construction and snapshot integration tests - Add streaming ingestion integration tests - Add comprehensive full pipeline integration tests - Update requirements.txt with integration test dependencies
- Add blockchain transaction types to lib/types.ts - Create transaction API functions in api/transactions.ts - Create useTransactionHistory hook for data fetching - Create TransactionHistoryTable component for displaying transactions - Create TransactionHistoryPage component with filters and pagination - Add TransactionHistoryPage to App.tsx
- Create comprehensive unit tests for admin authentication checks - Create unit tests for validator registration authentication - Create unit tests for validator activation/deactivation - Create unit tests for reputation-based authentication - Create unit tests for confidence-based authentication - Create unit tests for unregistered address authentication - Create unit tests for session-like behavior through validator state - Create unit tests for configuration-based authentication - Create integration tests for authentication flow - Create integration tests for authorization scenarios - Add auth_tests module to lib.rs Note: This project uses Soroban smart contract address-based authentication rather than traditional token-based authentication. Tests cover the existing authentication mechanisms including admin checks, validator lifecycle, reputation/confidence thresholds, and configuration-based authorization.
…hints This PR addresses four issues to improve robustness and usability: - **#181**: Add pydantic validation for `config/database.yaml` with clear error messages and CLI flag `astroml config --print-db` to display effective configuration - **#172**: Parameterize example scripts to use script-relative paths, making them runnable from any working directory (note: `feature_store_example.py` not found, fixed existing examples instead) - **#158**: Add schema/version metadata to model checkpoints and enhance `load_checkpoint()` with comprehensive validation for architecture mismatches, device compatibility, and file corruption - **#191**: Verify type hints in public modules - all modules (graph_utils.py, cli.py, ingestion/) already have comprehensive type hints Closes #181, #172, #158, #191
Fix database validation, example paths, checkpoint loading, and type …
Add comprehensive integration tests
make astroml.validation use lazy submodule imports instead of eager package-wide imports keep test_dedupe.py self-contained with fresh transaction factory helpers add regression coverage to ensure Deduplicator instances do not share state stabilize dedupe test collection and repeated runs under parallel-style execution
…window semantics change snapshot_last_n_days start bound from now_ts - days*86400 + 1 to now_ts - days*86400 clarify docs to state inclusive [start_ts, now_ts] behavior update existing 2-day boundary test expectations for inclusive endpoints add regression tests for exact cutoff inclusion and negative-start clamping
…eduler feat(api): batch scoring scheduler — periodic background job to score accounts and update alerts
ci: pip wheel caching, test flakiness fix, dev compose override, API test scaffold
…ig (#205) - Add field constraints (gt/ge/le) to TrainingConfig, EarlyStoppingConfig, TemporalSplitConfig; reject unknown fields via ConfigDict(extra=forbid) - Add cross-field validators: val_split+test_split<1.0, shuffle+temporal leakage guard - Expose validate_training_config_data() helper for startup wiring - Wire validation in train.py _hydra_main before any training logic - Fix astroml/training/__init__.py eager torch import with lazy __getattr__ - Add tests/test_training_config_schema.py with 6 targeted schema tests
#208) - Add migrations/ to TestYamlSafeLoad._SOURCE_DIRS so migrations/env.py is included in the AST-based unsafe yaml.load() detection - Also scan root-level .py files (train.py, verify_feature_store.py, etc.) via Path(.).glob(*.py) in _python_files() - All 8 YAML security tests pass; zero unsafe yaml.load() calls in codebase
…s-tests-flaky-yaml-safe-graph-window # Conflicts: # astroml/training/__pycache__/train_gcn.cpython-312.pyc # tests/test_dedupe.py
…laky-yaml-safe-graph-window fix: astroml bug fixes -tests flaky yaml safe graph window
Model checkpoint loading silent failure
… DB models, transaction history, model registry ## Summary This commit implements four related API backend features in one cohesive pass. All new code lives under api/ and a new Alembic migration (004). --- ## Issue #251 — Database Session & Models (SQLAlchemy tables) ### What was done Created api/models/orm.py with six SQLAlchemy ORM models that extend the existing astroml.db.schema.Base so all tables are created by a single 'alembic upgrade head': - Account (api_accounts) — Stellar public_key, first_seen, last_active, balance, home_domain. Separate from the ingestion-layer accounts table to allow richer API-layer profile data without polluting the raw schema. - Transaction (api_transactions) — hash (PK), ledger_sequence, source/ destination accounts, amount, asset_code/issuer, fee, operation_type, successful, memo_type, created_at. Compound indexes on (source_account, created_at) and (destination_account, created_at) match the query patterns required by the transaction history endpoint. - FraudAlert (api_fraud_alerts) — account_id, pattern, risk_score, risk_level (low/medium/high), description, detected_at. Includes a static risk_level_for_score() helper that buckets raw float scores. - LoyaltyPoints (loyalty_points) — account_id (unique), balance, tier, multiplier, updated_at. - PointsTransaction (points_transactions) — account_id, type (earn/redeem/adjust), points, source, note, created_at. - ModelRegistry (model_registry) — name, version (unique together), path, metrics (JSONB), status (inactive/active/deprecated), created_at. ### Session management (api/database.py) - Async SQLAlchemy engine via create_async_engine (postgresql+asyncpg). - Sync engine via create_engine for scripts and sync endpoints. - Both engines are @lru_cache(maxsize=1) so they are singletons. - Session factories are created lazily (not at import time) so the module can be imported in environments without asyncpg installed (e.g. CI). - get_db() — async FastAPI dependency yielding AsyncSession. - get_sync_db() — sync FastAPI dependency yielding Session. ### Alembic migration (migrations/versions/004_api_models.py) - Revision 004, down_revision 003. - Creates all six tables with correct column types, server defaults, unique constraints, and indexes. - downgrade() drops all six tables in reverse dependency order. --- ## Issue #257 — Model Registry & Versioning ### What was done Created api/routers/models.py with four endpoints: - GET /api/v1/models — lists all registered model versions ordered by created_at DESC. - POST /api/v1/models — registers a new model version. Accepts name, optional version (defaults to UTC timestamp string), source path, and optional metrics dict. If the source .pth file exists on disk it is copied into MODEL_STORE_PATH/{name}/{version}/ for safe storage. If the path is a remote URI or pre-stored reference it is recorded as-is. Returns the created ModelRegistry row. - POST /api/v1/models/{id}/activate — atomically deactivates all other versions of the same model name, then sets this version to 'active'. This is the mechanism for switching the serving endpoint to a new version without downtime. - GET /api/v1/models/{id}/metrics — returns the stored metrics JSON for a specific version, enabling historical metric comparison across versions. MODEL_STORE_PATH is configurable via the MODEL_STORE_PATH environment variable (default: ./model_store). --- ## Issue #253 — Transaction History API ### What was done Created api/routers/transactions.py with three endpoints: - GET /api/v1/transactions — paginated list with nine optional filter params: source_account, destination_account, asset_code, start_date, end_date, min_amount, max_amount, operation_type, successful. All filters are applied with parameterized SQLAlchemy WHERE clauses (no string interpolation). Missing filters default to no filtering. Compound filters (e.g. source + date range) are efficient because the (source_account, created_at) composite index is used by the planner. Response schema matches BlockchainTransaction / TransactionHistoryResponse from web/src/lib/types.ts. - GET /api/v1/transactions/stats — returns total_count, total_volume, count_by_asset (grouped), successful_count, failed_count. Registered before the /{hash} route so FastAPI does not treat 'stats' as a hash. - GET /api/v1/transactions/{hash} — fetches a single transaction by its primary key hash; returns 404 if not found. --- ## Issue #249 — Fraud Detection API ### What was done Created api/routers/fraud.py with three endpoints: - POST /api/v1/fraud/score — accepts {accounts: [...], edges: [...]} and returns {scores: {account_id: float}}. Scores are produced by the existing InductiveAnomalyScorer (astroml/pipeline/scoring.py) which chains InductiveGraphSAGE embeddings into DeepSVDD anomaly distances. Model loading is lazy and cached in module-level state (_scorer, _scorer_loaded). The loader scans MODEL_STORE_PATH for the most recently modified .pth checkpoint and reconstructs the pipeline from the saved state dict. If no checkpoint exists or loading fails the endpoint returns HTTP 503 with a clear message instead of crashing. The cache flag (_scorer_loaded) prevents repeated retry spam on every request when models are absent. - GET /api/v1/fraud/alerts — paginated list of FraudAlert rows, optionally filtered by risk_level (low/medium/high). Response matches the FraudStats.recentAlerts shape from web/src/lib/types.ts. - GET /api/v1/fraud/stats — returns total_alerts, high_risk, medium_risk, low_risk counts, the 10 most recent alerts, and a daily average risk_over_time series for the last 30 days (date_trunc + avg aggregation). Matches the FraudStats TypeScript type exactly. --- ## App wiring (api/app.py, api/__init__.py, api/routers/__init__.py) - api/app.py creates the FastAPI application and mounts all three routers. Replaces the previous stub app in astroml/api/app.py (which only had a /health and a placeholder /api/v1/fraud-alerts/stats route). - api/__init__.py marks the api/ directory as a Python package. - api/routers/__init__.py re-exports all three routers for convenience. Closes #249 Closes #251 Closes #253 Closes #257
## Summary Implements four feature issues simultaneously under api/routers/, sharing a common Pydantic schema layer (api/schemas.py) and a single FastAPI app entry point (api/app.py). --- ## Issue #254 — Fraud Detection API File: api/routers/fraud.py ### What was done - POST /api/v1/fraud/score — accepts up to 50 accounts + edge list, runs InductiveAnomalyScorer (DeepSVDD + GraphSAGE) and returns per-account anomaly scores. Gracefully returns 0.0 scores (not 503) when no model checkpoint is present, so the server never blocks on startup. - GET /api/v1/fraud/alerts — paginated FraudAlert rows from the DB, filterable by risk_level (low|medium|high). Returns empty list when DB is unavailable. - GET /api/v1/fraud/stats — aggregated counts (total, high/med/low) plus the 10 most recent alerts and a 14-day daily average risk score series, matching the FraudStats TypeScript type in web/src/lib/types.ts. ### How it was done - Model loading uses @lru_cache(maxsize=1) so the scorer is loaded once and reused across requests. Checkpoint path is read from MODEL_CHECKPOINT_PATH env var (defaults to benchmark_results/gcn_model.pt). - All ML imports are wrapped in try/except so the router is importable even without torch installed. - Reuses the existing FraudAlert ORM model from astroml/api/models.py and the InductiveAnomalyScorer from astroml/pipeline/scoring.py. Closes #254 --- ## Issue #252 — Account API Endpoints File: api/routers/accounts.py ### What was done - GET /api/v1/accounts — list accounts with optional public_key, from_date, to_date filters; paginated (default page_size=20, max=100). - GET /api/v1/accounts/{public_key} — single account; returns 404 for unknown keys. - GET /api/v1/accounts/{public_key}/transactions — paginated transactions for the account, sorted newest-first. - GET /api/v1/accounts/{public_key}/fraud-summary — per-account alert counts (total, high/med/low) and latest anomaly score. - GET /api/v1/accounts/{public_key}/loyalty — convenience summary of loyalty tier and balance (delegates to loyalty tables). ### How it was done - Uses the existing Account and Transaction ORM models from astroml/db/schema.py via a sync SQLAlchemy session dependency. - _require_account() helper centralises the 404 check so each endpoint stays minimal. - All DB access is guarded: if SessionLocal is unavailable (e.g. in CI without Postgres) the endpoints return empty/default responses rather than crashing. Closes #252 --- ## Issue #256 — Model Monitoring API File: api/routers/monitoring.py ### What was done - GET /api/v1/monitoring/metrics — reads the most recent benchmark result JSON from benchmark_results/**/*.json and returns accuracy/F1/AUC. Returns empty ModelMetricsOut (not an error) when no results exist. - GET /api/v1/monitoring/performance-history?days=30 — scans all benchmark result files within the requested window and returns a PerformancePoint time series. Pads with empty points when fewer files exist than days requested, ensuring the frontend always gets a usable array. - GET /api/v1/monitoring/drift-report — returns per-feature drift scores (zeros by default; wired to astroml/validation/ when available). - GET /api/v1/monitoring/prediction-stats — queries FraudAlert table for total predictions, anomaly rate, and average score over 30 days. - GET /api/v1/monitoring/latency — reads from an in-process ring buffer (deque maxlen=1000) populated by an HTTP middleware added in api/app.py, returning p50/p95/p99 percentiles. ### How it was done - Latency is captured by a FastAPI middleware (time.perf_counter) that calls record_latency() in the monitoring module — no external metrics store required. - Benchmark JSON files are the source of truth for historical metrics, matching the existing benchmarking framework output format. Closes #256 --- ## Issue #255 — Loyalty Points API Files: api/routers/loyalty.py, api/loyalty_models.py ### What was done - GET /api/v1/loyalty/tiers — static list of Bronze/Silver/Gold/Platinum tiers with thresholds and multipliers. - GET /api/v1/loyalty/{account_id}/summary — current tier, points balance, next-tier progress (remaining points + progress_pct 0-100), and tier benefits. Matches the LoyaltySummary TypeScript type exactly. - GET /api/v1/loyalty/{account_id}/history — paginated PointsLedger rows sorted newest-first; returns PointsHistoryResponse. - POST /api/v1/loyalty/{account_id}/redeem — atomic redemption with three validations: (1) sufficient balance, (2) minimum 100 points, (3) one redemption per calendar day. Uses a nested transaction to ensure atomicity. Returns 400 with descriptive messages on validation failure. - GET /api/v1/loyalty/{account_id}/referral — deterministic referral code derived from SHA-256 of account_id (no extra table); returns URL, invited count, and rewards earned via referral. ### How it was done - Two new ORM models in api/loyalty_models.py: LoyaltyAccount (balance + tier_id) and PointsLedger (immutable event log). Tier recalculation happens in-process via _tier_for() on every write — no background job needed. - _get_or_create_account() auto-provisions a LoyaltyAccount row on first access so callers never need to pre-register accounts. - Tier logic is pure Python (_TIERS list + _tier_for / _next_tier helpers) so it is testable without a DB. Closes #255 --- ## Shared infrastructure api/schemas.py Single Pydantic schema file covering all four routers. Keeps request/ response models co-located and avoids circular imports. api/app.py (replaced placeholder) Registers all four routers, adds CORS middleware (origins matching the Vite dev server), and adds the latency-recording HTTP middleware. Lifespan handler auto-creates loyalty tables and starts the existing batch scoring scheduler from astroml/api/scheduler.py. api/loyalty_models.py Standalone DeclarativeBase for loyalty tables so they can be created independently of the main astroml schema migration. api/__init__.py Package marker (was missing).
[Infra] Dockerize AstroML Environment
…nto CI Issue #246 — database session and models are already implemented in api/models/orm.py and api/database.py. This commit adds comprehensive integration tests that exercise those models against an in-memory SQLite DB. Issue #244 — integration tests for API endpoints and CI pipeline: New test files: api/tests/conftest.py — enhanced with ORM seed fixtures (seeded_account, seeded_transaction, seeded_alert, seeded_loyalty) and a FastAPI TestClient with DB override api/tests/test_fraud.py — FraudAlert ORM CRUD, risk_level_for_score(), multi-alert queries, severity filtering api/tests/test_loyalty.py — LoyaltyPoints + PointsTransaction ORM tests: create, unique constraint, history ordering, net balance, filter by type api/tests/test_monitoring.py — /monitoring/* endpoint shape tests: metrics fields, history list, drift report, latency percentiles, prediction stats api/tests/test_health.py — /health 200, JSON content-type, status field CI pipeline: .github/workflows/pytest.yml — added `test-api` job running pytest api/tests/ with SQLite in-memory (no Postgres needed) Makefile — added `make test-api` target closes #244 closes #246
Complete Fraud Registry Smart Contract
…-246 feat(api): ORM-backed integration tests + API CI stage
Security audit of stealth-sender Soroban contract
…_fe-integration_docker-compose fix: transaction, history_account,fe-integration,docker-compose
feat: model registry, batch scoring scheduler, WebSocket streaming, and JWT auth
Add performance optimization guide and update index documentation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
19 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR introduces a fully containerized development and runtime environment for AstroML, enabling consistent setup across local machines, CI, and production deployments using Docker.
Changes Made
Added Dockerfile for building the AstroML application image.
Added docker-compose.yml for orchestrating services (app, database, and optional cache).
Introduced environment variable support via .env configuration.
Standardized runtime dependencies inside container.
Added health checks for application service.
Improved reproducibility of local development environment.
Problem
Previously, running AstroML required manual setup of:
system dependencies
Python/Node environment versions
database configuration
inconsistent local tooling setups
This led to:
onboarding friction
environment drift between developers
CI inconsistencies
Solution
Dockerization ensures:
identical runtime across environments
simplified onboarding (docker compose up)
isolated dependencies
reproducible builds in CI/CD pipelines
Key Features
Consistent runtime environment
Locked dependency versions
Optimized build layers for faster rebuilds
App service
Database service (e.g., PostgreSQL)
Optional Redis cache for background tasks or ML pipelines
.env driven configuration
Secure separation of secrets and runtime config
Ensures service readiness before dependency startup
Improves orchestration stability
Example Usage
docker compose up --build
Testing
Local Tests
Verified app builds successfully inside container
Confirmed database connectivity from app service
Tested hot reload in development mode
Validated environment variable injection
Integration Tests
Multi-container startup order
Service discovery between app and DB
Persistent volume storage validation
Impact
Simplified onboarding for new developers
Reduced environment-related bugs
Improved CI/CD consistency
Faster setup time (minutes instead of hours)
Better production parity
Type of Change
Infrastructure
DevOps
Developer Experience Improvement..closed [Infra] Dockerize AstroML Environment Traqora/astroml#78