Automation

Enterprise-grade infrastructure-as-code repository for deploying and managing OSDU (Open Subsurface Data Universe) and Trackonomy platforms on Microsoft Azure.

Overview

This repository serves as the central hub for infrastructure provisioning, CI/CD pipelines, Helm charts, Terraform modules, and operational scripts. It supports multi-environment deployments (development, staging, pre-production, and production) with multi-tenant capabilities.

Repository Structure

automation/
├── infra-azure-provisioning/     # Main OSDU infrastructure IaC
│   ├── infra/                    # Terraform modules & templates
│   ├── tools/                    # Initialization & data seeding tools
│   ├── charts/                   # Helm charts for Kubernetes
│   ├── devops/                   # CI/CD pipeline definitions
│   ├── source/                   # Custom tools and microservices
│   └── docs/                     # Extensive documentation
│
├── cloud-devops/                 # Cloud-specific DevOps configurations
│   ├── bicep/                    # Azure Bicep templates
│   ├── terraform/                # Terraform configurations
│   ├── charts/                   # Helm charts
│   └── scripts/                  # Utility scripts
│
├── multi-tenant-devops/          # Multi-tenant platform automation
│   ├── iaac/                     # Infrastructure as Code
│   ├── azure-pipelines/          # Azure DevOps pipelines
│   └── charts/                   # Helm charts
│
├── mtmaps/                       # Architecture diagrams & visualization
│   └── [C4 Model diagrams, technical documentation]
│
├── adops/                        # Azure DevOps specific automation
│
└── scripting/                    # Azure management shell scripts

Key Features

• Infrastructure as Code - Terraform modules for all Azure resources with reusable templates
• Multi-Environment Support - Development, staging, pre-production, and production configurations
• Kubernetes Deployment - AKS cluster provisioning with Helm chart management and auto-scaling
• Security & Access Control - Azure AD integration, RBAC, Key Vault secrets management
• Multi-Tenant Architecture - Tenant isolation with per-partition entitlements
• CI/CD Automation - Support for Azure DevOps, GitHub Actions, and GitLab CI
• Monitoring & Observability - Azure Monitor, Log Analytics, and Application Insights integration
• Disaster Recovery - Automated backup policies for CosmosDB and Storage

Technologies

Category	Technologies
IaC	Terraform, Azure Bicep, HCL
Cloud	Microsoft Azure (AKS, CosmosDB, PostgreSQL, Key Vault, Service Bus, Event Grid)
Containers	Docker, Kubernetes, Helm
Service Mesh	Istio
Databases	PostgreSQL, CosmosDB, Redis, SQL Server
CI/CD	Azure DevOps, GitHub Actions, GitLab CI
Languages	Go, Python, Bash

Prerequisites

• Terraform >= 1.11.2
• Go >= 1.24 (for testing)
• Python >= 3.5.3 (for initialization tools)
• Azure CLI with appropriate permissions
• kubectl and Helm for Kubernetes operations

Getting Started

1. Clone the Repository

git clone <repository-url>
cd automation

2. Configure Environment

Set up environment variables using direnv:

cp .envrc.template .envrc
# Edit .envrc with your Azure subscription details
direnv allow

3. Initialize Terraform

Navigate to the desired infrastructure template and initialize:

cd infra-azure-provisioning/infra/templates/osdu-r3-mvp/central_resources
terraform init

4. Deploy Infrastructure

terraform plan -var-file=terraform.tfvars
terraform apply -var-file=terraform.tfvars

Infrastructure Templates

The main templates in infra-azure-provisioning/infra/templates/osdu-r3-mvp/:

Template	Purpose
central_resources	Common infrastructure (networking, storage, vaults)
service_resources	Shared service infrastructure
data_partition	Multi-tenant partition resources
monitoring_resources	Observability stack (optional)

Terraform Modules

Core Azure modules available in infra/modules/providers/azure/:

• aks - Azure Kubernetes Service
• cosmosdb - Document database
• keyvault - Secrets management
• network - VNet and networking
• postgreSQL - Relational database
• redis-cache - In-memory caching
• service-bus - Messaging queue
• storage-account - Blob storage

Helm Charts

Charts are organized across modules for deploying:

• osdu-common - Common OSDU services
• osdu-istio - Service mesh configuration
• osdu-istio-auth - Authentication for service mesh
• airflow / airflow2 - Workflow orchestration

Scripts

Utility scripts in scripting/ for common Azure operations:

Script	Purpose
az-aks-admin.sh	AKS admin role assignments
az-kv.sh	Key Vault management
az-storage.sh	Storage account operations
az-contri.sh	Contributor role management

Architecture Documentation

The mtmaps/ directory contains enterprise-grade architecture diagrams following the C4 Model:

• Context diagrams
• Container diagrams
• Deployment architecture
• Data flow architecture
• Network & security architecture

CI/CD Pipelines

Pipelines are available for multiple platforms:

• GitLab CI - infra-azure-provisioning/.gitlab-ci.yml
• Azure DevOps - multi-tenant-devops/azure-pipelines/
• GitHub Actions - cloud-devops/.github/workflows/

Documentation

Detailed documentation is available in:

• infra-azure-provisioning/README.md - Deployment guide
• infra-azure-provisioning/docs/ - Setup guides, migrations, troubleshooting
• mtmaps/ENTERPRISE_DIAGRAMS_README.md - Architecture diagrams guide

Contributing

Please read infra-azure-provisioning/CONTRIBUTING.md for contribution guidelines.

License

This project is licensed under the Apache 2.0 License - see the LICENSE file for details.