Document updates & License headers

.env.example

@@ -0,0 +1,61 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# Sample environment file for the Custos backend.
+# Copy to .env, fill in real values, and `source .env` (or use a dotenv loader).
+# The portal has its own env template at web/.env.example.
+
+# --- Server: required ----------------------------------------------------
+
+# MySQL/MariaDB DSN. Must allow multi-statements for migrations.
+DATABASE_DSN=admin:admin@tcp(localhost:3306)/custos?parseTime=true&charset=utf8mb4&multiStatements=true
+
+# OIDC issuer + audience, into config/custos.yaml.
+# Server refuses to start without these once auth is wired.
+OIDC_ISSUER_URL=
+OIDC_AUDIENCE=
+
+# --- Server: dev convenience ---------------------------------------------
+
+# UUID of the compute cluster this Custos instance serves.
+# Shared across connectors that need to know which cluster they belong to.
+CUSTOS_CLUSTER_ID=00000000-0000-0000-0000-000000000001
+
+# --- AMIE connector (ACCESS-CI) ------------------------------------------
+
+AMIE_BASE_URL=http://localhost:8180
+AMIE_SITE_CODE=TESTSITE
+AMIE_API_KEY=dev
+AMIE_CLUSTER_ID=00000000-0000-0000-0000-000000000001
+
+# --- COmanage connector --------------------------------------------------
+
+COMANAGE_REGISTRY_URL=
+COMANAGE_CO_ID=
+COMANAGE_API_USER=
+COMANAGE_API_KEY=
+COMANAGE_UNIX_CLUSTER_ID=
+COMANAGE_PERSON_ID_TYPE=
+
+# --- SLURM connector -----------------------------------------------------
+
+SLURM_API=
+SLURM_USER=
+SLURM_TOKEN=
+SLURM_API_VERSION=
+# Used only by the Usage-Monitor connector.
+SLURM_MONITOR_CLUSTER_ID=

.gitignore

@@ -66,4 +66,11 @@ raft.db
 config.yaml
 
 # Binaries
-bin/
+bin/
+
+# Python virtualenvs (e.g. AMIE mock-server)
+venv/
+.venv/
+
+# Local working drafts
+docs/drafts/

CONFIG.md

@@ -1,3 +1,22 @@
+<!--
+    Licensed to the Apache Software Foundation (ASF) under one
+    or more contributor license agreements.  See the NOTICE file
+    distributed with this work for additional information
+    regarding copyright ownership.  The ASF licenses this file
+    to you under the Apache License, Version 2.0 (the
+    "License"); you may not use this file except in compliance
+    with the License.  You may obtain a copy of the License at
+
+      http://www.apache.org/licenses/LICENSE-2.0
+
+    Unless required by applicable law or agreed to in writing,
+    software distributed under the License is distributed on an
+    "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+    KIND, either express or implied.  See the License for the
+    specific language governing permissions and limitations
+    under the License.
+-->
+
 # Configuration Guide
 
 Custos server can be configured using a YAML configuration file instead of environment variables. This guide explains how to set up and use the configuration system.

CONTRIBUTING.md

@@ -1,3 +1,22 @@
+<!--
+    Licensed to the Apache Software Foundation (ASF) under one
+    or more contributor license agreements.  See the NOTICE file
+    distributed with this work for additional information
+    regarding copyright ownership.  The ASF licenses this file
+    to you under the Apache License, Version 2.0 (the
+    "License"); you may not use this file except in compliance
+    with the License.  You may obtain a copy of the License at
+
+      http://www.apache.org/licenses/LICENSE-2.0
+
+    Unless required by applicable law or agreed to in writing,
+    software distributed under the License is distributed on an
+    "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+    KIND, either express or implied.  See the License for the
+    specific language governing permissions and limitations
+    under the License.
+-->
+
 # Contributing
 
 Thanks for your interest in contributing to Apache Airavata Custos. This document covers the repository layout, build commands, coding conventions, and the pull request workflow.
@@ -16,7 +35,7 @@ For instructions on running Custos locally against a database, see [INSTALL.md](
 - `pkg/` — public packages (models, service, events)
 - `connectors/` — protocol/site-specific connectors (ACCESS, SLURM, …)
 - `extensions/` — out-of-process extensions (PAM module, SSH cert signer, …)
-- `dev-ops/compose/` — local Docker Compose stack (MariaDB, Keycloak, Vault, Prometheus, Grafana)
+- `dev-ops/compose/` — local Docker Compose stack (MariaDB, Adminer, Prometheus, Grafana, Vault)
 
 ## Build
 
@@ -53,18 +72,50 @@ Every SQL file must carry the Apache 2.0 license header (see existing files for
 
 ## Tests
 
-Run the full test suite from the repo root:
+Two tiers: unit tests and integration tests, separated by a build tag.
+
+### Unit tests (default, no external services)
 
 ```bash
 go test ./...
 ```
 
-Connector packages (for example `connectors/SLURM/Association-Mapper/...`) use `httptest` and do not require external services.
+Runs every package's unit tests. Handlers and stores use `httptest` and mocks.
+No DB, no network. This is what should pass on every commit.
+
+### Integration tests (build tag, real services)
+
+Integration test files carry `//go:build integration` at the top, so the default
+`go test ./...` skips them silently. To run them, pass the tag and provide the
+required env vars:
+
+```bash
+# General integration tests (server, core service, identity resolver).
+# Reuses the dev compose stack on :3306.
+export CORE_TEST_DATABASE_DSN='admin:admin@tcp(localhost:3306)/custos?parseTime=true&charset=utf8mb4&multiStatements=true'
+go test -tags integration ./...
+```
+
+The AMIE connector ships its own integration stack because its tests mutate a
+lot of state and we don't want to corrupt the dev DB. It brings up an isolated
+MariaDB on `:3307` and a mock AMIE server on `:8181`, runs the suite, then
+tears it down:
+
+```bash
+make integration-test-amie
+```
+
+(equivalent to `scripts/run-amie-integration-tests.sh`). The `:3307` is
+deliberate. It's a separate stack defined in `dev-ops/local-amie/`, independent
+of the `:3306` dev compose.
+
+If you see a test file you expect to run but `go test ./...` reports zero tests
+for the package, check the first line for `//go:build integration`.
 
 ## Submitting changes
 
 1. Open an issue describing the change, if one does not already exist.
-2. Create a topic branch off `main`.
+2. Create a topic branch off `master`.
 3. Make focused, well-scoped commits with clear messages.
 4. Ensure `go build ./...`, `go vet ./...`, and `go test ./...` all pass.
 5. Open a pull request and link the related issue.

INSTALL.md

@@ -1,3 +1,22 @@
+<!--
+    Licensed to the Apache Software Foundation (ASF) under one
+    or more contributor license agreements.  See the NOTICE file
+    distributed with this work for additional information
+    regarding copyright ownership.  The ASF licenses this file
+    to you under the Apache License, Version 2.0 (the
+    "License"); you may not use this file except in compliance
+    with the License.  You may obtain a copy of the License at
+
+      http://www.apache.org/licenses/LICENSE-2.0
+
+    Unless required by applicable law or agreed to in writing,
+    software distributed under the License is distributed on an
+    "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+    KIND, either express or implied.  See the License for the
+    specific language governing permissions and limitations
+    under the License.
+-->
+
 # Local Installation
 
 This guide covers running Apache Airavata Custos locally for development and testing. For coding conventions, build commands, and contribution workflow, see [CONTRIBUTING.md](CONTRIBUTING.md).
@@ -35,7 +54,6 @@ From the repository root:
 
 ```bash
 export DATABASE_DSN='admin:admin@tcp(localhost:3306)/custos?parseTime=true&charset=utf8mb4&multiStatements=true'
-export HTTP_ADDR=':8080'
 go run ./cmd/server
 ```
 
@@ -44,20 +62,30 @@ On startup, the server:
 1. Opens the database connection (`internal/db`).
 2. Runs the embedded migrations (`internal/db/migrations/`).
 3. Wires the event bus, service layer, and HTTP router.
-4. Listens on `HTTP_ADDR`.
+4. Listens on the port from `config/custos.yaml` (`core.api.port`, default `8080`).
+
+### 3. Seed dev users (optional)
+
+The schema is created by the server's migrations on first run. After that, apply
+the dev seed to populate sample identity data (org, users, roles, privileges):
+
+```bash
+docker exec -i custos_db mariadb -uadmin -padmin custos \
+  < dev-ops/compose/seeds/dev_users_and_roles.sql
+```
+
+The seed uses `INSERT IGNORE`, so re-running is safe. See the seed file for
+exactly what it inserts.
 
 ### Environment variables
 
-| Variable             | Default                                                            | Description                                  |
-|----------------------|--------------------------------------------------------------------|----------------------------------------------|
-| `DATABASE_DSN`       | _(required)_                                                       | Go MySQL DSN; must allow multi-statements    |
-| `HTTP_ADDR`          | `:8080`                                                            | Listen address for the HTTP API              |
-| `DB_MAX_OPEN_CONNS`  | `25`                                                               | Max open DB connections                      |
-| `DB_MAX_IDLE_CONNS`  | `5`                                                                | Max idle DB connections                      |
+See [.env.example](.env.example) for the template. Copy it to `.env`, fill in
+the required values (`DATABASE_DSN`, `OIDC_ISSUER_URL`, `OIDC_AUDIENCE`), and
+source it before running the server.
 
 ### Optional services
 
-The Compose file also defines Keycloak, Vault, Adminer, Prometheus, and Grafana. Start them as needed, for example:
+The Compose file also defines Adminer, Prometheus, Grafana, and Vault. Start them as needed, for example:
 
 ```bash
 docker compose up -d adminer       # DB UI at http://localhost:18080

README.md

@@ -65,34 +65,25 @@ The shape generalizes: a new connector that needs to record connector-specific r
 
 ## Quick Start
 
-Clone the repository:
-
 ```sh
 git clone https://github.com/apache/airavata-custos.git
 cd airavata-custos
 ```
 
-Start the backing services (MariaDB, Prometheus, Grafana, Vault):
-
-```sh
-cd dev-ops/compose
-docker compose up -d
-```
+See [INSTALL.md](INSTALL.md) to bring up the dev stack and run the server. See each connector's and extension's README for run and configuration details.
 
-Build and test a connector, e.g. ACCESS-CI AMIE:
-
-```sh
-cd connectors/ACCESS/AMIE-Processor
-go build ./...
-go test ./...
-```
+## Documentation
 
-See each connector's and extension's README for run and configuration details.
+- [INSTALL.md](INSTALL.md): run the server locally against the dev compose stack
+- [CONTRIBUTING.md](CONTRIBUTING.md): coding conventions, build, and test workflow
+- [docs/API-Docs.md](docs/API-Docs.md): REST API reference
+- [docs/Allocation-Data-Models.md](docs/Allocation-Data-Models.md): domain model overview
+- [docs/ACCESS-HPC-Reference.md](docs/ACCESS-HPC-Reference.md): ACCESS-CI integration reference
 
 ## Questions or Need Help?
 
 * Open a [GitHub issue](https://github.com/apache/airavata-custos/issues)
-* Subscribe to the Custos mailing list: `custos-subscribe@airavata.apache.org`
+* Join the [Airavata dev mailing list](https://airavata.apache.org/mailing-list.html)
 
 ## Publications
 

config/custos.yaml

@@ -1,3 +1,20 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
 core:
   database:
     url: "${DATABASE_DSN}"