feat(docker): establish a Z3 Docker Compose setup and guidelines

.cursor/rules/docker-compose-configuration-guidelines.mdc

@@ -0,0 +1,48 @@
+---
+description: 
+globs: 
+alwaysApply: true
+---
+## Docker Compose Configuration Guidelines
+
+This document outlines best practices for configuring multi-service applications using Docker Compose, drawing from the setup for Zebra, Zaino, and Zallet.
+
+**Key Files:**
+- Environment Variables: [`z3/.env`](mdc:zaino/z3/.env)
+- Docker Compose Configuration: [`z3/docker-compose.yml`](mdc:zaino/z3/docker-compose.yml)
+- Zebra Entrypoint Script: [`zebra/docker/entrypoint.sh`](mdc:zaino/zebra/docker/entrypoint.sh)
+- Zaino Entrypoint Script: [`zaino/docker/entrypoint.sh`](mdc:zaino/zaino/docker/entrypoint.sh)
+- Placeholder for Docker `configs`: [`z3/.cookie`](mdc:zaino/z3/.cookie) (example)
+
+**Core Principles:**
+
+1.  **Centralized Environment Variables (`.env`)**:
+    *   Store common and service-specific configurable parameters in a single `.env` file (e.g., [`z3/.env`](mdc:zaino/z3/.env)). This includes network settings, ports, log levels, and feature flags.
+    *   For shared resources like an RPC authentication cookie directory, define a single global variable in the `.env` file (e.g., `COOKIE_AUTH_FILE_DIR=/var/run/auth`). This directory should be an FHS-compliant path for transient runtime data.
+
+2.  **Respecting Application Entrypoints in `docker-compose.yml`**:
+    *   Each service in [`z3/docker-compose.yml`](mdc:zaino/z3/docker-compose.yml) should have its environment variables set according to what its entrypoint script (e.g., [`zebra/docker/entrypoint.sh`](mdc:zaino/zebra/docker/entrypoint.sh), [`zaino/docker/entrypoint.sh`](mdc:zaino/zaino/docker/entrypoint.sh)) expects.
+    *   If an entrypoint script requires a specific variable (e.g., `ZEBRA_COOKIE_DIR` for Zebra, `ZAINO_VALIDATOR_COOKIE_PATH` for Zaino), these should be set in the `environment` section of the service definition.
+    *   The values for these service-specific variables should be derived from the global variables in the `.env` file. For example:
+        *   Zebra's `ZEBRA_COOKIE_DIR` could be `${COOKIE_AUTH_FILE_DIR}`.
+        *   Zaino's `ZAINO_VALIDATOR_COOKIE_PATH` could be `${COOKIE_AUTH_FILE_DIR}/.cookie`.
+
+3.  **Using Docker `configs` for Shared Runtime Data**:
+    *   For runtime-generated files that need to be shared between services (like RPC cookie files), use Docker `configs`.
+    *   Define a top-level `configs:` block in [`z3/docker-compose.yml`](mdc:zaino/z3/docker-compose.yml), sourcing from a placeholder file on the host (e.g., `file: ./.cookie`).
+        ```yaml
+        configs:
+          cookie_auth_config: # Descriptive name for the shared config object
+            file: ./.cookie
+        ```
+    *   Mount this Docker `config` into each relevant service:
+        *   The service generating the file (e.g., Zebra) mounts the config to the full path where its entrypoint expects to write the file (e.g., `target: ${COOKIE_AUTH_FILE_DIR}/.cookie`).
+        *   The service consuming the file (e.g., Zaino) mounts the same config to the full path where its entrypoint expects to read the file (e.g., `target: ${COOKIE_AUTH_FILE_DIR}/.cookie`).
+    *   This ensures both services are looking at the same underlying data, managed by Docker. The initial placeholder file allows Docker Compose to create the config object.
+
+4.  **FHS-Compliant Paths**:
+    *   For transient runtime data generated within containers (like cookie files or PID files), prefer FHS-compliant directories such as `/var/run/<appname>/` or `/run/<appname>/`.
+
+5.  **Clear Naming Conventions**:
+    *   Use descriptive names for environment variables (e.g., `COOKIE_AUTH_FILE_DIR` instead of generic names).
+    *   Name Docker `config` objects clearly to reflect their purpose and shared nature (e.g., `cookie_auth_config`).

.cursor/rules/z3_docker_orchestration.mdc

@@ -0,0 +1,26 @@
+---
+description: 
+globs: 
+alwaysApply: true
+---
+# Z3 Docker Orchestration Overview
+
+This project uses Docker Compose to orchestrate the Zebra, Zaino, and Zallet services. The main orchestration file is [z3/docker-compose.yml](mdc:z3/docker-compose.yml).
+
+## Key Components:
+
+*   **`docker-compose.yml`**: Defines the services (Zebra, Zaino, Zallet), their build contexts (if building locally), images, volumes for data persistence, networks, and mechanisms for specific configuration data (like Docker `configs` for TLS certificates and volume mounts for identity files). It also specifies dependencies between services to manage startup order.
+*   **`z3/config/` Directory**: This host directory is for user-provided files that are explicitly used by the Docker setup. This includes:
+    *   `[z3/config/zallet_identity.txt](mdc:z3/config/zallet_identity.txt)`: A crucial `age` identity file for Zallet wallet encryption, which must be generated by the user. This file is volume-mounted into the Zallet container.
+    *   TLS certificates (e.g., `[z3/config/tls/zaino.crt](mdc:z3/config/tls/zaino.crt)` and `[z3/config/tls/zaino.key](mdc:z3/config/tls/zaino.key)`): These are sourced by Docker `configs` and made available to services like Zaino.
+    *   It's important to note that general service configuration files like `zebra.toml`, `zaino.toml`, or `zallet.toml` are **not** placed in `z3/config/` by the user to be mounted for overriding base service configurations in this Z3 setup. Services use their internal defaults, and operational parameters are set via environment variables.
+*   **`.env` File**: The `[z3/.env](mdc:z3/.env)` file is the **exclusive method for users to customize operational parameters** for the Zebra, Zaino, and Zallet services. This includes settings like network choice, ports, log levels, and feature flags. Docker Compose injects these variables into the respective service containers.
+*   **README**: The main instructions for setup and running are in `[z3/README.md](mdc:z3/README.md)`.
+
+## Service Interaction:
+
+*   **Zebra** acts as the consensus node.
+*   **Zaino** connects to Zebra to index blockchain data.
+*   **Zallet** connects primarily to Zaino for indexed data and wallet operations. It might also interact directly with Zebra for certain functionalities.
+
+Networking between services is handled by a custom Docker bridge network (`z3_net`) defined in `[z3/docker-compose.yml](mdc:z3/docker-compose.yml)`. Services can reach each other using their service names as hostnames (e.g., `zebra`, `zaino`, `zallet`).

.cursor/rules/z3_gitignore_config_rules.mdc

@@ -0,0 +1,50 @@
+---
+description: 
+globs: 
+alwaysApply: false
+---
+# .gitignore Rules for the z3/config/ Directory
+
+The `[z3/.gitignore](mdc:z3/z3/.gitignore)` file contains specific rules to manage the contents of the `z3/config/` directory and its subdirectories (like `z3/config/tls/`). The goal is to generally ignore user-generated configuration files or sensitive data placed here, while ensuring that the directory structure itself (including empty directories needed for Docker mounts or `configs`) can be version-controlled using `.gitkeep` files.
+
+## Key Patterns in `z3/.gitignore` for `config/`:
+
+1.  **General Ignore for `config/` Contents:**
+    ```gitignore
+    config/**
+    ```
+    This rule ignores all files and subdirectories recursively within `z3/config/`.
+
+2.  **Un-ignoring the `tls` Subdirectory (to look inside):**
+    ```gitignore
+    !config/tls/
+    ```
+    This allows Git to consider rules pertaining to the contents of `z3/config/tls/`.
+
+3.  **Ignoring Contents of `tls` (before specific un-ignores):**
+    ```gitignore
+    config/tls/*
+    ```
+    After un-ignoring the `tls` directory itself, this rule ensures its contents are ignored by default, allowing for specific un-ignores next.
+
+4.  **Un-ignoring `.gitkeep` in `config/tls/`:**
+    ```gitignore
+    !config/tls/.gitkeep
+    ```
+    This explicitly prevents `[z3/config/tls/.gitkeep](mdc:z3/z3/config/tls/.gitkeep)` from being ignored, allowing the `z3/config/tls/` directory to be tracked even if empty.
+
+5.  **Un-ignoring `.gitkeep` in `config/`:**
+    ```gitignore
+    !config/.gitkeep
+    ```
+    This explicitly prevents `[z3/config/.gitkeep](mdc:z3/z3/config/.gitkeep)` from being ignored, allowing the `z3/config/` directory to be tracked even if empty.
+
+**Summary of Behavior:**
+
+*   All files and folders within `z3/config/` are ignored by default.
+*   The `z3/config/tls/` directory is specifically managed to allow its `.gitkeep` file to be tracked.
+*   The `z3/config/` directory itself has its `.gitkeep` file tracked.
+*   Any other files (e.g., `config/my_secret.txt`, `config/tls/extra_cert.pem`) will be ignored by Git and should not be committed unless a specific negation rule (`!config/...`) is added for them.
+*   Note: The `[z3/config/zallet_identity.txt](mdc:z3/z3/config/zallet_identity.txt)` file, while crucial, is covered by `config/**` and will be ignored. It is user-generated and should not be committed to the repository.
+
+These rules ensure that the necessary directory structure for configurations (especially for Docker `configs` which might require specific source paths) can be maintained in the Git repository without tracking actual sensitive configuration files or runtime data within them.

.cursor/rules/z3_security_and_networking.mdc

@@ -0,0 +1,52 @@
+---
+description: 
+globs: 
+alwaysApply: false
+---
+# Z3 Service Security and Networking Configuration Guide
+
+This document outlines the security policies and networking configurations for the Z3 services (Zebra, Zaino, Zallet) as implemented in the `z3/docker-compose.yml` and related entrypoint scripts.
+
+## Core Security Policies:
+
+1.  **Zebra <-> Zaino (Validator Connection):**
+    *   Zaino connects to Zebra's RPC. If Zebra's address (`ZAINO_VALIDATOR_LISTEN_ADDRESS`) is not a loopback address from Zaino's perspective (e.g., using the service name `zebra:${ZEBRA_RPC_PORT}`), Zaino's internal policy mandates that cookie authentication must be used.
+    *   To comply:
+        *   The `ENABLE_COOKIE_AUTH` environment variable (in `[.env](mdc:zaino/zaino/zaino/z3/.env)`) must be set to `true`.
+        *   Zebra (`[entrypoint.sh](mdc:zaino/zaino/zaino/z3/entrypoint.sh)`) must be configured to enable cookie authentication and generate a `.cookie` file.
+        *   Zaino (`[entrypoint.sh](mdc:zaino/zaino/zaino/zaino/docker/entrypoint.sh)`) must be configured to use cookie authentication and read the shared cookie file.
+        *   A shared cookie mechanism is used:
+            *   A host file `./.cookie` is defined as a Docker `config` named `cookie_auth_config`.
+            *   This config is mounted into both `zebra` and `zaino` containers at `${COOKIE_AUTH_FILE_DIR}/.cookie`. The `COOKIE_AUTH_FILE_DIR` variable (defined in `[.env](mdc:zaino/zaino/zaino/z3/.env)`) should specify an FHS-compliant path (e.g., `/var/run/auth`).
+            *   **Note on Writability**: If Zebra generates the cookie, it needs to write to this file. Docker `configs` are typically read-only. The user must ensure this setup allows Zebra to manage the cookie file when `ENABLE_COOKIE_AUTH=true`.
+    *   **Authentication:**
+        *   **Cookie-based (Primary method if `ENABLE_COOKIE_AUTH=true` in `[.env](mdc:zaino/z3/.env)`):**
+            *   If `ZAINO_VALIDATOR_COOKIE_AUTH_ENABLE` is `true` (derived from `${ENABLE_COOKIE_AUTH}`), Zaino reads Zebra's RPC cookie from the path specified by `ZAINO_VALIDATOR_COOKIE_PATH` (e.g., `${COOKIE_AUTH_FILE_DIR}/.cookie`).
+            *   Zaino expects this file to contain the cookie string, typically in the format `__cookie__:<base64_encoded_random_string>`.
+            *   Zaino then uses this cookie to authenticate to Zebra by sending an HTTP `Authorization` header with the value `Basic __cookie__:<base64_encoded_random_string>`. The `__cookie__:` part acts as the username, and the rest as the password in Basic Authentication terms.
+            *   This mechanism relies on Zebra generating the cookie file and Zaino having read access to it via a shared Docker `config` mount.
+        *   **User/Password-based (Basic Auth):** If `ZAINO_VALIDATOR_COOKIE_AUTH_ENABLE` is `false`, Zaino can use HTTP Basic Authentication with `ZAINO_VALIDATOR_USER` and `ZAINO_VALIDATOR_PASSWORD` (if these are provided).
+        *   **No Authentication:** If cookie auth is disabled and user/pass are empty, Zaino attempts to connect without authentication, relying on Zebra to permit such connections.
+
+2.  **Zaino's gRPC Server (Zaino <-> Zallet Connection):**
+    *   Zaino's gRPC server listens on `ZAINO_GRPC_LISTEN_ADDRESS`. If this address is `0.0.0.0` (or any non-loopback/non-private IP from Zaino's perspective), Zaino's internal policy requires its gRPC service to use TLS.
+    *   To comply:
+        *   The `ZAINO_GRPC_TLS_ENABLE` environment variable (in `[.env](mdc:zaino/zaino/zaino/z3/.env)`) must be set to `true`.
+        *   TLS certificate and key must be provided to Zaino. These are supplied as Docker `configs` (e.g., `zaino_tls_cert` and `zaino_tls_key`), sourced from host files (e.g., `./config/tls/zaino.crt` and `./config/tls/zaino.key`).
+        *   The Docker configs are mounted into Zaino at FHS-compliant paths defined by environment variables in `[.env](mdc:zaino/zaino/zaino/z3/.env)` (e.g., `ZAINO_TLS_CERT_CONTAINER_PATH=/var/run/zaino/tls/zaino.crt`, `ZAINO_TLS_KEY_CONTAINER_PATH=/var/run/zaino/tls/zaino.key`).
+        *   Zaino's internal configuration then uses these paths, set via `ZAINO_GRPC_TLS_CERT_PATH=${ZAINO_TLS_CERT_CONTAINER_PATH}` and `ZAINO_GRPC_TLS_KEY_PATH=${ZAINO_TLS_KEY_CONTAINER_PATH}` in `docker-compose.yml`.
+        *   The user is responsible for generating these TLS certificate files on the host.
+
+3.  **Zallet's Connection to Zaino's gRPC Server:**
+    *   Since Zaino's gRPC service will be TLS-enabled, Zallet must connect to it using TLS and must trust Zaino's server certificate.
+    *   **Challenge**: Standard Zallet configuration (`[entrypoint.sh](mdc:zaino/zaino/zaino/zaino/zaino/zaino/wallet/docker/entrypoint.sh)`) does not have explicit, documented options for configuring TLS trust (e.g., specifying a CA certificate or trusting a self-signed server certificate) for its indexer connection.
+    *   The user needs to investigate and implement how Zallet will trust Zaino's TLS certificate. This might involve custom modifications to Zallet's container or finding undocumented configuration methods.
+
+## Docker Compose Configuration (`[docker-compose.yml](mdc:zaino/zaino/zaino/zaino/zaino/zaino/z3/docker-compose.yml)`):
+
+*   Environment variables from `[.env](mdc:zaino/zaino/zaino/zaino/zaino/zaino/z3/.env)` are used extensively to control these security features and other service parameters.
+*   Zaino's healthcheck uses `curl` and requires additional environment variables (`ZAINO_GRPC_TLS_ENABLE_SCHEME_SUFFIX`, `ZAINO_GRPC_TLS_ENABLE_CURL_OPT`) to adapt to TLS settings. These should be defined in `[.env](mdc:zaino/zaino/zaino/zaino/zaino/zaino/z3/.env)`.
+
+## Entrypoint Scripts:
+
+*   The entrypoint scripts for `zebra` (`[entrypoint.sh](mdc:zaino/zaino/zaino/zaino/zaino/zaino/z3/entrypoint.sh)`), `zaino` (`[entrypoint.sh](mdc:zaino/zaino/zaino/zaino/zaino/zaino/zaino/docker/entrypoint.sh)`), and `zallet` (`[entrypoint.sh](mdc:zaino/zaino/zaino/zaino/zaino/zaino/wallet/docker/entrypoint.sh)`) have been modified to handle directory ownership (`chown`) failures more gracefully by warning instead of exiting, particularly for directories that might contain read-only mounted files (like configs). However, `mkdir -p` failures will still cause an exit.

.cursor/rules/z3_service_zaino.mdc

@@ -0,0 +1,52 @@
+---
+description: 
+globs: 
+alwaysApply: false
+---
+# Z3 Service: Zaino
+
+This rule describes the Zaino service configuration within the Z3 Docker stack, focusing on its interaction with Zebra.
+
+## Docker Configuration (`docker-compose.yml`):
+
+*   **Image:** `ghcr.io/zcashfoundation/zaino:sha-54e0da6` (specific SHA).
+*   **Container Name:** `z3_zaino`.
+*   **Depends On:** `zebra` (waits for Zebra to be healthy).
+*   **Environment Variables (relevant to Zebra connection):
+    *   `ZAINO_NETWORK`: Specifies the Zcash network, must match Zebra (e.g., `${NETWORK_NAME:-Testnet}`).
+    *   `ZAINO_VALIDATOR_LISTEN_ADDRESS`: Address of the Zebra RPC service (e.g., `zebra:${ZEBRA_RPC_PORT:-18232}`).
+    *   `ZAINO_VALIDATOR_COOKIE_AUTH_ENABLE`: Boolean to enable/disable cookie auth for connecting to Zebra (e.g., `${ENABLE_COOKIE_AUTH:-false}`).
+    *   `ZAINO_VALIDATOR_COOKIE_PATH`: Path to Zebra's RPC cookie file *if cookie auth is enabled*. Default is `${ZEBRA_DATA_DIR}/.cookie` which resolves to `/home/zebra/.cache/zebra/.cookie` inside the Zaino container if `ZEBRA_DATA_DIR` is not overridden from its default. (e.g. `/zebra_rpc_cookies/.cookie` if using a shared volume for cookies).
+    *   `ZAINO_VALIDATOR_USER`: RPC username for Zebra (if using basic auth).
+    *   `ZAINO_VALIDATOR_PASSWORD`: RPC password for Zebra (if using basic auth).
+*   **Volumes:**
+    *   `zaino_data:/home/zaino/.cache/zaino`: Persists Zaino's indexer database.
+    *   `./config/zaino.toml:/home/zaino/.config/zaino/zindexer.toml:ro`: Mounts the Zaino configuration file [z3/config/zaino.toml](mdc:zaino/z3/config/zaino.toml).
+    *   (Conditional) If Zebra uses cookie auth, a shared volume for the cookie file would be mounted here, for example: `- zebra_cookies:/zebra_rpc_cookies:ro`.
+*   **Healthcheck:** Uses `curl` to check its own gRPC port (`http://127.0.0.1:8137/`).
+
+## Source and Build (if building locally):
+
+*   **Dockerfile:** [zaino/docker/Dockerfile](mdc:zaino/zaino/docker/Dockerfile)
+*   **Entrypoint Script:** [zaino/docker/entrypoint.sh](mdc:zaino/zaino/docker/entrypoint.sh)
+    *   Generates `zindexer.toml` (by default at `/home/zaino/.config/zaino/zindexer.toml`) based on environment variables if the file is not present.
+    *   Handles setting up default paths and user permissions.
+
+## Configuration File (`config/zaino.toml` or generated by entrypoint):
+
+*   Contains settings for connecting to the validator (Zebra), its own gRPC/JSON-RPC server settings, database paths, and network type.
+*   Key section for Zebra connection: `validator_listen_address`, `validator_cookie_auth`, `validator_cookie_path`, `validator_user`, `validator_password`.
+
+## Connection to Zebra:
+
+*   Zaino connects to Zebra primarily via **JSON-RPC over HTTP**.
+*   The connection details are specified in Zaino's configuration (`zindexer.toml`), typically generated or influenced by environment variables like `ZAINO_VALIDATOR_LISTEN_ADDRESS`.
+*   **Authentication:**
+    *   **Cookie-based:** If `ZAINO_VALIDATOR_COOKIE_AUTH_ENABLE` is `true`, Zaino reads Zebra's RPC cookie from `ZAINO_VALIDATOR_COOKIE_PATH` and uses it for an `Authorization: Basic __cookie__:<base64_cookie_content>` header.
+    *   **User/Password-based (Basic Auth):** If `ZAINO_VALIDATOR_COOKIE_AUTH_ENABLE` is `false`, Zaino uses HTTP Basic Authentication with `ZAINO_VALIDATOR_USER` and `ZAINO_VALIDATOR_PASSWORD`.
+    *   **No Authentication:** If cookie auth is disabled and user/pass are empty, Zaino attempts to connect without authentication, relying on Zebra to permit such connections.
+*   **Core Logic:** The connection is managed by `JsonRpSeeConnector` in [zaino/zaino-fetch/src/jsonrpsee/connector.rs](mdc:zaino/zaino/zaino-fetch/src/jsonrpsee/connector.rs), which uses `reqwest`.
+
+## Role in Z3 Stack:
+
+Zaino indexes blockchain data fetched from Zebra, providing an efficient query layer for clients like Zallet. It relies on Zebra for raw block data, transaction information, and tree states.