Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions ASSETS.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,7 @@ Run the command from the root of your bundle project. The CLI will prompt for an
|---|---|---|---|
| `sdp-checkpoint-recovery` | Reset checkpoint selection on a Lakeflow Spark Declarative Pipeline after a source table has been dropped and recreated. | Stable | [README](assets/sdp-checkpoint-recovery/README.md) |
| `dbx-ro-query` | Dependency-free Python wrapper around `databricks experimental aitools tools query` that gives LLM agents a guarded read-only SQL window into a Databricks workspace. Ships a `SKILL.md` for agent integration. | Stable | [README](assets/dbx-ro-query/README.md) |
| `monitoring-sql-warehouse` | Small, dedicated serverless SQL warehouse (2X-Small, `auto_stop_mins: 1`) for scheduled Databricks Alerts and monitoring queries. Keeps cost proportional to actual query time instead of idle warm-up. | Stable | [README](assets/monitoring-sql-warehouse/README.md) |

## What an asset is not

Expand Down
10 changes: 10 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,15 @@ and this project adheres to [Semantic Versioning](https://semver.org/).

## [Unreleased]

## [1.8.0] - 2026-05-30

### Added
- **Asset `monitoring-sql-warehouse`**: Declarative Automation Bundle resource defining a small, dedicated serverless SQL warehouse tuned for bursty workloads (scheduled Databricks Alerts, monitoring queries, ad-hoc health checks).
- Defaults to `cluster_size: 2X-Small`, `warehouse_type: PRO`, `enable_serverless_compute: true`, and `auto_stop_mins: 1` (the serverless minimum verified end-to-end via DABs deploy; sub-minute values silently revert to the platform default).
- Channel pinned to `CHANNEL_NAME_CURRENT` so Databricks SQL Preview-channel rollouts cannot change query behavior under monitoring workloads. Cost-attribution tags `workload=monitoring-alerts` and `created_by=dabs-asset/monitoring-sql-warehouse` ship by default for traceability in usage reports.
- Five prompts (`target_dir`, `warehouse_resource_key`, `warehouse_name`, `cluster_size`, `auto_stop_mins`) with safe defaults; the resource file lands at `<target_dir>/<warehouse_resource_key>.sql_warehouse.yml` and is picked up by the conventional `resources/*.yml` include glob without any `databricks.yml` change.
- In-bundle usage doc at `docs/monitoring-sql-warehouse/README.md` covers the auto-stop nuance, cross-resource ID reference pattern, and `databricks warehouses edit --auto-stop-mins` / REST API paths for editing warehouses created outside DABs.

## [1.7.1] - 2026-05-13

### Fixed
Expand Down Expand Up @@ -176,6 +185,7 @@ Initial public release.
- L2 tests: YAML syntax, environment targets, content validation
- CI/CD tests: pipeline generation, auth patterns, branch references

[1.8.0]: https://github.com/vmariiechko/databricks-bundle-template/releases/tag/v1.8.0
[1.7.0]: https://github.com/vmariiechko/databricks-bundle-template/releases/tag/v1.7.0
[1.6.0]: https://github.com/vmariiechko/databricks-bundle-template/releases/tag/v1.6.0
[1.5.0]: https://github.com/vmariiechko/databricks-bundle-template/releases/tag/v1.5.0
Expand Down
2 changes: 2 additions & 0 deletions ROADMAP.md
Original file line number Diff line number Diff line change
Expand Up @@ -47,6 +47,8 @@ Modular sub-templates installable via `databricks bundle init <repo-url> --templ

**Shipped:**
- `sdp-checkpoint-recovery`: Reset checkpoint selection on a Lakeflow Spark Declarative Pipeline after a source table has been dropped and recreated
- `dbx-ro-query`: Dependency-free Python wrapper around `databricks experimental aitools tools query` that gives LLM agents a guarded read-only SQL window into a Databricks workspace
- `monitoring-sql-warehouse`: Dedicated serverless SQL warehouse tuned for bursty workloads (scheduled Alerts, monitoring queries) with `auto_stop_mins: 1`

**Possible future assets** (ideas for contributors, not commitments):
- `etl-pipeline`: Medallion-layered Declarative Pipeline with Bronze/Silver layers and DLT expectations
Expand Down
33 changes: 33 additions & 0 deletions assets/monitoring-sql-warehouse/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
# monitoring-sql-warehouse

A small, dedicated serverless SQL warehouse tuned for bursty workloads: scheduled Databricks Alerts, monitoring queries, and ad-hoc health checks. Defaults to `2X-Small` with `auto_stop_mins: 1` so cost stays proportional to actual query time.

## Install

```bash
databricks bundle init https://github.com/vmariiechko/databricks-bundle-template \
--template-dir assets/monitoring-sql-warehouse
```

You will be prompted for:

| Prompt | Default | Notes |
|---|---|---|
| `target_dir` | `resources` | Where the resource YAML lands. Default matches the `resources/*.yml` glob most bundles already include. |
| `warehouse_resource_key` | `monitoring_sql_warehouse` | DABs key under `resources.sql_warehouses.<key>`. Also the resource filename. |
| `warehouse_name` | `Monitoring SQL Warehouse` | Display name in the workspace SQL Warehouses list. |
| `cluster_size` | `2X-Small` | One of: 2X-Small, X-Small, Small, Medium, Large, X-Large, 2X-Large, 3X-Large, 4X-Large. |
| `auto_stop_mins` | `1` | Integer. `1` is the serverless minimum; `0` disables auto-stop; `10+` for Pro/Classic. |

Two files are installed:

- `<target_dir>/<warehouse_resource_key>.sql_warehouse.yml`: the DABs resource definition (serverless PRO, single cluster, channel pinned to `CHANNEL_NAME_CURRENT`, tagged `workload=monitoring-alerts` and `created_by=dabs-asset/monitoring-sql-warehouse`).
- `docs/monitoring-sql-warehouse/README.md`: usage notes, the `auto_stop_mins: 1` serverless nuance, and CLI/API paths for editing existing warehouses outside DABs.

## Usage

After install, open `docs/monitoring-sql-warehouse/README.md` in your project for deploy steps, the cross-resource reference pattern (`${resources.sql_warehouses.<key>.id}`), and the auto-stop behavior reference table.

## What this asset is

A standalone sub-template in the [databricks-bundle-template](https://github.com/vmariiechko/databricks-bundle-template) asset library. It does not depend on the core template; it can be installed into any Databricks bundle. See [ASSETS.md](../../ASSETS.md) for the full catalog.
55 changes: 55 additions & 0 deletions assets/monitoring-sql-warehouse/databricks_template_schema.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,55 @@
{
"welcome_message": "\nMonitoring SQL Warehouse: Asset Installer\n\nInstalls a single Declarative Automation Bundle resource file that\ndefines a small, dedicated serverless SQL warehouse tuned for bursty\nworkloads (scheduled Databricks Alerts, monitoring queries, ad-hoc\nhealth checks).\n\nWhy it exists: Alerts and monitoring queries are sub-second, but a\ngeneral-purpose warehouse on a long auto-stop window stays warm for\nminutes after each run. Idle time dominates the bill. A dedicated\n2X-Small serverless warehouse with auto_stop_mins=1 keeps the cost\nproportional to actual query time.\n\nThe asset is opinionated about serverless PRO: the 1-minute auto-stop\nis only valid on serverless. Pro/Classic stay at the 10-minute floor.\n\nLet's pick the install settings...\n",

"properties": {
"target_dir": {
"type": "string",
"default": "resources",
"description": "\nTarget directory for the warehouse resource file (relative to your bundle root).\nMost bundles already include `resources/*.yml` from databricks.yml, so the\ndefault `resources` lands the file where the bundle will auto-pick it up.\n\nIf your bundle uses a different convention (e.g., `resources/sql_warehouses/`),\nset that here. The directory will be created if it does not exist.\ntarget_dir",
"order": 1,
"pattern": "^[A-Za-z0-9_][A-Za-z0-9_./-]*$",
"pattern_match_failure_message": "Target directory must start with a letter, number, or underscore and contain only letters, numbers, underscores, slashes, hyphens, or dots."
},

"warehouse_resource_key": {
"type": "string",
"default": "monitoring_sql_warehouse",
"description": "\n========================================\n\nDABs resource key for the warehouse.\n\nThis key appears in the bundle as `resources.sql_warehouses.<key>` and is\nalso the filename: `<target_dir>/<key>.sql_warehouse.yml`. Other bundle\nresources reference this warehouse by `${resources.sql_warehouses.<key>.id}`.\n\nDefault `monitoring_sql_warehouse` is safe for most projects. Pick another\nname only if your bundle already defines a warehouse with this key.\nMust be lowercase snake_case (letters, numbers, underscores).\nwarehouse_resource_key",
"order": 2,
"pattern": "^[a-z][a-z0-9_]*$",
"pattern_match_failure_message": "Resource key must be lowercase, start with a letter, and contain only letters, numbers, and underscores."
},

"warehouse_name": {
"type": "string",
"default": "Monitoring SQL Warehouse",
"description": "\n========================================\n\nDisplay name shown in the Databricks workspace SQL Warehouses list.\n\nThis is what users see in the UI when picking a warehouse for an Alert\nor query. Free-form; spaces and mixed case are fine.\nwarehouse_name",
"order": 3,
"pattern": "^.+$",
"pattern_match_failure_message": "Warehouse name cannot be empty."
},

"cluster_size": {
"type": "string",
"default": "2X-Small",
"description": "\n========================================\n\nCluster size for the warehouse.\n\nValid Databricks SQL warehouse sizes (smallest to largest):\n 2X-Small, X-Small, Small, Medium, Large, X-Large, 2X-Large, 3X-Large, 4X-Large, 5X-Large\n\nDefault `2X-Small` is the smallest and cheapest. It is sufficient for\nDatabricks Alerts and monitoring queries that complete in under a second.\nScale up only if your queries scan large tables.\ncluster_size",
"order": 4,
"pattern": "^(2X-Small|X-Small|Small|Medium|Large|X-Large|2X-Large|3X-Large|4X-Large|5X-Large)$",
"pattern_match_failure_message": "Cluster size must be one of: 2X-Small, X-Small, Small, Medium, Large, X-Large, 2X-Large, 3X-Large, 4X-Large, 5X-Large."
},

"auto_stop_mins": {
"type": "string",
"default": "1",
"description": "\n========================================\n\nIdle minutes before the warehouse auto-stops (integer).\n\nDefault `1` is the serverless minimum verified via DABs deploy: the UI\nshows 'After 1 minute of inactivity'. Sub-minute values are NOT supported\n(the field is modeled as an integer).\n\nSpecial values:\n 0 = disable auto-stop entirely (warehouse runs until manually stopped)\n 1 = serverless minimum (recommended for Alerts / bursty workloads)\n 10+ = required floor for Pro/Classic warehouses; safe upper bound for serverless too\n\nKeep at 1 unless you have a reason to override.\nauto_stop_mins",
"order": 5,
"pattern": "^(0|[1-9][0-9]*)$",
"pattern_match_failure_message": "auto_stop_mins must be a non-negative integer (0 to disable, 1 for serverless minimum, 10+ for Pro/Classic)."
}
},

"success_message": "\n========================================\n\nMonitoring SQL warehouse installed:\n - Resource: {{.target_dir}}/{{.warehouse_resource_key}}.sql_warehouse.yml\n - Usage doc: docs/monitoring-sql-warehouse/README.md\n\nNext steps:\n\n1. Confirm your `databricks.yml` includes the resource file. Most bundles\n already include `resources/*.yml` by default; verify:\n include:\n - resources/*.yml\n If you installed to a custom subdirectory, add the matching glob.\n\n2. Validate and deploy:\n databricks bundle validate -t <your-target>\n databricks bundle deploy -t <your-target>\n\n3. After deploy, the warehouse appears in the workspace SQL Warehouses list\n as '{{.warehouse_name}}'. Point your Alerts / monitoring queries at it.\n\n4. Reference the warehouse ID from other bundle resources via:\n ${resources.sql_warehouses.{{.warehouse_resource_key}}.id}\n\nUsage notes, the auto_stop_mins=1 nuance, and the CLI/API paths for\nexisting warehouses live in 'docs/monitoring-sql-warehouse/README.md'.\n",

"min_databricks_cli_version": "v0.296.0",
"version": 1
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,133 @@
# Monitoring SQL Warehouse

A small, dedicated serverless SQL warehouse for scheduled Databricks Alerts, monitoring queries, and ad-hoc health checks. Optimized for bursty, sub-second workloads where idle time would otherwise dominate the bill.

## When to use this

Use a dedicated warehouse like this when:

- You run scheduled Databricks Alerts (every minute, every five minutes, every hour) whose underlying SQL completes in well under a second.
- The default warehouse those queries land on has a long auto-stop window (10 minutes is the platform default for Pro/Classic; serverless allows 1 minute).
- You see the warehouse staying warm for the full auto-stop window after a single sub-second query, inflating monthly cost.

A separate `2X-Small` serverless warehouse with `auto_stop_mins: 1` keeps cost proportional to actual query time: the warehouse starts on the Alert run, executes in under a second, and stops one minute after the last query.

## What this asset installs

| Path | Purpose |
|---|---|
| `<target_dir>/<warehouse_resource_key>.sql_warehouse.yml` | The DABs SQL warehouse resource definition. |
| `docs/monitoring-sql-warehouse/README.md` | This file. |

Default install paths:

- Resource: `resources/monitoring_sql_warehouse.sql_warehouse.yml`
- Docs: `docs/monitoring-sql-warehouse/README.md`

## Bundle integration

Most generated bundles already include `resources/*.yml` from `databricks.yml`:

```yaml
include:
- resources/*.yml
```

If your bundle uses that pattern and you accepted the default `target_dir`, the warehouse is picked up automatically. No `databricks.yml` change is needed.

If you installed to a custom subdirectory (e.g., `resources/sql_warehouses/`), add the matching glob to `databricks.yml`:

```yaml
include:
- resources/sql_warehouses/*.yml
```

## Deploy

```bash
databricks bundle validate -t <your-target>
databricks bundle deploy -t <your-target>
```

After deploy, the warehouse appears in the workspace SQL Warehouses list under the display name you chose at install time.

## Referencing the warehouse from other bundle resources

Other resources (Alerts, jobs, dashboards) can pin to this warehouse by ID:

```yaml
${resources.sql_warehouses.<warehouse_resource_key>.id}
```

For example, an Alert resource:

```yaml
resources:
alerts:
my_alert:
warehouse_id: ${resources.sql_warehouses.monitoring_sql_warehouse.id}
# ... rest of the alert definition
```

## The `auto_stop_mins: 1` nuance

The proven configuration was verified by deploying this asset to a real workspace (Databricks SQL channel v2026.10). The notes below capture behavior that contradicts or refines parts of the public API reference.

| Setting | Behavior |
|---|---|
| `auto_stop_mins: 0` | Disables auto-stop entirely. The warehouse runs until manually stopped. Does NOT mean "stop immediately." |
| `auto_stop_mins: 1` | Valid and effective on serverless via DABs/API. The workspace UI shows "After 1 minute of inactivity." This is the recommended value for Alerts and monitoring workloads. |
| `auto_stop_mins: 0.5` (or any sub-minute value) | Not supported. The field is modeled as an integer; the bundle CLI emits an integer-coercion warning and the workspace silently reverts to the platform default (120 minutes). |
| `auto_stop_mins: 10+` | Per the public [Warehouses API reference][create-warehouse-api], the documented contract is `auto_stop_mins` must be `0` or `>= 10`. This floor applies to Pro/Classic warehouses. Safe upper bound for serverless too. |

The public [Warehouses API reference][create-warehouse-api] states that `auto_stop_mins` must be `0` or `>= 10`. That floor applies to Pro/Classic warehouses; the serverless path accepts `1` end-to-end via DABs and via the CLI (confirmed by deploying this resource and inspecting the workspace UI).

The manual UI editor for an existing warehouse may enforce a higher floor on serverless than 1 minute. If you see that, fall back to the DABs path or the `databricks warehouses edit` CLI path, both of which apply `1` on serverless.

[create-warehouse-api]: https://docs.databricks.com/api/workspace/warehouses/create
[edit-warehouse-api]: https://docs.databricks.com/api/workspace/warehouses/edit

## Applying the same tuning to a warehouse created outside DABs

If you have an existing warehouse that you do not want to recreate as a DABs resource, you can edit it in place to the same auto-stop behavior.

### Via the Databricks CLI

```bash
databricks warehouses edit <warehouse-id> --auto-stop-mins 1
```

The full flag set is visible via `databricks warehouses edit --help`. For serverless, also ensure `--enable-serverless-compute` is on and `--warehouse-type PRO` is set.

### Via the REST API

Use the [Edit Warehouse endpoint][edit-warehouse-api] (`POST /api/2.0/sql/warehouses/{id}/edit`) with `auto_stop_mins: 1`. For serverless, include `enable_serverless_compute: true` and `warehouse_type: PRO` in the request body. The endpoint reference still documents the `0 or >= 10` contract; the serverless `1` value is accepted in practice and is what DABs and the CLI emit.

```bash
curl -X POST \
-H "Authorization: Bearer $DATABRICKS_TOKEN" \
-H "Content-Type: application/json" \
https://$DATABRICKS_HOST/api/2.0/sql/warehouses/<warehouse-id>/edit \
-d '{
"auto_stop_mins": 1,
"enable_serverless_compute": true,
"warehouse_type": "PRO"
}'
```

### Via the workspace UI

The "Edit warehouse" form may enforce a higher minimum than 1 minute for serverless. If the UI rejects `1`, use the CLI or DABs path.

## Cost intuition

A `2X-Small` serverless warehouse billed per second of running time, kept warm by a 1-minute auto-stop after each sub-second query, costs a small constant per scheduled run plus the query duration itself. Compared to a 10-minute auto-stop window on the same size warehouse, the per-run dwell time drops by an order of magnitude. Apply this to dozens of scheduled Alerts and the monthly delta is meaningful.

## References

1. [Databricks Docs: Configure SQL warehouse settings (auto-stop)](https://docs.databricks.com/aws/en/compute/sql-warehouse/warehouse-behavior)
2. [Declarative Automation Bundles: `sql_warehouses` resource reference](https://docs.databricks.com/aws/en/dev-tools/bundles/resources.html#sql_warehouse)
3. [Databricks API: Create Warehouse][create-warehouse-api] (`auto_stop_mins` contract)
4. [Databricks API: Edit Warehouse][edit-warehouse-api] (in-place edits to an existing warehouse)
5. [Databricks CLI: `databricks warehouses edit`](https://docs.databricks.com/aws/en/dev-tools/cli/reference/warehouses-commands.html#edit)
6. [Databricks Alerts overview](https://docs.databricks.com/aws/en/sql/user/alerts/)
Original file line number Diff line number Diff line change
@@ -0,0 +1,40 @@
# Monitoring SQL Warehouse
#
# Small, dedicated serverless SQL warehouse tuned for bursty workloads:
# scheduled Databricks Alerts, monitoring queries, ad-hoc health checks.
#
# Why a dedicated warehouse: a general-purpose warehouse on a long
# auto-stop window stays warm for minutes after a sub-second Alert
# query, and idle time dominates the bill. A 2X-Small serverless
# warehouse with auto_stop_mins=1 keeps cost proportional to actual
# query time.
#
# Reference this warehouse from other bundle resources via:
# ${resources.sql_warehouses.{{.warehouse_resource_key}}.id}
#
# See docs/monitoring-sql-warehouse/README.md for usage notes,
# the auto_stop_mins=1 serverless nuance, and CLI/API alternatives
# for warehouses created outside DABs.

resources:
sql_warehouses:
{{.warehouse_resource_key}}:
name: {{.warehouse_name}}
cluster_size: {{.cluster_size}}
warehouse_type: PRO
enable_serverless_compute: true
auto_stop_mins: {{.auto_stop_mins}}
min_num_clusters: 1
max_num_clusters: 1
# Pin to the stable Databricks SQL channel. Prevents Preview-channel
# feature flags from changing query behavior under monitoring queries
# that are assumed-stable. Override to CHANNEL_NAME_PREVIEW only if
# you are intentionally validating an upcoming release.
channel:
name: CHANNEL_NAME_CURRENT
tags:
custom_tags:
- key: workload
value: monitoring-alerts
- key: created_by
value: dabs-asset/monitoring-sql-warehouse
Loading
Loading