Add sdp-quarantine-pattern asset (reference pipeline + companion agent skill)

[vmariiechko]

Summary

Adds sdp-quarantine-pattern, a dual-door asset to the asset library: a validated reference Lakeflow Spark Declarative Pipeline that demonstrates the inverse-expectations quarantine pattern on the public samples.nyctaxi.trips dataset, plus a companion agent skill that adapts the pattern to the user's own dataset and verifies it. Critical (drop) expectations route violating rows to a quarantine_trips table while valid rows flow to clean_trips; advisory (warn) expectations annotate the event log without dropping.

The central lesson the asset teaches is the NULL trap: expect_all_or_drop keeps a row whose predicate evaluates to SQL NULL, so a naive inverse expectation double-counts NULL rows into both tables and breaks the partition. Every drop predicate is written NULL-safe so clean and quarantine partition the input exactly once.

Changes

• New asset assets/sdp-quarantine-pattern/: pipeline notebook, declarative expectations.json, pure expectations.py helper (loads rules, derives the clean predicate and its inverse), DABs pipeline resource with a published event log and event_log_queries.sql, source parameterization via quarantine.source, and an in-bundle usage doc.
• Offline unit-test suite shipped inside the asset (<target_dir>/tests/): a real local-Spark pytest that proves the partition invariant and NULL routing on crafted edge rows, modeling expect_all_or_drop keep-on-NULL semantics with IS NOT FALSE.
• Companion agent skill at <skill_dir>/skills/sdp-quarantine-pattern/ (SKILL.md + references/adapt-the-pattern.md + references/self-verify.md): enforces NULL-safe drop predicates, splits work at a deploy/run trust boundary (Phase A adapt + offline unit tests with no workspace; Phase B gated deploy + live-verify, default human-in-the-middle), and ships a three-tier verification ladder.
• Repo-level tests tests/assets/test_sdp_quarantine_pattern.py and config tests/configs/assets/sdp_quarantine_pattern.json.
• Catalog and docs: ASSETS.md, ROADMAP.md, CHANGELOG.md ([1.9.0]), pyproject.toml (Google docstring convention).

Change Area

• Asset Library (assets/<name>/)

Configuration Axes Affected

• Unity Catalog / schemas
• Template schema (databricks_template_schema.json)
• Asset Library (new asset, asset schema, or framework changes)

Testing

• All tests pass (pytest tests/ -V)
• Manual template generation tested (databricks bundle init . --template-dir assets/sdp-quarantine-pattern)
• New tests added for new functionality (if applicable)

Additionally, the offline unit-test suite was run with a real local Spark session (PySpark 4.x, JDK 17) and the pattern was validated live on serverless SDP: the partition invariant held exactly (21,847 clean + 85 quarantine = 21,932 raw at baseline) and a NULL drop column routed to quarantine without leaking into clean.

Asset Changes (if applicable)

• Asset installs standalone via databricks bundle init . --template-dir assets/<name> --output-dir <dir>
• Asset is self-contained (no references to library/helpers.tmpl or other assets)
• tests/configs/assets/<name>.json added
• Asset appears in ASSETS.md catalog

Checklist

• Go template syntax is valid (no unclosed {{ }} blocks)
• No .tmpl files appear in generated output
• Generated YAML files are valid
• Documentation updated (if behavior changed)