skills(aibi-dashboards): backport widget index + theme guidance + audit fixes from a-d-k #558 and #561

[QuentinAmbard]

Why this PR exists

Backports two upstream PRs and their on-tree follow-up commits to the DAS databricks-aibi-dashboards experimental skill:

• ai-dev-kit#558 — "Dashboard skill: widget index + dataset measures + new widgets + forecast partial-bucket fix."
• ai-dev-kit#561 — "aibi-dashboards: theme/color guidance, audit fixes, runnable example." (tip: 18ba9f2)

Both PRs and their follow-up commits are folded into a single backport commit. The runnable verification harness (.tests/) from #561 is intentionally excluded — only the skill content is in scope here.

Upstream commit map

Upstream	What it adds
#558	Widget-index table, dataset MEASURE(), new advanced widgets, forecast partial-bucket fix.
da45f3a	queryLines verbatim-join warning + widgetBorderColor borderless theme trick.
38c4491	Cross-filtering UX rule (fewer wider datasets).
084db5f	Pure-white light-mode canvas (#FCFCFC → #FFFFFF).
76103a0	Drop format block from MEASURE()-backed counters.
#561	Theme & color section, audit fixes from 4 real dashboards, fully-grounded worked example.
a427af7	Densify palette rules + promote symbol-map to core widgets.
dce929d	Fix stale line numbers in widget index.
32aae76	Soften narrative prescription in 4-examples.md.
58d9b31	60/30/10 color-budget framing, layout sync, bare-hex mappings fix.
e6fa8c8	showDescription: true on the pie example.
18ba9f2	decimalPlaces "exact" — fix typo + add example usage.

What this PR improves

Widget index — one-stop lookup. SKILL.md opens with a table mapping every supported widget type (counter, table, line/bar, pie, area, scatter, combo, forecast-line, pivot, histogram, heatmap, funnel, sankey, box, waterfall, choropleth-map, symbol-map, all filter widgets) to its spec version and the exact reference file + section + line number where it's documented.

Theme & color guidance (from #561). New "Theme & Color (always set this — it makes or breaks the dashboard)" section: one coherent palette per dashboard, semantic colors pinned as literal hex (not palette positions), adjacent-stop contrast rule, single-color-faded-toward-white = quantitative ramp not categorical, pastel-tweak guidance, 60/30/10 color-budget framing, 8 starter palettes for first-shot color selection. Fix: palette positions are 0-indexed (was incorrectly documented as 1-indexed). Default theme matches the live Customer Support warm-sunset palette.

Dataset measures + MEASURE(). Top-level reusable aggregations on the dataset rather than per-widget. Reduces query bloat when many widgets share the same aggregation.

Cross-filtering UX rule (from 38c4491). Clicking a value cascades to every widget sharing the dataset. Splitting widgets across many narrow datasets silently breaks this. Skill leads with this so the agent designs around one wide dataset per domain.

Audit findings backfilled (from #561). frame.description / showDescription; value-formatting (number-plain, currencyCode, suffix, formatTemplate, decimalPlaces "exact" mode); MEASURE() + plain \"number\" format warning; pie label.show + showDescription:true; symbol-map bubble-map encouragement, colorRamp modes (custom-sequential preferred), mark.opacity; heatmap axis.hideTitle; forecast-line 3-CTE pattern (actuals / bridge / forecast) with explicit column-list UNION ALL to avoid NUM_COLUMNS_MISMATCH.

Advanced widgets backfilled (from #558). Forecast-line, pivot with conditional cell rules, sankey, choropleth-map, symbol-map, histogram with bin(col, binWidth=N), box, waterfall — all gain dedicated sections.

Forecast partial-bucket fix. Last forecast bucket rendering as a half-height bar — fix documented in 5-troubleshooting.md.

queryLines verbatim-join warning (from da45f3a). Array elements joined character-for-character — a line ending in -- comment with no trailing newline swallows the next line. PR #561 also adopts explicit \\n in the worked example.

Borderless theme trick (from da45f3a). widgetBorderColor set equal to widgetBackgroundColor hides the busy default border (matches the dbdemos customer-support reference dashboard).

Light-mode canvas tweak (from 084db5f). #FCFCFC (reads beige next to white widget cards) → #FFFFFF (one flat modern surface). Dark-mode pair unchanged.

MEASURE-counter format-block warning (from 76103a0). A counter whose value is a MEASURE() plus a format: {type: \"number\", ...} block triggers the browser's "unsupported widget definition was automatically fixed" warning. Omit the format for a clean integer.

Fully grounded worked example (from #561). End-to-end "Product release 4.1" incident narrative: forecast chart with vertical-line annotation at the release date, reopen-rate measure outputting 0–1 (was 0–100) so number-percent renders correctly, queryLines using explicit \\n, theme aligned to the live customer-support palette, symbol-map colorRamp with family-aligned gradient (coral → mint), semantic colors pinned as literal hex.

Cohort retention = pivot with color-scale. Explicit note that there is no cohort widget type.

Summary of changes

File	What changed
SKILL.md	Widget-index table; new Theme & Color section with palette-design rules + 60/30/10 framing + 8 starter palettes + 0-indexed-position fix; cross-filtering rationale; widgetBorderColor theme trick; light-mode canvas #FCFCFC → #FFFFFF; cohort-retention-via-pivot clarification; internal links updated to DAS's references/ layout.
references/1-widget-specifications.md	frame.description + showDescription documented; value-formatting (number-plain, currencyCode, suffix, formatTemplate, decimalPlaces "exact"); pie label.show.
references/2-advanced-widget-specifications.md	Symbol-map bubble-map / colorRamp modes / mark.opacity; heatmap axis.hideTitle; forecast-line 3-CTE pattern with column-list UNION ALL; pivot, sankey, choropleth-map, histogram, box, waterfall sections.
references/3-filters.md	Filter-widget structure section added (unchanged from #558).
references/4-examples.md	Replaced fragmented examples with one grounded "Product release 4.1" narrative dashboard; queryLines uses explicit \\n; reopen-rate measure in 0–1 range; theme aligned to live customer-support palette; semantic hex colors; pie showDescription: true; decimalPlaces "exact" example.
references/5-troubleshooting.md	Forecast partial-bucket trap (unchanged from #558).

Scope note: tests excluded

PR #561 ships a databricks-skills/databricks-aibi-dashboards/.tests/ subdirectory (test-data generator, runnable dashboard JSON, Lakeview deploy script, README, .gitignore) used to verify the worked example against synthetic data on e2-demo-field-eng. That harness is upstream-only — not included here. Only the four skill content files are in this backport.

Path remapping

Upstream files live at databricks-skills/databricks-aibi-dashboards/N-widget-foo.md; DAS uses experimental/databricks-aibi-dashboards/references/N-widget-foo.md. Content is identical; only the file location differs. SKILL.md sibling-style links to reference files rewritten to references/-prefixed. Cross-skill link to databricks-spark-declarative-pipelines (a-d-k name) renamed to databricks-pipelines (DAS name).

Reviewer aid

Upstream PRs: ai-dev-kit#558, ai-dev-kit#561. All commits taken at the 18ba9f2 tip of databricks-solutions/ai-dev-kit/dashboard-skill-polish-2. Diff is a straight content backport minus the .tests/ harness.

This pull request and its description were written by Isaac.

[simonfaltum]

Reviewed as a skill gate -- verified the load-bearing claims against Databricks docs and a local scripts/skills.py validate run, not just a read. Net: in good shape, fix-then-merge. Two items worth fixing before merge, the rest are polish -- all left inline.

Fix before merge

1. The frontmatter drops parent / compatibility / metadata.version, which rolls the published manifest version backward (0.1.0 -> 0.0.1) and breaks consistency with every other experimental skill. (inline on SKILL.md)
2. BIN_FLOOR is presented as raw SQL but isn't in the SQL function reference -- needs confirmation it's a valid dashboard expression, since the skill tells agents to CLI-test all SQL. (inline on 2-advanced)

Polish (inline): hard-coded (L###) line numbers in the widget index (one already wrong), unverified symbol-map / range-slider widgetType strings, one cloud-specific docs URL.

Verified clean

• scripts/skills.py validate passes; manifest / Codex metadata / icons / plugin manifests all in sync (the version flip is a regen side effect, not a hand-edit).
• ai_forecast signature + output columns (*_forecast / _upper / _lower, horizon / time_col / value_col) match the docs.
• Dataset columns[] + MEASURE() is a real feature (dataset-scoped custom calculations).
• The new widget types (pivot, histogram, heatmap, sankey, funnel, box, waterfall, choropleth, combo, area, scatter) are all real per the visualization-types doc.
• All widget-index anchors and cross-file links resolve.
• The databricks-spark-declarative-pipelines -> databricks-pipelines rename is correct.
• Deleting the old "do NOT use mappings / colorRamp on bar/line" rule looks right -- per-value categorical color mapping is a documented feature (2024 AI/BI release notes). Worth a quick render check of the JSON shape.

Useful backport overall -- the widget index and the single reference dashboard are genuinely good additions.

[simonfaltum]

Frontmatter dropped (fix before merge). This change removes three frontmatter fields that exist on main: parent: databricks-core, compatibility: Requires databricks CLI (>= v1.0.0), and metadata.version: "0.1.0". The PR description doesn't mention it, so it looks like the upstream ai-dev-kit frontmatter was copied over the destination's.

Two consequences:

• Version regression. scripts/skills.py derives the manifest version from metadata.version and falls back to a 0.0.1 floor when it's absent, which is why manifest.json flips 0.1.0 -> 0.0.1 (a backward version for consumers).
• Consistency. Every other experimental skill carries parent + compatibility (backfilled in experimental: backfill metadata.version + parent + compatibility frontmatter #105 six days ago); this drops them for one skill and removes the databricks-core hierarchy link.

Suggest restoring the three lines, then python3 scripts/skills.py generate so the manifest version returns to 0.1.0 (or bump to 0.2.0 for this feature release).

[simonfaltum]

This 0.1.0 -> 0.0.1 downgrade is a side effect of dropping metadata.version from the SKILL.md frontmatter (see my comment there). validate passes because the manifest faithfully regenerates from the now-missing field, but the published version still moves backward. Restoring the frontmatter and re-running scripts/skills.py generate fixes it.

[simonfaltum]

BIN_FLOOR unverified (fix before merge). It isn't in the Databricks SQL function reference (only bin, floor, histogram_numeric), so calling it "the actual SQL expression" is misleading -- and this skill's headline rule is "test ALL SQL queries via CLI before deploying," which would fail on BIN_FLOOR(...). It's plausibly a valid dashboard field-expression function (it lives in a widget expression, not dataset SQL). Could you confirm it renders (a working dashboard from the upstream PR is enough) and, if it's dashboard-expression-only, reword this so it doesn't claim to be raw SQL? Same expression is used at 4-examples.md:342.

[simonfaltum]

This replaces the previous marker-map entry. The visualization-types doc calls it a "point map," and I couldn't confirm the exact serialized widgetType string from public docs. Worth double-checking symbol-map is what the dashboard JSON actually uses (vs marker-map / point-map), since a wrong widgetType silently breaks the widget.

[simonfaltum]

Every other filter in this skill is filter--prefixed (filter-single-select, filter-multi-select, filter-date-range-picker), but this one is bare range-slider. The range-slider filter is real (confirmed in the 2024 AI/BI release notes); please just confirm the serialized widgetType is actually un-prefixed and the version is 2.

[simonfaltum]

The hard-coded (L###) hints in this index will drift on the next edit to any reference file -- and this one is already wrong: the Range Slider heading is at line 282 of 3-filters.md, not 239 (L239 is the Multi-Select Parameters section). The anchor itself resolves, so the link works; consider dropping the (L###) hints and relying on the anchors, which are stable.

[simonfaltum]

House style is cloud-neutral docs URLs -- strip the /aws/en/ prefix: https://docs.databricks.com/dashboards/manage/visualizations/types. (The line this replaced was already neutral.)

[QuentinAmbard]

@simonfaltum thanks for the deep review — all addressed in the force-pushed commit (b3ea6a5).

Fix-before-merge

1. ✅ Frontmatter restored. parent: databricks-core, compatibility: Requires databricks CLI (>= v1.0.0), and metadata.version are back. Per your suggestion bumped to "0.2.0" to reflect the feature release. Manifest regenerated; version goes 0.0.1 → 0.2.0. Root cause was that the file-recopy from a-d-k overwrote the DAS-side frontmatter — I should have merged it rather than overwriting.
2. ✅ BIN_FLOOR reworded. Confirmed against the live dashboard.lvdash.json from upstream PR #561's .tests/ (deployed to e2-demo-field-eng) — it appears as "expression": "BIN_FLOOR(...)" inside a widget, i.e. a Lakeview field-expression, not raw SQL. Updated the line in 2-advanced-widget-specifications.md accordingly so the skill no longer claims it's SQL.

Polish

3. ✅ (L###) line hints dropped across all 20 rows of the widget index. Anchors stay; line numbers go.
4. ✅ symbol-map widgetType verified — live dashboard JSON has "widgetType": "symbol-map". No change.
5. ✅ range-slider widgetType verified — live JSON has bare "widgetType": "range-slider" (no filter- prefix). It really is the odd one out. No change.
6. ✅ Cloud-prefixed URL fixed — docs.databricks.com/aws/en/... → docs.databricks.com/....

scripts/skills.py validate passes; the manifest version regression is resolved.

[dustinvannoy-db]

LGTM