|
| 1 | +Diagram a SQL query and explain what it shows — either its execution steps or its column lineage. |
| 2 | + |
| 3 | +## Steps |
| 4 | + |
| 5 | +1. Get the SQL. If the user named a file, use it. If the query is embedded in Python (most of this |
| 6 | + repo's SQL lives in f-strings under `scripts/`), extract it to a scratch `.sql` file first and |
| 7 | + **replace the interpolated placeholders with literals** — `sqlglot` parses SQL, not f-strings. |
| 8 | +2. Pick the mode. `mode=plan` (the default) answers "what does this query *do*, step by step"; |
| 9 | + `mode=lineage` answers "where does this output column come from". When the user asks about |
| 10 | + joins, stages, filters or ordering, they want `plan`. |
| 11 | +3. Run `make sql-diagram sql=<path> name=<basename> comments=1` via Bash. It writes three files to |
| 12 | + `reports/sql-diagram/`: `<basename>.sql` (the query as analysed), `.mmd` and `.svg` — all |
| 13 | + gitignored, so `git add -f` them only if they are meant to be a committed example. Pass |
| 14 | + `--stdout` to `scripts/sql_diagram.py` for a throwaway look with no files written. |
| 15 | +4. Read the `.mmd`, show it in a ```mermaid fence, and explain it (see below). The `.svg` is the |
| 16 | + same graph for linking from prose where no Mermaid renderer is available. |
| 17 | + |
| 18 | +The emitted `.sql` is what makes the diagram auditable: it is the query *after* any f-string |
| 19 | +placeholders were filled in, so `make sql-diagram sql=reports/sql-diagram/<basename>.sql` reproduces |
| 20 | +the diagram exactly. When you commit a diagram as an example, commit its `.sql` with it. |
| 21 | + |
| 22 | +Both diagrams come from the parsed AST, so they are exactly what the query says — do not "improve" |
| 23 | +one by adding a node or edge you believe should be there. If it looks wrong, the query is the thing |
| 24 | +to question. |
| 25 | + |
| 26 | +## Reading `mode=plan` |
| 27 | + |
| 28 | +Nodes are the query's steps, bottom-up: `SCAN` per table, one `JOIN n` per individual join, then |
| 29 | +`WHERE`, `AGGREGATE`, `SORT`, `OUTPUT`. A CTE appears as its own sub-pipeline feeding the `SCAN` |
| 30 | +that reads it. |
| 31 | + |
| 32 | +- **Each join is numbered in the order the query writes it** and carries its side and keys. |
| 33 | + `sqlglot` models a multi-table join as one n-ary step; the script splits it back apart. An |
| 34 | + `INNER JOIN` silently drops rows where a `LEFT JOIN` keeps them — always say which, because it |
| 35 | + changes what a blank in the output means. |
| 36 | +- **Extra `ON` predicates beyond the equality keys** are listed under the keys as `and …`. On a |
| 37 | + slowly-changing dimension those range predicates are what stop the join fanning out; call them |
| 38 | + out rather than treating them as noise. |
| 39 | +- **This is the logical plan, not the physical one.** Databricks reorders joins, chooses broadcast |
| 40 | + versus shuffle, and prunes columns. Say "as written" — and if the real execution matters, point |
| 41 | + at the query profile in the UI or `EXPLAIN FORMATTED`, which is the only authority on what ran. |
| 42 | +- `AGGREGATE` may show synthetic operand names (`_a_0`) for `DISTINCT`/expression arguments that |
| 43 | + `sqlglot` lifted out. Read the intent off the original SQL rather than repeating the placeholder. |
| 44 | + |
| 45 | +## Reading `mode=lineage` |
| 46 | + |
| 47 | +- **Subgraphs are source tables**, one node per source column actually read. A column the query |
| 48 | + never touches does not appear — that is the point. |
| 49 | +- **The `output` subgraph** is the projected column list, in select order. |
| 50 | +- **`(unqualified)`** collects columns referenced without a table prefix in a multi-table join. |
| 51 | + `sqlglot` will not guess which side they came from without the table schemas, and neither should |
| 52 | + you. Call it out: it is usually a readability defect in the query worth fixing at the source. |
| 53 | +- **Struct columns collapse to their root.** `u.usage_metadata.job_id` traces back to |
| 54 | + `usage_metadata`, not to the leaf field. Say so rather than implying field-level precision. |
| 55 | +- Columns in `WHERE`/`GROUP BY` but not in the output do **not** appear. Use `mode=plan` when |
| 56 | + filtering is the point. |
| 57 | + |
| 58 | +## In either mode |
| 59 | + |
| 60 | +**The grey line under a table name** is its Unity Catalog comment, present only when the run passed |
| 61 | +`comments=1` and the profile could read the table. It is fetched, never written by you — if a table |
| 62 | +has no comment the space is blank, and that absence is itself worth reporting. |
| 63 | + |
| 64 | +## What to say about it |
| 65 | + |
| 66 | +- The **shape** of the query first: how many tables, how many joins, what it groups by. A reader |
| 67 | + who cannot restate the query after your first paragraph has learned nothing. |
| 68 | +- Any source column or table feeding **many** outputs — the query's hub, where a schema change has |
| 69 | + the widest blast radius. |
| 70 | +- Any table contributing **only one or two** columns, especially through a `LEFT JOIN`. That is |
| 71 | + often a lookup that could be a smaller subquery, and a join whose only job is one column is a |
| 72 | + cheap thing to get wrong. |
| 73 | +- Join predicates that look under-constrained. A join on a slowly-changing dimension without a |
| 74 | + time-range predicate fans rows out and silently multiplies aggregates — this repo has been bitten |
| 75 | + by exactly that (see the `#47` entry in `specs/CHANGELOG.md`). |
| 76 | + |
| 77 | +## Limits worth stating rather than hiding |
| 78 | + |
| 79 | +- `SELECT *` errors out in lineage mode by design — tracing it needs the table schemas, which the |
| 80 | + script does not have. Plan mode draws it fine. |
| 81 | +- Lineage mode also refuses a query whose output projects the same column name twice |
| 82 | + (`SELECT a.id, b.id`): `sqlglot` resolves lineage by name and would trace both to the first |
| 83 | + match, drawing a confident wrong graph. Alias them, or use plan mode. |
| 84 | +- `CREATE TABLE … AS SELECT` and `INSERT … SELECT` are unwrapped to their SELECT and diagrammed. |
| 85 | + Anything with no SELECT at all (a `DELETE`, a DDL statement) exits with a one-line message. |
| 86 | +- Dialect defaults to `databricks`; pass `--dialect` to `scripts/sql_diagram.py` directly for others. |
| 87 | +- CTEs resolve through to their base tables, but a query reading a **view** stops at the view name; |
| 88 | + the view's own definition is not expanded. |
| 89 | +- `--comments` is the only part that touches the network, and it uses the `dev` profile: the MCP |
| 90 | + service principal lacks `USE SCHEMA` on `system.billing`. |
0 commit comments