feat(bigquery): use native UNPIVOT for pivot_longer

[dlstadther]

Description of changes

Table.pivot_longer currently compiles to a generic struct-pack / array / unnest expression on every backend, including BigQuery, even though BigQuery has a native UNPIVOT operator for exactly this shape of query. This PR adds a BigQuery-specific compilation path that uses UNPIVOT when possible, matching the existing pattern of backend-specific rewrites via LOWERED_OPS.

• ops.PivotLonger is now a real relational op (instead of being eagerly expanded into struct/array/unnest at expression-build time), so a backend can special-case it during compilation. to_generic() on the op still produces the original struct/array/unnest tree and remains the implementation for every other backend.
• On BigQuery, a pivot_longer call with a single names_to column compiles to a native UNPIVOT. A call with more than one names_to column (e.g. from a multi-group names_pattern) still falls back to the generic implementation, since UNPIVOT (in both its single-column and multi-column forms) only ever produces a single name column — it can't express multiple names_to columns.
• Two BigQuery-specific correctness issues had to be handled to make this a safe drop-in replacement:
  • UNPIVOT's IN (...) list requires every column to share the exact same type (no implicit widening), so pivot columns of different-but-unifiable types (e.g. int64 + float64) are now explicitly cast to the pivoted column's unified type before entering the IN list.
  • UNPIVOT's physical output column order is always [...passthrough, value_col, name_col], which does not match PivotLonger's declared schema order of [...passthrough, name_col, value_col]. Since results are mapped to schema positionally, this silently swapped the name/value columns' data. The generated SQL now selects columns explicitly in schema order rather than relying on SELECT *.
• Added values_drop_na: bool = False to pivot_longer. When True, rows where the resulting values_to column is NULL are dropped (result.drop_null(values_to) in the generic implementation). This is implemented once in to_generic(), so every backend gets it for free. On BigQuery it maps to UNPIVOT's native EXCLUDE NULLS modifier instead. pivot_longer keeps NULLs by default (values_drop_na=False) on every backend, which is the opposite of UNPIVOT's own default (EXCLUDE NULLS) — the generated BigQuery SQL always states INCLUDE NULLS or EXCLUDE NULLS explicitly, so it never silently relies on BigQuery's default.

All of the above (native UNPIVOT compilation, the type-cast fix, the column-order fix, and values_drop_na) were also verified against a live BigQuery project, comparing results row-for-row against pandas.melt(), in addition to the unit tests included here.

This is a drop-in replacement: output is identical to the previous struct/array/unnest implementation for BigQuery users, and there is no behavior change for any other backend.

---

This addresses an active query performance issue I'm hitting in production use of Ibis against BigQuery. AI (Claude) was used to aid in scoping and implementing these changes, including the live BigQuery testing referenced above.

[deepyaman]

This looks reasonable at a high level. Can you do me a favor and try and check that the added code is idiomatic and aligned in style with the rest of the repo? Some of the AI-coding things stick out even at a glance, like the massive comments and notes (which you won't see throughout the rest of the repo); haven't looked deeply, but this will help reduce some churn.

Other question that will come up with is, which over backends could benefit from switching to a native UNPIVOT? I imagine it's not just BigQuery; haven't looked, and not asking you to change other backends in this PR (if anything, maybe this PR scoped to BigQuery will be easier to review), but it will be helpful to have an answer for which backends adding the native op could potentially benefit.

I'll try to get to reviewing this properly once these changes are in, but preemptively forgive the fact that there will probably be some delay.

[dlstadther]

Thanks for your quick initial response @deepyaman . I'll go back through this proposal and ensure it aligns in style with rest of Ibis. I'll also trim unnecessary comments at the same time.

As for scope of changes and other areas that could benefit from UNPIVOT, I suspect there are other candidates, but I wish to keep this PR scoped to what is minimally required to support the backend override + BigQuery + address failing tests (which is why there are adjustments to polars here).

[dlstadther]

Went back through and trimmed the AI-generated bloat:

• Comments/docstrings: cut the multi-paragraph "notes" style comments in bigquery/__init__.py, expr/operations/relations.py, expr/types/relations.py, and backends/polars/compiler.py down to match the terse, why-only comment convention used elsewhere in the codebase.
• Naming: renamed lower_pivot_longer_bigquery -> pivot_longer_to_unpivot. No other backend-specific @replace rewrite in the codebase uses a lower_ prefix (reserved for shared rewrites.py/base.py factories) or a redundant backend-name suffix — local ones are named for what they do (offset_to_filter, rewrite_rows_range_order_by_window, etc).
• Imports: PivotLonger.to_generic() did import ibis then ibis.struct(...)/ibis.array(...). Narrowed to from ibis.expr.types import array, struct (the functions' actual home), matching the existing precedent in ops/core.py for this same circular-import constraint (confirmed a true top-level import isn't possible here — ibis.expr.api pulls in ibis.expr.types -> ibis.expr.builders -> ops before ops finishes loading).

Left PivotLonger.schema as self.to_generic().schema() rather than computing dtypes directly — happy to discuss further if you'd rather see it changed.

Still scoped to BigQuery only per the earlier discussion; the "which other backends could benefit" question is open for whenever you get to a full review.

---

I have not checked on usefulness for other backends yet. Though, i still would NOT advocate to expand the scope of this PR beyond BQ + necessary for tests/compatibility.

[dlstadther]

Hi @deepyaman , checking in to see if you (or any other maintainer) have opportunity to review

[deepyaman]

Sorry, it's on my list, but honestly I will try to get to it next week (doubtful this week, due to work travel).

…

On Thu, Sep 17, 2026, 3:00 PM Dillon Stadther ***@***.***> wrote: *dlstadther* left a comment (ibis-project/ibis#12091) <#12091 (comment)> Hi @deepyaman <https://github.com/deepyaman> , checking in to see if you (or any other maintainer) have opportunity to review — Reply to this email directly, view it on GitHub <#12091?email_source=notifications&email_token=ADK3W3QGH2D3XI3E47AUODL5PQYFTA5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTKNZRHE3DSMRRHAY2M4TFMFZW63VHNVSW45DJN5XKKZLWMVXHJLDGN5XXIZLSL5RWY2LDNM#issuecomment-5719692181>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADK3W3VLJRAFKFZPPUMUDZD5PQYFTAVCNFSNUABEKJSXA33TNF2G64TZHMZTIMJTHEZDGMB3JFZXG5LFHM2TENZSGI2DEMJTGSQXMAQ> . Triage notifications, keep track of coding agent tasks and review pull requests on the go with GitHub Mobile for iOS <https://github.com/notifications/mobile/ios/ADK3W3WSAR3DSGMMZMHWBBD5PQYFTA5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTKNZRHE3DSMRRHAY2M4TFMFZW63VHNVSW45DJN5XKKZLWMVXHJKTGN5XXIZLSL5UW64Y> and Android <https://github.com/notifications/mobile/android/ADK3W3RW76PRINPVC3W34TL5PQYFTA5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTKNZRHE3DSMRRHAY2M4TFMFZW63VHNVSW45DJN5XKKZLWMVXHJLTGN5XXIZLSL5QW4ZDSN5UWI>. Download it today! You are receiving this because you were mentioned.Message ID: ***@***.***>