fix(trino): parse string-to-json casts with JSON_PARSE - #12100
Open
gabrielecalvo wants to merge 1 commit into
Open
gabrielecalvo wants to merge 1 commit into
gabrielecalvo wants to merge 1 commit into
Conversation
`CAST(varchar AS JSON)` in Trino wraps the string as a JSON string
scalar rather than parsing its contents, so `.cast("json")` on a
column of JSON text produced a JSON string and every downstream field
access silently returned NULL. JSON_PARSE parses the varchar as JSON.
String-to-JSON casts now compile to JSON_PARSE; other casts, including
json-to-json, are unchanged. Athena inherits the fix since its
compiler falls through to Trino's for casts.
Added a compiler-level regression test asserting the generated SQL,
and an execution-level test in the shared cross-backend JSON suite
(ibis/backends/tests/test_json.py) so it runs against every
JSON-capable backend, not just Trino.
Verified locally against a real Trino container: the execution test
fails on the pre-fix compiler with the same symptom as the issue
(None instead of the parsed value), and passes after the fix. Also
verified each backend-skip mark individually against real
duckdb/sqlite/mysql/polars/flink execution rather than copying them
from the unrelated JSON-literal test; mysql turned out to already
work and its mark was dropped, and flink's mark's reason was corrected
to the actual Calcite error observed. athena and pyspark could not be
exercised in this environment (no AWS credentials; local Java version
too old for pyspark) and are left unmarked pending CI.
Closes ibis-project#12073
github-actions
Bot
added
tests
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description of changes
CAST(varchar AS JSON)in Trino wraps the string as a JSON string scalar rather than parsing its contents, so.cast("json")on a column of JSON text produced a JSON string, and every downstream field access silently returned NULL.JSON_PARSEparses the varchar as JSON.String-to-JSON casts now compile to
JSON_PARSE; other casts, including json-to-json, are unchanged. Athena inherits the fix since its compiler falls through to Trino's for casts.Added a compiler-level regression test asserting the generated SQL, and an execution-level test in the shared cross-backend JSON suite (
ibis/backends/tests/test_json.py) so it runs against every JSON-capable backend, not just Trino.Verified locally against a real Trino container: the execution test fails on the pre-fix compiler with the same symptom as the issue (
Noneinstead of the parsed value), and passes after the fix. Also verified each backend-skip mark individually against real duckdb/sqlite/mysql/polars/flink execution rather than copying them from the unrelated JSON-literal test; mysql turned out to already work and its mark was dropped, and flink's mark's reason was corrected to the actual Calcite error observed. Athena and pyspark could not be exercised in this environment (no AWS credentials; local Java version too old for pyspark) and are left unmarked pending CI.Issues closed
.cast(\"json\")on a string column silently produces NULL instead of parsing the JSON #12073