Allow keeping PostgresHook SQLAlchemy engines on psycopg2 - #72000
Open
sehajsandhu123 wants to merge 2 commits into
Open
Allow keeping PostgresHook SQLAlchemy engines on psycopg2#72000sehajsandhu123 wants to merge 2 commits into
sehajsandhu123 wants to merge 2 commits into
Conversation
Provider 7.0.0 switched hook-built SQLAlchemy engines to psycopg (v3) whenever SQLAlchemy 2.x is installed, with no opt-out. The psycopg dialect renders typed bind casts, so string parameters PostgreSQL previously coerced implicitly now fail server-side — most visibly pandas.DataFrame.to_sql into uuid columns (apache#71977). Honor the existing DbApiHook sqlalchemy_scheme connection extra (and hook parameter) so connections that rely on psycopg2 behaviour can keep it.
|
Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contributors' Guide
|
2 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Since postgres provider 7.0.0,
PostgresHookbuilds its SQLAlchemy engines on psycopg (v3) whenever SQLAlchemy 2.x is installed, and the 7.0.0 changelog notes there is no connection or configuration option to keep hooks on psycopg2.This silently changes semantics for Dag code that uses
hook.get_uri()orhook.get_sqlalchemy_engine(). The psycopg (v3) dialect renders typed bind casts, so string parameters that PostgreSQL used to coerce implicitly now fail server-side. The most common casualty ispandas.DataFrame.to_sqlinto a table with auuidcolumn, which worked for years on psycopg2 and now fails with:This PR adds the missing opt-out by honoring the
sqlalchemy_schemeconnection extra (and an equivalent hook parameter), the same conventionOdbcHookandDbApiHook.dialect_namealready follow.PostgresHookeven listssqlalchemy_schemeinignored_extra_options, so it is already excluded from libpq connect args — it just wasn't used when building the URL. With this change, a connection that needs the old behaviour can set:{"sqlalchemy_scheme": "postgresql+psycopg2"}The value is validated to be
postgresqlorpostgresql+<driver>with no:or/, so a connection extra can't smuggle in a different URL. Nothing changes for connections that don't set it.Tested against PostgreSQL 17 with pandas 3.0.5 / SQLAlchemy 2.0.51:
df.to_sqlinto auuidcolumn reproduces the error above on the default psycopg3 engine and succeeds with the extra set. Unit tests cover the override, parameter precedence,get_uripropagation, and rejection of invalid schemes.related: #71977
Was generative AI tooling used to co-author this PR?
Generated-by: Claude Code (Fable 5) following the guidelines