Limit pandas to < 3 for DataFrame XComs (#70791)

* Limit pandas to < 3 for DataFrame XComs

Under pandas 3 a DataFrame does not round trip through an XCom the way it does
under pandas 2. The public classes moved to the `pandas` namespace, so the same
object qualifies as `pandas.DataFrame` rather than `pandas.core.frame.DataFrame`
and the serde registry -- keyed on the latter -- does not dispatch to the pandas
serializer at all. The caller gets serde's generic "cannot serialize object of
type" with nothing pointing at pandas. Beyond that, the dtypes differ: an
`object` column reads back as `str` and missing values as `nan` rather than
`None`, so the reader's pandas version, not the writer's, decides what a task
receives.

Declare `pandas<3` where Airflow declares pandas, and back it with a runtime
check, since pandas is an optional dependency and the constraint alone cannot be
relied on -- a deployment may install pandas 3 directly or pull it in through
another package.

The check needs the pandas 3 qualname registered to be reachable at all:
without `pandas.DataFrame` in the registry the request never reaches this module.
It is registered for that reason only, to refuse the value with a message that
names pandas and says what to do, not to support it.

Reads are refused as well as writes. A payload written under pandas 2 comes back
with pandas 3 dtypes, so accepting it would hand the task different data than was
pushed, with no signal.

Moving from pandas 2 to pandas 3 is a deliberate migration for users to make.
Support for it is not dropped, only deferred.

Generated-by: Claude Opus 5 (1M context) following the guidelines at
https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions

* Sync uv.lock with the pandas < 3 constraint

* Add significant newsfragment for the pandas < 3 limit

* Apply the pandas < 3 limit to the remaining providers

The XCom limit only binds where Airflow declares pandas. Thirteen providers pin
pandas independently of common-sql, so a deployment installing any of them could
still resolve pandas 3 and reach the runtime refusal rather than being held on a
supported version by the constraint.

pandas-gbq is deliberately untouched -- a different package, unaffected by this.

Provider READMEs and uv.lock are regenerated to match.

* Regenerate provider docs index for the pandas < 3 pins

The update-providers-build-files hook renders the extras table in each
provider's docs/index.rst from its pyproject dependencies. Fourteen providers
show the pandas extra, so the pin change widens those tables.

* Explain the pandas upper bound at every site and drop a vacuous test

contributing-docs/13_airflow_dependencies_and_extras.rst asks for a comment
saying why whenever a dependency is upper-bound, and AGENTS.md asks for the
tracking URL in the file rather than only in a PR comment. All 45 specifiers
across 15 files landed without either. Add the rationale and a pointer to the
pandas 3 support PR above each block, so whoever decides when the cap comes off
does not have to reconstruct it.

test_pandas_2_dataframe_xcom_still_round_trips passed unchanged on main: nothing
read pd.__version__ before this PR, so monkeypatching it to a 2.x value while the
installed pandas is already 2.x changed nothing observable, and the round trip is
already covered by test_pandas_serializers. Dropped.

---------

Co-authored-by: Rahul Vats <43964496+vatsrahul1001@users.noreply.github.com>

Author: potiuk

airflow-core/newsfragments/70791.significant.rst

@@ -0,0 +1,35 @@
+pandas is limited to ``< 3``; DataFrame XComs are refused on pandas 3
+
+A ``pandas.DataFrame`` does not survive an XCom round trip under pandas 3 the way it does
+under pandas 2, so this version of Airflow constrains pandas to ``< 3`` and refuses DataFrame
+XComs at runtime when a newer pandas is installed anyway.
+
+Two things change under pandas 3. Its public classes are exposed from the ``pandas``
+namespace, so a DataFrame qualifies as ``pandas.DataFrame`` rather than
+``pandas.core.frame.DataFrame`` and the XCom serializer registry -- keyed on the latter --
+never dispatches to the pandas serializer. And the dtypes differ: an ``object`` column reads
+back as ``str``, and missing values as ``nan`` rather than ``None``, so the *reader's* pandas
+version decides what a task receives rather than the writer's.
+
+Because pandas is an optional dependency, the ``< 3`` constraint alone cannot be relied on --
+a deployment may install pandas 3 directly, or pull it in through another package. It is
+therefore backed by a runtime check.
+
+**Behaviour changes:**
+
+- With pandas 3 or newer installed, pushing **or** pulling a ``DataFrame`` through XCom now
+  raises ``RuntimeError`` naming the installed pandas version and the supported range.
+  Reads are refused as well as writes, because a payload written under pandas 2 reads back
+  with pandas 3 dtypes and would otherwise hand the task different data than was pushed,
+  with no signal.
+- Previously the same situation produced an opaque
+  ``TypeError: cannot serialize object of type <class 'pandas.DataFrame'>`` from the
+  serializer registry, with nothing identifying pandas as the cause.
+- Deployments that pass DataFrames through XCom must pin ``pandas < 3`` in the environment
+  that runs their tasks. Deployments that do not use DataFrame XComs are unaffected at
+  runtime.
+- Nothing changes for pandas 2 users.
+
+**This defers pandas 3 support rather than dropping it.** Moving from pandas 2 to pandas 3
+is a deliberate migration for users to make, given the dtype changes it brings to data that
+has already been written. Support for pandas 3 will be enabled in a future Airflow release.

providers/amazon/docs/index.rst

@@ -171,9 +171,9 @@ Install them when installing from PyPI. For example:
     pip install apache-airflow-providers-amazon[aiobotocore]
 
 
-====================  ============================================================================================================================================================
+====================  =====================================================================================================================================================================
 Extra                 Dependencies
-====================  ============================================================================================================================================================
+====================  =====================================================================================================================================================================
 ``aiobotocore``       ``aiobotocore>=3.0.0``
 ``cncf.kubernetes``   ``apache-airflow-providers-cncf-kubernetes>=7.2.0``
 ``s3fs``              ``s3fs>=2023.10.0``
@@ -186,14 +186,14 @@ Extra                 Dependencies
 ``imap``              ``apache-airflow-providers-imap``
 ``microsoft.azure``   ``apache-airflow-providers-microsoft-azure``
 ``mongo``             ``apache-airflow-providers-mongo``
-``pandas``            ``pandas>=2.1.2; python_version <"3.13"``, ``pandas>=2.2.3; python_version >="3.13" and python_version <"3.14"``, ``pandas>=2.3.3; python_version >="3.14"``
+``pandas``            ``pandas>=2.1.2,<3; python_version <"3.13"``, ``pandas>=2.2.3,<3; python_version >="3.13" and python_version <"3.14"``, ``pandas>=2.3.3,<3; python_version >="3.14"``
 ``openlineage``       ``apache-airflow-providers-openlineage>=2.3.0``
 ``salesforce``        ``apache-airflow-providers-salesforce``
 ``ssh``               ``apache-airflow-providers-ssh``
 ``standard``          ``apache-airflow-providers-standard``
 ``common.messaging``  ``apache-airflow-providers-common-messaging>=2.0.0``
 ``sqlalchemy``        ``sqlalchemy>=1.4.54``
-====================  ============================================================================================================================================================
+====================  =====================================================================================================================================================================
 
 Downloading official packages
 -----------------------------

providers/amazon/pyproject.toml

@@ -130,9 +130,18 @@ dependencies = [
     "apache-airflow-providers-mongo"
 ]
 "pandas" = [
-    'pandas>=2.1.2; python_version <"3.13"',
-    'pandas>=2.2.3; python_version >="3.13" and python_version <"3.14"',
-    'pandas>=2.3.3; python_version >="3.14"',
+    # pandas 3 changes the dtypes a DataFrame reads back with, so DataFrame XComs do not
+    # round trip; capped until pandas 3 support lands.
+    # Tracked at https://github.com/apache/airflow/pull/70558
+    'pandas>=2.1.2,<3; python_version <"3.13"',
+    # pandas 3 changes the dtypes a DataFrame reads back with, so DataFrame XComs do not
+    # round trip; capped until pandas 3 support lands.
+    # Tracked at https://github.com/apache/airflow/pull/70558
+    'pandas>=2.2.3,<3; python_version >="3.13" and python_version <"3.14"',
+    # pandas 3 changes the dtypes a DataFrame reads back with, so DataFrame XComs do not
+    # round trip; capped until pandas 3 support lands.
+    # Tracked at https://github.com/apache/airflow/pull/70558
+    'pandas>=2.3.3,<3; python_version >="3.14"',
 ]
 "openlineage" = [
     "apache-airflow-providers-openlineage>=2.3.0"

providers/apache/hdfs/README.rst

@@ -51,19 +51,19 @@ The package supports the following python versions: 3.10,3.11,3.12,3.13,3.14
 Requirements
 ------------
 
-==========================================  ==================================================================
+==========================================  ====================================================================
 PIP package                                 Version required
-==========================================  ==================================================================
+==========================================  ====================================================================
 ``apache-airflow``                          ``>=2.11.0``
 ``apache-airflow-providers-common-compat``  ``>=1.12.0``
 ``hdfs[avro,dataframe,kerberos]``           ``>=2.5.4; python_version < "3.12"``
 ``hdfs[avro,dataframe,kerberos]``           ``>=2.7.3; python_version >= "3.12"``
 ``fastavro``                                ``>=1.10.0; python_version >= "3.13" and python_version < "3.14"``
 ``fastavro``                                ``>=1.12.1; python_version >= "3.14"``
-``pandas``                                  ``>=2.1.2; python_version < "3.13"``
-``pandas``                                  ``>=2.2.3; python_version >= "3.13" and python_version < "3.14"``
-``pandas``                                  ``>=2.3.3; python_version >= "3.14"``
-==========================================  ==================================================================
+``pandas``                                  ``>=2.1.2,<3; python_version < "3.13"``
+``pandas``                                  ``>=2.2.3,<3; python_version >= "3.13" and python_version < "3.14"``
+``pandas``                                  ``>=2.3.3,<3; python_version >= "3.14"``
+==========================================  ====================================================================
 
 The changelog for the provider package can be found in the
 `changelog <https://airflow.apache.org/docs/apache-airflow-providers-apache-hdfs/4.12.1/changelog.html>`_.

providers/apache/hdfs/docs/index.rst

@@ -85,19 +85,19 @@ Requirements
 
 The minimum Apache Airflow version supported by this provider distribution is ``2.11.0``.
 
-==========================================  ==================================================================
+==========================================  ====================================================================
 PIP package                                 Version required
-==========================================  ==================================================================
+==========================================  ====================================================================
 ``apache-airflow``                          ``>=2.11.0``
 ``apache-airflow-providers-common-compat``  ``>=1.12.0``
 ``hdfs[avro,dataframe,kerberos]``           ``>=2.5.4; python_version < "3.12"``
 ``hdfs[avro,dataframe,kerberos]``           ``>=2.7.3; python_version >= "3.12"``
 ``fastavro``                                ``>=1.10.0; python_version >= "3.13" and python_version < "3.14"``
 ``fastavro``                                ``>=1.12.1; python_version >= "3.14"``
-``pandas``                                  ``>=2.1.2; python_version < "3.13"``
-``pandas``                                  ``>=2.2.3; python_version >= "3.13" and python_version < "3.14"``
-``pandas``                                  ``>=2.3.3; python_version >= "3.14"``
-==========================================  ==================================================================
+``pandas``                                  ``>=2.1.2,<3; python_version < "3.13"``
+``pandas``                                  ``>=2.2.3,<3; python_version >= "3.13" and python_version < "3.14"``
+``pandas``                                  ``>=2.3.3,<3; python_version >= "3.14"``
+==========================================  ====================================================================
 
 Downloading official packages
 -----------------------------

providers/apache/hdfs/pyproject.toml

@@ -65,9 +65,18 @@ dependencies = [
     'hdfs[avro,dataframe,kerberos]>=2.7.3;python_version>="3.12"',
     'fastavro>=1.10.0; python_version>="3.13" and python_version<"3.14"',
     'fastavro>=1.12.1; python_version>="3.14"',
-    'pandas>=2.1.2; python_version <"3.13"',
-    'pandas>=2.2.3; python_version >="3.13" and python_version <"3.14"',
-    'pandas>=2.3.3; python_version >="3.14"',
+    # pandas 3 changes the dtypes a DataFrame reads back with, so DataFrame XComs do not
+    # round trip; capped until pandas 3 support lands.
+    # Tracked at https://github.com/apache/airflow/pull/70558
+    'pandas>=2.1.2,<3; python_version <"3.13"',
+    # pandas 3 changes the dtypes a DataFrame reads back with, so DataFrame XComs do not
+    # round trip; capped until pandas 3 support lands.
+    # Tracked at https://github.com/apache/airflow/pull/70558
+    'pandas>=2.2.3,<3; python_version >="3.13" and python_version <"3.14"',
+    # pandas 3 changes the dtypes a DataFrame reads back with, so DataFrame XComs do not
+    # round trip; capped until pandas 3 support lands.
+    # Tracked at https://github.com/apache/airflow/pull/70558
+    'pandas>=2.3.3,<3; python_version >="3.14"',
 ]
 
 [dependency-groups]

providers/apache/hive/README.rst

@@ -50,19 +50,19 @@ The package supports the following python versions: 3.10,3.11,3.12,3.13,3.14
 Requirements
 ------------
 
-==========================================  =================================================================
+==========================================  ====================================================================
 PIP package                                 Version required
-==========================================  =================================================================
+==========================================  ====================================================================
 ``apache-airflow``                          ``>=2.11.0``
 ``apache-airflow-providers-common-compat``  ``>=1.12.0``
 ``apache-airflow-providers-common-sql``     ``>=1.32.0``
 ``hmsclient``                               ``>=0.1.0``
-``pandas``                                  ``>=2.1.2; python_version < "3.13"``
-``pandas``                                  ``>=2.2.3; python_version >= "3.13" and python_version < "3.14"``
-``pandas``                                  ``>=2.3.3; python_version >= "3.14"``
+``pandas``                                  ``>=2.1.2,<3; python_version < "3.13"``
+``pandas``                                  ``>=2.2.3,<3; python_version >= "3.13" and python_version < "3.14"``
+``pandas``                                  ``>=2.3.3,<3; python_version >= "3.14"``
 ``pyhive[hive_pure_sasl]``                  ``>=0.7.0``
 ``jmespath``                                ``>=0.7.0``
-==========================================  =================================================================
+==========================================  ====================================================================
 
 Optional cross provider package dependencies
 --------------------------------------------

providers/apache/hive/docs/index.rst

@@ -99,19 +99,19 @@ Requirements
 
 The minimum Apache Airflow version supported by this provider distribution is ``2.11.0``.
 
-==========================================  =================================================================
+==========================================  ====================================================================
 PIP package                                 Version required
-==========================================  =================================================================
+==========================================  ====================================================================
 ``apache-airflow``                          ``>=2.11.0``
 ``apache-airflow-providers-common-compat``  ``>=1.12.0``
 ``apache-airflow-providers-common-sql``     ``>=1.32.0``
 ``hmsclient``                               ``>=0.1.0``
-``pandas``                                  ``>=2.1.2; python_version < "3.13"``
-``pandas``                                  ``>=2.2.3; python_version >= "3.13" and python_version < "3.14"``
-``pandas``                                  ``>=2.3.3; python_version >= "3.14"``
+``pandas``                                  ``>=2.1.2,<3; python_version < "3.13"``
+``pandas``                                  ``>=2.2.3,<3; python_version >= "3.13" and python_version < "3.14"``
+``pandas``                                  ``>=2.3.3,<3; python_version >= "3.14"``
 ``pyhive[hive_pure_sasl]``                  ``>=0.7.0``
 ``jmespath``                                ``>=0.7.0``
-==========================================  =================================================================
+==========================================  ====================================================================
 
 Optional cross provider package dependencies
 --------------------------------------------

providers/apache/hive/pyproject.toml

@@ -63,9 +63,18 @@ dependencies = [
     "apache-airflow-providers-common-compat>=1.12.0",
     "apache-airflow-providers-common-sql>=1.32.0",
     "hmsclient>=0.1.0",
-    'pandas>=2.1.2; python_version <"3.13"',
-    'pandas>=2.2.3; python_version >="3.13" and python_version <"3.14"',
-    'pandas>=2.3.3; python_version >="3.14"',
+    # pandas 3 changes the dtypes a DataFrame reads back with, so DataFrame XComs do not
+    # round trip; capped until pandas 3 support lands.
+    # Tracked at https://github.com/apache/airflow/pull/70558
+    'pandas>=2.1.2,<3; python_version <"3.13"',
+    # pandas 3 changes the dtypes a DataFrame reads back with, so DataFrame XComs do not
+    # round trip; capped until pandas 3 support lands.
+    # Tracked at https://github.com/apache/airflow/pull/70558
+    'pandas>=2.2.3,<3; python_version >="3.13" and python_version <"3.14"',
+    # pandas 3 changes the dtypes a DataFrame reads back with, so DataFrame XComs do not
+    # round trip; capped until pandas 3 support lands.
+    # Tracked at https://github.com/apache/airflow/pull/70558
+    'pandas>=2.3.3,<3; python_version >="3.14"',
     "pyhive[hive_pure_sasl]>=0.7.0",
     "jmespath>=0.7.0",
 ]

providers/common/sql/docs/index.rst

@@ -141,18 +141,18 @@ Install them when installing from PyPI. For example:
     pip install apache-airflow-providers-common-sql[pandas]
 
 
-==================  =======================================================================================================================================================================
+==================  ================================================================================================================================================================================
 Extra               Dependencies
-==================  =======================================================================================================================================================================
-``pandas``          ``pandas[sql-other]>=2.1.2; python_version <"3.13"``, ``pandas>=2.2.3; python_version >="3.13" and python_version <"3.14"``, ``pandas>=2.3.3; python_version >="3.14"``
+==================  ================================================================================================================================================================================
+``pandas``          ``pandas[sql-other]>=2.1.2,<3; python_version <"3.13"``, ``pandas>=2.2.3,<3; python_version >="3.13" and python_version <"3.14"``, ``pandas>=2.3.3,<3; python_version >="3.14"``
 ``openlineage``     ``apache-airflow-providers-openlineage``
 ``polars``          ``polars>=1.26.0``
 ``sqlalchemy``      ``sqlalchemy>=1.4.54``
 ``amazon``          ``apache-airflow-providers-amazon``
 ``datafusion``      ``datafusion>=50.0.0,<52.0.0``
 ``pyiceberg-core``  ``pyiceberg-core>=0.8.0``
 ``apache.iceberg``  ``apache-airflow-providers-apache-iceberg``
-==================  =======================================================================================================================================================================
+==================  ================================================================================================================================================================================
 
 Downloading official packages
 -----------------------------