Skip to content

Commit 0cbebb7

Browse files
committed
Document pandas 3 impact on DataFrame XComs
Deployments need to know that every component has to carry the pandas 3 support before pandas 3 reaches any worker, that a rollback strands the XComs written in the meantime, and that a pulled DataFrame now takes its dtypes from the reader's pandas version.
1 parent 03f29c4 commit 0cbebb7

1 file changed

Lines changed: 33 additions & 0 deletions

File tree

Lines changed: 33 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,33 @@
1+
pandas 3 changes how DataFrame XComs are stored and read back
2+
3+
pandas 3 exposes its public classes from the ``pandas`` namespace, so a DataFrame is qualified as
4+
``pandas.DataFrame`` instead of ``pandas.core.frame.DataFrame``. XComs record that name alongside the
5+
serialized value, so the name written into the metadata database depends on the pandas version of the
6+
component that pushed the value. Airflow registers both names, and a DataFrame written by either
7+
pandas version can be read by either — no configuration change is needed, and existing XComs stay
8+
readable.
9+
10+
What you should do:
11+
12+
* **Roll this Airflow version out to every component before pandas 3 reaches any of them** — workers
13+
in particular. A component that predates this change cannot read a DataFrame XCom written under
14+
pandas 3, and fails the pull with:
15+
16+
.. code-block:: text
17+
18+
ImportError: pandas.DataFrame was not found in allow list for deserialization imports.
19+
To allow it, add it to allowed_deserialization_classes in the configuration
20+
21+
The message points at configuration, but the allow list is not the cause and changing it does not
22+
help. The rows are not corrupt: they become readable again as soon as the reader is upgraded.
23+
24+
* **Treat a downgrade as a one-way door for those XComs.** Rolling back to an Airflow version without
25+
this change strands any DataFrame XCom written while on pandas 3, with the same error, until you
26+
roll forward again.
27+
28+
* **Review Dags that inspect the dtypes of a pulled DataFrame.** The pandas version of the *reader*
29+
determines what a pulled DataFrame looks like, not the version that wrote it. Under pandas 3, a
30+
column of strings comes back with the ``str`` dtype rather than ``object``, and its missing values
31+
come back as ``nan`` rather than ``None``. Values are unchanged, but downstream code that branches
32+
on ``dtype == "object"``, checks cells with ``is None``, or compares against a reference frame with
33+
``DataFrame.equals()`` can behave differently after the upgrade.

0 commit comments

Comments
 (0)