|
| 1 | +pandas 3 changes how DataFrame XComs are stored and read back |
| 2 | + |
| 3 | +pandas 3 exposes its public classes from the ``pandas`` namespace, so a DataFrame is qualified as |
| 4 | +``pandas.DataFrame`` instead of ``pandas.core.frame.DataFrame``. XComs record that name alongside the |
| 5 | +serialized value, so the name written into the metadata database depends on the pandas version of the |
| 6 | +component that pushed the value. Airflow registers both names, and a DataFrame written by either |
| 7 | +pandas version can be read by either — no configuration change is needed, and existing XComs stay |
| 8 | +readable. |
| 9 | + |
| 10 | +What you should do: |
| 11 | + |
| 12 | +* **Roll this Airflow version out to every component before pandas 3 reaches any of them** — workers |
| 13 | + in particular. A component that predates this change cannot read a DataFrame XCom written under |
| 14 | + pandas 3, and fails the pull with: |
| 15 | + |
| 16 | + .. code-block:: text |
| 17 | +
|
| 18 | + ImportError: pandas.DataFrame was not found in allow list for deserialization imports. |
| 19 | + To allow it, add it to allowed_deserialization_classes in the configuration |
| 20 | +
|
| 21 | + The message points at configuration, but the allow list is not the cause and changing it does not |
| 22 | + help. The rows are not corrupt: they become readable again as soon as the reader is upgraded. |
| 23 | + |
| 24 | +* **Treat a downgrade as a one-way door for those XComs.** Rolling back to an Airflow version without |
| 25 | + this change strands any DataFrame XCom written while on pandas 3, with the same error, until you |
| 26 | + roll forward again. |
| 27 | + |
| 28 | +* **Review Dags that inspect the dtypes of a pulled DataFrame.** The pandas version of the *reader* |
| 29 | + determines what a pulled DataFrame looks like, not the version that wrote it. Under pandas 3, a |
| 30 | + column of strings comes back with the ``str`` dtype rather than ``object``, and its missing values |
| 31 | + come back as ``nan`` rather than ``None``. Values are unchanged, but downstream code that branches |
| 32 | + on ``dtype == "object"``, checks cells with ``is None``, or compares against a reference frame with |
| 33 | + ``DataFrame.equals()`` can behave differently after the upgrade. |
0 commit comments