BUG: `fillna` cannot convert NaN to None

[david-cortes]

Pandas version checks

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pandas.

• I have confirmed this bug exists on the main branch of pandas.

Reproducible Example

import numpy as np, pandas as pd
pd.Series([np.nan], dtype="O").fillna(value=None)

Issue Description

If I have a series of generic object type with missing values represented as NaN from numpy, I cannot use fillna to conver them to python None.

A related issue says this should have been fixed in 1.4, but it isn't: #45490

Expected Behavior

Code example should convert the NaN to None.

Installed Versions

INSTALLED VERSIONS

commit : 4bfe3d0
python : 3.9.12.final.0
python-bits : 64
OS : Linux
OS-release : 5.18.0-2-amd64
Version : #1 SMP PREEMPT_DYNAMIC Debian 5.18.5-1 (2022-06-16)
machine : x86_64
processor :
byteorder : little
LC_ALL : None
LANG : en_US.UTF-8
LOCALE : en_US.UTF-8

pandas : 1.4.2
numpy : 1.22.3
pytz : 2022.1
dateutil : 2.8.2
pip : 21.2.4
setuptools : 61.2.0
Cython : 0.29.28
pytest : 7.1.2
hypothesis : None
sphinx : None
blosc : None
feather : None
xlsxwriter : None
lxml.etree : None
html5lib : None
pymysql : None
psycopg2 : None
jinja2 : None
IPython : 7.28.0
pandas_datareader: None
bs4 : None
bottleneck : None
brotli : None
fastparquet : None
fsspec : None
gcsfs : None
markupsafe : None
matplotlib : 3.5.2
numba : None
numexpr : 2.8.3
odfpy : None
openpyxl : None
pandas_gbq : None
pyarrow : None
pyreadstat : None
pyxlsb : None
s3fs : None
scipy : 1.7.3
snappy : None
sqlalchemy : None
tables : None
tabulate : None
xarray : None
xlrd : None
xlwt : None
zstandard : None

[dicristina]

None is used as a sentinel value by fillna so that will not work. Use Series.mask instead:

import pandas as pd
import numpy as np

s = pd.Series([1,2,3,np.nan,5,6], dtype=object)
s.mask(s.isna(), None)

Using a series full of None with fillna did not work as expected either:

import pandas as pd
import numpy as np

s = pd.Series([1,2,3,np.nan,5,6], dtype=object)
all_nones = pd.Series([None]*len(s), s.index, dtype=object)
s.fillna(all_nones)  # Not as expected

[MichaelTiemannOSC]

When the Series has the dtype Float64 instead of object, Series.mask doesn't work:

(Pdb) ss = pd.Series([1,2,3,pd.NA,5,6], dtype="Float64")
(Pdb) ss.mask(ss.isna(), np.nan)
0     1.0
1     2.0
2     3.0
3    <NA>
4     5.0
5     6.0
dtype: Float64

Changing back to "object" it does work:

(Pdb) ss = pd.Series([1,2,3,pd.NA,5,6], dtype="object")
(Pdb) ss.mask(ss.isna(), np.nan)
0      1
1      2
2      3
3    NaN
4      5
5      6
dtype: object

[jbrockmendel]

The OP example now correctly shows None. Could use a test (first check to see if one already exists)

[vijmeister]

I could not find a test (looked in pandas/pandas/tests/series/methods
/test_fillna.py).

I can try to add one.

[vijmeister]

I was about to submit a PR with a test in the series directory but saw in the whatsnew file that issue # 57723 (PR #58085) fixed the issue and added a test for the frame directory. Upon rereading at the documentation for contributing, I see that tests only need to be added to frame in the case of a shared test between a DataFrame/Series.

@jbrockmendel, is a test not needed then and should this issue be closed?

[jbrockmendel]

Good find. That test parametrizes over as_frame so covers the Series case as well. So you're right, another test is not needed. Closing.