How to proceed with `None` value in pandas fillna
Setup
Consider the sample dataframe df
df = pd.DataFrame(dict(A=[1, None], B=[None, 2], C=[None, 'D']))
df
A B C
0 1.0 NaN None
1 NaN 2.0 D
I can confirm the error
df.fillna(dict(A=1, B=None, C=4))
ValueError: must specify a fill method or value
This happens because pandas is cycling through keys in the dictionary and executing a fillna
for each relevant column. If you look at the signature of the pd.Series.fillna
method
Series.fillna(value=None, method=None, axis=None, inplace=False, limit=None, downcast=None, **kwargs)
You'll see the default value is None
. So we can replicate this error with
df.A.fillna(None)
Or equivalently
df.A.fillna()
I'll add that I'm not terribly surprised considering that you are attempting to fill a null value with a null value.
What you need is a work around
Solution
Use pd.DataFrame.fillna
over columns that you want to fill with non-null values. Then follow that up with a pd.DataFrame.replace
on the specific columns you want to swap one null value with another.
df.fillna(dict(A=1, C=2)).replace(dict(B={np.nan: None}))
A B C
0 1.0 None 2
1 1.0 2 D
In case you want to normalize all of the nulls with python's None.
df.fillna(np.nan).replace([np.nan], [None])
The first fillna
will replace all of (None, NAT, np.nan, etc) with Numpy's NaN, then replace Numpy's NaN with python's None.