Pandas fill missing values in dataframe from another dataframe
If you have two DataFrames of the same shape, then:
df[df.isnull()] = d2
Will do the trick.
Only locations where df.isnull()
evaluates to True
(highlighted in green) will be eligible for assignment.
In practice, the DataFrames aren't always the same size / shape, and transforming methods (especially .shift()
) are useful.
Data coming in is invariably dirty, incomplete, or inconsistent. Par for the course. There's a pretty extensive pandas tutorial and associated cookbook for dealing with these situations.
As I just learned, there is a DataFrame.combine_first()
method, which does precisely this, with the additional property that if your updating data frame d2
is bigger than your original df
, the additional rows and columns are added, as well.
df = df.combine_first(d2)