Python fill missing values according to frequency
A generic answer in case you have more than 2 valid values in your column is to find the distribution and fill based on that. For example,
dist = df.sex.value_counts(normalize=True)
print(list)
1.0 0.666667
0.0 0.333333
Name: sex, dtype: float64
Then get the rows with missing values
nan_rows = df['sex'].isnull()
Finally, fill the those rows with randomly selected values based on the above distribution
df.loc[nan_rows,'sex'] = np.random.choice(dist.index, size=len(df[nan_rows]),p=dist.values)
Check with value_counts
+ np.random.choice
s = df.sex.value_counts(normalize=True)
df['sex_fillna'] = df['sex']
df.loc[df.sex.isna(), 'sex_fillna'] = np.random.choice(s.index, p=s.values, size=df.sex.isna().sum())
df
Out[119]:
sex sex_fillna
0 1.0 1.0
1 1.0 1.0
2 1.0 1.0
3 1.0 1.0
4 0.0 0.0
5 0.0 0.0
6 NaN 0.0
7 NaN 1.0
8 NaN 1.0
The output for s
index is the category and the value is the probability
s
Out[120]:
1.0 0.666667
0.0 0.333333
Name: sex, dtype: float64