Python - Pandas, Resample dataset to have balanced classes
A very simple approach. Taken from sklearn documentation and Kaggle.
from sklearn.utils import resample
df_majority = df[df.label==0]
df_minority = df[df.label==1]
# Upsample minority class
df_minority_upsampled = resample(df_minority,
replace=True, # sample with replacement
n_samples=20, # to match majority class
random_state=42) # reproducible results
# Combine majority class with upsampled minority class
df_upsampled = pd.concat([df_majority, df_minority_upsampled])
# Display new class counts
df_upsampled.label.value_counts()
Provided that each name
is labeled by exactly one label
(e.g. all A
are 1
) you can use the following:
- Group the
name
s bylabel
and check which label has an excess (in terms of unique names). - Randomly remove names from the over-represented label class in order to account for the excess.
- Select the part of the data frame which does not contain the removed names.
Here is the code:
labels = df.groupby('label').name.unique()
# Sort the over-represented class to the head.
labels = labels[labels.apply(len).sort_values(ascending=False).index]
excess = len(labels.iloc[0]) - len(labels.iloc[1])
remove = np.random.choice(labels.iloc[0], excess, replace=False)
df2 = df[~df.name.isin(remove)]
Using imblearn (pip install imblearn
), this is as simple as:
from imblearn.under_sampling import RandomUnderSampler
rus = RandomUnderSampler(sampling_strategy='not minority', random_state=1)
df_balanced, balanced_labels = rus.fit_resample(df, df['label'])
There are many methods other than RandomUnderSampler
, so I suggest you read the documentation.