Stratified splitting of pandas dataframe in training, validation and test set
np.array_split
If you want to generalise to n
splits, np.array_split
is your friend (it works with DataFrames well).
fractions = np.array([0.6, 0.2, 0.2])
# shuffle your input
df = df.sample(frac=1)
# split into 3 parts
train, val, test = np.array_split(
df, (fractions[:-1].cumsum() * len(df)).astype(int))
train_test_split
A windy solution using train_test_split
for stratified splitting.
y = df.pop('diagnosis').to_frame()
X = df
X_train, X_test, y_train, y_test = train_test_split(
X, y,stratify=y, test_size=0.4)
X_test, X_val, y_test, y_val = train_test_split(
X_test, y_test, stratify=y_test, test_size=0.5)
Where X
is a DataFrame of your features, and y
is a single-columned DataFrame of your labels.