Convert pandas column of lists into matrix representation (One Hot Encoding)
If performance is important use MultiLabelBinarizer
:
test_hot = pd.Series([[1,2,3],[3,4,5],[1,6]])
from sklearn.preprocessing import MultiLabelBinarizer
mlb = MultiLabelBinarizer()
df = pd.DataFrame(mlb.fit_transform(test_hot),columns=mlb.classes_)
print (df)
1 2 3 4 5 6
0 1 1 1 0 0 0
1 0 0 1 1 1 0
2 1 0 0 0 0 1
Your solution should be changed with create DataFrame
, reshape and DataFrame.stack
, last using get_dummies
with DataFrame.max
for aggregate:
df = pd.get_dummies(pd.DataFrame(test_hot.values.tolist()).stack().astype(int))
.max(level=0, axis=0)
print (df)
1 2 3 4 5 6
0 1 1 1 0 0 0
1 0 0 1 1 1 0
2 1 0 0 0 0 1
Details:
Created MultiIndex Series
:
print(pd.DataFrame(test_hot.values.tolist()).stack().astype(int))
0 0 1
1 2
2 3
1 0 3
1 4
2 5
2 0 1
1 6
dtype: int32
Call pd.get_dummies
:
print (pd.get_dummies(pd.DataFrame(test_hot.values.tolist()).stack().astype(int)))
1 2 3 4 5 6
0 0 1 0 0 0 0 0
1 0 1 0 0 0 0
2 0 0 1 0 0 0
1 0 0 0 1 0 0 0
1 0 0 0 1 0 0
2 0 0 0 0 1 0
2 0 1 0 0 0 0 0
1 0 0 0 0 0 1
And last aggregate max
per first level.
Fixing your get_dummies
code, you can use:
df['lists'].map(lambda x: ','.join(map(str, x))).str.get_dummies(sep=',')
1 2 3 4 5
0 1 0 1 1 1
1 0 1 0 0 0
2 0 0 1 0 1
3 0 1 1 0 1