Convert pandas column of lists into matrix representation (One Hot Encoding)

If performance is important use MultiLabelBinarizer:

test_hot = pd.Series([[1,2,3],[3,4,5],[1,6]])

from sklearn.preprocessing import MultiLabelBinarizer

mlb = MultiLabelBinarizer()
df = pd.DataFrame(mlb.fit_transform(test_hot),columns=mlb.classes_)
print (df)
   1  2  3  4  5  6
0  1  1  1  0  0  0
1  0  0  1  1  1  0
2  1  0  0  0  0  1

Your solution should be changed with create DataFrame, reshape and DataFrame.stack, last using get_dummies with DataFrame.max for aggregate:

df = pd.get_dummies(pd.DataFrame(test_hot.values.tolist()).stack().astype(int))
       .max(level=0, axis=0)

print (df)
   1  2  3  4  5  6
0  1  1  1  0  0  0
1  0  0  1  1  1  0
2  1  0  0  0  0  1

Details:

Created MultiIndex Series:

print(pd.DataFrame(test_hot.values.tolist()).stack().astype(int))
0  0    1
   1    2
   2    3
1  0    3
   1    4
   2    5
2  0    1
   1    6
dtype: int32

Call pd.get_dummies:

print (pd.get_dummies(pd.DataFrame(test_hot.values.tolist()).stack().astype(int)))
     1  2  3  4  5  6
0 0  1  0  0  0  0  0
  1  0  1  0  0  0  0
  2  0  0  1  0  0  0
1 0  0  0  1  0  0  0
  1  0  0  0  1  0  0
  2  0  0  0  0  1  0
2 0  1  0  0  0  0  0
  1  0  0  0  0  0  1

And last aggregate max per first level.


Fixing your get_dummies code, you can use:

df['lists'].map(lambda x: ','.join(map(str, x))).str.get_dummies(sep=',')

   1  2  3  4  5
0  1  0  1  1  1
1  0  1  0  0  0
2  0  0  1  0  1
3  0  1  1  0  1