Pandas drop duplicates on elements made of lists

Call drop_duplicates on tuplized data:

df[0].apply(tuple, 1).drop_duplicates().apply(list).to_frame()

        0
0  [1, 0]
1  [0, 0]

However, I'd much prefer something that doesn't involve apply...

from collections import OrderedDict
pd.Series(map(
    list, (OrderedDict.fromkeys(map(tuple, df[0].tolist()))))
).to_frame()

Or,

pd.Series(
    list(k) for k in OrderedDict.fromkeys(map(tuple, df[0].tolist()))
).to_frame()

        0
0  [1, 0]
1  [0, 0]

You can use numpy.unique() function:

>>> df = pandas.DataFrame([[[1,0]],[[0,0]],[[1,0]]])
>>> pandas.DataFrame(np.unique(df), columns=df.columns)
        0
0  [0, 0]
1  [1, 0]

If you want to preserve the order checkout: numpy.unique with order preserved

Related