pandas get unique values from column of lists
If you only want to find the unique values, I'd recommend using itertools.chain.from_iterable
to concatenate all those lists
import itertools
>>> np.unique([*itertools.chain.from_iterable(df.Genre)])
array(['action', 'crime', 'drama'], dtype='<U6')
Or even faster
>>> set(itertools.chain.from_iterable(df.Genre))
{'action', 'crime', 'drama'}
Timings
df = pd.DataFrame({'Genre':[['crime','drama'],['action','crime','drama']]})
df = pd.concat([df]*10000)
%timeit set(itertools.chain.from_iterable(df.Genre))
100 loops, best of 3: 2.55 ms per loo
%timeit set([x for y in df['Genre'] for x in y])
100 loops, best of 3: 4.09 ms per loop
%timeit np.unique([*itertools.chain.from_iterable(df.Genre)])
100 loops, best of 3: 12.8 ms per loop
%timeit np.unique(df['Genre'].sum())
1 loop, best of 3: 1.65 s per loop
%timeit set(df['Genre'].sum())
1 loop, best of 3: 1.66 s per loop
You can use explode
:
data = pd.DataFrame([
{
"title": "The Godfather: Part II",
"genres": ["crime", "drama"],
"director": "Fracis Ford Coppola"
},
{
"title": "The Dark Knight",
"genres": ["action", "crime", "drama"],
"director": "Christopher Nolan"
}
])
# Changed from data.explode("genres")["genres"].unique() as suggested by rafaelc
data["genres"].explode().unique()
Results in:
array(['crime', 'drama', 'action'], dtype=object)