Pandas dataframe select rows where a list-column contains any of a list of strings

IIUC Re-create your df then using isin with any should be faster than apply

df[pd.DataFrame(df.species.tolist()).isin(selection).any(1).values]
Out[64]: 
  molecule            species
0        a              [dog]
2        c         [cat, dog]
3        d  [cat, horse, pig]

You can use mask with apply here.

selection = ['cat', 'dog']

mask = df.species.apply(lambda x: any(item for item in selection if item in x))
df1 = df[mask]

For the DataFrame you've provided as an example above, df1 will be:

molecule    species
0   a   [dog]
2   c   [cat, dog]
3   d   [cat, horse, pig]

Using Numpy would be much faster than using Pandas in this case,

Option 1: Using numpy intersection,

mask =  df.species.apply(lambda x: np.intersect1d(x, selection).size > 0)
df[mask]
450 µs ± 21.5 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)

    molecule    species
0   a   [dog]
2   c   [cat, dog]
3   d   [cat, horse, pig]

Option2: A similar solution as above using numpy in1d,

df[df.species.apply(lambda x: np.any(np.in1d(x, selection)))]
420 µs ± 17.5 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)

Option 3: Interestingly, using pure python set is quite fast here

df[df.species.apply(lambda x: bool(set(x) & set(selection)))]
305 µs ± 5.22 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)