find the set of column indices for non-zero values in each row in pandas' data frame
It seems you have to traverse the DataFrame by row.
cols = df.columns
bt = df.apply(lambda x: x > 0)
bt.apply(lambda x: list(cols[x.values]), axis=1)
and you will get:
0 [c1, c2]
1 [c1]
2 [c2]
3 [c1]
4 [c2]
5 []
6 [c2, c3, c4, c5, c6, c7, c9]
7 [c1, c2, c3, c6, c8, c9]
8 [c1, c2, c4, c5, c6, c7, c8, c9]
9 [c1, c2, c3, c4, c5, c6, c7, c8, c9]
10 [c1, c2, c4]
11 [c1, c2, c3, c5, c7, c8]
dtype: object
If performance is matter, try to pass raw=True
to boolean DataFrame creation like below:
%timeit df.apply(lambda x: x > 0, raw=True).apply(lambda x: list(cols[x.values]), axis=1)
1000 loops, best of 3: 812 µs per loop
It brings you a better performance gain. Following is raw=False
(which is default) result:
%timeit df.apply(lambda x: x > 0).apply(lambda x: list(cols[x.values]), axis=1)
100 loops, best of 3: 2.59 ms per loop
Potentially a better data structure (rather than a Series of lists) is to stack:
In [11]: res = df[df!=0].stack()
In [12]: res
Out[12]:
0 c1 1
c2 1
1 c1 1
2 c2 1
3 c1 1
...
And you can iterate over the original rows:
In [13]: res.loc[0]
Out[13]:
c1 1
c2 1
dtype: float64
In [14]: res.loc[0].index
Out[14]: Index(['c1', 'c2'], dtype='object')
Note: I thought you used to be able to return a list in an apply (to create a DataFrame which has list elements) this no longer appears to be the case.
How about this approach?
#create a True / False data frame
df_boolean = df>0
#a little helper method that uses boolean slicing internally
def bar(x,columns):
return ','.join(list(columns[x]))
#use an apply along the column axis
df_boolean['result'] = df_boolean.apply(lambda x: bar(x,df_boolean.columns),axis=1)
# filter out the empty "rows" adn grab the result column
df_result = df_boolean[df_boolean['result'] != '']['result']
#append an axis, just so each line will will output a list
lst_result = df_result.values[:,np.newaxis]
print '\n'.join([ str(myelement) for myelement in lst_result])
and this produces:
['c1,c2']
['c1']
['c2']
['c1']
['c2']
['c2,c3,c4,c5,c6,c7,c9']
['c1,c2,c3,c6,c8,c9']
['c1,c2,c4,c5,c6,c7,c8,c9']
['c1,c2,c3,c4,c5,c6,c7,c8,c9']
['c1,c2,c4']
['c1,c2,c3,c5,c7,c8']