python pandas: pivot_table silently drops indices with nans

This is currently not supported, see this issue for the enhancement: https://github.com/pydata/pandas/issues/3729.

Workaround to fill the index with a dummy, pivot, and replace

In [28]: df = df.reset_index()

In [29]: df['b'] = df['b'].fillna('dummy')

In [30]: df['dummy'] = np.nan

In [31]: df
Out[31]: 
   a      b       c    d   e  dummy
0  a      b   12.00   12  12    NaN
1  a  dummy   12.30  233  12    NaN
2  b      a  123.23  123   1    NaN
3  a      b    1.00    1   1    NaN

In [32]: df.pivot_table(index=['a', 'b'], values=['c', 'd', 'e'], aggfunc=sum)
Out[32]: 
              c    d   e
a b                     
a b       13.00   13  13
  dummy   12.30  233  12
b a      123.23  123   1

In [33]: df.pivot_table(index=['a', 'b'], values=['c', 'd', 'e'], aggfunc=sum).reset_index().replace('dummy',np.nan).set_index(['a','b'])
Out[33]: 
            c    d   e
a b                   
a b     13.00   13  13
  NaN   12.30  233  12
b a    123.23  123   1

Currently the option "dropna=False" is supported by pivot_table:

df.pivot_table(rows=['a', 'b'], values=['c', 'd', 'e'], aggfunc=sum, dropna=False)

Tags:

Python

Pandas