pandas: DataFrame.mean() very slow. How can I calculate means of columns faster?

Here's a similar sized from , but without an object column

In [10]: nrows = 10000000

In [11]: df = pd.concat([DataFrame(randn(int(nrows),34),columns=[ 'f%s' % i for i in range(34) ]),DataFrame(randint(0,10,size=int(nrows*19)).reshape(int(nrows),19),columns=[ 'i%s' % i for i in range(19) ])],axis=1)

In [12]: df.iloc[1000:10000,0:20] = np.nan

In [13]: df.info()
<class 'pandas.core.frame.DataFrame'>
Int64Index: 10000000 entries, 0 to 9999999
Data columns (total 53 columns):
f0     9991000  non-null values
f1     9991000  non-null values
f2     9991000  non-null values
f3     9991000  non-null values
f4     9991000  non-null values
f5     9991000  non-null values
f6     9991000  non-null values
f7     9991000  non-null values
f8     9991000  non-null values
f9     9991000  non-null values
f10    9991000  non-null values
f11    9991000  non-null values
f12    9991000  non-null values
f13    9991000  non-null values
f14    9991000  non-null values
f15    9991000  non-null values
f16    9991000  non-null values
f17    9991000  non-null values
f18    9991000  non-null values
f19    9991000  non-null values
f20    10000000  non-null values
f21    10000000  non-null values
f22    10000000  non-null values
f23    10000000  non-null values
f24    10000000  non-null values
f25    10000000  non-null values
f26    10000000  non-null values
f27    10000000  non-null values
f28    10000000  non-null values
f29    10000000  non-null values
f30    10000000  non-null values
f31    10000000  non-null values
f32    10000000  non-null values
f33    10000000  non-null values
i0     10000000  non-null values
i1     10000000  non-null values
i2     10000000  non-null values
i3     10000000  non-null values
i4     10000000  non-null values
i5     10000000  non-null values
i6     10000000  non-null values
i7     10000000  non-null values
i8     10000000  non-null values
i9     10000000  non-null values
i10    10000000  non-null values
i11    10000000  non-null values
i12    10000000  non-null values
i13    10000000  non-null values
i14    10000000  non-null values
i15    10000000  non-null values
i16    10000000  non-null values
i17    10000000  non-null values
i18    10000000  non-null values
dtypes: float64(34), int64(19)

Timings (similar machine specs to you)

In [14]: %timeit df.mean()
1 loops, best of 3: 21.5 s per loop

You can get a 2x speedup by pre-converting to floats (mean does this, but does it in a more general way, so slower)

In [15]: %timeit df.astype('float64').mean()
1 loops, best of 3: 9.45 s per loop

You problem is the object column. Mean will try to calculate for all of the columns, but because of the object column everything is upcast to object dtype which is not efficient for calculating.

Best bet is to do

 df._get_numeric_data().mean()

There is an option to do this numeric_only, at the lower level, but for some reason we don't directly support this via the top-level functions (e.g. mean). I think will create an issue to add this parameter. However will prob be False by default (to not-exclude).