quantile normalization on pandas dataframe
One thing worth noticing is that both ayhan and shawn's code use the smaller rank mean for ties, but if you use R package processcore's normalize.quantiles()
, it would use the mean of rank means for ties.
Using the above example:
> df
C1 C2 C3
A 5 4 3
B 2 1 4
C 3 4 6
D 4 2 8
> normalize.quantiles(as.matrix(df))
C1 C2 C3
A 5.666667 5.166667 2.000000
B 2.000000 2.000000 3.000000
C 3.000000 5.166667 4.666667
D 4.666667 3.000000 5.666667
Ok I implemented the method myself of relatively high efficiency.
After finishing, this logic seems kind of easy but, anyway, I decided to post it here for any one feels confused like I was when I couldn't googled the available code.
The code is in github: Quantile Normalize
Using the example dataset from Wikipedia article:
df = pd.DataFrame({'C1': {'A': 5, 'B': 2, 'C': 3, 'D': 4},
'C2': {'A': 4, 'B': 1, 'C': 4, 'D': 2},
'C3': {'A': 3, 'B': 4, 'C': 6, 'D': 8}})
df
Out:
C1 C2 C3
A 5 4 3
B 2 1 4
C 3 4 6
D 4 2 8
For each rank, the mean value can be calculated with the following:
rank_mean = df.stack().groupby(df.rank(method='first').stack().astype(int)).mean()
rank_mean
Out:
1 2.000000
2 3.000000
3 4.666667
4 5.666667
dtype: float64
Then the resulting Series, rank_mean
, can be used as a mapping for the ranks to get the normalized results:
df.rank(method='min').stack().astype(int).map(rank_mean).unstack()
Out:
C1 C2 C3
A 5.666667 4.666667 2.000000
B 2.000000 2.000000 3.000000
C 3.000000 4.666667 4.666667
D 4.666667 3.000000 5.666667