Calculating pairwise correlation among all columns
Assuming the data you have is in a pandas DataFrame.
df.corr('pearson') # 'kendall', and 'spearman' are the other 2 options
will provide you a correlation matrix between each column.
from pandas import *
import numpy as np
from libraries.settings import *
from scipy.stats.stats import pearsonr
import itertools
Creating random sample data:
df = DataFrame(np.random.random((5, 5)), columns=['gene_' + chr(i + ord('a')) for i in range(5)])
print(df)
gene_a gene_b gene_c gene_d gene_e
0 0.471257 0.854139 0.781204 0.678567 0.697993
1 0.292909 0.046159 0.250902 0.064004 0.307537
2 0.422265 0.646988 0.084983 0.822375 0.713397
3 0.113963 0.016122 0.227566 0.206324 0.792048
4 0.357331 0.980479 0.157124 0.560889 0.973161
correlations = {}
columns = df.columns.tolist()
for col_a, col_b in itertools.combinations(columns, 2):
correlations[col_a + '__' + col_b] = pearsonr(df.loc[:, col_a], df.loc[:, col_b])
result = DataFrame.from_dict(correlations, orient='index')
result.columns = ['PCC', 'p-value']
print(result.sort_index())
PCC p-value
gene_a__gene_b 0.461357 0.434142
gene_a__gene_c 0.177936 0.774646
gene_a__gene_d -0.854884 0.064896
gene_a__gene_e -0.155440 0.802887
gene_b__gene_c -0.575056 0.310455
gene_b__gene_d -0.097054 0.876621
gene_b__gene_e 0.061175 0.922159
gene_c__gene_d -0.633302 0.251381
gene_c__gene_e -0.771120 0.126836
gene_d__gene_e 0.531805 0.356315
- Get unique combinations of
DataFrame
columns usingitertools.combination(iterable, r)
- Iterate through these combinations and calculate pairwise correlations using
scipy.stats.stats.personr
- Add results (PCC and p-value tuple) to
dictionary
- Build
DataFrame
fromdictionary
You could then also save result.to_csv()
. You might find it convenient to use a MultiIndex
(two columns containing the names of each columns) instead of the created names for the pairwise correlations.
To get pairs, it is a combinations
problem. You can concat
all the rows into one the result dataframe
.
from pandas import *
from itertools import combinations
df = pandas.read_csv('gene.csv')
# get the column names as list, which are gene names
column_list = df.columns.values.tolist()
result = []
for c in combinations(column_list, 2):
firstGene, secondGene = c
firstGeneData = df[firstGene].tolist()
secondGeneData = df[secondGene].tolist()
# now to get the PCC, P-value using scipy
pcc = ...
p-value = ...
result.append(pandas.DataFrame([{'PCC': pcc, 'P-value': p-value}], index=str(firstGene)+ '_' + str(secondGene), columns=['PCC', 'P-value'])
result_df = pandas.concat(result)
#result_df.to_csv(...)
A simple solution is to use the pairwise_corr function of the Pingouin package (which I created):
import pingouin as pg
pg.pairwise_corr(data, method='pearson')
This will give you a DataFrame with all combinations of columns, and, for each of those, the r-value, p-value, sample size, and more.
There are also a number of options to specify one or more columns (e.g. one-vs-all behavior), as well as covariates for partial correlation and different methods to calculate the correlation coefficient. Please see this example Jupyter Notebook for a more in-depth demo.