Python pairwise comparison of elements in a array or list
Perhaps you want:
[x >= y for i,x in enumerate(a) for j,y in enumerate(a) if i != j]
This will not compare any item against itself, but compare each of the others against each other.
I'd like to apply @Divakar's solution to pandas objects. Here are two approaches for calculating pairwise absolute differences.
(IPython 6.1.0 on Python 3.6.2)
In [1]: import pandas as pd
...: import numpy as np
...: import itertools
In [2]: n = 256
...: labels = range(n)
...: ser = pd.Series(np.random.randn(n), index=labels)
...: ser.head()
Out[2]:
0 1.592248
1 -1.168560
2 -1.243902
3 -0.133140
4 -0.714133
dtype: float64
Loops
In [3]: %%time
...: result = dict()
...: for pair in itertools.combinations(labels, 2):
...: a, b = pair
...: a = ser[a] # retrieve values
...: b = ser[b]
...: result[pair] = a - b
...: result = pd.Series(result).abs().reset_index()
...: result.columns = list('ABC')
...: df1 = result.pivot('A', 'B, 'C').reindex(index=labels, columns=labels)
...: df1 = df1.fillna(df1.T).fillna(0.)
CPU times: user 18.2 s, sys: 468 ms, total: 18.7 s
Wall time: 18.7 s
NumPy broadcast
In [4]: %%time
...: arr = ser.values
...: arr = arr[:, None] - arr
...: df2 = pd.DataFrame(arr, labels, labels).abs()
CPU times: user 816 µs, sys: 432 µs, total: 1.25 ms
Wall time: 675 µs
Verify they're equal:
In [5]: df1.equals(df2)
Out[5]: True
Using loops is about 20000 times slower than the clever NumPy approach. NumPy has many optimizations, but sometimes they need a different way of thinking. :-)
You could use NumPy broadcasting
-
# Get the mask of comparisons in a vectorized manner using broadcasting
mask = a[:,None] >= a
# Select the elements other than diagonal ones
out = mask[~np.eye(a.size,dtype=bool)]
If you rather prefer to set the diagonal elements as False
in mask
and then mask
would be the output, like so -
mask[np.eye(a.size,dtype=bool)] = 0
Sample run -
In [56]: a
Out[56]: array([3, 7, 5, 8])
In [57]: mask = a[:,None] >= a
In [58]: mask
Out[58]:
array([[ True, False, False, False],
[ True, True, True, False],
[ True, False, True, False],
[ True, True, True, True]], dtype=bool)
In [59]: mask[~np.eye(a.size,dtype=bool)] # Selecting non-diag elems
Out[59]:
array([False, False, False, True, True, False, True, False, False,
True, True, True], dtype=bool)
In [60]: mask[np.eye(a.size,dtype=bool)] = 0 # Setting diag elems as False
In [61]: mask
Out[61]:
array([[False, False, False, False],
[ True, False, True, False],
[ True, False, False, False],
[ True, True, True, False]], dtype=bool)
Runtime test
Reasons to use NumPy broadcasting
? Performance! Let's see how with a large dataset -
In [34]: def pairwise_comp(A): # Using NumPy broadcasting
...: a = np.asarray(A) # Convert to array if not already so
...: mask = a[:,None] >= a
...: out = mask[~np.eye(a.size,dtype=bool)]
...: return out
...:
In [35]: a = np.random.randint(0,9,(1000)).tolist() # Input list
In [36]: %timeit [x >= y for i,x in enumerate(a) for j,y in enumerate(a) if i != j]
1 loop, best of 3: 185 ms per loop # @Sixhobbits's loopy soln
In [37]: %timeit pairwise_comp(a)
100 loops, best of 3: 5.76 ms per loop