How to calculate percentage of sparsity for a numpy array/matrix?
Measuring the percentage of missing values has already explained by 'hpaulj'.
I am taking the first part of your question, Assuming array has Zero's and Non-Zero's...
Sparsity refers to Zero values and density refers to Non-Zero values in array. Suppose your array is X, get count of non-zero values:
non_zero = np.count_nonzero(X)
total values in X:
total_val = np.product(X.shape)
Sparsity will be -
sparsity = (total_val - non_zero) / total_val
And Density will be -
density = non_zero / total_val
The sum of Sparsity and Density must equal to 100%...
Definition:
Code for a general case:
from numpy import array
from numpy import count_nonzero
import numpy as np
# create dense matrix
A = array([[1, 1, 0, 1, 0, 0], [1, 0, 2, 0, 0, 1], [99, 0, 0, 2, 0, 0]])
#If you have Nan
A = np.nan_to_num(A,0)
print(A)
#[[ 1 1 0 1 0 0]
# [ 1 0 2 0 0 1]
# [99 0 0 2 0 0]]
# calculate sparsity
sparsity = 1.0 - ( count_nonzero(A) / float(A.size) )
print(sparsity)
Results:
0.555555555556
np.isnan(a).sum()
gives the number of nan
values, in this example 8.
np.prod(a.shape)
is the number of values, here 50. Their ratio should give the desired value.
In [1081]: np.isnan(a).sum()/np.prod(a.shape)
Out[1081]: 0.16
You might also find it useful to make a masked array from this
In [1085]: a_ma=np.ma.masked_invalid(a)
In [1086]: print(a_ma)
[[0.0 0.0 0.0 0.0 1.0]
[1.0 1.0 0.0 -- --]
[0.0 -- 1.0 -- --]
[1.0 1.0 1.0 1.0 0.0]
[0.0 0.0 0.0 1.0 0.0]
[0.0 0.0 0.0 0.0 --]
[-- -- 1.0 1.0 1.0]
[0.0 1.0 0.0 1.0 0.0]
[1.0 0.0 1.0 0.0 0.0]
[0.0 1.0 0.0 0.0 0.0]]
The number of valid values then is:
In [1089]: a_ma.compressed().shape
Out[1089]: (42,)