Pandas: Count the first consecutive True values
Question:
Find the count of the first consecutive True
s
Consider a
a = np.array([True, True, True, False, True, False, True, True, True, True])
Answer 1numpy
: Use np.logical_and.accumulate
on the negation of a
and take the negation of that to make a mask that eliminates the first series of False
s if they should exist. Then append a False
at the end to ensure we have a non True
min. Finally, use np.argmin
to locate the first minimum value. If it's found a position 3
, that will indicate 3
True
values before it.
np.argmin(np.append(a[~np.logical_and.accumulate(~a)], False))
3
Answer 2numba.njit
I'd like to use numba
so I can loop and make sure I get to short circuit when we want/need to. This is a problem that is sure to be answered early in the array. There isn't need to evaluate things along the entire array for no reason.
from numba import njit
@njit
def first_true(a):
true_started = False
c = 0
for i, j in enumerate(a):
if true_started and not j:
return c
else:
c += j
true_started = true_started or j
return c
first_true(a)
3
Answer 3numpy
smarter use of argmin
and argmax
. I surround a
with False
then use argmax
to find the first True
then from that point on, use argmin
to find the first False
after that.
Note: @Divakar made an improvement on this answer that eliminates the use of np.concatenate
and uses if/then/else
instead. That cut this already very fast solution by a factor of 3
!
def first_true2(a):
a = np.concatenate([[False], a, [False]])
return np.argmin(a[np.argmax(a):])
first_true2(a)
3
How fast are these answers?
See @Divakar's Answer for source code of other functions being timed
%timeit first_true(a)
%timeit np.argmin(np.append(a[~np.logical_and.accumulate(~a)], False))
%timeit np.diff(np.flatnonzero(np.diff(np.r_[0,a,0])))[0]
%timeit first_True_island_len(a)
%timeit first_true2(a)
%timeit first_True_island_len_IFELSE(a)
a = np.array([True, True, True, False, True, False, True, True, True, True])
1000000 loops, best of 3: 353 ns per loop
100000 loops, best of 3: 8.32 µs per loop
10000 loops, best of 3: 27.4 µs per loop
100000 loops, best of 3: 5.48 µs per loop
100000 loops, best of 3: 5.38 µs per loop
1000000 loops, best of 3: 1.35 µs per loop
a = np.array([False] * 100000 + [True] * 10000)
10000 loops, best of 3: 112 µs per loop
10000 loops, best of 3: 127 µs per loop
1000 loops, best of 3: 513 µs per loop
10000 loops, best of 3: 110 µs per loop
100000 loops, best of 3: 13.9 µs per loop
100000 loops, best of 3: 4.55 µs per loop
a = np.array([False] * 100000 + [True])
10000 loops, best of 3: 102 µs per loop
10000 loops, best of 3: 115 µs per loop
1000 loops, best of 3: 472 µs per loop
10000 loops, best of 3: 108 µs per loop
100000 loops, best of 3: 14 µs per loop
100000 loops, best of 3: 4.45 µs per loop
Using NumPy funcs, one solution would be -
np.diff(np.flatnonzero(np.diff(np.r_[0,s,0])))[0]
Sample run -
In [16]: s
Out[16]:
0 True
1 True
2 True
3 False
4 True
5 False
6 True
7 True
8 True
9 True
dtype: bool
In [17]: np.diff(np.flatnonzero(np.diff(np.r_[0,s,0])))[0]
Out[17]: 3
For performance, we need to use np.concatenate
in place np.r_
and then slicing to replace the last differentiation -
def first_True_island_len(a): # a is NumPy array
v = np.concatenate(([False],a,[False]))
idx = np.flatnonzero(v[1:] != v[:-1])
if len(idx)>0:
return idx[1] - idx[0]
else:
return 0
Inspired by @piRSquared's argmax
and argmin
trickery, here's one more with a bunch of IF-ELSE
's -
def first_True_island_len_IFELSE(a): # a is NumPy array
maxidx = a.argmax()
pos = a[maxidx:].argmin()
if a[maxidx]:
if pos==0:
return a.size - maxidx
else:
return pos
else:
return 0
Try this way will find the first consecutive occurrences for True
or False
, and only for True
:
import pandas as pd
df = pd.DataFrame([True, True, True, False, True, False, True, True, True, True],columns=["Boolean"])
df['consecutive'] = df.Boolean.groupby((df.Boolean != df.Boolean.shift()).cumsum()).transform('size')
count_true_false = df['consecutive'][df['consecutive']>1].iloc[0] # count first consecutive occurrences for True or False
count_true = df['consecutive'][(df.Boolean == True) & (df.consecutive>1)].iloc[0] # count first consecutive occurrences for True
print count_true_false
print count_true
Output:
3
3