how to efficiently exhaust an iterator in a oneliner?
From the itertools
recipes:
# feed the entire iterator into a zero-length deque
collections.deque(iterator, maxlen=0)
Note that your suggestion can also be formulated as a one-liner:
for _ in it: pass
And I just made:
def exhaust(it):
for _ in it:
pass
It's not as fast as the deque
solution (10% slower on my laptop), but I find it cleaner.
2022 update (bounty asks): There's no "dedicated function" for it in the standard library, and deque(it, 0)
is still the most efficient. That's why it's used in itertools's consume
recipe and more-itertools's consume
function (click on [source] there).
Benchmark of the various proposals, iteration time per element, iterating itertools.repeat(None, 10**5)
(with CPython 3.10):
2.7 ns ± 0.1 ns consume_deque
6.5 ns ± 0.0 ns consume_loop
6.5 ns ± 0.0 ns consume_all_if_False
13.9 ns ± 0.3 ns consume_object_in
27.0 ns ± 0.1 ns consume_all_True
29.4 ns ± 0.3 ns consume_sum_0
44.8 ns ± 0.1 ns consume_reduce
The deque
one wins due to being C and having a fast path for maxlen == 0
which does nothing with the elements.
The simple loop gets second place, fastest with Python iteration. The other solutions previously proposed here waste more or less time by doing more or less work with each element. I added consume_all_if_False
to show how to do an all
/sum
efficiently: have an if False
clause so your generator doesn't produce anything.
Benchmark code (Try it online!):
def consume_loop(it):
for _ in it:
pass
def consume_deque(it):
deque(it, 0)
def consume_object_in(it):
object() in it
def consume_all_True(it):
all(True for _ in it)
def consume_all_if_False(it):
all(_ for _ in it if False)
def consume_sum_0(it):
sum(0 for _ in it)
def consume_reduce(it):
reduce(lambda x, y: y, it)
funcs = [
consume_loop,
consume_deque,
consume_object_in,
consume_all_True,
consume_all_if_False,
consume_sum_0,
consume_reduce,
]
from timeit import default_timer as timer
from itertools import repeat
from collections import deque
from functools import reduce
from random import shuffle
from statistics import mean, stdev
times = {f: [] for f in funcs}
def stats(f):
ts = [t * 1e9 for t in sorted(times[f])[:5]]
return f'{mean(ts):5.1f} ns ± {stdev(ts):3.1f} ns'
for _ in range(25):
shuffle(funcs)
for f in funcs:
n = 10**5
it = repeat(None, n)
t0 = timer()
f(it)
t1 = timer()
times[f].append((t1 - t0) / n)
for f in sorted(funcs, key=stats):
print(stats(f), f.__name__)