Passing generator expressions to any() and all()
The problem that you are having is that you are using the generator after it has produced all the values.
You can verify this by running the following code:
>>> bools = (b for b in (True, False, True, True))
>>> all(bools) # once the False is found it will stop producing values
True
>>> next(bools) # next value after False which is True
True
>>> next(bools) # next value after True which is True
True
>>> next(bools)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
StopIteration
This will work:
>>> bools = (b for b in (True, False, True, True))
>>> all(bools)
False
>>> bools = (b for b in (True, False, True, True))
>>> any(bools)
True
The behaviour of all()
and any()
are documented in the official documentation.
From the pseudo-code:
def all(iterable):
for element in iterable:
if not element:
return False
return True
all()
only consumes True
elements, it terminates when it finds the first element that evaluates to False
.
def any(iterable):
for element in iterable:
if element:
return True
return False
any()
consumes only False
elements, it terminates when it finds the first element that evaluates to True
.
Note that generators are not reset to their initial position when passed around. They stay at their current position unless more items are consumed. Hence,
>>> bools = (b for b in (True, False, True, True))
The following will consume the first two items. Since the second item is False
, the iteration stops after that. This leaves the generator at a position after the second element.
>>> all(bools)
False
At this point the generator has (True, True)
as the remaining values. You point that out correctly in your question. The following only consumes a single element.
>>> any(bools)
True
Note that there is still another True
value obtainable
from the generator after calling any()
.
And of course, if you call list()
on a generator, all items from the generator are consumed and the generator will not yield any more items (it is "empty").
A couple things are at play here.
The first thing is that generators can run exactly once for each element they're given. Unlike lists, or tuples, or any other objects with a fixed state, generators know what the __next__
value is, how to generate the value after that, and basically nothing else. When you call next(generator)
, you get that next value, the generator figures out a new __next__
, and it completely loses memory of the value you just obtained. In essence, generators can't be used multiple times in a row.
The second thing is how all()
, any()
, and list()
work internally, especially vis-a-vis generators. all()
's implementation looks something like this, only more complicated:
def all(iterable):
for element in iterable:
if bool(element) is False:
return False
return True
That is, the all()
function short-circuits when it first finds a non-truthy element (and any()
does the same thing, except the reverse). This is to save on processing time - why process the rest of the iterable if just the first element is unacceptable? In the case of a generator (e.g. your last example), this means it consumes all elements up until it finds a False
. The generator still has elements left, but since it's already yielded those first two, it will behave in the future as though they never existed.
list()
is simpler, and just calls next(generator)
until the generator stops producing values. This makes the generator give up any values it hasn't yet consumed.
So the explanation for your last example is that
- You create a generator that will spit out the elements
True, False, True, True
in order - You call
all()
on that generator, and it consumes the first two elements of the generator before it terminates, having found a falsey value. - You call
list()
on that generator, and it consumes all remaining elements of the generator (that is, the last two) to create a list. It produces[2, 2]
.