Guzzle Pool : Wait for Requests

It seems from the question, that you are able to move aggregation callback directly to the query. In this case the pool will always wait for your processing code, so you will be able to add new requests at any point.

A generator can return either a request or a promise, and promises can be combined together in different ways.

$generator = function () {
    while ($request = array_shift($this->requests)) {
        if (isset($request['page'])) {
            $key = 'page_' . $request['page'];
        } else {
            $key = 'listing_' . $request['listing'];
        }

        yield $this->client->sendAsync('GET', $request['url'])
            ->then(function (Response $response) use ($key) {
            /*
             * The fullfillment callback is now connected to the query, so the 
             * pool will wait for it.
             * 
             * $key is also available, because it's just a closure, so 
             * no $index needed as an argument.
             */
        });
    }
    echo "Exiting...\n";
    flush();
};

$promise = \GuzzleHttp\Promise\each_limit($generator(), [
    'concurrency' => function () {
        return max(1, min(count($this->requests), 2));
    },
    //...
]);

$promise->wait();

As I said before, the full example is in the gist, with MapIterator and ExpectingIterator

Iterators dont become valid again on php < 7, your example with arrayIterator and sample with MapIterator all stop after initial pool is exhausted...

On other hand it all works on earlier versions of php provided you use ->append method on iterator instead of [] push.

Unfortunately, you cannot do that with a generator, only with a custom iterator.

I prepared a gist with the full example, but the main idea is just to create an Iterator that will change its state in both ways (it can became valid again after the end).

An example with ArrayIterator in psysh:

>>> $a = new ArrayIterator([1, 2])
=> ArrayIterator {#186
     +0: 1,
     +1: 2,
   }
>>> $a->current()
=> 1
>>> $a->next()
=> null
>>> $a->current()
=> 2
>>> $a->next()
=> null
>>> $a->valid()
=> false
>>> $a[] = 2
=> 2
>>> $a->valid()
=> true
>>> $a->current()
=> 2

With this idea in mind we can pass such dynamic iterator to Guzzle and leave it do the work:

// MapIterator mainly needed for readability.
$generator = new MapIterator(
    // Initial data. This object will be always passed as the second parameter to the callback below
    new \ArrayIterator(['http://google.com']),
    function ($request, $array) use ($httpClient, $next) {
        return $httpClient->requestAsync('GET', $request)
            ->then(function (Response $response) use ($request, $array, $next) {
                // The status code for example.
                echo $request . ': ' . $response->getStatusCode() . PHP_EOL;
                // New requests.
                $array->append($next->shift());
                $array->append($next->shift());
            });
    }
);
// The "magic".
$generator = new ExpectingIterator($generator);
// And the concurrent runner.
$promise = \GuzzleHttp\Promise\each_limit($generator, 5);
$promise->wait();

As I said before, the full example is in the gist, with MapIterator and ExpectingIterator.