How to iterate over files in an S3 bucket?
I found out that boto3
has a Paginator class to deal with truncated results. The following worked for me:
paginator = client.get_paginator('list_objects')
page_iterator = paginator.paginate(Bucket='iper-apks')
after which I can use the page_iterator
generator in a for
loop.
As kurt-peek notes, boto3
has a Paginator
class, which allows you to iterator over pages of s3 objects, and can easily be used to provide an iterator over items within the pages:
import boto3
def iterate_bucket_items(bucket):
"""
Generator that iterates over all objects in a given s3 bucket
See http://boto3.readthedocs.io/en/latest/reference/services/s3.html#S3.Client.list_objects_v2
for return data format
:param bucket: name of s3 bucket
:return: dict of metadata for an object
"""
client = boto3.client('s3')
paginator = client.get_paginator('list_objects_v2')
page_iterator = paginator.paginate(Bucket=bucket)
for page in page_iterator:
if page['KeyCount'] > 0:
for item in page['Contents']:
yield item
for i in iterate_bucket_items(bucket='my_bucket'):
print i
Which will output something like:
{u'ETag': '"a8a9ee11bd4766273ab4b54a0e97c589"',
u'Key': '2017-06-01-10-17-57-EBDC490AD194E7BF',
u'LastModified': datetime.datetime(2017, 6, 1, 10, 17, 58, tzinfo=tzutc()),
u'Size': 242,
u'StorageClass': 'STANDARD'}
{u'ETag': '"03be0b66e34cbc4c037729691cd5efab"',
u'Key': '2017-06-01-10-28-58-732EB022229AACF7',
u'LastModified': datetime.datetime(2017, 6, 1, 10, 28, 59, tzinfo=tzutc()),
u'Size': 238,
u'StorageClass': 'STANDARD'}
...
Note that list_objects_v2
is recommended instead of list_objects
: https://docs.aws.amazon.com/AmazonS3/latest/API/RESTBucketGET.html
You can also do this at a lower level by calling list_objects_v2()
directly and passing in the NextContinuationToken
value from the response as ContinuationToken
while isTruncated
is true in the response.