How to list _all_ objects in Amazon S3 bucket?
Be aware that the answer above is not using the recommended API to List Objects: http://docs.aws.amazon.com/AmazonS3/latest/API/v2-RESTBucketGET.html
The following snippet shows how it looks with the new API:
using (var s3Client = new AmazonS3Client(Amazon.RegionEndpoint.USEast1))
{
ListObjectsV2Request request = new ListObjectsV2Request
{
BucketName = bucketName,
MaxKeys = 10
};
ListObjectsV2Response response;
do
{
response = await s3Client.ListObjectsV2Async(request);
// Process response.
// ...
request.ContinuationToken = response.NextContinuationToken;
} while (response.IsTruncated);
}
Use a paginator, introduced here. Amazon recommends these as a cleaner syntax compared to a do-while, and they take advantage of C# 8's IAsyncEnumerable
.
IAmazonS3 S3;
CancellationToken cancellationToken;
var results = new List<S3Object>();
var paginator = S3.Paginators.ListObjectsV2(new ListObjectsV2Request
{
BucketName = "my-bucket-name",
Prefix = "my/prefix",
});
await foreach (var response in paginator.Responses.WithCancellation(cancellationToken))
{
results.AddRange(response.S3Objects);
}
return results;
As stated already, Amazon S3 indeed requires Listing Keys Using the AWS SDK for .NET:
As buckets can contain a virtually unlimited number of keys, the complete results of a list query can be extremely large. To manage large result sets, Amazon S3 uses pagination to split them into multiple responses. Each list keys response returns a page of up to 1,000 keys with an indicator indicating if the response is truncated. You send a series of list keys requests until you have received all the keys.
The mentioned indicator is the NextMarker property from the ObjectsResponse Class - its usage is illustrated in the complete example Listing Keys Using the AWS SDK for .NET, with the relevant fragment being:
static AmazonS3 client;
client = Amazon.AWSClientFactory.CreateAmazonS3Client(
accessKeyID, secretAccessKeyID);
ListObjectsRequest request = new ListObjectsRequest();
request.BucketName = bucketName;
do
{
ListObjectsResponse response = client.ListObjects(request);
// Process response.
// ...
// If response is truncated, set the marker to get the next
// set of keys.
if (response.IsTruncated)
{
request.Marker = response.NextMarker;
}
else
{
request = null;
}
} while (request != null);
API version has been changed; so need to do as below:
ListObjectsV2Request request = new ListObjectsV2Request
{
BucketName = bucketName,
MaxKeys = 10
};
ListObjectsV2Response response;
do
{
response = await client.ListObjectsV2Async(request);
// Process the response.
foreach (S3Object entry in response.S3Objects)
{
Console.WriteLine("key = {0} size = {1}",
entry.Key, entry.Size);
}
Console.WriteLine("Next Continuation Token: {0}", response.NextContinuationToken);
request.ContinuationToken = response.NextContinuationToken;
} while (response.IsTruncated);
For full details, please see https://docs.aws.amazon.com/AmazonS3/latest/dev/ListingObjectKeysUsingNetSDK.html