How to list _all_ objects in Amazon S3 bucket?

Be aware that the answer above is not using the recommended API to List Objects: http://docs.aws.amazon.com/AmazonS3/latest/API/v2-RESTBucketGET.html

The following snippet shows how it looks with the new API:

using (var s3Client = new AmazonS3Client(Amazon.RegionEndpoint.USEast1))
{
    ListObjectsV2Request request = new ListObjectsV2Request
    {
          BucketName = bucketName,
          MaxKeys = 10
    };
    ListObjectsV2Response response;
    do
    {
         response = await s3Client.ListObjectsV2Async(request);

         // Process response.
         // ...

         request.ContinuationToken = response.NextContinuationToken;

    } while (response.IsTruncated);        
}

Use a paginator, introduced here. Amazon recommends these as a cleaner syntax compared to a do-while, and they take advantage of C# 8's IAsyncEnumerable.

IAmazonS3 S3;
CancellationToken cancellationToken;

var results = new List<S3Object>();

var paginator = S3.Paginators.ListObjectsV2(new ListObjectsV2Request
{
    BucketName = "my-bucket-name",
    Prefix = "my/prefix",
});
await foreach (var response in paginator.Responses.WithCancellation(cancellationToken))
{
    results.AddRange(response.S3Objects);
}

return results;

As stated already, Amazon S3 indeed requires Listing Keys Using the AWS SDK for .NET:

As buckets can contain a virtually unlimited number of keys, the complete results of a list query can be extremely large. To manage large result sets, Amazon S3 uses pagination to split them into multiple responses. Each list keys response returns a page of up to 1,000 keys with an indicator indicating if the response is truncated. You send a series of list keys requests until you have received all the keys.

The mentioned indicator is the NextMarker property from the ObjectsResponse Class - its usage is illustrated in the complete example Listing Keys Using the AWS SDK for .NET, with the relevant fragment being:

static AmazonS3 client;
client = Amazon.AWSClientFactory.CreateAmazonS3Client(
                    accessKeyID, secretAccessKeyID);

ListObjectsRequest request = new ListObjectsRequest();
request.BucketName = bucketName;
do
{
   ListObjectsResponse response = client.ListObjects(request);

   // Process response.
   // ...

   // If response is truncated, set the marker to get the next 
   // set of keys.
   if (response.IsTruncated)
   {
        request.Marker = response.NextMarker;
   }
   else
   {
        request = null;
   }
} while (request != null);

API version has been changed; so need to do as below:

            ListObjectsV2Request request = new ListObjectsV2Request
            {
                BucketName = bucketName,
                MaxKeys = 10
            };
            ListObjectsV2Response response;
            do
            {
                response = await client.ListObjectsV2Async(request);

                // Process the response.
                foreach (S3Object entry in response.S3Objects)
                {
                    Console.WriteLine("key = {0} size = {1}",
                        entry.Key, entry.Size);
                }
                Console.WriteLine("Next Continuation Token: {0}", response.NextContinuationToken);
                request.ContinuationToken = response.NextContinuationToken;
            } while (response.IsTruncated);

For full details, please see https://docs.aws.amazon.com/AmazonS3/latest/dev/ListingObjectKeysUsingNetSDK.html