Amazon S3: How to get a list of folders in the bucket?

For the sake of example, assume I have a bucket in the USEast1 region called MyBucketName, with the following keys:

 temp/
 temp/foobar.txt
 temp/txt/
 temp/txt/test1.txt
 temp/txt/test2.txt
 temp2/

Working with folders can be confusing because S3 does not natively support a hierarchy structure -- rather, these are simply keys like any other S3 object. Folders are simply an abstraction available in the S3 web console to make it easier to navigate a bucket. So when we're working programatically, we want to find keys matching the dimensions of a 'folder' (delimiter '/', size = 0) because they will likely be 'folders' as presented to us by the S3 console.

Note for both examples: I'm using the AWSSDK.S3 version 3.1 NuGet package.

Example 1: All folders in a bucket

This code is modified from this basic example in the S3 documentation to list all keys in a bucket. The example below will identify all keys that end with the delimiter character /, and are also empty.

IAmazonS3 client;
using (client = new AmazonS3Client(Amazon.RegionEndpoint.USEast1))
{
    // Build your request to list objects in the bucket
    ListObjectsRequest request = new ListObjectsRequest
    {
        BucketName = "MyBucketName"
    };

    do
    {
        // Build your call out to S3 and store the response
        ListObjectsResponse response = client.ListObjects(request);

        // Filter through the response to find keys that:
        // - end with the delimiter character '/' 
        // - are empty. 
        IEnumerable<S3Object> folders = response.S3Objects.Where(x =>
            x.Key.EndsWith(@"/") && x.Size == 0);

        // Do something with your output keys.  For this example, we write to the console.
        folders.ToList().ForEach(x => System.Console.WriteLine(x.Key));

        // If the response is truncated, we'll make another request 
        // and pull the next batch of keys
        if (response.IsTruncated)
        {
            request.Marker = response.NextMarker;
        }
        else
        {
            request = null;
        }
    } while (request != null);
}

Expected output to console:

temp/
temp/txt/
temp2/

Example 2: Folders matching a specified prefix

You could further limit this to only retrieve folders matching a specified Prefix by setting the Prefix property on ListObjectsRequest.

ListObjectsRequest request = new ListObjectsRequest
    {
        BucketName = "MyBucketName",
        Prefix = "temp/"
    };

When applied to Example 1, we would expect the following output:

temp/
temp/txt/

Further reading:

• S3 Documentation - Working With Folders
• .NET SDK Documentation - ListObjects

Using prefix of the/path/to/read/ (note that there is no leading slash, but there is a trailing slash), and delimiter of /, you'll find all the folders within that folder inside <CommonPrefixes>.

CommonPrefixes

A response can contain CommonPrefixes only if you specify a delimiter. When you do, CommonPrefixes contains all (if there are any) keys between Prefix and the next occurrence of the string specified by delimiter. In effect, CommonPrefixes lists keys that act like subdirectories in the directory specified by Prefix. For example, if prefix is notes/ and delimiter is a slash (/), in notes/summer/july, the common prefix is notes/summer/. All of the keys rolled up in a common prefix count as a single return when calculating the number of returns. See MaxKeys.

http://docs.aws.amazon.com/AmazonS3/latest/API/RESTBucketGET.html