How to use AWS S3 CLI to dump files to stdout in BASH?

If you are using a version of the AWS CLI that doesn't support copying to "-" you can also use /dev/stdout:

$ aws s3 cp --quiet s3://mybucket/stream.txt /dev/stdout

You also may want the --quiet flag to prevent a summary line like the following from being appended to your output:

download: s3://mybucket/stream.txt to ../../dev/stdout

dump the contents of all of the file objects to stdout.

You can accomplish this if you pass - for destination of aws s3 cp command. For example, $ aws s3 cp s3://mybucket/stream.txt -.

What you're trying to do is something like this? ::

#!/bin/bash

BUCKET=YOUR-BUCKET-NAME
for key in `aws s3api list-objects --bucket $BUCKET --prefix bucket/path/to/files/ | jq -r '.Contents[].Key'`
do
  echo $key
  aws s3 cp s3://$BUCKET/$key - | md5sum
done

You can try using s3streamcat, it supports bzip, gzip and xz formats as well.

Install with

sudo pip install s3streamcat

Usage:

s3streamcat s3://bucketname/dir/file_path
s3streamcat s3://bucketname/dir/file_path | more
s3streamcat s3://bucketname/dir/file_path | grep something