AWS S3 Sync very slow when copying to large directories

The sync command will need to enumerate all of the files in the bucket to determine whether a local file already exists in the bucket and if it is the same as the local file. The more documents you have in the bucket, the longer it's going to take.

If you don't need this sync behavior just use a recursive copy command like:

aws s3 cp --recursive . s3://mybucket/

and this should copy all of the local files in the current directory to the bucket in S3.


If you use the unofficial s3cmd from S3 Tools, you can use the --no-check-md5 option while using sync to disable the MD5 sums comparison to significantly speed up the process.

--no-check-md5        Do not check MD5 sums when comparing files for [sync].
                        Only size will be compared. May significantly speed up
                        transfer but may also miss some changed files.

Source: https://s3tools.org/usage

Example: s3cmd --no-check-md5 sync /directory/to/sync s3://mys3bucket/