How to generate a temporary url to upload file to Amazon S3 with boto library?

Here's what it looks like in boto3(tested with version 1.2.3).

First, create a presigned url with s3.generate_presigned_url method:

>>> import boto3
>>> s3 = boto3.client('s3')
>>> s3.generate_presigned_url('put_object', Params={'Bucket':'YourBucket', 'Key':'YourKey'}, ExpiresIn=3600, HttpMethod='PUT')
u'https://s3-ap-northeast-1.amazonaws.com/YourBucket/YourKey?AWSAccessKeyId=AKIAXXXXXXXXXXXXXXXX&Expires=1451061671&Signature=%2FtyAyCd5vrp13p%2FqLdoPkox7yTM%3D'

PUT to S3 with a presigned URL

$ curl \
  --request PUT \
  --upload-file path/to/file \
  "https://s3-ap-northeast-1.amazonaws.com/YourBucket/YourKey?AWSAccessKeyId=AKIAXXXXXXXXXXXXXXXX&Expires=1451061671&Signature=%2FtyAyCd5vrp13p%2FqLdoPkox7yTM%3D"

I found some time to experiment with this and here's what I found.

>>> import boto
>>> c =boto.connect_s3()
>>> fp = open('myfiletoupload.txt')
>>> content_length = len(fp.read())
>>> c.generate_url(300, 'PUT', 'test-1332789015', 'foobar', headers={'Content-Length': str(content_length)}, force_http=True)
'http://test-1332789015.s3.amazonaws.com/foobar?Signature=oUARG45mR95utXsiQYRJNiCI4x4%3D&Expires=1333731456&AWSAccessKeyId=AKIAJOTCCJRP4C3NSMYA&Content-Length=16'

I was then able to use curl to PUT the file to that url like this:

$ curl --request PUT --upload-file myfiletoupload.txt "http://test-1332789015.s3.amazonaws.com/foobar?Signature=oUARG45mR95utXsiQYRJNiCI4x4%3D&Expires=1333731456&AWSAccessKeyId=AKIAJOTCCJRP4C3NSMYA&Content-Length=16"

This resulted in the file being uploaded to the bucket. So, it appears that it is possible. You might want to see if you can calculate the content-md5 value and include that in the headers but then you also have to figure out how to get curl to send that header, as well. Also, you should be able to make this work over HTTPS rather than HTTP but I haven't tried that.

All the other answers assume the file will be uploaded with curl, which is really not convenient in most python scripts. In the following, a pre-signed url is generated with boto3 and the file is uploaded with the requests library:

session = boto3.Session(aws_access_key_id="XXX", aws_secret_access_key="XXX")
s3client = session.client('s3')
url = s3client.generate_presigned_url('put_object', Params={'Bucket': 'mybucket', 'Key': 'mykey'})

requests.put(url, data=open("/path/to/file").read())

This is a follow up to garnaat's answer from Apr 6 '12.

I am generating a signed URL server side, where I have credentials, and I pass it to a client such that a client can directly upload content. I trust the client far enough to allow it to upload arbitrary sized files, but not enough to give it security tokens. I wanted to avoid having the client tell the server how large its content would be as part of the request. Hence my follow up answer.

I was able to get the signed url for the PUT method working without specifying content length in the headers or specifying force_http=True.

Using Boto 2.31.1: as in garnaat's answere:

>>> import boto
>>> c =boto.connect_s3()

then instead I used:

>>> temp_url = c.generate_url(seconds_available, 'PUT', bucket_name, s3_key)

this produced a url in the following form:

https://s3_location/bucket_name/s3_key?Signature=Ew407JMktSIcFln%2FZe00VroCmTU%3D&Expires=1405647669&AWSAccessKeyId=kM__pEQo2AEVd_Juz4Qq

I was then able to use curl to post a file:

>>> os.system('curl --request PUT --upload-file true_measure/test_files/test_file_w_content.txt "'+temp_url+'"')

I did have a very difficult time figuring this out because I usually use python requests to write tests and debug; however I get an authentication failure when I try to use it to put a file to one of these boto generated signed urls using requests. I haven't fully debugged this, but I suspect it is because requests is adding a few additional headers as compared to what curl produces.

I hope this follow up answer spares someone else the debugging pain I went through.