Python and Django - How to use in memory and temporary files

If I understand correctly you are looking for a way to access the uploaded file before it is saved using instance.save() or similar.

If that is the case, you might try to read the file directly from request:

if my_form.is_valid():
    data = request.FILES['myfile'].read()

You can set which upload handler is used for a Django form: https://docs.djangoproject.com/en/1.11/ref/files/uploads/#module-django.core.files.uploadhandler

There are two built in options:

• in memory
• temporary file

The way Django seems to work is by walking through the list of all upload handlers: https://docs.djangoproject.com/en/1.11/topics/http/file-uploads/#upload-handlers

For each upload handler it will check for a specific condition and if that condition is true, it will activate and use that upload handler.

The InMemoryUploadHandler, for example, is activated and used when files are below a certain size. The TemporaryFileUploadHandler is used when files are very large.

You will be able to access the data and files from the request object.

If you're looking at Python specific (not Django), then it sounds like you may be interested in the tempfile module:

https://docs.python.org/3/library/tempfile.html

Adapting the example from the document above:

import tempfile

fp = tempfile.TemporaryFile()
fp.write(b'Hello world!')
# Closing automatically deletes the tempfile
fp.close()

You can see this works fairly similarly to generic read/write file operations.

When a file is uploaded Django will do one of two things: store it in memory if the file is small (< 2 MB last time I checked), or store it as a temporary file on disk if it's large. This behavior is configurable via the FILE_UPLOAD_HANDLERS setting. So, your web server and Django take care of the actual upload and storage, but it's your job to process the file before the request is over, otherwise the file is deleted.

Uploaded files are accessible through the request.FILES property. Each key in FILES will match the name of the file input on your <form>. The value is an UploadedFile object, which is a stream you can use to read the file data.

For example, say you have an <input name="img" type="file" /> and you want to detect if the image is completely white. You don't need to store the file for this, you just need to load it into memory, process it to get the result and then let it be discarded.

from PIL import Image

def some_view(request):
    if request.method == 'POST':
        img_file = request.FILES['img']

        if img_file.size > 2000000:
            return HttpResponseBadRequest()

        img = Image.open(img_file)

        # analyze the image...

Another possibility is that someone is uploading a backup file that is quite large (lets say 2 GB), and you need to store it somewhere. It's effectively the same thing, except we read the file into memory in chunks, then write each chunk to disk somewhere else so that it's saved after the request finishes.

def some_view(request):
    if request.method == 'POST':
        backup_file = request.FILES['backup_file']
        with open('some/file/name.bak', 'wb+') as destination:
            for chunk in backup_file.chunks():
                destination.write(chunk)

        # file is saved

When the request is over, the uploaded file is stored at some/file/name.bak.

Whether it's in memory or a temporary file is usually not important because the interface is the same. You can read a temporary file just like you can read an in memory file.