Creating large zip files in AWS S3 in chunks
What I need to do is to give the client the ability to download them all in a ZIP (or similar), but I can't do it in memory neither on the server storage as this is a serverless setup.
When you say server less, if what you mean is that you would like to use Lambda to create a zip file in S3, you will run into a few limitations:
- Lambda has a time limit on how long functions can execute.
- As Lambda has a memory limit, you may have trouble assembling a large file in a Lambda function
- Lambda has a limit on the maximum size of a PUT call.
For the above reasons, I think the following approach is better:
- When the files are required, create an EC2 instance on the fly. Perhaps your lambda function can trigger creation of the EC2 instance.
- copy all the files into the instance store of the machine or even EFS.
- Compress the files into a zip
- Upload the file back to S3 or serve the file directly
- Kill the EC2 instance.
In my opinion this would greatly simplify the code you have to write, as any code that runs on your laptop / desktop will probably work on the EC2 instance. You also won't have the time / space limitations of lambda.
As you can get rid of the EC2 instance once the zip file is uploaded back to S3, you don't have to worry about the cost of the server always running - just spin one up when you need it, and kill it when you're done.
The code for compressing multiple files in a folder could be as simple as :
From: https://code.tutsplus.com/tutorials/compressing-and-extracting-files-in-python--cms-26816
import os
import zipfile
fantasy_zip = zipfile.ZipFile('C:\\Stories\\Fantasy\\archive.zip', 'w')
for folder, subfolders, files in os.walk('C:\\Stories\\Fantasy'):
for file in files:
if file.endswith('.pdf'):
fantasy_zip.write(os.path.join(folder, file), os.path.relpath(os.path.join(folder,file), 'C:\\Stories\\Fantasy'), compress_type = zipfile.ZIP_DEFLATED)
fantasy_zip.close()
import io
class S3File(io.RawIOBase):
def __init__(self, s3_object):
self.s3_object = s3_object
self.position = 0
def __repr__(self):
return "<%s s3_object=%r>" % (type(self).__name__, self.s3_object)
@property
def size(self):
return self.s3_object.content_length
def tell(self):
return self.position
def seek(self, offset, whence=io.SEEK_SET):
if whence == io.SEEK_SET:
self.position = offset
elif whence == io.SEEK_CUR:
self.position += offset
elif whence == io.SEEK_END:
self.position = self.size + offset
else:
raise ValueError("invalid whence (%r, should be %d, %d, %d)" % (
whence, io.SEEK_SET, io.SEEK_CUR, io.SEEK_END
))
return self.position
def seekable(self):
return True
def read(self, size=-1):
if size == -1:
# Read to the end of the file
range_header = "bytes=%d-" % self.position
self.seek(offset=0, whence=io.SEEK_END)
else:
new_position = self.position + size
# If we're going to read beyond the end of the object, return
# the entire object.
if new_position >= self.size:
return self.read()
range_header = "bytes=%d-%d" % (self.position, new_position - 1)
self.seek(offset=size, whence=io.SEEK_CUR)
return self.s3_object.get(Range=range_header)["Body"].read()
def readable(self):
return True
if __name__ == "__main__":
import zipfile
import boto3
s3 = boto3.resource("s3")
s3_object = s3.Object(bucket_name="bukkit", key="bagit.zip")
s3_file = S3File(s3_object)
with zipfile.ZipFile(s3_file) as zf:
print(zf.namelist())
Your question is extremely complex, because solving it can send you down lots of rabbit holes.
I believe that Rahul Iyer is on the right track, because IMHO it would be easier to initiate a new EC2 instance and compress the files on this instance and move them back to a S3 bucket that only serves zip files to the client.
If your files were smaller you could use AWS Cloudfront to handling the zipping when a client requests a file.
During my research I did note that other languages, such as .Net and Java had APIs that handle streaming into zip files. I also looked at zipstream, which has been forked several times. It's unclear how zipstream can be used to stream a file for zipping.
The code below will chunk a file and write the chucks to a zip file. The input files were close to 12Gbs and the output file was almost 5Gbs.
During testing I didn't see any major issues with memory usage or big spikes.
I did add some pseudo S3 code to one of the posts below. I think more testing is required to understand how this code works on files in S3.
from io import RawIOBase
from zipfile import ZipFile
from zipfile import ZipInfo
from zipfile import ZIP_DEFLATED
# This module is needed for ZIP_DEFLATED
import zlib
class UnseekableStream(RawIOBase):
def __init__(self):
self._buffer = b''
def writable(self):
return True
def write(self, b):
if self.closed:
raise ValueError('The stream was closed!')
self._buffer += b
return len(b)
def get(self):
chunk = self._buffer
self._buffer = b''
return chunk
def zipfile_generator(path, stream):
with ZipFile(stream, mode='w') as zip_archive:
z_info = ZipInfo.from_file(path)
z_info.compress_type = ZIP_DEFLATED
with open(path, 'rb') as entry, zip_archive.open(z_info, mode='w') as dest:
for chunk in iter(lambda: entry.read(16384), b''): # 16384 is the maximum size of an SSL/TLS buffer.
dest.write(chunk)
yield stream.get()
yield stream.get()
stream = UnseekableStream()
# each on the input files was 4gb
files = ['input.txt', 'input2.txt', 'input3.txt']
with open("test.zip", "wb") as f:
for item in files:
for i in zipfile_generator(item, stream):
f.write(i)
f.flush()
stream.close()
f.close()
pseudocode s3/zip code
This code is strictly hypothetical, because it needs testing.
from io import RawIOBase
from zipfile import ZipFile
from zipfile import ZipInfo
from zipfile import ZIP_DEFLATED
import os
import boto3
# This module is needed for ZIP_DEFLATED
import zlib
session = boto3.Session(
aws_access_key_id='XXXXXXXXXXXXXXXXXXXXXXX',
aws_secret_access_key='XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX',
region_name='XXXXXXXXXX')
s3 = session.resource('s3')
bucket_name = s3.Bucket('bucket name')
class UnseekableStream(RawIOBase):
def __init__(self):
self._buffer = b''
def writable(self):
return True
def write(self, b):
if self.closed:
raise ValueError('The stream was closed!')
self._buffer += b
return len(b)
def get(self):
chunk = self._buffer
self._buffer = b''
return chunk
def zipfile_generator(path, stream):
with ZipFile(stream, mode='w') as zip_archive:
z_info = ZipInfo.from_file(path)
z_info.compress_type = ZIP_DEFLATED
with open(path, 'rb') as entry, zip_archive.open(z_info, mode='w') as dest:
for chunk in iter(lambda: entry.read(16384), b''):
dest.write(chunk)
yield stream.get()
yield stream.get()
stream = UnseekableStream()
with open("test.zip", "wb") as f:
for file in bucket_name.objects.all():
obj = s3.get_object(Bucket=bucket_name, Key=file.key)
for i in zipfile_generator(obj.get(), stream):
f.write(i)
f.flush()
stream.close()
f.close()