How to split a huge zip file into multiple volumes?
For what it's worth, I like to use try-with-resources everywhere. If you are into that design pattern, then you will like this. Also, this solves the problem of empty parts if the entries are larger than the desired part size. You will at least have as many parts as entries in the worst case.
In:
my-archive.zip
Out:
my-archive.part1of3.zip
my-archive.part2of3.zip
my-archive.part3of3.zip
Note: I'm using logging and Apache Commons FilenameUtils, but feel free to use what you have in your toolkit.
/**
* Utility class to split a zip archive into parts (not volumes)
* by attempting to fit as many entries into a single part before
* creating a new part. If a part would otherwise be empty because
* the next entry won't fit, it will be added anyway to avoid empty parts.
*
* @author Eric Draken, 2019
*/
public class Zip
{
private static final int DEFAULT_BUFFER_SIZE = 1024 * 4;
private static final String ZIP_PART_FORMAT = "%s.part%dof%d.zip";
private static final String EXT = "zip";
private static final Logger logger = LoggerFactory.getLogger( MethodHandles.lookup().lookupClass() );
/**
* Split a large archive into smaller parts
*
* @param zipFile Source zip file to split (must end with .zip)
* @param outZipFile Destination zip file base path. The "part" number will be added automatically
* @param approxPartSizeBytes Approximate part size
* @throws IOException Exceptions on file access
*/
public static void splitZipArchive(
@NotNull final File zipFile,
@NotNull final File outZipFile,
final long approxPartSizeBytes ) throws IOException
{
String basename = FilenameUtils.getBaseName( outZipFile.getName() );
Path basePath = outZipFile.getParentFile() != null ? // Check if this file has a parent folder
outZipFile.getParentFile().toPath() :
Paths.get( "" );
String extension = FilenameUtils.getExtension( zipFile.getName() );
if ( !extension.equals( EXT ) )
{
throw new IllegalArgumentException( "The archive to split must end with ." + EXT );
}
// Get a list of entries in the archive
try ( ZipFile zf = new ZipFile( zipFile ) )
{
// Silliness check
long minRequiredSize = zipFile.length() / 100;
if ( minRequiredSize > approxPartSizeBytes )
{
throw new IllegalArgumentException(
"Please select a minimum part size over " + minRequiredSize + " bytes, " +
"otherwise there will be over 100 parts."
);
}
// Loop over all the entries in the large archive
// to calculate the number of parts required
Enumeration<? extends ZipEntry> enumeration = zf.entries();
long partSize = 0;
long totalParts = 1;
while ( enumeration.hasMoreElements() )
{
long nextSize = enumeration.nextElement().getCompressedSize();
if ( partSize + nextSize > approxPartSizeBytes )
{
partSize = 0;
totalParts++;
}
partSize += nextSize;
}
// Silliness check: if there are more parts than there
// are entries, then one entry will occupy one part by contract
totalParts = Math.min( totalParts, zf.size() );
logger.debug( "Split requires {} parts", totalParts );
if ( totalParts == 1 )
{
// No splitting required. Copy file
Path outFile = basePath.resolve(
String.format( ZIP_PART_FORMAT, basename, 1, 1 )
);
Files.copy( zipFile.toPath(), outFile );
logger.debug( "Copied {} to {} (pass-though)", zipFile.toString(), outFile.toString() );
return;
}
// Reset
enumeration = zf.entries();
// Split into parts
int currPart = 1;
ZipEntry overflowZipEntry = null;
while ( overflowZipEntry != null || enumeration.hasMoreElements() )
{
Path outFilePart = basePath.resolve(
String.format( ZIP_PART_FORMAT, basename, currPart++, totalParts )
);
overflowZipEntry = writeEntriesToPart( overflowZipEntry, zf, outFilePart, enumeration, approxPartSizeBytes );
logger.debug( "Wrote {}", outFilePart );
}
}
}
/**
* Write an entry to the to the outFilePart
*
* @param overflowZipEntry ZipEntry that didn't fit in the last part, or null
* @param inZipFile The large archive to split
* @param outFilePart The part of the archive currently being worked on
* @param enumeration Enumeration of ZipEntries
* @param approxPartSizeBytes Approximate part size
* @return Overflow ZipEntry, or null
* @throws IOException File access exceptions
*/
private static ZipEntry writeEntriesToPart(
@Nullable ZipEntry overflowZipEntry,
@NotNull final ZipFile inZipFile,
@NotNull final Path outFilePart,
@NotNull final Enumeration<? extends ZipEntry> enumeration,
final long approxPartSizeBytes
) throws IOException
{
try (
ZipOutputStream zos =
new ZipOutputStream( new FileOutputStream( outFilePart.toFile(), false ) )
)
{
long partSize = 0;
byte[] buffer = new byte[DEFAULT_BUFFER_SIZE];
while ( overflowZipEntry != null || enumeration.hasMoreElements() )
{
ZipEntry entry = overflowZipEntry != null ? overflowZipEntry : enumeration.nextElement();
overflowZipEntry = null;
long entrySize = entry.getCompressedSize();
if ( partSize + entrySize > approxPartSizeBytes )
{
if ( partSize != 0 )
{
return entry; // Finished this part, but return the dangling ZipEntry
}
// Add the entry anyway if the part would otherwise be empty
}
partSize += entrySize;
zos.putNextEntry( entry );
// Get the input stream for this entry and copy the entry
try ( InputStream is = inZipFile.getInputStream( entry ) )
{
int bytesRead;
while ( (bytesRead = is.read( buffer )) != -1 )
{
zos.write( buffer, 0, bytesRead );
}
}
}
return null; // Finished splitting
}
}
Check: http://saloon.javaranch.com/cgi-bin/ubb/ultimatebb.cgi?ubb=get_topic&f=38&t=004618
I am not aware of any public API that will help you do that. (Although if you do not want to do it programatically, there are utilities like WinSplitter that will do it)
I have not tried it but, every ZipEntry while using ZippedInput/OutputStream has a compressed size. You may get a rough estimate of the size of the zipped file while creating it. If you need 2MB of zipped files, then you can stop writing to a file after the cumulative size of entries become 1.9MB, taking .1MB for Manifest file and other zip file specific elements. So, in a nutshell, you can write a wrapper over the ZippedInputStream as follows:
import java.io.File;
import java.io.FileNotFoundException;
import java.io.FileOutputStream;
import java.io.IOException;
import java.util.zip.ZipEntry;
import java.util.zip.ZipOutputStream;
public class ChunkedZippedOutputStream {
private ZipOutputStream zipOutputStream;
private final String path;
private final String name;
private long currentSize;
private int currentChunkIndex;
private final long MAX_FILE_SIZE = 16000000; // Whatever size you want
private final String PART_POSTFIX = ".part.";
private final String FILE_EXTENSION = ".zip";
public ChunkedZippedOutputStream(String path, String name) throws FileNotFoundException {
this.path = path;
this.name = name;
constructNewStream();
}
public void addEntry(ZipEntry entry) throws IOException {
long entrySize = entry.getCompressedSize();
if ((currentSize + entrySize) > MAX_FILE_SIZE) {
closeStream();
constructNewStream();
} else {
currentSize += entrySize;
zipOutputStream.putNextEntry(entry);
}
}
private void closeStream() throws IOException {
zipOutputStream.close();
}
private void constructNewStream() throws FileNotFoundException {
zipOutputStream = new ZipOutputStream(new FileOutputStream(new File(path, constructCurrentPartName())));
currentChunkIndex++;
currentSize = 0;
}
private String constructCurrentPartName() {
// This will give names is the form of <file_name>.part.0.zip, <file_name>.part.1.zip, etc.
return name + PART_POSTFIX + currentChunkIndex + FILE_EXTENSION;
}
}
The above program is just a hint of the approach and not a final solution by any means.
If the goal is to have the output be compatible with pkzip and winzip, I'm not aware of any open source libraries that do this. We had a similar requirement for one of our apps, and I wound up writing our own implementation (compatible with the zip standard). If I recall, the hardest thing for us was that we had to generate the individual files on the fly (the way that most zip utilities work is they create the big zip file, then go back and split it later - that's a lot easier to implement. Took about a day to write and 2 days to debug.
The zip standard explains what the file format has to look like. If you aren't afraid of rolling up your sleeves a bit, this is definitely doable. You do have to implement a zip file generator yourself, but you can use Java's Deflator class to generate the segment streams for the compressed data. You'll have to generate the file and section headers yourself, but they are just bytes - nothing too hard once you dive in.
Here's the zip specification - section K has the info you are looking for specifically, but you'll need to read A, B, C and F as well. If you are dealing with really big files (We were), you'll have to get into the Zip64 stuff as well - but for 24 MB, you are fine.
If you want to dive in and try it - if you run into questions, post back and I'll see if I can provide some pointers.