Is there a way to create ZIP OutputStream ?
As mentioned already it's easy to implement this functionality with JLink.
But once you use Java's ZipOtputStream
you will need to convert your data to list of bytes first. Borrowing java code from here.
<< JLink`
InstallJava[];
openStream[file_] := Module[{fos, bos, zos},
fos = JavaNew["java.io.FileOutputStream", file~StringJoin~".zip"];
bos = JavaNew["java.io.BufferedOutputStream", fos];
zos = JavaNew["java.util.zip.ZipOutputStream", bos];
zos@putNextEntry[JavaNew["java.util.zip.ZipEntry", file]];
zos];
closeStream[stream_] := Module[{},
stream@closeEntry[];
stream@close[];
];
stream = openStream["blah.dat"];
stream@write[{1, 2, 3}]
closeStream[stream];
As result you will get blah.dat.zip archive that will have blah.dat inside. This is not very useful. We need some sort of serialization. Let's take your example and define
doStuff[x_] := x*x
We will store squares of 1-1000000 in a zip file:
convertToBytes[list_] :=
Flatten[ToCharacterCode /@ (ToString[#]~StringJoin~"\n" & /@ list)];
stream = openStream["blah.dat"];
stream@write[convertToBytes[doStuff /@ Range[1000000]]]
closeStream[stream];
Update @rcollyer proposed to hook into native streams with
DefineOutputStreamMethod
and it actually worked:
DefineOutputStreamMethod["Zipped", {
"ConstructorFunction" ->
Function[{name, isAppend, caller, opts}, {True, openStream[name]}],
"WriteFunction" ->
Function[{state, bytes}, state@write[bytes]; {Length[bytes], state}],
"CloseFunction" -> Function[{state}, closeStream[state]]
}];
Now we can work with zipped streams using native methods:
starWars = OpenWrite["star-wars.dat", Method -> "Zipped"];
Write[starWars, "yoda forever!"];
Write[starWars, {"Luke", "Leia"}];
Write[starWars, doStuff /@ Range[1000]];
Close[starWars];
Neither the import format ZIP (nor any of those similar to it like GZIP, TAR or BZIP) nor the function CreateArchive
do have any documented functionality which would let you do what you want. So I see two possible ways to achieve what you want:
as george2079 mentioned in his comment, you could use
StartProcess
and an external program. That will of course become somewhat OS dependent.you could use JLink to access the java functionality you are referring to directly.
If you get that to work you might have a look at the new stream methods functionality which should in principle let you define a specific output stream method which would behave as you indicated. Unfortunately the documentation of stream methods is not very detailed so I guess it would mean some experimenting to get this running.
Some superfluos spelunking in the CreateArchive
code indicates that WRI uses JLink`
plus java.util.zip.ZipOutputStream
there so I think that is the most straightforward path. If you want to have a look at how they do things, you could have a look at the definition of CreateArchiveDump`compress
and CreateArchiveDump`addToZIPFile
which will be available after the first use of CreateArchive
.