Python equivalent of piping file output to gzip in Perl using a pipe

Try something like this:

from subprocess import Popen, PIPE
f = open('zipped.gz', 'w')
pipe = Popen('gzip', stdin=PIPE, stdout=f)
pipe.communicate('Hello world\n')
f.close()

ChristopheD's suggestion of using the subprocess module is an appropriate answer to this question. However, it's not clear to me that it will solve your performance problems. You would have to measure the performance of the new code to be sure.

To convert your sample code:

import subprocess

p = subprocess.Popen("gzip -c > zipped.gz", shell=True, stdin=subprocess.PIPE)
p.communicate("Hello World\n")

Since you need to send large amounts of data to the sub-process, you should consider using the stdin attribute of the Popen object. For example:

import subprocess

p = subprocess.Popen("gzip -c > zipped.gz", shell=True, stdin=subprocess.PIPE)
p.stdin.write("Some data")

# Write more data here...

p.communicate() # Finish writing data and wait for subprocess to finish

You may also find the discussion at this question helpful.