Virtual write-only file system for storing files in archive

A classic case of RTFM (all of it!). The -T option to GNU tar will read the files to be archived from another file (in my case, /dev/stdin, you can also use -), and there's even a --remove-files option:

alias magic_otf_compressor='tar --create -T - --remove-files -O | pixz'

(using the parallel version of xz for compression, but you can use your preferred compressor instead). To be used as:

arg_generating_process |
  parallel --gnu my_process |
  magic_otf_compressor > file.tar.xz

EDIT: As Ole points out, tar seems to read the entire list of files with the -T option for some reason. The following test confirms this:

for ((f = 0; $f < 1000; f++)); do
    touch $f; echo $f;
done | tar -c -f otf.tar -T - -v

There is a one second delay on my system before all files are printed at once; in contrast, if the tar command is replaced by cat, all files are printed as they are created. I have filed a support request with the tar folks, let's see.

EDIT^2: The most recent tar from source fixes this. It's not in Ubuntu 13.10 yet, but might be included with 14.04.