Virtual write-only file system for storing files in archive
A classic case of RTFM (all of it!). The -T
option to GNU tar
will read the files to be archived from another file (in my case, /dev/stdin
, you can also use -
), and there's even a --remove-files
option:
alias magic_otf_compressor='tar --create -T - --remove-files -O | pixz'
(using the parallel version of xz
for compression, but you can use your preferred compressor instead). To be used as:
arg_generating_process |
parallel --gnu my_process |
magic_otf_compressor > file.tar.xz
EDIT: As Ole points out, tar
seems to read the entire list of files with the -T
option for some reason. The following test confirms this:
for ((f = 0; $f < 1000; f++)); do
touch $f; echo $f;
done | tar -c -f otf.tar -T - -v
There is a one second delay on my system before all files are printed at once; in contrast, if the tar
command is replaced by cat
, all files are printed as they are created. I have filed a support request with the tar folks, let's see.
EDIT^2: The most recent tar
from source fixes this. It's not in Ubuntu 13.10 yet, but might be included with 14.04.