Read non-blocking from multiple fifos in parallel
This solution will only work if the number of fifos are fewer than the number of jobs that GNU parallel can run in parallel (which is limited by file handles and number of processes):
parallel -j0 --line-buffer cat ::: fifo*
It seems to be able to move up to 500 MB/s:
window1$ mkfifo {1..100}
window1$ parallel -j0 --line-buffer cat ::: {1..100} | pv >/dev/null
window2$ parallel -j0 'cat bigfile > ' ::: *
And it does not mix half-lines:
window1$ mkfifo {1..100}
window1$ parallel -j0 --line-buffer cat ::: {1..100} &
window2$ parallel -j0 'traceroute {}.1.1.1 > {}' ::: *
It reads jobs in parallel (it does not read one job completely before going to the next):
window1$ mkfifo {1..100}
window1$ parallel -j0 --line-buffer cat ::: * > >(tr -s ABCabc)
window2$ long_lines_with_pause() {
perl -e 'print STDOUT "a"x30000_000," "'
perl -e 'print STDOUT "b"x30000_000," "'
perl -e 'print STDOUT "c"x30000_000," "'
echo "$1"
sleep 2
perl -e 'print STDOUT "A"x30000_000," "'
perl -e 'print STDOUT "B"x30000_000," "'
perl -e 'print STDOUT "C"x30000_000," "'
echo "$1"
}
window2$ export -f long_lines_with_pause
window2$ parallel -j0 'long_lines_with_pause {} > {}' ::: *
Here a lot of 'a b c' (first half of a job) will be printed before 'A B C' (second half of the job).