Set pipe capacity in Linux
writes over the default pipe capacity will still require waking up the downstream command
If your goal is not to wake up the downstream command too often, did you try using the -p
option to buffer
? It should cause buffer
to hold writes until the buffer is filled to certain percentage. You might need the -s
option as well to write large chunks.
Update: D'oh, the pipes between the commands still limit things. Maybe try using the following adapter program:
#define _GNU_SOURCE
#include <stdlib.h>
#include <unistd.h>
#include <fcntl.h>
int main(int argc, char** argv)
{
fcntl(atoi(argv[1]), F_SETPIPE_SZ, atoi(argv[2]));
execvp(argv[3],argv+3);
while (1);
}
as in:
adapter 1 (BIGSIZE) cmd1 | cmd2
or even:
adapter 1 (BIGSIZE) cmd1 | adapter 1 (BIGSIZE) buffer [args] | cmd2
if cmd1
still makes small writes.
Based on DepressedDaniel and Stéphane Chazelas suggestions I settled on the closest thing to a oneliner I could find:
function hugepipe {
perl -MFcntl -e 'fcntl(STDOUT, 1031, 1048576) or die $!; exec { $ARGV[0] } @ARGV or die $!' "$@"
}
This allows to do:
hugepipe <command> | <command>
and the pipe between the two commands is going to have the capacity specified via the fcntl
in the perl script.