Can I parallelize sort?
As of coreutils 8.6 (2010-10-15), GNU sort
already sorts in parallel to make use of several processors where available. So, it cannot be further improved in that regard like pigz
or pbzip2
improve gzip
or bzip2
.
If your sort
is not parallel, you can try and install the GNU sort
from the latest version of GNU coreutils.
With GNU sort, you can limit the number of threads with the --parallel
option.
The one thing that always helps me most with sort is giving it as much memory as possible, so as to reduce swapping, e.g.:
sort -S 20G
If your file is large enough, sorting will cause disk swapping, either because the allocated virtual memory is growing too big, or because the sort
program itself is swapping chunks to disk and back. Older sort
implementations are more likely to have this "sort via disk buffer" sort of behavior, since it was the only way to sort large files in the old days.
sort
has a -m
option that may help you here. It might be faster to split the file into chunks — say with split -l
— sort them independently, then merge them back together.
Then again, it may be that this is exactly what "sort via disk buffer" does. The only way to find out if it helps is to benchmark it on your particular test load. The critical parameter will be the line count you give to split -l
.