How do I distribute a large download over multiple computers?
The common protocols HTTP, FTP and SFTP support range requests, so you can request part of a file. Note that this also requires server support, so it might or might not work in practice.
You can use curl
and the -r
or --range
option to specify the range and
eventually just cat
ting the files together. Example:
curl -r 0-104857600 -o distro1.iso 'http://files.cdn/distro.iso'
curl -r 104857601-209715200 -o distro2.iso 'http://files.cdn/distro.iso'
[…]
And eventually when you gathered the individual parts you concatenate them:
cat distro* > distro.iso
You can get further information about the file, including its size with the --head
option:
curl --head 'http://files.cdn/distro.iso'
You can retrieve the last chunk with an open range:
curl -r 604887601- -o distro9.iso 'http://files.cdn/distro.iso'
Read the curl man page for more options and explanations.
You can further leverage ssh and tmux to ease running and keeping track of the downloads on multiple servers.