Throttle the download speed of wget or curl while downloading
Yes both wget and curl support limiting your download rate. Both options are directly mentioned in the man page.
curl
--limit-rate <speed> Specify the maximum transfer rate you want curl to use. This feature is useful if you have a limited pipe and you'd like your transfer not to use your entire bandwidth. The given speed is measured in bytes/second, unless a suffix is appended. Appending 'k' or 'K' will count the number as kilobytes, 'm' or M' makes it megabytes, while 'g' or 'G' makes it gigabytes. Examples: 200K, 3m and 1G.
E.g: curl --limit-rate 423K
wget
--limit-rate=amount Limit the download speed to amount bytes per second. Amount may be expressed in bytes, kilobytes with the k suffix, or megabytes with the m suffix. For example, --limit-rate=20k will limit the retrieval rate to 20KB/s. This is useful when, for whatever reason, you don't want Wget to consume the entire available bandwidth.
E.g: wget --limit-rate=423k
2 years later I will throw this tidbit in, while wget
and curl
are not interactive, at least wget
(and possibly curl
but i do not know for sure) has the -c
switch (which stands for continue from where I left off downloading earlier). So if you need to change your speed in the middle of a download and you presumably used the -c
switch with the --limit-rate=x
then you could stop wget
and restart it with a different speed and it would change.
It is possible to limit the traffic rate using the tc
and netem
tools but this will limit the rate for the network interface of the computer. I am assuming that you use only wget
or curl
and no other application is exchanging traffic through the network interface.
tc
uses Token Bucket Filter (TBF) to control the rate.
One example of TBF would be as follows (ref. http://www.lartc.org/manpages/tc-tbf.html):
To attach a TBF with a sustained maximum rate of 0.5mbit/s, a peakrate of 1.0mbit/s, a 5kilobyte buffer, with a pre-bucket queue size limit calculated so the TBF causes at most 70ms of latency, with perfect peakrate behaviour, issue:
# tc qdisc add dev eth0 root tbf rate 0.5mbit \ burst 5kb latency 70ms peakrate 1mbit \ minburst 1540
Another example of usign tc and netem would be as follows (found in http://www.linuxfoundation.org/collaborate/workgroups/networking/netem):
There is no rate control built-in to the netem discipline, instead use one of the other disciplines that does do rate control. In this example, we use Token Bucket Filter (TBF) to limit output.
To add the delay of each packet going/coming through the interface eth0
# tc qdisc add dev eth0 root handle 1:0 netem delay 100ms
to add the data rate in tbf, packet buffer size and maximum burst limit
# tc qdisc add dev eth0 parent 1:1 handle 10: tbf rate 256kbit buffer 1600 limit 3000
To see the list of rules assigned in tc for interface eth0
# tc -s qdisc ls dev eth0
The output of the above command would be as below
qdisc netem 1: limit 1000 delay 100.0ms
Sent 0 bytes 0 pkts (dropped 0, overlimits 0 )
qdisc tbf 10: rate 256Kbit burst 1599b lat 26.6ms
Sent 0 bytes 0 pkts (dropped 0, overlimits 0 )
Check on the options for buffer and limit as you might find you need bigger defaults than these (they are in bytes)