How to set a maximum size limit to PHP cURL downloads?
The server does not honor the Range header. The best you can do is to cancel the connection as soon as you receive more data than you want. Example:
<?php
$curl_url = 'http://steamcommunity.com/id/edgen?xml=1';
$curl_handle = curl_init($curl_url);
$data_string = "";
function write_function($handle, $data) {
global $data_string;
$data_string .= $data;
if (strlen($data_string) > 1000) {
return 0;
}
else
return strlen($data);
}
curl_setopt ($curl_handle, CURLOPT_RETURNTRANSFER, 1);
curl_setopt ($curl_handle, CURLOPT_CONNECTTIMEOUT, 2);
curl_setopt ($curl_handle, CURLOPT_WRITEFUNCTION, 'write_function');
curl_exec($curl_handle);
echo $data_string;
Perhaps more cleanly, you could use the http wrapper (this would also use curl if it was compiled with --with-curlwrappers). Basically you would call fread in a loop and then fclose on the stream when you got more data than you wanted. You could also use a transport stream (open the stream with fsockopen, instead of fopen and send the headers manually) if allow_url_fopen is disabled.
I have another answer that addresses the situation even better to leave here for posterity.
CURLOPT_WRITEFUNCTION
is good for this but CURLOPT_PROGRESSFUNCTION
is the best.
// We need progress updates to break the connection mid-way
curl_setopt($cURL_Handle, CURLOPT_BUFFERSIZE, 128); // more progress info
curl_setopt($cURL_Handle, CURLOPT_NOPROGRESS, false);
curl_setopt($cURL_Handle, CURLOPT_PROGRESSFUNCTION, function(
$DownloadSize, $Downloaded, $UploadSize, $Uploaded
){
// If $Downloaded exceeds 1KB, returning non-0 breaks the connection!
return ($Downloaded > (1 * 1024)) ? 1 : 0;
});
Keep in mind that even if the PHP.net states^ for CURLOPT_PROGRESSFUNCTION
:
A callback accepting five parameters.
My local tests have featured only four (4) parameters as the 1st (handle) is not present.
There is. It's the PHP memory limit, I presume. As the download is done in memory...
But CURLOPT_FILE
and CURLOPT_WRITEHEADER
^ are your friends as they allow you to reroute the cURL
download to streams. This allows you to create tmpfile()
temporary streams (stream_get_meta_data()
gives you the file path) and download to them. And downloading directly to drive lifts the memory limitations.
Once the download completes, you get to read those files and do what you wish with them.