Curl: transfer closed with outstanding read data remaining
Okay, after some searching and IRC chat's I found solution, but not 100% sure what the cause is. Looks like the keep-alives weren't send enough to keep the connection going on. Will post the solution here, hopefully I can help someone out.
What helped for me is adding
--keepalive-time 2
An explanation of the curl option
--keepalive-time <seconds>
This option sets the time a connection needs to remain idle before sending keepalive probes and the time between individual keepalive probes. It is currently effective on operating systems offering the TCP_KEEPIDLE and TCP_KEEPINTVL socket options (meaning Linux, recent AIX, HP-UX and more). This option has no effect if --no-keepalive is used. (Added in 7.18.0)
If this option is used several times, the last one will be used. If unspecified, the option defaults to 60 seconds.
Looks like the default was too high to keep my connection open.
Here is the full command I used for my call
curl URL -H 'Accept-Encoding: gzip, deflate, sdch' -H 'Accept-Language: en-US,en;q=0.8,et;q=0.6,nl;q=0.4' -H 'Upgrade-Insecure-Requests: 1' -H 'User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/45.0.2454.101 Safari/537.36' -H 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8' -H 'Connection: keep-alive' --compressed -v --keepalive-time 2
And I'm running this version of curl on osx
curl 7.43.0 (x86_64-apple-darwin15.0) libcurl/7.43.0 SecureTransport zlib/1.2.5
Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtsp smb smbs smtp smtps telnet tftp
Features: AsynchDNS IPv6 Largefile GSS-API Kerberos SPNEGO NTLM NTLM_WB SSL libz UnixSockets
If someone would like to have this option used in PHP curl, the --keepalive-time option is available since PHP 5.5. You can use it as following:
curl_setopt($connection, CURLOPT_TCP_KEEPALIVE, 1);
curl_setopt($connection, CURLOPT_TCP_KEEPIDLE, 2);
Hope this helps someone struggling with the same issue!
Encountered similar issue, my server is behind nginx. Curl is able to receive a response if it connects to the server directly but if curl connects to the server via nginx, curl is throwing below error
Session
* transfer closed with outstanding read data remaining
* Closing connection 0 curl: (18) transfer closed with outstanding read data remaining
When I connect to same nginx URL using a browser, Response is showing up fine. It is very weird, When I tried to connect to the same nginx URL using curl, it was throwing above error.
After comparing headers sent by the browser and curl. I found that the browser is able to receive the response because of the below header which curl is not sending:
'Accept-Encoding: gzip'
Sending above header using curl is working fine. So, what the above header is doing is compressing response to gzip thus reducing the response size.
After some more digging, found that nginx is not able to send any payload > 80kb. After wasting a lot of time, found that issue is with nginx buffering and nginx worked like a charm after adding below proxy_buffering property in nginx.conf:
location / {
proxy_buffering off;
}
The accepted answer didn't resolve my issue. Writing this answer, so that anyone need not waste their time if they are facing the same issue as I did.
libcurl only tells you that the connection was cut by the server in an unclean way where it didn't deliver data it had promised to do. It looks like the chunked-encoding didn't signal the end of the transfer.
Browsers are notorious for being extremely liberal in what they receive so they overlook and are fine with all sorts of protocol violations to a much larger extent than libcurl is.