400 Bad Request - request header or cookie too large
It's just what the error says - Request Header Or Cookie Too Large
. One of your headers is really big, and nginx is rejecting it.
You're on the right track with large_client_header_buffers
. If you check the docs, you'll find it's only valid in http
or server
contexts. Bump it up to a server block and it will work.
server {
# ...
large_client_header_buffers 4 32k;
# ...
}
By the way, the default buffer number and size is 4
and 8k
, so your bad header must be the one that's over 8192 bytes. In your case, all those cookies (which combine to one header) are well over the limit. Those mixpanel cookies in particular get quite large.
Fixed by adding
server {
...
large_client_header_buffers 4 16k;
...
}
I get the error almost per 600 requests when web scraping. Firstly, assumed that a proxy server or remote ngix limits. I've tried to delete all cookies and other browser solutions that generally talked by related posts, but no luck. Remote server is not in my control.
In my case, I made a mistake about adding over and over new header to the httpClient object. After defined a global httpclient object, added header once and the problem doesn't appear again. It was a little mistake but unfortunately instead of try to understand the problem, jumped to the stackoverflow :) Sometimes, we should try to understand the problem own.
With respect to answers above, but there is client_header_buffer_size
needs to be mentioned:
http {
...
client_body_buffer_size 32k;
client_header_buffer_size 8k;
large_client_header_buffers 8 64k;
...
}