Grok Issue with Multiple IP's in NginX Logstash

Not sure if you're still having this issue, but if so, here's what will work for you.

Given this log format:

log_format custom '$remote_addr - $remote_user [$time_local] "$request" $status $body_bytes_sent "$http_referer" "$http_user_agent" "$host" "$http_x_forwarded_for"';

the grok pattern you've specified doesn't take into account the addition of the "$host" "$http_x_forwarded_for" portion.

Not sure why your grok isn't failing, but it should.

In any event, this pattern will work with the log format above:

%{IP:clientip} %{NOTSPACE:ident} %{NOTSPACE:auth} \[%{HTTPDATE:timestamp}\] "%{WORD:verb} %{DATA:request} HTTP/%{NUMBER:httpversion}" %{NUMBER:response} (?:%{NUMBER:bytes}|-) (?:"(?:%{URI:referrer}|-)"|%{QS:referrer})(?:;|) %{QS:agent} "%{NOTSPACE:host}" "(?<x_forwarded_for>%{IP:xff_clientip}, .*)"

And results in the following fields

httpversion      1.1
request          /api/filter/14928/content?api_key=apikey&site=website
timestamp        28/Sep/2015:12:39:56·+1000
auth             -
agent            "-"
bytes            101
response         403
ident            -
verb             GET

Note that you've got a couple of new fields than you would have had before.

The first ("x_forward_for" =>,,,, is the contents of the last set of quotes, or $http_x_forwarded_for from the log format.
The second ("xff_clientip" => is just the first IP in that list, which should translate to the actual source IP of the request.

If it were me, I'd also run the x_forwarded_for field through a mutate filter to break it into an array:

mutate {
  split  => { "x_forwarded_for" => ", " }