Is there any existing grok{} pattern for date format YYYY/MM/DD HH:mm:ss?
No. You find the included patterns on github. The comment to datestamp
seems to fit to your YYYY/MM/DD, but DATE_US
and DATE_EU
are different.
I suggest overload the DATE
pattern using grok option patterns_dir and go with DATESTAMP
.
DATE_YMD %{YEAR}/%{MONTHNUM}/%{MONTHDAY}
DATE %{DATE_US}|%{DATE_EU}|%{DATE_YMD}
or just add your pattern into a patterns-file and use grok's patterns_dir option.
Successful timestamp capture strategy comprised of 3 things
- Precision and timezone in the original log. Change your nginx timestamp log format.
Use $msec
to capture milliseconds. Otherwise you wouldn't be able to sort it precisely.
log_format custom '[$msec] [$remote_addr] [$remote_user] '
'"$request" $status '
'"$http_referer" "$http_user_agent"';
- Raw timestamp. Use greedy matching to capture raw data into a field.
Use GREEDYDATA:
grok {
match => { "message" => "\[%{GREEDYDATA:raw_timestamp}\] %{GREEDYDATA:message}" }
overwrite => [ "message" ]
}
- Parsed timestamp. Use
date
filter to parse raw timestamp.
reference
date {
match => [ "timestamp", "yyyy/MM/dd HH:mm:ss.S z" ]
target => "@timestamp"
}
To match 2015/08/30 05:55:20, use:
%{DATESTAMP:mytimestamp}
Tested on Logstash 6.5
Source: https://github.com/logstash-plugins/logstash-patterns-core/blob/master/patterns/grok-patterns