Nginx 错误日志

Nginx 错误日志是运维人员最常见但又极其容易忽略的日志类型之一。Nginx 错误日志即没有统一明确的分隔符，也没有特别方便的正则模式，但通过 logstash 不同插件的组合，还是可以轻松做到数据处理。

值得注意的是，Nginx 错误日志中，有一类数据是接收过大请求体时的报错，默认信息会把请求体的具体字节数记录下来。每次请求的字节数基本都是在变化的，这意味着常用的 topN 等聚合函数对该字段都没有明显效果。所以，对此需要做一下特殊处理。

最后形成的 logstash 配置如下所示：

  1. filter {
  2.     grok {
  3.         match => { "message" => "(?<datetime>\d\d\d\d/\d\d/\d\d \d\d:\d\d:\d\d) \[(?<errtype>\w+)\] \S+: \*\d+ (?<errmsg>[^,]+), (?<errinfo>.*)$" }
  4.     }
  5.     mutate {
  6.         rename => [ "host", "fromhost" ]
  7.         gsub => [ "errmsg", "too large body: \d+ bytes", "too large body" ]
  8.     }
  9.     if [errinfo]
  10.     {
  11.         ruby {
  12.             code => "
  13.                 new_event = LogStash::Event.new(Hash[event.get('errinfo').split(', ').map{|l| l.split(': ')}])
  14.                 new_event.remove('@timestamp')
  15.                 event.append(new_event)
  16.             "
  17.         }
  18.     }
  19.     grok {
  20.         match => { "request" => '"%{WORD:verb} %{URIPATH:urlpath}(?:\?%{NGX_URIPARAM:urlparam})?(?: HTTP/%{NUMBER:httpversion})"' }
  21.         patterns_dir => ["/etc/logstash/patterns"]
  22.         remove_field => [ "message", "errinfo", "request" ]
  23.     }
  24. }

经过这段 logstash 配置的 Nginx 错误日志生成的事件如下所示：

  1. {
  2.     "@version": "1",
  3.     "@timestamp": "2015-07-02T01:26:40.000Z",
  4.     "type": "nginx-error",
  5.     "errtype": "error",
  6.     "errmsg": "client intended to send too large body",
  7.     "fromhost": "web033.mweibo.yf.sinanode.com",
  8.     "client": "36.16.7.17",
  9.     "server": "api.v5.weibo.cn",
  10.     "host": "\"api.weibo.cn\"",
  11.     "verb": "POST",
  12.     "urlpath": "/2/client/addlog_batch",
  13.     "urlparam": "gsid=_2A254UNaSDeTxGeRI7FMX9CrEyj2IHXVZRG1arDV6PUJbrdANLROskWp9bXakjUZM5792FW9A5S9EU4jxqQ..&wm=3333_2001&i=0c6f156&b=1&from=1053093010&c=iphone&v_p=21&skin=default&v_f=1&s=8f14e573&lang=zh_CN&ua=iPhone7,1__weibo__5.3.0__iphone__os8.3",
  14.     "httpversion": "1.1"
  15. }