filebeat抓取nginx日志

目录

一、抓取普通的应用输出日志到elasticsearch

二、抓取nginx日志输出到ElasticSearch

2.1、nginx.conf设定日志输出为JSON格式

2.2、nginx.conf设定日志按天输出文件

2.3、抓取Nginx JSON到ElasticSearch配置

---

一、抓取普通的应用输出日志到elasticsearch

- type: log
  # 默认是 5s
  scan_frequency: 5s
  enabled: true
  encoding: utf-8
  paths:
    - /opt/mydomain.cn/log-file.log
  multiline.pattern: '^[0-9]{4}-[0-9]{2}-[0-9]{2} [0-9]{2}:[0-9]{2}:[0-9]{2},[0-9]{3}'
  multiline.negate: true
  multiline.match: after
  tags: ["IP地址","业务名","myapi","dev"]
  fields:
    app_from: myapi-172
  fields_under_root:  true

二、抓取nginx日志输出到ElasticSearch

2.1、nginx.conf设定日志输出为JSON格式

log_format main '{ "time_local": "$time_local",''"remote_addr": "$remote_addr",''"referer": "$http_referer",''"uri": "$host$uri",''"status": $status,''"bytes": $body_bytes_sent,''"up_addr": "$upstream_addr",''"upstream_time": "$upstream_response_time",''"request_time": "$request_time"''}';

2.2、nginx.conf设定日志按天输出文件

map $time_iso8601 $logdate {'~^(?<ymd>\d{4}-\d{2}-\d{2})' $ymd;default 'date-not-found';}access_log logs/access-$logdate.log main;

2.3、抓取Nginx JSON到ElasticSearch配置

- type: log# 默认是 5sscan_frequency: 10senabled: trueencoding: utf-8paths:- /usr/local/nginx/logs/*.logjson.keys_under_root: true  # Flase会将json解析的格式存储至messages，改为true则不存储至json.overwrite_keys: true  #覆盖默认message字段，使用自定义json格式的keytags: ["11.111.11.111","nginx","dev"]fields:app_from: nginx-111fields_under_root:  true

这一么一点点配置，搞了一整天，才整明白。