ElK 8 收集 Nginx 日志

1. 说明

elk 版本：8.15.0

2. 启个 nginx

有 nginx 可以直接使用。我这里是在之前环境下 docker-compose.yml 中启动了个 nginx：

nginx:restart: alwaysimage: nginx:1.26.1ports:- "80:80"- "443:443"volumes:#- ./nginx/html:/usr/share/nginx/html- ./nginx/logs:/var/log/nginx#- ./nginx/certs:/etc/nginx/certs- ./nginx/conf/nginx.conf:/etc/nginx/nginx.conf- ./nginx/conf/conf.d:/etc/nginx/conf.d

刚开始如果没有 nginx 配置，可以先不映射配置目录和文件，先启动，然后 docker cp 把配置和目录拷贝出来，再挂载进去。

配置 nginx/conf/conf.d/default.conf：

#
upstream kibana_servers {server kibana:5601 weight=1 max_fails=3 fail_timeout=30s;
}server{listen 80;server_name localhost;# root /usr/local/nginx/html ;# rewrite ^(.*) https://$server_name$1 permanent;location / {proxy_pass http://kibana_servers/;proxy_set_header Host $host:$server_port;proxy_set_header X-Forwarded-Host $host:$server_port;proxy_set_header X-Forwarded-Server $host;proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;proxy_set_header X-Forwarded-Proto $scheme;proxy_set_header X-Real-IP $remote_addr;proxy_connect_timeout  30s;proxy_read_timeout     90s;proxy_send_timeout     90s;client_max_body_size   50m;client_body_buffer_size 16k;  # 默认是 8k 或 16k，可以根据需要调整}
}

3. filebeat 接入到 es

根据指引 filebeat 直接接入到 es。

filebeat 安装参考 《Docker compose 安装 ELK》中 “6. 安装 filebeat”。

filebeat 的目录下，设置 nginx 日志路径 modules.d/nginx.yml：

- module: nginx# Access logsaccess:enabled: true# Set custom paths for the log files. If left empty,# Filebeat will choose the paths depending on your OS.var.paths: ["/data/docker/elk/nginx/logs/access.log*"]# Error logserror:enabled: true# Set custom paths for the log files. If left empty,# Filebeat will choose the paths depending on your OS.var.paths: ["/data/docker/elk/nginx/logs/error.log*"]

主配置 filebeat.yml：

filebeat.config.modules:# Glob pattern for configuration loadingpath: ${path.config}/modules.d/*.ymloutput.elasticsearch:hosts: ["https://10.1.205.165:9200"]username: "filebeat_writer"password: "YOUR_PASSWORD"ssl:enabled: true# fingerprint=$(openssl x509 -fingerprint -sha256 -noout -in certs/ca/ca.crt | awk -F"=" '{print $2}' | sed 's/://g')ca_trusted_fingerprint: "33CB5A3B3ECCA59FDF7333D9XXXXXXXXFD34D5386FF9205AB8E1"# certs/ca 目录从 es 中拷过来certificate_authorities: ["certs/ca/ca.crt"]# output.logstash:
#   hosts: ["10.1.205.165:5044", "10.1.205.166:5044"]setup.kibana:host: "10.1.205.165:5601"logging.level: warning

Dev Tools 创建角色：

PUT /_security/role/filebeat_writer_role
{"cluster": ["monitor"],"indices": [{"names": ["filebeat*"],"privileges": ["create_doc","create","delete","index","write","all","auto_configure","manage"],"allow_restricted_indices": false}],"applications": [],"run_as": [],"metadata": {},"transient_metadata": {"enabled": true}
}

Dev Tools 创建用户：

POST /_security/user/filebeat_writer
{"password" : "YOUR_PASSWORD","roles" : [ "filebeat_writer_role" ],"full_name" : "Filebeat Writer","email" : "[e-mail]","metadata" : {}
}

启用模块：

./filebeat modules enable nginx
./filebeat modules enable all-filesets#
./filebeat setup
./filebeat -e

成功后，在 kibana 的 nginx log dashboard 中可以看到数据

4. filebeat 接入 logstash

主配置 filebeat.yml：

filebeat.config.modules:# Glob pattern for configuration loadingpath: ${path.config}/modules.d/*.yml# output.elasticsearch:
#   hosts: ["https://10.1.205.165:9200"]
#   username: "filebeat_internal"
#   password: "YOUR_PASSWORD"
#   ssl:
#     enabled: true
#     # fingerprint=$(openssl x509 -fingerprint -sha256 -noout -in certs/ca/ca.crt | awk -F"=" '{print $2}' | sed 's/://g')
#     ca_trusted_fingerprint: "33CB5A3B3ECCA59FDF7333D9XXXXXXXXFD34D5386FF9205AB8E1"
#     # certs/ca 目录从 es 中拷过来
#     certificate_authorities: ["certs/ca/ca.crt"]output.logstash:hosts: ["10.1.205.165:5044", "10.1.205.166:5044"]setup.kibana:host: "10.1.205.165:5601"logging.level: warning

有问题的配置我就不贴了，最终配置在下面，注释有说明解决问题的关键。
启动 filebeat 后，在 kibana Dev Tools 中查询：

GET filebeat-*/_search
{"size": 1,"sort": [{}]
}

结果发现 ingest pipeline 处理异常，经过一段时间排查，找到了 相同问题链接

然后下面是最终的可用 Logstash Pipeline 设置 logstash/pipeline/filebeat.conf：

#
input {beats {port => 5044}
}filter {if [@metadata][pipeline] {# 这里解决上面 `[event][original]` 已存在问题ruby {code => "event.remove('[event][original]')"}} else if [log_type] == "k8s" {# 将嵌套字段提升为顶层字段mutate {rename => {"[kubernetes][namespace]" => "kubernetes_namespace""[kubernetes][node][name]" => "kubernetes_node_name""[kubernetes][pod][ip]" => "kubernetes_pod_ip""[kubernetes][pod][name]" => "kubernetes_pod_name""[container][runtime]" => "container_runtime"}}# 使用 prune 过滤器保留所需字段prune {whitelist_names => ["@timestamp", "log_type", "container_runtime", "kubernetes_namespace", "kubernetes_node_name", "kubernetes_pod_ip", "kubernetes_pod_name", "message"]}}
}output {if [@metadata][pipeline] {elasticsearch {hosts => ["https://es01:9200", "https://es02:9200", "https://es03:9200"]user => "logstash_writer"password => "logstash_password"ssl_enabled => truessl_certificate_authorities => "/usr/share/logstash/certs/ca/ca.crt"action => "create"pipeline => "%{[@metadata][pipeline]}"index => "%{[@metadata][beat]}-%{[@metadata][version]}"manage_template => false}} else if [log_type] == "k8s" {elasticsearch {hosts => ["https://es01:9200", "https://es02:9200", "https://es03:9200"]index => "k8s-%{+YYYY.MM.dd}"user => "logstash_writer"password => "logstash_password"ssl_enabled => truessl_certificate_authorities => "/usr/share/logstash/certs/ca/ca.crt"}} else {# 默认索引elasticsearch {hosts => ["https://es01:9200", "https://es02:9200", "https://es03:9200"]index => "logstash-%{+YYYY.MM.dd}"user => "logstash_writer"password => "logstash_password"ssl_enabled => truessl_certificate_authorities => "/usr/share/logstash/certs/ca/ca.crt"action => "create"manage_template => false}}
}

logstash 角色和用户权限同上面 filebeat 角色和用户。

5. 总结

Filebeat 收集 nginx、mysql 等日志，没有特别需求，最好直接接入 ES 中，少了一层 logstash 其实性能更好，问题也相对较少，官方文档推荐的。

参考资料：
[1] https://www.elastic.co/guide/en/logstash/8.15/use-ingest-pipelines.html
[2] https://www.elastic.co/guide/en/elasticsearch/reference/8.15/ingest.html
[3] https://www.elastic.co/guide/en/beats/filebeat/8.15/filebeat-installation-configuration.html
[4] https://discuss.elastic.co/t/aws-ingest-pipeline-error-in-processor-rename-message-to-event-original/341472/8
[5] https://www.elastic.co/guide/en/ecs-logging/overview/master/intro.html