docker filebeat 将日志多级目录和多维json数据日志同步到es

注

使用的时候先调试调试配置,调试成功在尝试写入es,如果es写入失败就是es账户.密码/白名单.和index未创建的问题,细节可以留言

setup.template.priority 模板优先级 调整这个可以配置一台机器多个filebeat 容器启动

多级目录日志和多维josn日志结构

filebeat.inputs:- type: logenabled: truepaths:- '/home/data/logs/test/*.log'- '/home/data/logs/upload/logs/**/*.log'json.keys_under_root: true  json.add_error_key: truejson.message_key: json  processors:- decode_json_fields:fields: ["json"]process_array: falsemax_depth: 1target: ""

docker 直接启动

以docker形式启动filebeat

docker run -d \--name=filebeat \--user=root \--volume="/home/golang/gopath/filebeat/test/filebeat.docker.yml:/usr/share/filebeat/filebeat.yml:ro" \--volume="/var/run/docker.sock:/var/run/docker.sock:ro" \--volume="/home/data/logs/test:/home/data/logs/test/logs:ro" \docker.elastic.co/beats/filebeat:8.6.2 filebeat -e --strict.perms=false #调试
docker run  \--name=filebeat \--user=root \--volume="/home/golang/gopath/filebeat/test/filebeat.docker.yml:/usr/share/filebeat/filebeat.yml:ro" \--volume="/var/run/docker.sock:/var/run/docker.sock:ro" \--volume="/home/data/logs/test:/home/data/logs/test/logs:ro" \docker.elastic.co/beats/filebeat:8.6.2 filebeat -e --strict.perms=false 

filebeat 配置 filebeat.docker.yml

filebeat.config:modules:path: ${path.config}/modules.d/*.ymlreload.enabled: falsefilebeat.inputs:- type: logenabled: truepaths:- '/home/data/logs/test/*.log'- '/home/data/logs/upload/logs/**/*.log'json.keys_under_root: true  json.add_error_key: truejson.message_key: json  processors:- decode_json_fields:fields: ["json"]process_array: falsemax_depth: 1target: ""output.elasticsearch:ssl.verification_mode: nonehosts: ["https://127.0.0.1:9200"]username: "test"password: "12345"index: "metricbeat-test"setup.template.name: "metricbeattest"
setup.template.pattern: "metricbeat-*"
setup.template.priority: 50

filebeat 调试配置

filebeat.config:modules:path: ${path.config}/modules.d/*.ymlreload.enabled: falsefilebeat.inputs:- type: logenabled: truepaths:- '/home/data/logs/test/*.log'- '/home/data/logs/upload/logs/**/*.log'json.keys_under_root: falsejson.add_error_key: truejson.message_key: logprocessors:- decode_json_fields:fields: ["log"]process_array: falsemax_depth: 1target: ""# 丢掉不需要的字段- drop_fields:fields: ["agent","input","ecs","message","host", "log","error","level"]# 配置输出到控制台
output.console:pretty: truesetup.template.name: "metricbeat-test"
setup.template.pattern: "metricbeat-*"
setup.template.priority: 50