2.3做logstash实验

收集apache日志输出到es

在真实服务器安装logstash，httpd

systemctl start httpd

echo 666 > /var/www/html/index.html

cat /usr/local/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-patterns-core-4.1.2/patterns/httpd #系统内置变量

cd /usr/local/logstash/config/

cp logstash-sample.conf httpd-access.conf

vim httpd-access.conf

input {stdin {}
}
filter {grok  {match => {"message" => "%{HTTPD_COMBINEDLOG}"}remove_field => ["message","auth","ident"]}date {match => ["timestamp","dd/MM/yyy:HH:mm:ss Z"]}
}output {stdout {}
}

cat /var/log/httpd/access_log

127.0.0.1 - - [21/Jun/2023:14:41:22 +0800] "GET / HTTP/1.1" 403 4897 "-" "curl/7.29.0"

logstash -f httpd-access.conf

127.0.0.1 - - [21/Jun/2023:14:41:22 +0800] "GET / HTTP/1.1" 403 4897 "-" "curl/7.29.0"

可以看出apache的日志输出带有\，咱可以改进一下：

cd /usr/local/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-patterns-core-4.1.2/patterns/

grep QS -R .

grep QUOTEDSTRING -R .

此时可以看出定义太复杂，咱自定义一个变量ALL，取代QS，双引号引起来。

vim /usr/local/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-patterns-core-4.1.2/patterns/httpd

ALL .* #空行新增

HTTPD_COMBINEDLOG %{HTTPD_COMMONLOG} "%{ALL:referrer}" "%{ALL:agent}"

此时filter的相关配置已经完善!

接下来配置input引用apache日志，output输出到es。

vim /usr/local/logstash/config/httpd-access.conf

input {file {path => ["/var/log/*.log","/var/log/message*"]type => "httpd_access"start_position => "beginning"}
}
filter {grok  {match => {"message" => "%{HTTPD_COMBINEDLOG}"}remove_field => ["message","auth","ident"]}date {match => ["timestamp","dd/MM/yyy:HH:mm:ss Z"]}
}output {elasticsearch {hosts => ["http://192.168.148.132:9200"]index => "%{type}-%{+YYYY.MM.dd}"}
}

logstash -f /usr/local/logstash/config/httpd-access.conf

127.0.0.1 - - [21/Jun/2023:14:41:22 +0800] "GET / HTTP/1.1" 403 4897 "-" "curl/7.29.0"

安装插件ElasticSearch Head可视化工具

（略）

测试：浏览器访问192.168.148.132:9100

http://localhost:9200/ > http://192.168.148.132:9200/ > 连接

收集nginx日志输出到es

方案一

nginx:

修改 nginx server 的配置文件

    log_format  json '{''"client":"$remote_addr",''"time":"$time_local",''"verb":"$request_method",''"url":"$request_uri",''"status":"$status",''"size":$body_bytes_sent,' '"referer": "$http_referer",''"agent": "$http_user_agent"''}';access_log  /var/log/nginx/access_json.log  json;   

logstash配置文件:

input {file {path => "/var/log/nginx/access_json.log"codec => "json"                           #输入预定义好的 JSON 数据, 可以省略掉 filter/grok 配置, 从而减轻logstash的负载start_position => "beginning"}
}
output {elasticsearch {hosts => ["192.168.10.11:9200"]index => "nginx-log-%{+YYYY.MM.dd}"}
}

方案二

Logstash 对nginx 标准日志的 grok 正则定义是:

MAINNGINXLOG %{COMBINEDAPACHELOG} %{QS:x_forwarded_for}

logstash:直接使用访问日志

input {file {path => "/var/log/nginx/access.log"start_position => beginning}
}
filter {grok {match => { "message" => "%{COMBINEDAPACHELOG} %{QS:x_forwarded_for}"}    }    date {match => [ "timestamp" , "dd/MMM/YYYY:HH:mm:ss Z" ]    }    geoip {source => "clientip"    }
}
output {elasticsearch {     hosts => "192.168.11.10:9200" }
}