filebeat

1、作用

1、可以在本机收集日志2、也可以远程收集日志3、轻量级的日志收集系统，可以在非java环境运行。logstash是在jmv环境中运行，资源消耗很大，启动一个logstash要消耗500M左右的内存，filebeat只消耗10M左右的内存。收集nginx的日志

2、收集nginx日志

systemctl restart nginx
systemctl stop firewalld
setenforce 0#解压，将filebeat移到/usr/local下
tar -xf filebeat-6.7.2-linux-x86_64.tar.gz
mv filebeat-6.7.2-linux-x86_64 /usr/local/filebeat#备份
cd /usr/local/filebeat
cp filebeat.yml filebeat.yml.bak#修改配置文件
vim filebeat.yml
type: log
enabled: true
paths:- /usr/local/nginx/logs/access.log- /usr/local/nginx/logs/error.log
#开启日志收集，以及确定日志文本的路径，指定标签和发送到目标主机的logstashtags: ["nginx"]fields: service_name: 192.168.230.21_nginxlog_type_ nginxfrom: 192.168.230.21#output.elasticsearch:# Array of hosts to connect to.#hosts: ["localhost:9200"]outout elasticsearch
output.logsatsh:
hosts: ["192.168.230.30:5045"]
#5044是logstash默认的端口，只要是logstash主机上没有被占用的端口都可以使用，端口号要大于1024#修改从21接收的nginx日志文件
vim nginx_21.confinput {beats { port => "5045"}
}output {if "nginx" in [tags] {elasticsearch {hosts => ["192.168.230.10:9200","192.168.230.230:9200"]index => "%{[fields][service_name]}-%{+YYYY.MM.dd}"}}
}#启动filebeat
nohup ./filebeat -e -c filebeat.yml > filebeat.out &
#-e：输出到标准输出-c：指定配置文件nohup：在系统的后台运行，不会因为终端的关闭导致程序停机运行；可以把运行的日志保存到指定文件#后后台运行
logstash -f nginx_61.conf --path.data /opt/test2 &

3、远程收集nginx、http、mysql日志

filebeat远程收集发送到logstash主机

展示的索引：

192.168.230.21_mysql-*

192.168.230.21_nginx-*

192.168.230.21_http-*

systemctl stop firewalld
setenforce 0#修改配置文件
vim /etc/my.cnf
添加：
general_log=ON
general_log_file=/usr/local/mysql/data/mysql_general.log#安装httpd、nginx
yum -y install httpd nginx#重启mysqld、httpd
systemctl restart mysqld
systemctl restart httpd#修改nginx的端口号
listen       85;
#listen       [::]:80;#重启nginx
systemctl restart nginx#浏览器访问nginx、httpd#将filebeat的安装包放到/opt目录下，并解压
tar -xf filebeat-6.7.2-linux-x86_64.tar.gz#将filebeat-6.7.2-linux-x86_64放到/opt命令下，改名为filebeat
mv filebeat-6.7.2-linux-x86_64 filebeat#修改filebeat的配置文件
vim filebeat.yml
- type: logenabled: truepaths:- /var/log/nginx/access.log- /var/log/nginx/error.logtags: ["nginx"]fields:service_name: 192.168.230.21_nginxlog_type: nginxfrom: 192.168.230.21- type: logenabled: truepaths:- /var/log/httpd/access_log- /var/log/httpd/error_logtags: ["httpd"]fields:service_name: 192.168.230.21_httpdlog_type: httpdfrom: 192.168.230.21- type: logenabled: truepaths:- /usr/local/mysql/data/mysql_general.logtags: ["mysqld"]fields:service_name: 192.168.230.21_mysqldlog_type: mysqldfrom: 192.168.230.21#output.elasticsearch:# Array of hosts to connect to.#hosts: ["localhost:9200"]output.logstash:# The Logstash hostshosts: ["192.168.230.30:5048"]vim nmh_21.confinput {beats { port => "5048"}
}output {if "nginx" in [tags] {elasticsearch {hosts => ["192.168.230.10:9200","192.168.230.230:9200"]index => "%{[fields][service_name]}-%{+YYYY.MM.dd}"}}if "httpd" in [tags] {elasticsearch {hosts => ["192.168.230.10:9200","192.168.230.230:9200"]index => "%{[fields][service_name]}-%{+YYYY.MM.dd}"}}if "mysqld" in [tags] {elasticsearch {hosts => ["192.168.230.10:9200","192.168.230.230:9200"]index => "%{[fields][service_name]}-%{+YYYY.MM.dd}"}}
}#开启日志收集
nohup ./filebeat -e -c filebeat.yml > filebeat.out &#后台运行
logstash -f nmh_21.conf --path.data /opt/test3 &#多了httpd、nginx、mysqld的日志#到kibana可视化界面创建索引模式

4、收集http日志

#安装httpd
yum -y install httpdcd /etc/logstash
cd conf.d#配置文件
vim http.confinput {file {path => "/etc/httpd/logs/access_log"type => "access"start_position => "beginning"} file {path => "/etc/httpd/logs/error_log"type => "error"start_position => "beginning"}
}output {if [type] == "access" {elasticsearch {hosts => ["192.168.230.10:9200","192.168.230.230:9200"]index => "apache_access-%{+YYYY.MM.dd}"} 
}if [type] == "error" {elasticsearch {hosts => ["192.168.230.10:9200","192.168.230.230:9200"]index => "apache_error-%{+YYYY.MM.dd}"}
}#启动logstash
logstash -f http.conf --path.data /opt/test1 &
显示：Successful started Logstash API endpoint {:port=>9601}
API接口：软件内部代码之间通信的接口，代码的连接点
端口是对外提供访问程序的内容接口#到kibana可视化界面创建索引模式