logstash入门学习

1、入门示例

1.1、安装

Redhat 平台

rpm --import http://packages.elasticsearch.org/GPG-KEY-elasticsearch
cat > /etc/yum.repos.d/logstash.repo <<EOF
[logstash-5.0]
name=logstash repository for 5.0.x packages
baseurl=http://packages.elasticsearch.org/logstash/5.0/centos
gpgcheck=1
gpgkey=http://packages.elasticsearch.org/GPG-KEY-elasticsearch
enabled=1
EOF
yum clean all
yum install logstash

yum install -y java-1.8.0-openjdk

1.2Hello World

运行

/usr/share/logstash/bin/logstash -e 'input{stdin{}}output{stdout{codec=>rubydebug}}'

结果

1.3、其他

[root@db01-84-31 conf.d]# /usr/share/logstash/bin/logstash -f logstash.conf
--- jar coordinate com.fasterxml.jackson.core:jackson-annotations already loaded with version 2.7.1 - omit version 2.7.0
--- jar coordinate com.fasterxml.jackson.core:jackson-databind already loaded with version 2.7.1 - omit version 2.7.1-1
Logstash has a new settings file which defines start up time settings. This file is typically located in $LS_HOME/config or /etc/logstash. If you installed Logstash through a package and are starting it manually please specify the location to this settings file by passing in "--path.settings=/path/.." in the command line options {:level=>:warn}
Failed to load settings file from "path.settings". Aborting... {"path.settings"=>"/usr/share/logstash/config", "exception"=>Errno::ENOENT, "message"=>"No such file or directory - /usr/share/logstash/config/logstash.yml", :level=>:fatal}
[root@db01-84-31 conf.d]# touch /usr/share/logstash/config/logstash.yml
touch: 无法创建"/usr/share/logstash/config/logstash.yml": 没有那个文件或目录
[root@db01-84-31 conf.d]# mkdir /usr/share/logstash/config/
[root@db01-84-31 conf.d]# touch /usr/share/logstash/config/logstash.yml

[root@db01-84-31 conf.d]# cat logstash.conf
input{stdin{}
}filter{}output{stdout{codec=>rubydebug}
}
[root@db01-84-31 conf.d]#

[root@db01-84-31 conf.d]# /usr/share/logstash/bin/logstash -f logstash.conf
--- jar coordinate com.fasterxml.jackson.core:jackson-annotations already loaded with version 2.7.1 - omit version 2.7.0
--- jar coordinate com.fasterxml.jackson.core:jackson-databind already loaded with version 2.7.1 - omit version 2.7.1-1
Pipeline main started
12345678910
{"@timestamp" => 2024-08-21T04:05:46.003Z,"@version" => "1","host" => "db01-84-31","message" => "12345678910"
}

2、插件配置

[root@db01-84-31 conf.d]# /usr/share/logstash/bin/logstash-plugin --help
Usage:bin/logstash-plugin [OPTIONS] SUBCOMMAND [ARG] ...Parameters:SUBCOMMAND                    subcommand[ARG] ...                     subcommand argumentsSubcommands:install                       Install a pluginuninstall                     Uninstall a pluginupdate                        Update a pluginpack                          Package currently installed pluginsunpack                        Unpack packaged pluginslist                          List all installed pluginsgenerate                      Create the foundation for a new pluginOptions:-h, --help                    print help

2.1、input详解

https://elkguide.elasticsearch.cn/logstash/plugins/input/

读取文件(File)

分析网站访问日志应该是一个运维工程师最常见的工作了。所以我们先学习一下怎么用 logstash 来处理日志文件。

配置示例

input {file {path => ["/var/log/*.log", "/var/log/message"]type => "system"start_position => "beginning"}
}

 elasticsearch {hosts => ["192.168.84.71:9200"]index => "nginxlog"flush_size => 20000idle_flush_time => 10}