
ElasticStack集群部署
(测试环境)
ElasticStack集群部署包含的技术栈:
- ElasticSearch :
数据库,用于数据存储。
- filebeat:
数据采集,将数据写入ES集群。
- Kibana:
数据展示,从ES集群获取数据,并提供图形化界面。
- Logstash:
做数据处理。
- zookeeper :
分布式协调服务。
- Kafka:
分布式消息队列。
在企业当中,有可能用到的架构: EFK,ELK,ELFK,ELFK+kafka
环境准备:
2 Core 4GB 50GB+
10.0.0.91 elk91
10.0.0.92 elk92
10.0.0.93 elk93
ElasticSearch单机部署
参考链接:
https://www.elastic.co/guide/en/elasticsearch/reference/7.17/install-elasticsearch.html
1.下载ES软件包
wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-7.17.28-amd64.deb
2.安装ES
[root@elk91 ~]# dpkg -i elasticsearch-7.17.28-amd64.deb
3.修改ES的配置文件
[root@elk91 ~]# vim /etc/elasticsearch/elasticsearch.yml
...
[root@elk91 ~]# egrep -v "^#|^$" /etc/elasticsearch/elasticsearch.yml
cluster.name: oldboyedu-linux98-single
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
network.host: 0.0.0.0
http.port: 9200
discovery.type: single-node
[root@elk91 ~]#
*********************
相关参数说明:
cluster.name:
指定集群的名称。
path.data:
数据的存储路径。
path.logs:
日志的存储路径。
network.host:
服务的监听地址。
http.port:
服务的监听端口。
discovery.type:
指定部署ES的模式,可以指定单点模式。
4.登录管理员用户
oldboy@elk93:~$ sudo su -
[sudo] password for oldboy: # 输入密码1
配置ps1
[root@elk93 ~]#vim .bashrc
[root@elk93 ~]#source .bashrc
5.启动ES服务
5[root@elk91 ~]# systemctl enable --now elasticsearch
[root@elk91 ~]# ss -ntl | egrep "92|300"
LISTEN 0 4096 *:9200 *:*
LISTEN 0 4096 *:9300 *:*
[root@elk91 ~]#
6.访问测试
[root@elk93 ~]# curl http://10.0.0.91:9200/
{
"name" : "elk91",
"cluster_name" : "oldboyedu-linux98-single",
"cluster_uuid" : "mOskmq27SdKvmyeHWGb55Q",
"version" : {
"number" : "7.17.28",
"build_flavor" : "default",
"build_type" : "deb",
"build_hash" : "139cb5a961d8de68b8e02c45cc47f5289a3623af",
"build_date" : "2025-02-20T09:05:31.349013687Z",
"build_snapshot" : false,
"lucene_version" : "8.11.3",
"minimum_wire_compatibility_version" : "6.8.0",
"minimum_index_compatibility_version" : "6.0.0-beta1"
},
"tagline" : "You Know, for Search"
}
[root@elk93 ~]#
7.查看集群的节点数量
[root@elk01 /var/lib/elasticsearch]#curl http://43.139.47.66:9200/_cat/nodes
10.1.24.4 62 97 1 0.06 0.04 0.01 cdfhilmrstw - elk03 # 这个-表示从节点
10.1.20.5 51 97 2 0.15 0.12 0.06 cdfhilmrstw - elk01 # 这个-表示从节点
10.1.24.13 14 96 1 0.00 0.02 0.02 cdfhilmrstw * elk02 # 这个*表示主节点
2、在这里插入一个ES故障排查技巧!!!!
ES故障排查技巧
#1、查看服务配置文件,所有服务都适用
systemctl cat elasticsearch.service
2、实时查看ElasticSearch服务的日志,所有服务都适用
journalctl -u elasticsearch.service -f
3、查看日志观察详细的日志信息
tail -f /var/log/elasticsearch/oldboyedu-linux98-single.log
4、手动启动ES服务
观察是否有错误信息输出,如果直接kill,则可能是内存不足导致
具体操作如下:
[root@elk91 ~]# vim /etc/elasticsearch/jvm.options
...
-Xms256m
-Xmx256m
3、ElasticSearch集群部署
1.停止旧集群服务
[root@elk91 ~]# systemctl disable --now elasticsearch.service
2.清空原始数据
[root@elk91 ~]# rm -rf /var/{log,lib}/elasticsearch/*
3.修改ES的配置文件
[
4.拷贝ES程序到其他节点
[root@elk91 ~]# scp elasticsearch-7.17.28-amd64.deb 10.0.0.92:~
[root@elk91 ~]# scp elasticsearch-7.17.28-amd64.deb 10.0.0.93:~
5.其他节点安装ES服务
[root@elk92 ~]# dpkg -i elasticsearch-7.17.28-amd64.deb
[root@elk93 ~]# dpkg -i elasticsearch-7.17.28-amd64.deb
6.拷贝配置文件到其他节点
[root@elk91 ~]# scp /etc/elasticsearch/elasticsearch.yml 10.0.0.92:/etc/elasticsearch
[root@elk91 ~]# scp /etc/elasticsearch/elasticsearch.yml 10.0.0.93:/etc/elasticsearch
7.集群启动
[root@elk91 ~]# systemctl enable --now elasticsearch.service
[root@elk91 ~]# ss -ntl | egrep "92|300"
LISTEN 0 4096 *:9200 *:*
LISTEN 0 4096 *:9300 *:*
[root@elk91 ~]#
[root@elk92 ~]# systemctl enable --now elasticsearch.service
[root@elk92 ~]# ss -ntl | egrep "92|300"
LISTEN 0 4096 *:9200 *:*
LISTEN 0 4096 *:9300 *:*
[root@elk92 ~]#
[root@elk93 ~]# systemctl enable --now elasticsearch.service
[root@elk93 ~]# ss -ntl | egrep "92|300"
LISTEN 0 4096 *:9300 *:*
LISTEN 0 4096 *:9200 *:*
[root@elk93 ~]#
8.测试验证集群是否正常
[root@elk91 ~]# for i in `seq 91 93`; do curl -s 10.0.0.${i}:9200 | egrep "cluster_uuid";done
"cluster_uuid" : "rvKtmLJURKaYV5mUI3LTAg",
"cluster_uuid" : "rvKtmLJURKaYV5mUI3LTAg",
"cluster_uuid" : "rvKtmLJURKaYV5mUI3LTAg",
[root@elk91 ~]#
[root@elk91 ~]#
[root@elk91 ~]# curl 10.0.0.93:9200/_cat/nodes
10.0.0.93 5 97 4 0.07 0.14 0.06 cdfhilmrstw - elk93
10.0.0.92 5 97 5 0.05 0.12 0.04 cdfhilmrstw * elk92
10.0.0.91 25 97 5 0.04 0.10 0.03 cdfhilmrstw - elk91
[root@elk91 ~]#
测试集群是否可以正常读写
1.写入数据
curl --location --request POST 'http://10.0.0.91:9200/_bulk' \
--header 'Content-Type: application/json' \
--data-raw '{ "create" : { "_index" : "shaopeng-linux996", "_id" : "1001" } }
{ "name" : "猪八戒","hobby": ["猴哥","高老庄"] }
{ "create" : { "_index" : "shaopeng-linux996", "_id" : "1002" } }
{ "name" : "沙和尚","hobby": ["流沙河","挑行李"] }
{ "create" : { "_index" : "shaopeng-linux996", "_id" : "1003" } }
{ "name" : "白龙马","hobby": ["大师兄,师傅被妖怪抓走啦"] }
'
2.查询数据
apt -y install jq
curl -s --location --request GET '10.0.0.93:9200/shaopeng-linux996/_search' \
--header 'Content-Type: application/json' \
--data-raw '{
"query": {
"match": {
"hobby": "猴哥"
}
}
}' | jq
3.删除索引
curl --location --request DELETE '10.0.0.92:9200/curl --location --request DELETE '10.0.0.92:9200/oldboyedu-linux98''
综上所述:
我们不难发现,直接使用ES集群的DSL语句查询数据,发现对于新手并不友好,于是我们采用kibana来解决此问题。
- ES集群的常用术语:
- 索引: Index
客户端对ES进行数据读写的逻辑单元
- 分片: Shard
一个索引最少有1个或多个分片,是数据的实际存储载体。
分片不可切分,隶属于某个ES节点,分片可以从某个节点迁移到其他节点。
如果说一个索引只有1个分片的话,该索引将无法充分利用集群资源。
- 副本: replica
副本是针对分片而言的,用于对分片的数据进行备份。
一个分片可以有0个或多个副本。
当分片的副本数量为0时,则可能会存在数据丢失的风险。
- 文档: document
文档是用户进行数据存储的最小单元。文档包含元数据和源数据。
元数据:
用于描述源数据的数据。
源数据:
用户实际存储的数据。
举个例子:
源数据:
{"name": "孙悟空","hobby": "紫霞仙子"}
ES中存储的样子:
{
_index: "shaopeng-linux996",
_type: "_doc",
_id: "XXXXXX"
...
_source: {"name": "孙悟空","hobby": "紫霞仙子"}
}
其中源数据就是"_source"字段的内容,而"_source","_type","_index","_id"都是用来描述源数据的数据,这些字段称之为"元数据"。
ES集群故障排查思路
1.检查配置文件是否正确
egrep -v "^#|^$" /etc/elasticsearch/elasticsearch.yml
2.尝试清空数据
systemctl disable --now elasticsearch.service
rm -rf /var/{log,lib}/elasticsearch/*
3.启动服务
systemctl enable --now elasticsearch.service
快速部署kibana
1.下载kibana 软件包
wget https://artifacts.elastic.co/downloads/kibana/kibana-7.17.28-amd64.deb
2.安装kibana
[root@elk91 ~]# dpkg -i kibana-7.17.28-amd64.deb
3.修改kibana的配置文件
[root@elk91 ~]# vim /etc/kibana/kibana.yml
...
[root@elk91 ~]# egrep -v "^#|^$" /etc/kibana/kibana.yml
server.port: 5601
server.host: "0.0.0.0"
elasticsearch.hosts: ["http://10.0.0.91:9200","http://10.0.0.92:9200","http://10.0.0.93:9200"]
i18n.locale: "zh-CN"
[root@elk91 ~]#
对相关参数进行说明:
server.port:
服务的监听端口。
server.host:
服务的监听地址。
elasticsearch.hosts:
指定ES集群地址。
i18n.locale: "zh-CN"
kibana图形化展示使用的语言。
4.启动服务
[root@elk91 ~]# systemctl enable --now kibana.service
[root@elk91 ~]# ss -ntl | grep 5601
LISTEN 0 511 0.0.0.0:5601 0.0.0.0:*
[root@elk91 ~]#
5.访问测试
http://10.0.0.91:5601/
filebeat环境部署
1.下载软件包
wget https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-7.17.28-amd64.deb
2.安装filebeat
[root@elk92 ~]# dpkg -i filebeat-7.17.28-amd64.deb
3.编写配置文件
root@elk92 ~]# mkdir /etc/filebeat/config
[root@elk92 ~]# cd /etc/filebeat
[root@elk92 filebeat]# cat config/01-stdin-to-console.yaml
filebeat.inputs:
- type: stdin
output.console:
pretty: true
4.启动服务
[root@elk92 filebeat]# filebeat -e -c config/01-stdin-to-console.yaml
...
111111111111111111111111111111111111
{
"@timestamp": "2025-06-19T08:37:35.519Z",
"@metadata": {
"beat": "filebeat",
"type": "_doc",
"version": "7.17.28"
},
"agent": {
"ephemeral_id": "60ff7608-8f0f-4cf1-abdb-bb7ed555c0b0",
"id": "d2d0d6a3-a362-4400-b47a-0658acabe034",
"name": "elk92",
"type": "filebeat",
"version": "7.17.28",
"hostname": "elk92"
},
"log": {
"offset": 0,
"file": {
"path": ""
}
},
"message": "111111111111111111111111111111111111",
"input": {
"type": "stdin"
},
"ecs": {
"version": "1.12.0"
},
"host": {
"name": "elk92"
}
}
filebeat采集日志文件细节分析
1.编写filebeat配置文件
[root@elk92 filebeat]# cat config/02-log-to-console.yaml
filebeat.inputs:
- type: log
paths:
- /tmp/xixi.log
output.console:
pretty: true
[root@elk92 filebeat]#
2.启动服务
[root@elk92 filebeat]# filebeat -e -c config/02-log-to-console.yaml
3.发送测试数据【观察filebeat采集效果】
[root@elk92 ~]# echo ABCD > /tmp/xixi.log
[root@elk92 ~]#
[root@elk92 ~]# cat /tmp/xixi.log
ABCD
[root@elk92 ~]#
[root@elk92 ~]# echo -n abc >> /tmp/xixi.log
[root@elk92 ~]#
[root@elk92 ~]# echo -n 1234 >> /tmp/xixi.log
[root@elk92 ~]#
[root@elk92 ~]# cat /tmp/xixi.log
ABCD
abc1234[root@elk92 ~]#
[root@elk92 ~]#
[root@elk92 ~]# echo def >> /tmp/xixi.log
[root@elk92 ~]# cat /tmp/xixi.log
ABCD
abc1234def
[root@elk92 ~]#
温馨提示喔:
- filebeat默认按行采集数据;
- filebeat会保留采集源文件的offset记录。
- /var/lib/filebeat/registry/filebeat/log.json
- 首次采集,或数据目录被删除时,则默认从头采集数据。
filebeat采集nginx日志并写入ES集群
1.安装nginx服务并启动
[root@elk92 ~]# apt -y install nginx
[root@elk92 ~]# systemctl enable --now nginx
2.filebeat采集nginx日志
root@elk92 filebeat]# cat
filebeat.inputs:
- type: log
paths:
- /var/log/nginx/access.log*
#output.console:
# pretty: true
output.elasticsearch:
hosts:
- 10.0.0.91:9200
- 10.0.0.92:9200
- 10.0.0.93:9200
# 自定义索引名称
index: "linux996-nginx-accesslog-%{+yyyy.MM.dd}"
# 关闭索引生命周期,如果开启,则不支持filebeat自定义索引名称
setup.ilm.enabled: false
# 定义索引模板的名称
setup.template.name: "shaopeng-linux996"
# 定义索引的匹配模式,该模式可以匹配写入ES的索引。匹配的索引必须遵循该索引模板的配置。
setup.template.pattern: "linux996*"
# 如果索引模板已经存在是否覆盖。
setup.template.overwrite: false
# 设置索引模板
setup.template.settings:
# 自定义索引的分片数量
index.number_of_shards: 3
# 自定义索引的副本数量
index.number_of_replicas: 0
[root@elk92 filebeat]#
3.采集日志到ES集群
[root@elk92 filebeat]# rm -rf /var/lib/filebeat/
[root@elk92 filebeat]# filebeat -e -c config/03-nginx-to-es.yaml
4.kibana查看数据
5.模拟请求
[root@elk91 ~]# for i in `seq 10`; do curl 10.0.0.92 ;done
filebeat的模块管理
1.什么是filebeat模块?
其实就是filebeat针对不同主流中间件日志采集的预定方案。
2.查看模块列表
[root@elk92 ~]# filebeat modules list
Enabled:
Disabled:
activemq
apache
auditd
aws
awsfargate
azure
barracuda
bluecoat
cef
checkpoint
cisco
coredns
...
3.启用模块
[root@elk92 ~]# filebeat modules enable nginx tomcat mysql traefik
Enabled nginx
Enabled tomcat
Enabled mysql
Enabled traefik
[root@elk92 ~]#
[root@elk92 ~]# filebeat modules list
Enabled:
mysql
nginx
tomcat
traefik
Disabled:
activemq
apache
auditd
aws
awsfargate
azure
...
[root@elk92 ~]# ll /etc/filebeat/modules.d/*.yml
-rw-r--r-- 1 root root 472 Feb 14 00:58 /etc/filebeat/modules.d/mysql.yml
-rw-r--r-- 1 root root 784 Feb 14 00:58 /etc/filebeat/modules.d/nginx.yml
-rw-r--r-- 1 root root 623 Feb 14 00:58 /etc/filebeat/modules.d/tomcat.yml
-rw-r--r-- 1 root root 303 Feb 14 00:58 /etc/filebeat/modules.d/traefik.yml
[root@elk92 ~]#
4.禁用模块
[root@elk92 ~]# filebeat modules disable mysql traefik
Disabled mysql
Disabled traefik
[root@elk92 ~]#
[root@elk92 ~]# ll /etc/filebeat/modules.d/*.yml
-rw-r--r-- 1 root root 784 Feb 14 00:58 /etc/filebeat/modules.d/nginx.yml
-rw-r--r-- 1 root root 623 Feb 14 00:58 /etc/filebeat/modules.d/tomcat.yml
[root@elk92 ~]#
[root@elk92 ~]# filebeat modules list
Enabled:
nginx
tomcat
Disabled:
activemq
apache
auditd
aws
awsfargate
azure
...
5.验证模块启用和禁用的原理
[root@elk92 ~]# ll /etc/filebeat/modules.d/*.yml
-rw-r--r-- 1 root root 784 Feb 14 00:58 /etc/filebeat/modules.d/nginx.yml
-rw-r--r-- 1 root root 623 Feb 14 00:58 /etc/filebeat/modules.d/tomcat.yml
[root@elk92 ~]#
[root@elk92 ~]# mv /etc/filebeat/modules.d/tomcat.yml{,.disabled}
[root@elk92 ~]#
[root@elk92 ~]# ll /etc/filebeat/modules.d/*.yml
-rw-r--r-- 1 root root 784 Feb 14 00:58 /etc/filebeat/modules.d/nginx.yml
[root@elk92 ~]#
[root@elk92 ~]# filebeat modules list
Enabled:
nginx
Disabled:
activemq
apache
auditd
aws
awsfargate
azure
...
filebeat的模块案例之nginx
1.准备Nginx访问日志
[root@elk92 ~]# cat /var/log/nginx/access.log
123.117.19.236 - - [19/Jun/2025:17:27:13 +0800] "GET / HTTP/1.1" 200 612 "-" "curl/7.81.0"
123.117.19.236 - - [19/Jun/2025:17:36:41 +0800] "GET / HTTP/1.1" 200 612 "-" "curl/7.81.0"
123.117.19.236 - - [19/Jun/2025:17:36:41 +0800] "GET / HTTP/1.1" 200 612 "-" "curl/7.81.0"
123.117.19.236 - - [19/Jun/2025:17:36:41 +0800] "GET / HTTP/1.1" 200 612 "-" "curl/7.81.0"
123.117.19.236 - - [19/Jun/2025:17:36:41 +0800] "GET / HTTP/1.1" 200 612 "-" "curl/7.81.0"
123.117.19.236 - - [19/Jun/2025:17:36:41 +0800] "GET / HTTP/1.1" 200 612 "-" "curl/7.81.0"
123.117.19.236 - - [19/Jun/2025:17:36:41 +0800] "GET / HTTP/1.1" 200 612 "-" "curl/7.81.0"
123.117.19.236 - - [19/Jun/2025:17:36:41 +0800] "GET / HTTP/1.1" 200 612 "-" "curl/7.81.0"
123.117.19.236 - - [19/Jun/2025:17:36:41 +0800] "GET / HTTP/1.1" 200 612 "-" "curl/7.81.0"
123.117.19.236 - - [19/Jun/2025:17:36:41 +0800] "GET / HTTP/1.1" 200 612 "-" "curl/7.81.0"
123.117.19.236 - - [19/Jun/2025:17:36:41 +0800] "GET / HTTP/1.1" 200 612 "-" "curl/7.81.0"
23.117.19.236 - - [20/Jun/2025:09:31:30 +0800] "GET / HTTP/1.1" 200 396 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/137.0.0.0 Safari/537.36"
24.117.19.236 - - [20/Jun/2025:09:31:30 +0800] "GET /favicon.ico HTTP/1.1" 404 197 "http://10.0.0.92/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/137.0.0.0 Safari/537.36"
25.117.19.236 - - [20/Jun/2025:09:31:58 +0800] "GET / HTTP/1.1" 200 396 "-" "Mozilla/5.0 (iPhone; CPU iPhone OS 16_6 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/16.6 Mobile/15E148 Safari/604.1"
30.117.19.236 - - [20/Jun/2025:09:32:17 +0800] "GET / HTTP/1.1" 200 396 "-" "Mozilla/5.0 (iPhone; CPU iPhone OS 16_6 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/16.6 Mobile/15E148 Safari/604.1"
31.117.19.236 - - [20/Jun/2025:09:32:30 +0800] "GET / HTTP/1.1" 200 396 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/16.0 Safari/605.1.15"
32.117.19.236 - - [20/Jun/2025:09:32:51 +0800] "GET / HTTP/1.1" 200 396 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/16.0 Safari/605.1.15"
40.117.19.236 - - [20/Jun/2025:09:33:04 +0800] "GET / HTTP/1.1" 200 396 "-" "Mozilla/5.0 (iPad; CPU OS 16_6 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/16.6 Mobile/15E148 Safari/604.1"
41.117.19.236 - - [20/Jun/2025:09:33:12 +0800] "GET / HTTP/1.1" 200 396 "-" "Mozilla/5.0 (iPad; CPU OS 16_6 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/16.6 Mobile/15E148 Safari/604.1"
42.117.19.236 - - [20/Jun/2025:09:33:13 +0800] "GET / HTTP/1.1" 200 396 "-" "Mozilla/5.0 (iPad; CPU OS 16_6 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/16.6 Mobile/15E148 Safari/604.1"
51.117.19.236 - - [20/Jun/2025:09:33:23 +0800] "GET / HTTP/1.1" 200 396 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/16.0 Safari/605.1.15"
52.117.19.236 - - [20/Jun/2025:09:33:42 +0800] "GET / HTTP/1.1" 200 396 "-" "Mozilla/5.0 (Linux; Android 8.0.0; SM-G955U Build/R16NW) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/137.0.0.0 Mobile Safari/537.36"
53.117.19.236 - - [20/Jun/2025:09:33:49 +0800] "GET / HTTP/1.1" 200 396 "-" "Mozilla/5.0 (Linux; Android 8.0.0; SM-G955U Build/R16NW) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/137.0.0.0 Mobile Safari/537.36"
80.117.19.236 - - [20/Jun/2025:09:33:54 +0800] "GET / HTTP/1.1" 200 396 "-" "Mozilla/5.0 (Linux; Android 8.0.0; SM-G955U Build/R16NW) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/137.0.0.0 Mobile Safari/537.36"
82.117.19.236 - - [20/Jun/2025:09:33:54 +0800] "GET / HTTP/1.1" 200 396 "-" "Mozilla/5.0 (Linux; Android 8.0.0; SM-G955U Build/R16NW) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/137.0.0.0 Mobile Safari/537.36"
92.117.19.236 - - [20/Jun/2025:09:33:54 +0800] "GET / HTTP/1.1" 200 396 "-" "Mozilla/5.0 (Linux; Android 8.0.0; SM-G955U Build/R16NW) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/137.0.0.0 Mobile Safari/537.36"
78.117.19.236 - - [20/Jun/2025:09:33:55 +0800] "GET / HTTP/1.1" 200 396 "-" "Mozilla/5.0 (Linux; Android 8.0.0; SM-G955U Build/R16NW) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/137.0.0.0 Mobile Safari/537.36"
110.117.19.236 - - [20/Jun/2025:09:33:55 +0800] "GET / HTTP/1.1" 404 396 "-" "Mozilla/5.0 (Linux; Android 8.0.0; SM-G955U Build/R16NW) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/137.0.0.0 Mobile Safari/537.36"
[root@elk92 ~]#
2.启用模块
[root@elk92 ~]# ll /etc/filebeat/modules.d/*.yml
-rw-r--r-- 1 root root 784 Feb 14 00:58 /etc/filebeat/modules.d/nginx.yml
[root@elk92 ~]#
[root@elk92 ~]# egrep -v "^.*#|^$" /etc/filebeat/modules.d/nginx.yml
- module: nginx
access:
enabled: true
error:
enabled: false
ingress_controller:
enabled: false
[root@elk92 ~]#
3.编写filebeat配置文件
[root@elk92 filebeat]# cat config/04-module-to-es.yaml
filebeat.config.modules:
path: ${path.config}/modules.d/nginx.yml
reload.enabled: true
#output.console:
# pretty: true
output.elasticsearch:
hosts:
- 10.0.0.91:9200
- 10.0.0.92:9200
- 10.0.0.93:9200
# 自定义索引名称
index: "linux98-modeules-nginx-accesslog-%{+yyyy.MM.dd}"
# 关闭索引生命周期,如果开启,则不支持filebeat自定义索引名称
setup.ilm.enabled: false
# 定义索引模板的名称
setup.template.name: "shaopeng-liux996"
# 定义索引的匹配模式,该模式可以匹配写入ES的索引。匹配的索引必须遵循该索引模板的配置。
setup.template.pattern: "linux996*"
# 如果索引模板已经存在是否覆盖。
setup.template.overwrite: false
# 设置索引模板
setup.template.settings:
# 自定义索引的分片数量
index.number_of_shards: 3
# 自定义索引的副本数量
index.number_of_replicas: 0
[root@elk92 filebeat]#
[root@elk92 filebeat]# rm -rf /var/lib/filebeat/
[root@elk92 filebeat]#
[root@elk92 filebeat]# filebeat -e -c config/04-module-to-es.yaml
5.kibana查询数据
略,见视频
4.启动实例
[root@elk92 filebeat]# rm -rf /var/lib/filebeat/
[root@elk92 filebeat]#
[root@elk92 filebeat]# filebeat -e -c config/04-module-to-es.yaml
5.kibana查询数据
2.启动filebeat实例
[root@elk92 filebeat]# cat config/05-efk-to-es.yaml
filebeat.config.modules:
path: ${path.config}/modules.d/nginx.yml
reload.enabled: true
output.elasticsearch:
hosts:
- 10.0.0.91:9200
- 10.0.0.92:9200
- 10.0.0.93:9200
index: "linux98-efk-nginx-accesslog-%{+yyyy.MM.dd}"
setup.ilm.enabled: false
setup.template.name: "shaopeng-linux996"
setup.template.pattern: "linux996*"
setup.template.overwrite: false
setup.template.settings:
index.number_of_shards: 3
index.number_of_replicas: 0
[root@elk92 filebeat]#
[root@elk92 filebeat]# rm -rf /var/lib/filebeat/
[root@elk92 filebeat]# filebeat -e -c config/05-efk-to-es.yaml
3.统计PV
4.IP统计
5.带宽统计(相关字段)
http.request.body.bytes
6.全球用户分布图(相关字段)
source.geo.location
7.操作系统统计(相关字段)
user_agent.os.full
8.用户设备类型分析(相关字段)
user_agent.device.name
Kibana如果查询不到数据,可能是由什么原因呢?
温馨提示:
- Filebeat端存在问题的可能性:
- filebeat挂掉无法采集数据;
- 配置文件和实际采集的数据不对应;
- 源数据文件为空,未能写入;
- 数据已经采集过了,本地缓存offset未清空;
- logstash和Filebeat同理,也会存在类似的问题。
- ES集群挂掉,导致kibana无法查询数据;
- kibana的时间选择有问题,也会查询不到数据;
- kibana做了KQL数据过滤,也可能导致数据查询不到;
- kibana的索引被删除,索引模式不生效;
filebeat的多实例实战案例
1.什么是多实例?
一台服务器运行多个filebeat实例。多个实例共同同一套程序。
2.实战案例
2.1 启动第一个实例
[root@elk92 filebeat]# filebeat -e -c config/03-nginx-to-es.yaml
2.2 启动第二个实例
[root@elk92 filebeat]# filebeat -e -c config/02-log-to-console.yaml --path.data /tmp/xixi
2.3 测试验证
[root@elk92 ~]# ps -ef | grep filebeat
root 111460 109818 0 14:42 pts/0 00:00:00 /usr/share/filebeat/bin/filebeat --path.home /usr/share/filebeat --path.config /etc/filebeat --path.data /var/lib/filebeat --path.logs /var/log/filebeat -e -c config/03-nginx-to-es.yaml
root 111513 110130 1 14:44 pts/1 00:00:00 /usr/share/filebeat/bin/filebeat --path.home /usr/share/filebeat --path.config /etc/filebeat --path.data /var/lib/filebeat --path.logs /var/log/filebeat -e -c config/02-log-to-console.yaml --path.data /tmp/xixi
root 111640 111620 0 14:44 pts/2 00:00:00 grep --color=auto filebeat
[root@elk92 ~]#
[root@elk92 ~]#
3.实战案例
3.1 实例1-采集系统日志文件
[root@elk92 filebeat]#
[root@elk92 filebeat]# cat config/06-systlog-to-es.yaml
filebeat.inputs:
- type: log
paths:
- /var/log/syslog*
# 排除以"*.gz"结尾的文件
exclude_files: ['\.gz$']
output.elasticsearch:
hosts:
- 10.0.0.91:9200
- 10.0.0.92:9200
- 10.0.0.93:9200
index: "linux98-efk-syslog-%{+yyyy.MM.dd}"
setup.ilm.enabled: false
setup.template.name: "shaopeng-linux996"
setup.template.pattern: "linux996*"
setup.template.overwrite: false
setup.template.settings:
index.number_of_shards: 3
index.number_of_replicas: 0
[root@elk92 filebeat]#
[root@elk92 filebeat]#
[root@elk92 filebeat]# filebeat -e -c config/06-systlog-to-es.yaml
3.2 实例2-采集auth日志文件
[root@elk92 filebeat]# cat config/07-auth-to-es.yaml
filebeat.inputs:
- type: log
paths:
- /var/log/auth.log
exclude_files: ['\.gz$']
output.elasticsearch:
hosts:
- 10.0.0.91:9200
- 10.0.0.92:9200
- 10.0.0.93:9200
index: "linux98-efk-auth-%{+yyyy.MM.dd}"
setup.ilm.enabled: false
setup.template.name: "shaopeng-linux996"
setup.template.pattern: "linux996*"
setup.template.overwrite: false
setup.template.settings:
index.number_of_shards: 3
index.number_of_replicas: 0
[root@elk92 filebeat]#
[root@elk92 filebeat]# filebeat -e -c config/07-auth-to-es.yaml --path.data /var/lib/filebeat-auth
3.3 kibana查看数据
syslog日志过滤:
message :"Successfully "
auth日志过滤:
message :"10.0.0.1 "
filebeat写入数据到不同的ES索引
1.编写filebeat配置文件
[root@elk92 filebeat]# cat config/08-multiple_input-to-es.yaml
filebeat.inputs:
- type: log
paths:
- /var/log/auth.log
exclude_files: ['\.gz$']
# 给数据打标签
tags: "auth"
- type: log
tags: "syslog"
paths:
- /var/log/syslog*
exclude_files: ['\.gz$']
output.elasticsearch:
hosts:
- 10.0.0.91:9200
- 10.0.0.92:9200
- 10.0.0.93:9200
# index: "linux98-efk-auth-%{+yyyy.MM.dd}"
indices:
- index: "linux98-efk-contains-auth-%{+yyyy.MM.dd}"
# 当tags的值为"auth"时,则写入当前索引。
when.contains:
tags: "auth"
- index: "linux98-efk-contains-syslog-%{+yyyy.MM.dd}"
when.contains:
tags: "syslog"
setup.ilm.enabled: false
setup.template.name: "shaopeng-linux996"
setup.template.pattern: "linux996*"
# 如果索引模板已经存在,是否覆盖原有的索引模板
# setup.template.overwrite: false
setup.template.overwrite: true
setup.template.settings:
index.number_of_shards: 5
index.number_of_replicas: 0
[root@elk92 filebeat]#
2.启动filebeat实例
[root@elk92 filebeat]# rm -rf /var/lib/filebeat
[root@elk92 filebeat]#
[root@elk92 filebeat]# filebeat -e -c config/08-multiple_input-to-es.yaml
3.kibana出图展示
略
syslog日志过滤:
message :"Successfully "
auth日志过滤:
message :"10.0.0.1 "
filebeat采集json数据
1.测试文件
[root@elk92 filebeat]# cat /tmp/student.json
{"name":"张锋","hobby":["玩手机","俯卧撑","看美女"],"gender": "boy"}
{"name":"常义朝","hobby":["打台球","吹牛","喝啤酒"],"gender": "boy","age":18}
{"name":"刘志松","hobby":["打游戏","看动漫"],"gender":"boy","class": "linux98"}
{"name":"李鑫","hobby":["听音乐","打飞机"]}
{"name":"杨晓东","hobby":["学习","打飞机"]}
[root@elk92 filebeat]#
2.准备配置文件
[root@elk92 filebeat]# cat config/09-log_json-to-es.yaml
filebeat.inputs:
- type: log
paths:
- /tmp/student.json
# 将message字段进行解析,解析后的数据放在顶级字段中。
# 如果解析正确,则message字段就删除,如果解析错误,则message字段保留。
json.keys_under_root: true
#output.console:
# pretty: true
output.elasticsearch:
hosts:
- 10.0.0.91:9200
- 10.0.0.92:9200
- 10.0.0.93:9200
index: "linux98-efk-log-json-%{+yyyy.MM.dd}"
setup.ilm.enabled: false
setup.template.name: "shaopeng-linux996"
setup.template.pattern: "linux996*"
setup.template.overwrite: false
setup.template.settings:
index.number_of_shards: 3
index.number_of_replicas: 0
[root@elk92 filebeat]#
3.启动实例
[root@elk92 filebeat]# rm -rf /var/lib/filebeat
[root@elk92 filebeat]#
[root@elk92 filebeat]# filebeat -e -c config/09-log_json-to-es.yaml
4.kibana验证数据
略
filestream类型采集tomcat日志案例
1.安装tomcat
wget https://dlcdn.apache.org/tomcat/tomcat-11/v11.0.8/bin/apache-tomcat-11.0.8.tar.gz
1.2 安装tomcat
[root@elk92 ~]# tar xf apache-tomcat-11.0.8.tar.gz -C /usr/local/
[root@elk92 ~]#
1.3 配置环境变量
[root@elk92 ~]# cat /etc/profile.d/tomcat.sh
#!/bin/bash
export JAVA_HOME=/usr/share/elasticsearch/jdk
export TOMCAT_HOME=/usr/local/apache-tomcat-11.0.8
export PATH=$PATH:$TOMCAT_HOME/bin:$JAVA_HOME/bin
[root@elk92 ~]#
[root@elk92 ~]# source /etc/profile.d/tomcat.sh
[root@elk92 ~]#
1.4 启动tomcat
[root@elk92 ~]# startup.sh
Using CATALINA_BASE: /usr/local/apache-tomcat-11.0.8
Using CATALINA_HOME: /usr/local/apache-tomcat-11.0.8
Using CATALINA_TMPDIR: /usr/local/apache-tomcat-11.0.8/temp
Using JRE_HOME: /usr/share/elasticsearch/jdk
Using CLASSPATH: /usr/local/apache-tomcat-11.0.8/bin/bootstrap.jar:/usr/local/apache-tomcat-11.0.8/bin/tomcat-juli.jar
Using CATALINA_OPTS:
Tomcat started.
[root@elk92 ~]#
[root@elk92 ~]# ss -ntl | grep 8080
LISTEN 0 100 *:8080 *:*
[root@elk92 ~]#
1.5 访问tomcat的webUI
1.6 查看访问日志
[root@elk92 ~]# cat /usr/local/apache-tomcat-11.0.8/logs/localhost_access_log.2025-06-20.txt
10.0.0.1 - - [20/Jun/2025:16:49:43 +0800] "GET / HTTP/1.1" 200 11235
10.0.0.1 - - [20/Jun/2025:16:49:43 +0800] "GET /tomcat.css HTTP/1.1" 200 5584
10.0.0.1 - - [20/Jun/2025:16:49:43 +0800] "GET /tomcat.svg HTTP/1.1" 200 67795
10.0.0.1 - - [20/Jun/2025:16:49:43 +0800] "GET /asf-logo-wide.svg HTTP/1.1" 200 27235
10.0.0.1 - - [20/Jun/2025:16:49:43 +0800] "GET /bg-nav.png HTTP/1.1" 200 1401
10.0.0.1 - - [20/Jun/2025:16:49:43 +0800] "GET /bg-button.png HTTP/1.1" 200 713
10.0.0.1 - - [20/Jun/2025:16:49:43 +0800] "GET /bg-upper.png HTTP/1.1" 200 3103
10.0.0.1 - - [20/Jun/2025:16:49:43 +0800] "GET /bg-middle.png HTTP/1.1" 200 1918
10.0.0.1 - - [20/Jun/2025:16:49:43 +0800] "GET /favicon.ico HTTP/1.1" 200 21630
[root@elk92 ~]#
2.filebeat采集tomcat日志案例
2.1 编写filebeat配置文件
[root@elk92 filebeat]# cat config/10-filestream-to-es.yaml
filebeat.inputs:
- type: filestream
paths:
- /usr/local/apache-tomcat-11.0.8/logs/localhost_access_log*.txt
output.elasticsearch:
hosts:
- 10.0.0.91:9200
- 10.0.0.92:9200
- 10.0.0.93:9200
index: "linux98-efk-filestream-tomcat-%{+yyyy.MM.dd}"
setup.ilm.enabled: false
setup.template.name: "shaopeng-linux996"
setup.template.pattern: "linux996*"
setup.template.overwrite: false
setup.template.settings:
index.number_of_shards: 3
index.number_of_replicas: 0
[root@elk92 filebeat]#
2.2 启动filebeat实例
[root@elk92 filebeat]# filebeat -e -c config/10-filestream-to-es.yaml
2.3 kibana查看数据
略,
filestream解析json格式
1.修改tomcat的访问日志格式
[root@elk92 ~]# vim /usr/local/apache-tomcat-11.0.8/conf/server.xml
...
<Host name="tomcat.shaopeng.com" appBase="webapps"
unpackWARs="true" autoDeploy="true">
<Valve className="org.apache.catalina.valves.AccessLogValve" directory="logs"
prefix="tomcat.oldboyedu.com_access_log" suffix=".json"
pattern="{"clientip":"%h","ClientUser":"%l","authenticated":"%u","AccessTime":"%t","request":"%r","status":"%s","SendBytes":"%b","Query?string":"%q","partner":"%{Referer}i","http_user_agent":"%{User-Agent}i"}"/>
</Host>
2.重启tomcat服务
[root@elk92 ~]# source /etc/profile.d/tomcat.sh
[root@elk92 ~]#
[root@elk92 ~]# shutdown.sh
Using CATALINA_BASE: /usr/local/apache-tomcat-11.0.8
Using CATALINA_HOME: /usr/local/apache-tomcat-11.0.8
Using CATALINA_TMPDIR: /usr/local/apache-tomcat-11.0.8/temp
Using JRE_HOME: /usr/share/elasticsearch/jdk
Using CLASSPATH: /usr/local/apache-tomcat-11.0.8/bin/bootstrap.jar:/usr/local/apache-tomcat-11.0.8/bin/tomcat-juli.jar
Using CATALINA_OPTS:
[root@elk92 ~]#
[root@elk92 ~]# startup.sh
Using CATALINA_BASE: /usr/local/apache-tomcat-11.0.8
Using CATALINA_HOME: /usr/local/apache-tomcat-11.0.8
Using CATALINA_TMPDIR: /usr/local/apache-tomcat-11.0.8/temp
Using JRE_HOME: /usr/share/elasticsearch/jdk
Using CLASSPATH: /usr/local/apache-tomcat-11.0.8/bin/bootstrap.jar:/usr/local/apache-tomcat-11.0.8/bin/tomcat-juli.jar
Using CATALINA_OPTS:
Tomcat started.
[root@elk92 ~]#
[root@elk92 ~]# ss -ntl | grep 8080
LISTEN 0 100 *:8080 *:*
[root@elk92 ~]#
3.访问测试
http://tomcat.oldboyedu.com:8080/
温馨提示:
windows添加解析:"10.0.0.92 tomcat.oldboyedu.com"