loki日志监控
主旨
- 基于Loki+promtail+grafana日志采集监控展示
- 部署配置使用配置解析
- 深入研究不断优化方案
说明
1、目录结构
$ tree -LN 1
.
├── image # 文档截图存放
├── loki_in_docker # docker部署的存放路径
├── loki_in_k8s # k8s部署的存放路径
├── loki_in_vm # 虚拟机部署的存放路径
└── promtail+loki+grafana日志监控系统.md #主文档
4 directories, 1 file
2、环境说明
192.168.1.100 /usr/local/loki /usr/local/promtail
192.168.1.102 /root/loki_in_docker
192.168.1.100 /root/k8s/loki/loki_in_k8s
192.168.1.101
192.168.1.102
3、任务分解
-
虚拟机服务器搭建Promtail+Loki+Grafana轻量日志采集
-
Docker搭建Promtail+Loki+Grafana轻量日志采集
-
k8s搭建搭建Promtail+Loki+Grafana轻量日志采集
-
数据存储配置优化
-
增加告警配置
-
方案优化
环境部署
虚拟机部署loki
## 这里的grafana我没有再安装，直接用下面容器部署的grafana
## 版本说明，配置文件链接
# ./loki-linux-amd64 -version
loki, version 2.5.0 (branch: HEAD, revision: 2d9d0ee23)
build user: root@4779f4b48f3a
build date: 2022-04-07T21:50:00Z
go version: go1.17.6
platform: linux/amd64
# ll
总用量 53280
-rwxr-xr-x 1 root root 54554624 4月 8 2022 loki-linux-amd64
-rw-r--r-- 1 root root 1323 4月 12 12:10 loki-local-config.yaml
# ./promtail-linux-amd64 -version
promtail, version 2.5.0 (branch: HEAD, revision: 2d9d0ee23)
build user: root@4779f4b48f3a
build date: 2022-04-07T21:50:00Z
go version: go1.17.6
platform: linux/amd64
# ll
总用量 78352
-rwxr-xr-x 1 root root 80224856 4月 8 2022 promtail-linux-amd64
-rw-r--r-- 1 root root 313 4月 12 12:11 promtail-local-config.yaml
## 二进制文件下载地址，配置文件地址
https://github.com/grafana/loki/releases/tag/v2.5.0
https://github.com/grafana/loki/blob/v2.6.1/cmd/loki/loki-local-config.yaml
https://github.com/grafana/loki/blob/v2.5.0/clients/cmd/promtail/promtail-local-config.yaml
## 启动测试
# pwd
/usr/local/loki
# ./loki-linux-amd64 -config.file=./loki-local-config.yaml
# pwd
/usr/local/promtail
# ./promtail-linux-amd64 -config.file=./promtail-local-config.yaml
## 若上面无日志报错，则可以执行后台启动
nohup ./loki-linux-amd64 -config.file=./loki-local-config.yaml &
nohup ./promtail-linux-amd64 -config.file=./promtail-local-config.yaml &
# ./loki-linux-amd64 -config.file=./loki-local-config.yaml
failed parsing config: ./loki-local-config.yaml: yaml: unmarshal errors:
line 22: field embedded_cache not found in type cache.Config
该错误的原因是下载了v2.5.0配置文件，有问题，我换成v2.6.1启动OK
https://github.com/grafana/loki/blob/v2.6.1/cmd/loki/loki-local-config.yaml
Docker方式部署loki
搭建Promtail + Loki + Grafana 轻量日志系统
如下针对宿主机服务器/var/log/messages做采集展示
# ip addr | grep ens | grep inet
inet 192.168.1.101/24 brd 192.168.1.255 scope global ens160
# cat /etc/redhat-release
CentOS Linux release 7.9.2009 (Core)
# pwd
/root/loki_in_docker
# ll
总用量 12
-rwxr-xr-x 1 root root 488 4月 12 10:05 install.sh ## 基于如下配置，简单封装脚本
-rw-r--r-- 1 root root 1298 4月 12 10:00 loki-config.yaml ## loki配置文件
-rw-r--r-- 1 root root 302 4月 12 10:00 promtail-config.yaml ## promtail配置文件
# cat install.sh
#!/bin/bash
## Pull loki image and start loki container
docker run --name loki -d -v $(pwd):/mnt/config -p 3100:3100 grafana/loki:2.7.4 -config.file=/mnt/config/loki-config.yaml
## Pull promtail image and start promtail container
docker run --name promtail -d -v $(pwd):/mnt/config -v /var/log:/var/log --link loki grafana/promtail:2.7.4 -config.file=/mnt/config/promtail-config.yaml
## Pull grafana image and start grafana container
docker run -d -p 3000:3000 grafana/grafana-oss
wget https://raw.githubusercontent.com/grafana/loki/v2.7.4/cmd/loki/loki-local-config.yaml -O loki-config.yaml
docker run --name loki -d -v $(pwd):/mnt/config -p 3100:3100 grafana/loki:2.7.4 -config.file=/mnt/config/loki-config.yaml
wget https://raw.githubusercontent.com/grafana/loki/v2.7.4/clients/cmd/promtail/promtail-docker-config.yaml -O promtail-config.yaml
docker run --name promtail -d -v $(pwd):/mnt/config -v /var/log:/var/log --link loki grafana/promtail:2.7.4 -config.file=/mnt/config/promtail-config.yaml
docker run -d -p 3000:3000 grafana/grafana-oss
# cat loki-config.yaml
auth_enabled: false
server:
http_listen_port: 3100
grpc_listen_port: 9096
common:
path_prefix: /tmp/loki
storage:
filesystem:
chunks_directory: /tmp/loki/chunks
rules_directory: /tmp/loki/rules
replication_factor: 1
ring:
instance_addr: 127.0.0.1
kvstore:
store: inmemory
query_range:
results_cache:
cache:
embedded_cache:
enabled: true
max_size_mb: 100
schema_config:
configs:
- from: 2020-10-24
store: boltdb-shipper
object_store: filesystem
schema: v11
index:
prefix: index_
period: 24h
ruler:
alertmanager_url: http://localhost:9093
# By default, Loki will send anonymous, but uniquely-identifiable usage and configuration
# analytics to Grafana Labs. These statistics are sent to https://stats.grafana.org/
#
# Statistics help us better understand how Loki is used, and they show us performance
# levels for most users. This helps us prioritize features and documentation.
# For more information on what's sent, look at
# https://github.com/grafana/loki/blob/main/pkg/usagestats/stats.go
# Refer to the buildReport method to see what goes into a report.
#
# If you would like to disable reporting, uncomment the following lines:
#analytics:
# reporting_enabled: false
# cat promtail-config.yaml
server:
http_listen_port: 9080
grpc_listen_port: 0
positions:
filename: /tmp/positions.yaml
clients:
- url: http://loki:3100/loki/api/v1/push
scrape_configs:
- job_name: system
static_configs:
- targets:
- localhost
labels:
job: varlogs
__path__: /var/log/messages
宿主机服务器/var/log/messages做采集展示：
1、新建loki数据源；
2、explore匹配；
3、run query；
测试环境k8s部署loki
注意：
k8s集群环境搭建这里不再赘述，可参考kubernetes环境搭建篇！
该操作涉及到的配置文件存放在loki_in_k8s目录！
这里只做了数据接入loki以及采集展示，做数据持久化不是该小结的重点，因此初学者使用此demo.yaml.bak_nostorage文件即可！
[root@master-1 loki_in_k8s]# cat /etc/hosts
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.1.100 master-1 server1
192.168.1.101 master-2 server2
192.168.1.102 master-3 server3
[root@master-1 loki_in_k8s]# ip addr | grep inet | grep ens
inet 192.168.1.100/24 brd 192.168.1.255 scope global ens160
inet 192.168.1.99/24 scope global secondary ens160 ## keepalived虚拟IP
[root@master-1 loki_in_k8s]# cat /etc/redhat-release
CentOS Linux release 7.9.2009 (Core)
[root@master-1 loki_in_k8s]# kubectl get node
NAME STATUS ROLES AGE VERSION
master-1 Ready control-plane,master 5d12h v1.22.1
master-2 Ready <none> 5d12h v1.22.1
master-3 Ready <none> 5d12h v1.22.1
[root@master-1 loki_in_k8s]# kubectl get pod -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
alertmanager-0 2/2 Running 2 (36h ago) 2d13h 10.244.2.9 master-3 <none> <none>
alertmanager-1 2/2 Running 2 (36h ago) 2d11h 10.244.1.22 master-2 <none> <none>
alertmanager-2 2/2 Running 2 (36h ago) 2d11h 10.244.1.27 master-2 <none> <none>
demo-567b47d5f6-tf22j 2/2 Running 0 11h 10.244.1.30 master-2 <none> <none>
grafana-0 1/1 Running 4 (36h ago) 5d11h 10.244.1.24 master-2 <none> <none>
loki-0 1/1 Running 3 (36h ago) 5d11h 10.244.2.8 master-3 <none> <none>
pvc-test-pod 1/1 Running 4 (3h30m ago) 2d11h 10.244.1.28 master-2 <none> <none>
test-deployment-7cff4666d5-8blw9 1/1 Running 1 (36h ago) 2d11h 10.244.2.7 master-3 <none> <none>
[root@master-1 loki_in_k8s]# kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
alertmanager NodePort 10.96.109.205 <none> 80:31575/TCP 2d13h
alertmanager-cluster ClusterIP None <none> 9094/TCP 2d13h
demo NodePort 10.103.51.114 <none> 8080:31110/TCP 5d11h
glusterfs ClusterIP 10.104.156.182 <none> 49152/TCP 11h
grafana NodePort 10.101.93.163 <none> 3000:30710/TCP 5d11h
kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 5d12h
loki ClusterIP None <none> 3100/TCP 5d11h
loki-nodeport NodePort 10.102.199.2 <none> 3100:32111/TCP 5d11h
test-deployment NodePort 10.99.215.167 <none> 80:32000/TCP 2d11h
# pwd
/root/k8s/loki/loki_in_k8s
[root@master-1 loki_in_k8s]# ll
总用量 44
-rw-r--r-- 1 root root 605 4月 5 16:24 alertmanager-configmap.yaml
-rw-r--r-- 1 root root 754 4月 5 16:24 alertmanager-service.yaml
-rw-r--r-- 1 root root 2826 4月 5 16:24 alertmanager-sts.yaml
-rw-r--r-- 1 root root 1144 4月 11 23:04 demo.yaml ## 存储到基于nfs的pv数据
-rw-r--r-- 1 root root 1078 4月 11 22:57 demo.yaml.bak_nostorage ## 无持久化存储配置文件
-rw-r--r-- 1 root root 1472 4月 5 16:24 grafana-sts.yaml
-rw-r--r-- 1 root root 52 4月 5 16:24 labels.sh
-rw-r--r-- 1 root root 2473 4月 5 16:24 loki-configmap.yaml
-rw-r--r-- 1 root root 209 4月 5 16:24 loki-nodeport-svc.yaml
-rw-r--r-- 1 root root 1584 4月 5 16:24 loki-rules.yaml
-rw-r--r-- 1 root root 2727 4月 5 16:24 loki-server-sts.yaml
# kubectl get node --show-labels
NAME STATUS ROLES AGE VERSION LABELS
master-1 Ready control-plane,master 5d12h v1.22.1 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=master-1,kubernetes.io/os=linux,loki=node,n
ode-role.kubernetes.io/control-plane=,node-role.kubernetes.io/master=,node.kubernetes.io/exclude-from-external-load-balancers=master-2 Ready <none> 5d12h v1.22.1 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=master-2,kubernetes.io/os=linux,loki=node
master-3 Ready <none> 5d12h v1.22.1 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=master-3,kubernetes.io/os=linux,loki=node
这里仅作数据源-Loki配置及其数据展示，告警邮件发送等后续补充！
# 打标签
kubectl label node master-1 loki=node --overwrite
kubectl label node master-2 loki=node --overwrite
kubectl label node master-3 loki=node --overwrite
# 启动loki
kubectl apply -f loki-server-sts.yaml -f loki-configmap.yaml -f loki-nodeport-svc.yaml -f loki-rules.yaml
# 启动grafana
kubectl apply -f grafana-sts.yaml
# 启动sidecar应用程序，初学者请将demo.yaml.bak_nostorage命名为demo.yaml执行即可
kubectl apply -f demo.yaml
至此，一个sidecar应用的日志采集就展示完成了！
数据持久化配置
方案说明
集群基于glusterfs和nfs共享的pv及pvc操作，建议直接使用glusterfs
这里侧重对pv和pvc的分配，以及应用业务日志的挂载配置！
两个环境的具体搭建及其使用参考数据持久化相关参考文档！
## 文件存放在loki_in_k8s中的pv目录
# pwd
/root/k8s/pv
[root@master-1 pv]# ll
总用量 16
-rw-r--r-- 1 root root 187 4月 11 23:08 demo-pvc.yml
-rw-r--r-- 1 root root 219 4月 11 22:53 endpoints.yaml
-rw-r--r-- 1 root root 271 4月 11 22:55 pv.yaml
-rw-r--r-- 1 root root 199 4月 11 22:54 service.yam
# kubectl get pv
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
demo-pv 1Gi RWX Retain Available 15h
# kubectl get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
demo-pvc Bound pvc-4567db81-67ee-4bee-8417-8d014c42f8ac 200m RWX managed-nfs-storage 14h
## 如下可以看到，将pod中sidecar日志都挂载到了当前的pv存储中
# ll /data/nfs-volume/prometheus/default-demo-pvc-pvc-4567db81-67ee-4bee-8417-8d014c42f8ac/
总用量 28
-rw-r--r-- 1 root root 6135 4月 11 23:09 catalina.2023-04-11.log
-rw-r--r-- 1 root root 7007 4月 12 10:44 catalina.2023-04-12.log
-rw-r--r-- 1 root root 0 4月 11 23:08 host-manager.2023-04-11.log
-rw-r--r-- 1 root root 0 4月 12 10:44 host-manager.2023-04-12.log
-rw-r--r-- 1 root root 459 4月 11 23:09 localhost.2023-04-11.log
-rw-r--r-- 1 root root 735 4月 12 10:44 localhost.2023-04-12.log
-rw-r--r-- 1 root root 0 4月 11 23:08 localhost_access_log.2023-04-11.txt
-rw-r--r-- 1 root root 0 4月 12 10:44 localhost_access_log.2023-04-12.txt
-rw------- 1 root root 171 4月 12 14:06 loki_positions.yaml
-rw-r--r-- 1 root root 0 4月 11 23:08 manager.2023-04-11.log
-rw-r--r-- 1 root root 0 4月 12 10:44 manager.2023-04-12.log
日志监控告警……待补充
官网参考链接
容器部署loki:
https://grafana.com/docs/loki/latest/installation/docker/
容器部署grafana:
https://grafana.com/docs/grafana/latest/setup-grafana/installation/docker/
GitHub参考：
https://github.com/grafana/loki/releases
参考引用
第二种-sidecar模式日志收集
https://www.cnblogs.com/huningfei/p/17122929.html
多集群thanos sidecar+MinIO监控告警实践
https://blog.51cto.com/starsliao/5763149
promtail收集标准输出日志
https://www.isolves.com/it/wlyx/fwq/2021-07-30/42209.html
Linux搭建Promtail + Loki + Grafana 轻量日志监控系统
https://blog.csdn.net/weixin_52270081/article/details/124801308?utm_medium=distribute.pc_relevant.none-task-blog-2~default~baidujs_baidulandingword~default-0-124801308-blog-111883107.235^v27^pc_relevant_recovery_v2&spm=1001.2101.3001.4242.1&utm_relevant_index=2
k8s安装loki-promtail（非helm安装）
https://www.cnblogs.com/danny-djy/p/15523349.html
Loki & Promtail 详解
https://cloud.tencent.com/developer/article/1889851?from=15425&areaSource=102001.2&traceId=h1E1q5tpdQD1DSEHFau4A
部署grafana:
https://blog.csdn.net/u011943534/article/details/102906030
部署loki:
https://blog.csdn.net/u011943534/article/details/120871258?utm_medium=distribute.pc_relevant.none-task-blog-2~default~baidujs_baidulandingword~default-4-120871258-blog-112691053.235^v27^pc_relevant_recovery_v2&spm=1001.2101.3001.4242.3&utm_relevant_index=7
loki数据存储相关：
https://blog.51cto.com/u_15820126/5738632
京东到家Loki日志系统实践
https://mp.weixin.qq.com/s/-IvWfhXbm3WXFIBtbrhX-A
使用 loki 作为 k8s 应用日志收集器(下篇)
https://blog.cong.moe/post/2020-08-08-use_loki_as_k8s_log_collector_2/
https://gist.githubusercontent.com/zcong1993/2ed197b97a3286dd958e4c8cfd81e5ea/raw/2c8504b5f1e3aecf9c874c637be1d5c774c936a3/loki2.yaml
含AWS的部分配置：
https://blog.csdn.net/u014331650/article/details/127889779
https://grafana.com/docs/loki/latest/configuration/examples/
官方loki生产环境集群kubernetes方案
https://github.com/LinkMaq/loki-cluster-deploy/tree/master/production/loki-system
存储示例：
https://blog.csdn.net/Java_Mr_Jin/article/details/125643455?ops_request_misc=%257B%2522request%255Fid%2522%253A%2522168076003416800197087695%2522%252C%2522scm%2522%253A%252220140713.130102334..%2522%257D&request_id=168076003416800197087695&biz_id=0&utm_medium=distribute.pc_search_result.none-task-blog-2~all~top_positive~default-1-125643455-null-null.142^v81^pc_new_rank,201^v4^add_ask,239^v2^insert_chatgpt&utm_term=minio&spm=1018.2226.3001.4187
历史事件
2023-04-12｜容器、虚拟机轻量部署采集
2023-04-10 ｜k8s轻量部署采集
2023-04-08 ｜创建文档