Static File Service Discovery

Static File Service Discovery

How Prometheus discovery monitoring targets with static files

When prometheus_sd_method == ‘static’, Prometheus will use static conf files to manage monitoring targets.

Prometheus FHS

Monitoring Targets are placed @ /etc/prometheus/targets/, And divided by modules into several sub directories: infra, nodes, pgsql, redis, ….

1. #------------------------------------------------------------------------------
2. # Config FHS
3. #------------------------------------------------------------------------------
4. # /etc/prometheus/
5. #  ^-----prometheus.yml              # prometheus main config file
6. #  ^-----alertmanager.yml            # alertmanger main config file
7. #  ^-----@bin                        # util scripts: check,reload,status,new
8. #  ^-----@rules                      # record & alerting rules definition
9. #            ^-----@infra            # infrastructure rules & alert
10. #            ^-----@nodes            # nodes rules & alert
11. #            ^-----@pgsql            # pgsql rules & alert
12. #            ^-----@redis            # redis rules & alert
13. #            ^-----@..........       # etc...
14. #  ^-----@targets                    # file based service discovery targets definition
15. #            ^-----@infra            # infra static targets definition
16. #            ^-----@nodes            # nodes static targets definition
17. #            ^-----@pgsql            # pgsql static targets definition
18. #            ^-----@redis            # redis static targets definition
19. #            ^-----@.....            # other targets
20. #------------------------------------------------------------------------------

Targets

1. /etc/prometheus/targets
2. ├── nodes
3. │   ├── 10.10.10.13.yml
4. │   ├── 10.10.10.10.yml
5. │   ├── 10.10.10.11.yml
6. │   ├── 10.10.10.12.yml
7. ├── pgsql
8. │   ├── pg-meta-1.yml
9. │   ├── pg-test-2.yml
10. │   ├── pg-test-3.yml
11. │   ├── pg-test-1.yml
12. ├── infra
13. │   ├── 10.10.10.10.yml
14. ├── redis
15. │   ├── redis-common-1.yml
16. │   ├── redis-test-2.yml
17. │   ├── redis-meta-1.yml
18. │   ├── redis-test-1.yml

INFRA Targets

INFRA监控对象以元节点的IP地址作为文件名，内容如下：

1. ---
2. #------------------------------------------------------------------------------
3. # Prometheus (9090)
4. #------------------------------------------------------------------------------
5. - labels: { ip: 10.10.10.10, type: prometheus }
6. targets:
7. - 10.10.10.10:9090

9. #------------------------------------------------------------------------------
10. # AlertManager (9093)
11. #------------------------------------------------------------------------------
12. - labels: { ip: 10.10.10.10, type: alertmanager }
13. targets:
14. - 10.10.10.10:9093

16. #------------------------------------------------------------------------------
17. # Grafana (3000)
18. #------------------------------------------------------------------------------
19. - labels: { ip: 10.10.10.10, type: grafana }
20. targets:
21. - 10.10.10.10:3000

23. #------------------------------------------------------------------------------
24. # Loki (3100)
25. #------------------------------------------------------------------------------
26. - labels: { ip: 10.10.10.10, type: loki }
27. targets:
28. - 10.10.10.10:3100

30. #------------------------------------------------------------------------------
31. # Nginx (Exporter @ 9113)
32. #------------------------------------------------------------------------------
33. - labels: { ip: 10.10.10.10, type: nginx }
34. targets:
35. - 10.10.10.10:9113

DCS Targets

DCS监控对象直接定于 /etc/prometheus/prometheus.yml 主配置文件中：

1. - job_name: consul
2. metrics_path: /v1/agent/metrics
3. params:
4. format: ['prometheus']
5. static_configs: [ { targets: [ 127.0.0.1:8500 ] , labels: { ip: 10.10.10.10, type: consul , job: infra } } ]

7. - job_name: etcd
8. metrics_path: /metrics
9. static_configs:
10. - { targets: [ 10.10.10.11:2379 ] , labels: { ip: 10.10.10.11, ins: meta-2, type: etcd , job: infra } }
11. - { targets: [ 10.10.10.12:2379 ] , labels: { ip: 10.10.10.12, ins: meta-3, type: etcd , job: infra } }
12. - { targets: [ 10.10.10.10:2379 ] , labels: { ip: 10.10.10.10, ins: meta-1, type: etcd , job: infra } }

NODES Targets

NODES监控对象以IP地址作为监控对象文件名，内容如下所示：

1. $ cat nodes/10.10.10.10.yml

1. # 10.10.10.10
2. - labels: { ip: 10.10.10.10 , ins: pg-meta-1 , cls: pg-meta }
3. targets:
4. - 10.10.10.10:9100   # node_exporter
5. - 10.10.10.10:9323   # docker
6. - 10.10.10.10:9080   # promtail

PGSQL Targets

PGSQL监控对象以实例名作为监控对象文件名，内容如下所示：

1. $ cat pgsql/pg-meta-1.yml

1. # pg-meta-1 [primary] @ 10.10.10.10
2. - labels: { cls: pg-meta, ins: pg-meta-1, ip: 10.10.10.10 }
3. targets:
4. - 10.10.10.10:9630  # postgres
5. - 10.10.10.10:9631  # pgbouncer
6. - 10.10.10.10:8008  # patroni
7. - 10.10.10.10:9101  # haproxy

REDIS Targets

REDIS监控对象以Redis节点名作为监控对象文件名，内容如下所示：

1. $ cat pgsql/redis-common-1.yml

1. # redis-common-1 @ 10.10.10.13

3. - labels: { cls: redis-common, ins: redis-common-1-6501, instance: 10.10.10.13:6501 }
4. targets: [ redis://10.10.10.13:6501 ]

6. - labels: { cls: redis-common, ins: redis-common-1-6502, instance: 10.10.10.13:6502 }
7. targets: [ redis://10.10.10.13:6502 ]

9. - labels: { cls: redis-common, ins: redis-common-1-6503, instance: 10.10.10.13:6503 }
10. targets: [ redis://10.10.10.13:6503 ]

Last modified 2022-06-04: fill en docs (5a858d3)

StaticFS SD