告警

告警

概览

IoTDB 告警功能预计支持两种模式：

写入触发：用户写入原始数据到原始时间序列，每插入一条数据都会触发 trigger 的判断逻辑，若满足告警要求则发送告警到下游数据接收器，数据接收器再转发告警到外部终端。这种模式：
适合需要即时监控每一条数据的场景。
由于触发器中的运算会影响数据写入性能，适合对原始数据写入性能不敏感的场景。
持续查询：用户写入原始数据到原始时间序列， ContinousQuery 定时查询原始时间序列，将查询结果写入新的时间序列，每一次写入触发 trigger 的判断逻辑，若满足告警要求则发送告警到下游数据接收器，数据接收器再转发告警到外部终端。这种模式：
适合需要定时查询数据在某一段时间内的情况的场景。
适合需要将原始数据降采样并持久化的场景。
由于定时查询几乎不影响原始时间序列的写入，适合对原始数据写入性能敏感的场景。

随着 trigger 模块和 sink 模块的引入，目前用户使用这两个模块，配合 AlertManager 可以实现写入触发模式的告警。

部署 AlertManager

安装与运行

二进制文件

预编译好的二进制文件可在这里 (opens new window) 下载。

运行方法：

1. ./alertmanager --config.file=<your_file>

Docker 镜像

可在 Quay.io (opens new window) 或 Docker Hub (opens new window) 获得。

运行方法：

1. docker run --name alertmanager -d -p 127.0.0.1:9093:9093 quay.io/prometheus/alertmanager

配置

如下是一个示例，可以覆盖到大部分配置规则，详细的配置规则参见这里 (opens new window)。

示例：

1. # alertmanager.yml

3. global:
4. # The smarthost and SMTP sender used for mail notifications.
5. smtp_smarthost: 'localhost:25'
6. smtp_from: 'alertmanager@example.org'

8. # The root route on which each incoming alert enters.
9. route:
10. # The root route must not have any matchers as it is the entry point for
11. # all alerts. It needs to have a receiver configured so alerts that do not
12. # match any of the sub-routes are sent to someone.
13. receiver: 'team-X-mails'

15. # The labels by which incoming alerts are grouped together. For example,
16. # multiple alerts coming in for cluster=A and alertname=LatencyHigh would
17. # be batched into a single group.
18. #
19. # To aggregate by all possible labels use '...' as the sole label name.
20. # This effectively disables aggregation entirely, passing through all
21. # alerts as-is. This is unlikely to be what you want, unless you have
22. # a very low alert volume or your upstream notification system performs
23. # its own grouping. Example: group_by: [...]
24. group_by: ['alertname', 'cluster']

26. # When a new group of alerts is created by an incoming alert, wait at
27. # least 'group_wait' to send the initial notification.
28. # This way ensures that you get multiple alerts for the same group that start
29. # firing shortly after another are batched together on the first
30. # notification.
31. group_wait: 30s

33. # When the first notification was sent, wait 'group_interval' to send a batch
34. # of new alerts that started firing for that group.
35. group_interval: 5m

37. # If an alert has successfully been sent, wait 'repeat_interval' to
38. # resend them.
39. repeat_interval: 3h

41. # All the above attributes are inherited by all child routes and can
42. # overwritten on each.

44. # The child route trees.
45. routes:
46. # This routes performs a regular expression match on alert labels to
47. # catch alerts that are related to a list of services.
48. - match_re:
49. service: ^(foo1|foo2|baz)$
50. receiver: team-X-mails

52. # The service has a sub-route for critical alerts, any alerts
53. # that do not match, i.e. severity != critical, fall-back to the
54. # parent node and are sent to 'team-X-mails'
55. routes:
56. - match:
57. severity: critical
58. receiver: team-X-pager

60. - match:
61. service: files
62. receiver: team-Y-mails

64. routes:
65. - match:
66. severity: critical
67. receiver: team-Y-pager

69. # This route handles all alerts coming from a database service. If there's
70. # no team to handle it, it defaults to the DB team.
71. - match:
72. service: database

74. receiver: team-DB-pager
75. # Also group alerts by affected database.
76. group_by: [alertname, cluster, database]

78. routes:
79. - match:
80. owner: team-X
81. receiver: team-X-pager

83. - match:
84. owner: team-Y
85. receiver: team-Y-pager

87. # Inhibition rules allow to mute a set of alerts given that another alert is
88. # firing.
89. # We use this to mute any warning-level notifications if the same alert is
90. # already critical.
91. inhibit_rules:
92. - source_match:
93. severity: 'critical'
94. target_match:
95. severity: 'warning'
96. # Apply inhibition if the alertname is the same.
97. # CAUTION:
98. #   If all label names listed in `equal` are missing
99. #   from both the source and target alerts,
100. #   the inhibition rule will apply!
101. equal: ['alertname']

103. receivers:
104. - name: 'team-X-mails'
105. email_configs:
106. - to: 'team-X+alerts@example.org, team-Y+alerts@example.org'

108. - name: 'team-X-pager'
109. email_configs:
110. - to: 'team-X+alerts-critical@example.org'
111. pagerduty_configs:
112. - routing_key: <team-X-key>

114. - name: 'team-Y-mails'
115. email_configs:
116. - to: 'team-Y+alerts@example.org'

118. - name: 'team-Y-pager'
119. pagerduty_configs:
120. - routing_key: <team-Y-key>

122. - name: 'team-DB-pager'
123. pagerduty_configs:
124. - routing_key: <team-DB-key>

在后面的示例中，我们采用的配置如下：

1. # alertmanager.yml

3. global:
4. smtp_smarthost: ''
5. smtp_from: ''
6. smtp_auth_username: ''
7. smtp_auth_password: ''
8. smtp_require_tls: false

10. route:
11. group_by: ['alertname']
12. group_wait: 1m
13. group_interval: 10m
14. repeat_interval: 10h
15. receiver: 'email'

17. receivers:
18. - name: 'email'
19. email_configs:
20. - to: ''

22. inhibit_rules:
23. - source_match:
24. severity: 'critical'
25. target_match:
26. severity: 'warning'
27. equal: ['alertname']

API

AlertManager API 分为 v1 和 v2 两个版本，当前 AlertManager API 版本为 v2 （配置参见 api/v2/openapi.yaml (opens new window))。

默认配置的前缀为 /api/v1 或 /api/v2，发送告警的 endpoint 为 /api/v1/alerts 或 /api/v2/alerts。如果用户指定了 --web.route-prefix，例如 --web.route-prefix=/alertmanager/，那么前缀将会变为 /alertmanager/api/v1 或 /alertmanager/api/v2，发送告警的 endpoint 变为 /alertmanager/api/v1/alerts 或 /alertmanager/api/v2/alerts。

创建 trigger

编写 trigger 类

用户通过自行创建 Java 类、编写钩子中的逻辑来定义一个触发器。具体配置流程以及 Sink 模块提供的 AlertManagerSink 相关工具类的使用方法参见 Triggers。

下面的示例创建了 org.apache.iotdb.trigger.AlertingExample 类，其 alertManagerHandler 成员变量可发送告警至地址为 http://127.0.0.1:9093/ 的 AlertManager 实例。

当 value > 100.0 时，发送 severity 为 critical 的告警；当 50.0 < value <= 100.0 时，发送 severity 为 warning 的告警。

1. package org.apache.iotdb.trigger;

3. /*
4. 此处省略包的引入
5. */

7. public class AlertingExample implements Trigger {

9. private final AlertManagerHandler alertManagerHandler = new AlertManagerHandler();

11. private final AlertManagerConfiguration alertManagerConfiguration =
12. new AlertManagerConfiguration("http://127.0.0.1:9093/api/v2/alerts");

14. private String alertname;

16. private final HashMap<String, String> labels = new HashMap<>();

18. private final HashMap<String, String> annotations = new HashMap<>();

20. @Override
21. public void onCreate(TriggerAttributes attributes) throws Exception {
22. alertManagerHandler.open(alertManagerConfiguration);

24. alertname = "alert_test";

26. labels.put("series", "root.ln.wf01.wt01.temperature");
27. labels.put("value", "");
28. labels.put("severity", "");

30. annotations.put("summary", "high temperature");
31. annotations.put("description", "{{.alertname}}: {{.series}} is {{.value}}");
32. }

34. @Override
35. public void onDrop() throws IOException {
36. alertManagerHandler.close();
37. }

39. @Override
40. public void onStart() {
41. alertManagerHandler.open(alertManagerConfiguration);
42. }

44. @Override
45. public void onStop() throws Exception {
46. alertManagerHandler.close();
47. }

49. @Override
50. public Double fire(long timestamp, Double value) throws Exception {
51. if (value > 100.0) {
52. labels.put("value", String.valueOf(value));
53. labels.put("severity", "critical");
54. AlertManagerEvent alertManagerEvent = new AlertManagerEvent(alertname, labels, annotations);
55. alertManagerHandler.onEvent(alertManagerEvent);
56. } else if (value > 50.0) {
57. labels.put("value", String.valueOf(value));
58. labels.put("severity", "warning");
59. AlertManagerEvent alertManagerEvent = new AlertManagerEvent(alertname, labels, annotations);
60. alertManagerHandler.onEvent(alertManagerEvent);
61. }

63. return value;
64. }

66. @Override
67. public double[] fire(long[] timestamps, double[] values) throws Exception {
68. for (double value : values) {
69. if (value > 100.0) {
70. labels.put("value", String.valueOf(value));
71. labels.put("severity", "critical");
72. AlertManagerEvent alertManagerEvent = new AlertManagerEvent(alertname, labels, annotations);
73. alertManagerHandler.onEvent(alertManagerEvent);
74. } else if (value > 50.0) {
75. labels.put("value", String.valueOf(value));
76. labels.put("severity", "warning");
77. AlertManagerEvent alertManagerEvent = new AlertManagerEvent(alertname, labels, annotations);
78. alertManagerHandler.onEvent(alertManagerEvent);
79. }
80. }
81. return values;
82. }
83. }

创建 trigger

如下的 sql 语句在 root.ln.wf01.wt01.temperature 时间序列上注册了名为 root-ln-wf01-wt01-alert、运行逻辑由 org.apache.iotdb.trigger.AlertingExample 类定义的触发器。

1. CREATE TRIGGER `root-ln-wf01-wt01-alert`
2. AFTER INSERT
3. ON root.ln.wf01.wt01.temperature
4. AS "org.apache.iotdb.trigger.AlertingExample"

写入数据

当我们完成 AlertManager 的部署和启动、Trigger 的创建，可以通过向时间序列写入数据来测试告警功能。

1. INSERT INTO root.ln.wf01.wt01(timestamp, temperature) VALUES (1, 0);
2. INSERT INTO root.ln.wf01.wt01(timestamp, temperature) VALUES (2, 30);
3. INSERT INTO root.ln.wf01.wt01(timestamp, temperature) VALUES (3, 60);
4. INSERT INTO root.ln.wf01.wt01(timestamp, temperature) VALUES (4, 90);
5. INSERT INTO root.ln.wf01.wt01(timestamp, temperature) VALUES (5, 120);

执行完上述写入语句后，可以收到告警邮件。由于我们的 AlertManager 配置中设定 severity 为 critical 的告警会抑制 severity 为 warning 的告警，我们收到的告警邮件中只包含写入 (5, 120) 后触发的告警。

告警机制

告警

概览

部署 AlertManager

安装与运行

二进制文件

Docker 镜像

配置

API

创建 trigger

编写 trigger 类

创建 trigger

写入数据