基于 RocketMQ Prometheus Exporter 打造定制化 DevOps 平台

tar -xzf prometheus-2.7.0-rc.1.linux-amd64.tar.gzcd prometheus-2.7.0-rc.1.linux-amd64/./prometheus --config.file=prometheus.yml --web.listen-address=:5555

Prometheus 默认监听端口号为 9090，为了不与系统上的其它进程监听端口冲突，我们在启动参数里面重新设置了监听端口号为 5555。然后通过浏览器访问 http://<服务器 IP 地址>:5555,就可以验证 Prometheus 是否已成功安装，显示界面如下：

由于 RocketMQ-Exporter 进程已启动，这个时候可以通过 Prometheus 来抓取 RocketMQ-Exporter 的数据，这个时候只需要更改 Prometheus 启动的配置文件即可。

整体配置文件如下：

my global config

global:

scrape_interval: 15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.

evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.

scrape_timeout is set to the global default (10s).

Load rules once and periodically evaluate them according to the global ‘evaluation_interval’.

rule_files:

- “first_rules.yml”

- “second_rules.yml”

scrape_configs:

job_name: ‘prometheus’

static_configs:

targets: [‘localhost:5555’]
job_name: ‘exporter’

static_configs:

targets: [‘localhost:5557’]

更改配置文件后，重启服务即可。重启后就可以在 Prometheus 界面查询 RocketMQ-Exporter 上报的指标，例如查询 rocketmq_broker_tps 指标，其结果如下：

5. 告警规则添加

在 Prometheus 可以展示 RocketMQ-Exporter 的指标后，就可以在 Prometheus 中配置 RocketMQ 的告警指标了。在 Prometheus 的配置文件中添加如下的告警配置项，*.rules 表示可以匹配多个后缀为 rules 的文件。

rule_files:

- “first_rules.yml”

- “second_rules.yml”

/home/prometheus/prometheus-2.7.0-rc.1.linux-amd64/rules/*.rules

当前设置的告警配置文件为 warn.rules，其文件具体内容如下所示。其中的阈值只起一个示例的作用，具体的阈值还需用户根据实际使用情况来自行设定。

Sample prometheus rules/alerts for rocketmq.

Galera Alerts

groups:

name: GaleraAlerts

rules:

alert: RocketMQClusterProduceHigh

expr: sum(rocketmq_producer_tps) by (cluster) >= 10

for: 3m

labels:

severity: warning

annotations:

description: ‘{{$labels.cluster}} Sending tps too high.’

summary: cluster send tps too high

alert: RocketMQClusterProduceLow

expr: sum(rocketmq_producer_tps) by (cluster) < 1

for: 3m

labels:

severity: warning

annotations:

description: ‘{{$labels.cluster}} Sending tps too low.’

summary: cluster send tps too low

alert: RocketMQClusterConsumeHigh

expr: sum(rocketmq_consumer_tps) by (cluster) >= 10

for: 3m

labels:

severity: warning

annotations:

description: ‘{{$labels.cluster}} consuming tps too high.’

summary: cluster consume tps too high

alert: RocketMQClusterConsumeLow

expr: sum(rocketmq_consumer_tps) by (cluster) < 1

for: 3m

labels:

severity: warning

annotations:

description: ‘{{$labels.cluster}} consuming tps too low.’

summary: cluster consume tps too low

alert: ConsumerFallingBehind

expr: (sum(rocketmq_producer_offset) by (topic) - on(topic) group_right sum(rocketmq_consumer_offset) by (group,topic)) - ignoring(group) group_left sum (avg_over_time(rocketmq_producer_tps[5m])) by (topic)560 > 0

for: 3m

labels:

severity: warning

annotations:

description: 'consumer {{ $KaTeX parse error: Expected 'EOF', got '}' at position 13: labels.group}̲} on {{$ labels.topic}} lag behind

and is falling behind (behind value {{$value}}).’

summary: consumer lag behind

alert: GroupGetLatencyByStoretime

expr: rocketmq_group_get_latency_by_storetime > 1000

for: 3m

labels:

severity: warning

annotations:

description: 'consumer {{ $KaTeX parse error: Expected 'EOF', got '}' at position 13: labels.group}̲} on {{$ labels.broker}}, {{$labels.topic}} consume time lag behind message store time

and (behind value is {{$value}}).’

summary: message consumes time lag behind message store time too much

最终，可以在 Prometheus 的看一下告警展示效果，红色表示当前处于告警状态的项，绿色表示正常状态。