【prometheus-operator】k8s监控redis

1、准备exporter

https://github.com/oliver006/redis_exporter

oliver006-redis_exporter-amd64.tar

# 安装镜像

docker load -i oliver006-redis_exporter-amd64.tar

# 上传镜像
docker tag oliver006/redis_exporter ip/monitor/redis_exporter:latest
docker push ip/monitor/redis_exporter:latest

2、部署exporter

kind: Deployment
apiVersion: apps/v1
metadata:
  name: redis-exporter
  namespace: devops
  labels:
    app: redis-exporter
  annotations:
    deployment.kubernetes.io/revision: '1'
    kubesphere.io/creator: admin
    kubesphere.io/description: 监控使用
spec:
  replicas: 1
  selector:
    matchLabels:
      app: redis-exporter
  template:
    metadata:
      labels:
        app: redis-exporter
      annotations:
        kubesphere.io/containerSecrets: '{"container-zsbagq":"harbor"}'
    spec:
      containers:
        - name: container-zsbagq
          image: 'ip:8888/monitor/redis_exporter:latest'
          args:
            - '-redis.addr'
            - 'redis-hip-redis-cluster-headless.hip-mid:6379'
            - '-redis.password'
            - '111111'
          ports:
            - name: tcp-9121
              containerPort: 9121
              protocol: TCP
          imagePullPolicy: IfNotPresent
      restartPolicy: Always
      imagePullSecrets:
        - name: harbor

3、创建serviceMonitor

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  labels:
    app: redis-exporter
  name: redis-exporter
  namespace: devops
spec:
  endpoints:
  - interval: 30s
    port: tcp-9121
    params:
        target:
          - redis-hip-redis-cluster-headless.hip-mid:6379
    relabelings:
      - sourceLabels: [__param_target]
        targetLabel: instance
  selector:
    matchLabels:
      app: redis-exporter
  namespaceSelector:
    matchNames:
    - devops

创建成功，访问prometheus targets界面，显示如图

4、告警规则

获取规则：

kubectl get prometheusrules prometheus-k8s-rules -n {namespace} -o yaml > prometheus-rule.yaml

  - name: redis_alters
    rules:
    - alert: redis_exporter_status
      annotations:
        description: job-{{$labels.job}}, instance-{{ $labels.instance }} 监控exporter下线，请检查！
        summary: 监控exporter下线预警
      expr: up{namespace="monitor",instance=~".*:9122"} == 0
      for: 3m
      labels:
        severity: emergency
    - alert: redis_status
      annotations:
        description: job-{{$labels.job}}, instance-{{ $labels.instance }} 节点下线，请检查！
        summary: 节点下线预警
      expr: redis_up{namespace="monitor"} == 0
      for: 3m
      labels:
        severity: emergency
    - alert: redis_memory_usage
      annotations:
        description: job-{{$labels.job}}, instance-{{ $labels.instance }} 内存超过90%，请检查！
        summary: 内存预警
      expr: redis_memory_used_bytes{namespace="monitor"} / redis_memory_max_bytes{namespace="monitor"}
        * 100 > 90
      for: 15s
      labels:
        severity: emergency
    - alert: redis_rejected_connection
      annotations:
        description: job-{{$labels.job}}, instance-{{ $labels.instance }} 一些连接被拒绝，请检查！
        summary: 连接被拒绝
      expr: increase(redis_rejected_connections_total{namespace="monitor"}[1m]) >
        0
      for: 1m
      labels:
        severity: emergency
    - alert: redis_max_client
      annotations:
        description: job-{{$labels.job}}, instance-{{ $labels.instance }} 连接总数达到总量的90%，请检查！
        summary: 连接数过多
      expr: redis_connected_clients{namespace="monitor"} > redis_config_maxclients{namespace="monitor"}
        * 0.90
      for: 15s
      labels:
        severity: emergency