Prometheus监控mysql nginx tomcat 黑盒监控

部署consul_exporter
https://github.com/prometheus/consul_exporter/releases/download/v0.9.0/consul_exporter-0.9.0.linux-amd64.tar.gz

oot@ubuntu20:~# cat consul_export.json 
root@ubuntu20:~# cat consul_export.json 
{
  "services": [{
      "id": "consul_exporter",
      "name": "192.168.1.50_cousul_expo",
      "address": "192.168.1.50",
      "port": 9107,
      "tags": ["consul_exporter"],
      "checks": [{
        "http": "http://192.168.1.50:9107/metrics",
        "interval": "5s"
      }]
    }
  ]

consul services register consul_export.json

Prometheus添加配置

 - job_name: "consul_exporter"
    metrics_path: '/metrics'
    scheme: "http"
    consul_sd_configs:
        - server: "192.168.1.50:8500"
          tags:
          - "consul_exporter"
          refresh_interval: 1m

重新加载Prometheus
root@consumer:/apps/prometheus# curl -XPOST localhost:9090/-/reload

在这里插入图片描述
export 获取数据转换为Prometheus数据

pushgetway

查询持久化

grafanf绘图

altermanager 发送告警系统
选择

应用程序不兼容Prometheus

mysqld export

nginx-export

https://github.com/nginxinc/nginx-prometheus-exporter

添加nginx配置，添加这个查看指标

       location /stub_status {
           # Turn on nginx stats
                stub_status on;
                access_log   off;
    # Only allow access from certain IP addresses
                allow all;
    #deny all;
                }

运行采集指标export
root@ubuntu20:/apps# ./nginx-prometheus-exporter -nginx.scrape-uri=http://192.168.1.50/stub_status

root@ubuntu20:/etc/nginx# netstat -antp |grep 9113
tcp6 0 0 :::9113 ::😗 LISTEN 168719/./nginx-prom


root@ubuntu20:~# cat nginx_export.json 
{
  "services": [{
      "id": "nginx_exporter",
      "name": "192.168.1.50_nginx",
      "address": "192.168.1.50",
      "port": 9113,
      "tags": ["nginx_exporter"],
      "checks": [{
        "http": "http://192.168.1.50:9113/metrics",
        "interval": "5s"
      }]
    }
  ]
}

consul services register  nginx_export.json

配置Prometheus添加

  - job_name: "nginx_exporter"
    metrics_path: '/metrics'
    scheme: "http"
    consul_sd_configs:
        - server: "192.168.1.50:8500"
          tags:
          - "nginx_exporter"
          refresh_interval: 1m

curl -XPOST localhost:9090/-/reload

在这里插入图片描述

监控tomcat

tomcat
mcat自身并不能提供监控指标数据，需要借助第三方exporter实现：https://github.com/nlighten/tomcat_exporter

JVM的export或者tomcat export

构建tomcat镜像，并暴露8082端口

cat docker-compose.yml 
version: '3.6'

volumes:
    tomcat_webapps: {}

networks:
  monitoring:
    driver: bridge
    ipam:
      config:
        - subnet: 172.31.130.0/24

services:
  tomcat:
    #image: tomcat:jdk11
    build:
      context: tomcat
      dockerfile: Dockerfile 
    hostname: tomcat.magedu.com
    expose:
      - 8080
    ports:
      - 8082:8080
    volumes:
      - tomcat_webapps:/usr/local/tomcat/webapps
      - ./tomcat/tomcat-users.xml:/usr/local/tomcat/conf/tomcat-users.xml
    networks:
      - monitoring
    environment:
      TZ: Asia/Shanghai

root@jenkins:~/learning-prometheus/08-prometheus-components-compose/tomcat-and-metrics# docker-compose up

注册到consul

root@ubuntu20:~# cat tomcat2.json 
{
  "services": [{
      "id": "tomcat",
      "name": "192.168.1.51_tomcat",
      "address": "192.168.1.51",
      "port": 8082,
      "tags": ["tomcat_exporter"],
      "checks": [{
        "http": "http://192.168.1.51:8082/metrics",
        "interval": "5s"
      }]
    }
  ]
}


consul services register   tomcat2.json

添加Prometheus配置并重新加载

  - job_name: "tomcat_exporter"
    metrics_path: '/metrics'
    scheme: "http"
    consul_sd_configs:
        - server: "192.168.1.50:8500"
          tags:
          - "tomcat_exporter"
          refresh_interval: 1m

curl -XPOST localhost:9090/-/reload

在这里插入图片描述

黑盒监控

https://github.com/prometheus/blackbox_exporter/releases/download/v0.24.0/blackbox_exporter-0.24.0.linux-amd64.tar.gz

通过容器运行
root@jenkins:~/learning-prometheus/08-prometheus-components-compose/blackbox-exporter# docker-compose up -d

Prometheus添加

  - job_name: "tomcat_exporter"
    metrics_path: '/metrics'
    scheme: "http"
    consul_sd_configs:
        - server: "192.168.1.50:8500"
          tags:
          - "tomcat_exporter"
          refresh_interval: 1m
  - job_name: 'blackbox'
    metrics_path: /probe
    params:
      module: [http_2xx]
    static_configs:
    - targets:
      - www.magedu.com
      - www.google.com
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: "192.168.1.51:9115"  # Blackbox exporter.
      - target_label: region
        replacement: "remote"