CoreDNS实战(七)-日志处理

本文主要用于介绍CoreDNS用来记录日志的几种方式以及在生产环境中遇到的一些问题和解决方案。

1 log插件

coredns的日志输出并不如nginx那么完善（并不能在配置文件中指定输出的文件目录，但是可以指定日志的格式），默认情况下不论是log插件还是error插件都会把所有的相关日志输出到程序的standard output中。使用systemd来管理coredns之后，默认情况下基本就是由rsyslog和systemd-journald这两个服务来管理日志。

1.1 log插件配置

前面的文章里我们有简单的介绍过coredns的日志处理，对于log插件也是类似的操作。考虑到CentOS8已经结束支持，这里仅介绍CentOS7的配置方法，更高版本的systemd可以参考CoreDNS实战(一)-构建高性能、插件化的DNS服务器-CSDN博客

对于centos7等系统而言，是不支持在高版本的systemd中的unit文件中使用append和file两个参数的，并且对于这种方式输出的日志缺少了前面的时间和主机名等信息，相对而言还是修改rsyslog的方式要更加的可靠。那么在开启了rsyslog.service服务的情况下，日志就会输出到/var/log/messages文件中，或者可以使用journalctl -u coredns命令来查看全部的日志。

如果想要将coredns的日志全部集中到一个文件进行统一管理，我们可以对负责管理systemd的日志的rsyslog服务的配置进行修改，然后再重启rsyslog服务即可生效。

 # 注意这段配置需要添加在其他日志配置规则之前
 # 否则日志会在/var/log/messages和/home/coredns/logs/coredns.log都写入一份
 [root@tiny-server coredns]# vim /etc/rsyslog.conf
 if $programname == 'coredns' then /home/coredns/logs/coredns.log
 & stop
 
 [root@tiny-server coredns]# systemctl restart rsyslog.service

但是有两点需要额外注意：

CoreDNS在开启日志记录之后的性能要差于不开启日志记录，这个很容易理解，因为记录日志需要耗费额外的资源，尤其是当QPS增大之后，记录日志组件的压力也会增大，需要注意两个服务对资源的抢占和分配是否合理；如果遇到写入日志占用大量资源的情况，可以考虑将其配置为只写入部分类型日志（如只写入错误日志）
rsyslog和systemd-journald这两个服务默认情况下都会对日志的写入频率进行限制，当QPS增大之后，记录的日志会不完整；当然这个问题可以通过调整rsyslog的参数解决

1.2 rsyslog配置

下面具体介绍一下如何配置rsyslog和systemd-journald解除相关的日志限制。

默认情况下我们查看systemd-journald.service的日志或者是查看/var/log/messages文件，如果看到类似下面的提示信息，就说明日志的输出频率太高被抑制了

 [root@tiny-server coredns]# systemctl status systemd-journald.service
 ● systemd-journald.service - Journal Service
    Loaded: loaded (/usr/lib/systemd/system/systemd-journald.service; static; vendor preset: disabled)
    Active: active (running) since Wed 2021-12-01 09:30:03 CST; 3 weeks 6 days ago
      Docs: man:systemd-journald.service(8)
            man:journald.conf(5)
  Main PID: 709 (systemd-journal)
    Status: "Processing requests..."
    CGroup: /system.slice/systemd-journald.service
            └─709 /usr/lib/systemd/systemd-journald
 
 Dec 28 15:36:27 tiny-server systemd-journal[709]: Suppressed 13174 messages from /system.slice/coredns.service
 Dec 28 15:36:57 tiny-server systemd-journal[709]: Suppressed 13185 messages from /system.slice/coredns.service
 Dec 28 15:37:27 tiny-server systemd-journal[709]: Suppressed 11600 messages from /system.slice/coredns.service
 Dec 28 15:37:57 tiny-server systemd-journal[709]: Suppressed 13896 messages from /system.slice/coredns.service
 Dec 28 15:38:27 tiny-server systemd-journal[709]: Suppressed 13653 messages from /system.slice/coredns.service
 Dec 28 15:38:57 tiny-server systemd-journal[709]: Suppressed 16503 messages from /system.slice/coredns.service
 Dec 28 15:39:27 tiny-server systemd-journal[709]: Suppressed 13300 messages from /system.slice/coredns.service
 Dec 28 15:39:57 tiny-server systemd-journal[709]: Suppressed 10491 messages from /system.slice/coredns.service
 Dec 28 15:40:27 tiny-server systemd-journal[709]: Suppressed 12079 messages from /system.slice/coredns.service
 Dec 28 15:40:57 tiny-server systemd-journal[709]: Suppressed 17182 messages from /system.slice/coredns.service
 
 [root@tiny-server coredns]# tail -f /var/log/messages
 Dec 28 15:43:55 tiny-server rsyslogd: imjournal: begin to drop messages due to rate-limiting

解决方案也比较简单，我们对这两个服务的配置参数进行调整，将限制日志写入频率和写入时间间隔都设为0（即不做限制），然后重启服务即可。

 echo "RateLimitInterval=0" >> /etc/systemd/journald.conf
 echo "RateLimitBurst=0" >> /etc/systemd/journald.conf
 systemctl restart systemd-journald.service 
 
 echo "\$imjournalRatelimitInterval 0" >> /etc/rsyslog.conf
 echo "\$imjournalRatelimitBurst 0" >> /etc/rsyslog.conf
 systemctl restart rsyslog.service

需要特别注意的是：上述两个参数对所有使用rsyslog的服务都会生效，如果有其他大量写入日志的服务，建议调整日志文件输出目录，同时关注/var/log等目录的硬盘空间情况。

2 dnstap插件

CoreDNS的原生日志功能对于一个DNS服务器的绝大部分应用场景来说是足够使用的，如果有更进阶的需求，可以考虑使用dnstap插件来实现。

dnstap是一个基于谷歌的缓冲区协议（Protocol buffers）实现的用于DNS的灵活的、结构化的、二进制日志格式记录软件，目前已经获得了bind9、unbound、coredns、knot等几乎所有主流dns软件的支持。dnstap目前支持二进制日志格式、text日志格式、yaml日志格式和json日志格式。

CoreDNS官网对于dnstap插件的介绍比较简单，不过官网页面有比较详细的介绍。

dnstap is a flexible, structured binary log format for DNS software. It uses Protocol Buffers to encode events that occur inside DNS software in an implementation-neutral format.
Currently dnstap can only encode wire-format DNS messages. It is planned to support additional types of DNS log information.
Protocol buffers are Google's language-neutral, platform-neutral, extensible mechanism for serializing structured data – think XML, but smaller, faster, and simpler. You define how you want your data to be structured once, then you can use special generated source code to easily write and read your structured data to and from a variety of data streams and using a variety of languages.
缓冲区协议（Protocol buffers）是 Google 用于序列化结构数据的语言中立、平台中立、可扩展机制。可以将其和XML相对比，但Protocol buffers更小、更快、更简单。使用者只需定义一次数据的结构化方式，然后就可以使用特殊生成的源代码轻松地将结构化数据写入和读取各种数据流，并使用各种语言。

这里需要额外注意的是：

dnstap和CoreDNS原生的log等日志记录插件可以同时使用
尽管CoreDNS默认内置dnstap插件，但是要使用dnstap还是需要我们自己手动安装dnstap，再到corefile中配置dnstap的socket路径或者通信端口才能使用

2.1 dnstap安装

dnstap官方在github上面开源了一个CoreDNS实战(一)-构建高性能、插件化的DNS服务器-CSDN博客的项目，可以使用go来快速安装dnstap

 # golang的安装配置比较简单，我们从https://go.dev/dl/直接下载最新的稳定版本即可。
 wget https://go.dev/dl/go1.17.5.linux-amd64.tar.gz
 tar -zxvf go1.17.5.linux-amd64.tar.gz -C /usr/local
 
 # 修改系统默认的go文件
 ln -s /usr/local/go/bin/go /usr/bin/go
 
 # 接下来的go环境变量同学们可以根据自己的实际需求进行配置。
 # 对于我个人而言，我直接在/etc/profile中添加下面的配置然后source生效即可。
 export GOROOT=/usr/local/go
 export GOBIN=$GOROOT/bin
 export PATH=$PATH:$GOBIN
 export GOPATH=/home/gopath

执行go version命令确定go安装无误之后就可以直接安装dnstap：

 # 接着就可以使用go命令来快速安装dnstap
 go get -u github.com/dnstap/golang-dnstap/dnstap
 
 # go版本高于v1.16建议使用下面这条指令
 # 关于go get和go install的区别可以查看go的官方文档https://golang.org/doc/go-get-install-deprecation
 go install github.com/dnstap/golang-dnstap/dnstap@latest
    
 # 如果不确定是否安装成功，可以执行下面的命令查看安装信息进行确定
 [root@coredns2 home]# dnstap --help
 Usage: dnstap [OPTION]...
   -T value
         write dnstap payloads to tcp/ip address
   -U value
         write dnstap payloads to unix socket
   -a    append to the given file, do not overwrite. valid only when outputting a text or YAML file.
   -j    use verbose JSON output
   -l value
         read dnstap payloads from tcp/ip
   -q    use quiet text output
   -r value
         read dnstap payloads from file
   -t duration
         I/O timeout for tcp/ip and unix domain sockets
   -u value
         read dnstap payloads from unix socket
   -w string
         write output to file
   -y    use verbose YAML output
 
 Quiet text output format mnemonics:
     AQ: AUTH_QUERY
     AR: AUTH_RESPONSE
     RQ: RESOLVER_QUERY
     RR: RESOLVER_RESPONSE
     CQ: CLIENT_QUERY
     CR: CLIENT_RESPONSE
     FQ: FORWARDER_QUERY
     FR: FORWARDER_RESPONSE
     SQ: STUB_QUERY
     SR: STUB_RESPONSE
     TQ: TOOL_QUERY
     TR: TOOL_RESPONSE

2.2 coredns配置

coredns中对dnstap的配置非常简单，可以选择配置socket通信或者是tcp通信，full参数则是可以记录更多的参数信息，如同样的一次查询，红框是开启了full参数，而绿框则是不开启，两者相差较大。

 # 使用tcp通信
 dnstap tcp://127.0.0.1:6000 full
 # 使用socket通信
 dnstap unix:///tmp/dnstap.sock full

2.3 dnstap配置

dnstap的配置也比较简单，基本可以分为三个部分：

指定通信方式：如-u /tmp/dnstap.sock或者-l 127.0.0.1:6000
指定日志文件：如-w /home/coredns/logs/dnstap.log
指定日志格式：默认为二进制格式，-q为text文本，-y为yaml格式，-j为json格式
其他参数：如-a追加写入内容到日志文件中而非覆盖原有的文件

 # 使用socket通信，并以默认的二进制格式记录到/home/coredns/logs/dnstap.log文件中
 dnstap -u /tmp/dnstap.sock -w /home/coredns/logs/dnstap.log
 # 此时无法直接查看日志文件中的内容
 [root@coredns2 logs]# tail -f /home/coredns/logs/dnstap.log
 "•••protobuf:dnstap.Dnstap
 ^C
 
 # 使用socket通信，并以text文本格式追加记录到/home/coredns/logs/dnstap.log文件中
 dnstap -u /tmp/dnstap.sock -w /home/coredns/logs/dnstap.log -q -a
 # 这里的coredns对tinychen.com的查询会forward出去，
 # 因此日志中会有CLIENT_QUERY CLIENT_RESPONSE FORWARDER_QUERY FORWARDER_RESPONSE四条记录
 [root@coredns2 logs]# tail -f /home/coredns/logs/dnstap.log
 16:14:39.329430 CQ 10.31.100.100 UDP 53b "tinychen.com." IN A
 16:14:39.329546 FQ 127.0.0.1 UDP 53b "tinychen.com." IN A
 16:14:39.330241 FR 127.0.0.1 UDP 69b "tinychen.com." IN A
 16:14:39.330441 CR 10.31.100.100 UDP 81b "tinychen.com." IN A
 
 
 
 
 # 使用socket通信，并以yaml格式追加记录到/home/coredns/logs/dnstap.log文件中
 dnstap -u /tmp/dnstap.sock -w /home/coredns/logs/dnstap.log -y -a
 # 这里的coredns对tinychen.com的查询会forward出去，
 # 因此日志中会有CLIENT_QUERY CLIENT_RESPONSE FORWARDER_QUERY FORWARDER_RESPONSE四条记录
 [root@coredns2 logs]# tail -f /home/coredns/logs/dnstap.log
 type: MESSAGE
 message:
   type: CLIENT_QUERY
   query_time: !!timestamp 2022-01-12 08:23:25.211126656
   socket_family: INET
   socket_protocol: UDP
   query_address: 10.31.100.100
   query_port: 58562
   query_message: |
     ;; opcode: QUERY, status: NOERROR, id: 30286
     ;; flags: rd ad; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1
 
     ;; QUESTION SECTION:
     ;tinychen.com.      IN       A
 
     ;; ADDITIONAL SECTION:
 
     ;; OPT PSEUDOSECTION:
     ; EDNS: version 0; flags: ; udp: 4096
     ; COOKIE: 22c8a1e55b27c1b9
 ---
 type: MESSAGE
 message:
   type: FORWARDER_QUERY
   query_time: !!timestamp 2022-01-12 08:23:25.211208675
   socket_family: INET
   socket_protocol: UDP
   query_address: 10.31.100.100
   response_address: 127.0.0.1
   query_port: 58562
   response_port: 35353
   query_message: |
     ;; opcode: QUERY, status: NOERROR, id: 30286
     ;; flags: rd ad; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1
 
     ;; QUESTION SECTION:
     ;tinychen.com.      IN       A
 
     ;; ADDITIONAL SECTION:
 
     ;; OPT PSEUDOSECTION:
     ; EDNS: version 0; flags: do; udp: 2048
     ; COOKIE: 22c8a1e55b27c1b9
 ---
 type: MESSAGE
 message:
   type: FORWARDER_RESPONSE
   query_time: !!timestamp 2022-01-12 08:23:25.211208675
   response_time: !!timestamp 2022-01-12 08:23:25.243335638
   socket_family: INET
   socket_protocol: UDP
   query_address: 10.31.100.100
   response_address: 127.0.0.1
   query_port: 58562
   response_port: 35353
   response_message: |
     ;; opcode: QUERY, status: NOERROR, id: 30286
     ;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
 
     ;; QUESTION SECTION:
     ;tinychen.com.      IN       A
 
     ;; ANSWER SECTION:
     tinychen.com.       589     IN      A       1.12.217.55
 
     ;; ADDITIONAL SECTION:
 
     ;; OPT PSEUDOSECTION:
     ; EDNS: version 0; flags: do; udp: 1232
 ---
 type: MESSAGE
 message:
   type: CLIENT_RESPONSE
   query_time: !!timestamp 2022-01-12 08:23:25.211126656
   response_time: !!timestamp 2022-01-12 08:23:25.243503207
   socket_family: INET
   socket_protocol: UDP
   query_address: 10.31.100.100
   query_port: 58562
   response_message: |
     ;; opcode: QUERY, status: NOERROR, id: 30286
     ;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
 
     ;; QUESTION SECTION:
     ;tinychen.com.      IN       A
 
     ;; ANSWER SECTION:
     tinychen.com.       589     IN      A       1.12.217.55
 
     ;; ADDITIONAL SECTION:
 
     ;; OPT PSEUDOSECTION:
     ; EDNS: version 0; flags: ; udp: 4096
     ; COOKIE: 22c8a1e55b27c1b9
 ---
 
 
 
 
 # 使用socket通信，并以yaml格式追加记录到/home/coredns/logs/dnstap.log文件中
 dnstap -u /tmp/dnstap.sock -w /home/coredns/logs/dnstap.log -j -a
 # 这里的coredns对tinychen.com的查询会forward出去，
 # 因此日志中会有CLIENT_QUERY CLIENT_RESPONSE FORWARDER_QUERY FORWARDER_RESPONSE四条记录
 [root@coredns2 logs]# tail -f /home/coredns/logs/dnstap.log
 {"type":"MESSAGE","message":{"type":"CLIENT_QUERY","query_time":"2022-01-12T08:36:41.745130199Z","socket_family":"INET","socket_protocol":"UDP","query_address":"10.31.100.100","query_port":58561,"query_message":";; opcode: QUERY, status: NOERROR, id: 5835\n;; flags: rd ad; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1\n\n;; QUESTION SECTION:\n;tinychen.com.\tIN\t A\n\n;; ADDITIONAL SECTION:\n\n;; OPT PSEUDOSECTION:\n; EDNS: version 0; flags: ; udp: 4096\n; COOKIE: fd6716688b43710a\n"}}
 {"type":"MESSAGE","message":{"type":"FORWARDER_QUERY","query_time":"2022-01-12T08:36:41.745243809Z","socket_family":"INET","socket_protocol":"UDP","query_address":"10.31.100.100","response_address":"127.0.0.1","query_port":58561,"response_port":35353,"query_message":";; opcode: QUERY, status: NOERROR, id: 5835\n;; flags: rd ad; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1\n\n;; QUESTION SECTION:\n;tinychen.com.\tIN\t A\n\n;; ADDITIONAL SECTION:\n\n;; OPT PSEUDOSECTION:\n; EDNS: version 0; flags: do; udp: 2048\n; COOKIE: fd6716688b43710a\n"}}
 {"type":"MESSAGE","message":{"type":"FORWARDER_RESPONSE","query_time":"2022-01-12T08:36:41.745243809Z","response_time":"2022-01-12T08:36:41.83691354Z","socket_family":"INET","socket_protocol":"UDP","query_address":"10.31.100.100","response_address":"127.0.0.1","query_port":58561,"response_port":35353,"response_message":";; opcode: QUERY, status: SERVFAIL, id: 5835\n;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1\n\n;; QUESTION SECTION:\n;tinychen.com.\tIN\t A\n\n;; ADDITIONAL SECTION:\n\n;; OPT PSEUDOSECTION:\n; EDNS: version 0; flags: do; udp: 1232\n"}}
 {"type":"MESSAGE","message":{"type":"CLIENT_RESPONSE","query_time":"2022-01-12T08:36:41.745130199Z","response_time":"2022-01-12T08:36:41.837144129Z","socket_family":"INET","socket_protocol":"UDP","query_address":"10.31.100.100","query_port":58561,"response_message":";; opcode: QUERY, status: SERVFAIL, id: 5835\n;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1\n\n;; QUESTION SECTION:\n;tinychen.com.\tIN\t A\n\n;; ADDITIONAL SECTION:\n\n;; OPT PSEUDOSECTION:\n; EDNS: version 0; flags: ; udp: 4096\n; COOKIE: fd6716688b43710a\n"}}

2.4 其他问题

目前在使用dnstap的过程中发现了两个小问题：

dnstap日志记录时间使用的是UTC时间记录，对于国内的用户，时间上就有八个小时的差距
dnstap限制缓冲区的消息条数为10000条，超出限制的消息不会被写入dnstap中

 Jan 13 20:41:56 coredns2 coredns: [WARNING] plugin/dnstap: Dropped dnstap messages: 196
 Jan 13 20:41:56 coredns2 coredns: [WARNING] plugin/dnstap: Dropped dnstap messages: 680
 Jan 13 20:41:58 coredns2 coredns: [WARNING] plugin/dnstap: Dropped dnstap messages: 1900
 Jan 13 20:41:58 coredns2 coredns: [WARNING] plugin/dnstap: Dropped dnstap messages: 10118
 Jan 13 20:41:59 coredns2 coredns: [WARNING] plugin/dnstap: Dropped dnstap messages: 1472
 Jan 13 20:41:59 coredns2 coredns: [WARNING] plugin/dnstap: Dropped dnstap messages: 9430
 Jan 13 20:42:02 coredns2 coredns: [WARNING] plugin/dnstap: Dropped dnstap messages: 3669
 Jan 13 20:42:02 coredns2 coredns: [WARNING] plugin/dnstap: Dropped dnstap messages: 23351
 Jan 13 20:42:06 coredns2 coredns: [WARNING] plugin/dnstap: Dropped dnstap messages: 5776
 Jan 13 20:42:06 coredns2 coredns: [WARNING] plugin/dnstap: Dropped dnstap messages: 35511
 Jan 13 20:42:17 coredns2 coredns: [WARNING] plugin/dnstap: Dropped dnstap messages: 13161
 Jan 13 20:42:17 coredns2 coredns: [WARNING] plugin/dnstap: Dropped dnstap messages: 81395
 Jan 13 20:42:41 coredns2 coredns: [WARNING] plugin/dnstap: Dropped dnstap messages: 29500
 Jan 13 20:42:41 coredns2 coredns: [WARNING] plugin/dnstap: Dropped dnstap messages: 180877

如果想要突破这个限制，可以考虑在编译的时候修改源码
 # github地址
 # https://github.com/coredns/coredns/blob/master/plugin/dnstap/io.go
 
 const (
  tcpWriteBufSize = 1024 * 1024 // there is no good explanation for why this number has this value.
  queueSize       = 10000       // idem.
 
  tcpTimeout   = 4 * time.Second
  flushTimeout = 1 * time.Second
 )

3 errors插件

3.1 errors插件简介

CoreDNS官方有对errors插件比较详细的介绍和使用说明，简单来说errors插件就是专门用来记录错误信息日志的一个组件。这里需要注意的是：

并不是不使用errors插件，日志中就不会输出[ERROR]级别的日志；其他的插件和CoreDNS本身也是可以输出[ERROR]级别的日志
errors插件默认情况下输出的日志级别都是[ERROR]，但是也可以通过consolidate命令进行定义

3.2 errors插件配置

errors的配置非常简单，位置和log插件一样

 .:53 {
     forward . 127.0.0.1:35353
     log
     errors
     cache {
         success 10240 600 60
         denial 5120 60 5
     }
 }

开启前后的对比如下：

 # 不开启error插件
 Jan 14 11:50:12 coredns2 coredns: [INFO] 10.31.53.2:36586 - 34882 "A IN tinychen.com. udp 38 false 4096" - - 0 1.001413555s
 
 # 开启error插件
 Jan 14 11:50:33 coredns2 coredns: [INFO] 10.31.53.2:33796 - 62387 "A IN tinychen.com. udp 38 false 4096" - - 0 1.001076641s
 Jan 14 11:50:33 coredns2 coredns: [ERROR] plugin/errors: 2 tinychen.com. A: read udp 127.0.0.1:57591->127.0.0.1:35353: read: connection refused

当然还可以有类似正则的配置方式可以简化日志输出，同时定义

    errors {
        consolidate 30s ".* connection refused$" warning
        consolidate 5m ".* i/o timeout$" warning
        consolidate 30s "^Failed to .+"
    }

这样子可以把多条错误日志合并输出，结果如下：

# 合并错误日志之前，每次错误日志都会单独输出，日志级别默认均为[ERROR]
Jan 14 14:29:31 coredns2 coredns: [ERROR] plugin/errors: 2 tinychen.com. A: read udp 127.0.0.1:53697->127.0.0.1:35353: read: connection refused
Jan 14 14:29:32 coredns2 coredns: [INFO] 10.31.53.2:38422 - 21429 "A IN tinychen.com. udp 38 false 4096" - - 0 0.000386185s
Jan 14 14:29:32 coredns2 coredns: [ERROR] plugin/errors: 2 tinychen.com. A: read udp 127.0.0.1:34774->127.0.0.1:35353: read: connection refused
Jan 14 14:29:33 coredns2 coredns: [INFO] 10.31.53.2:44775 - 40543 "A IN tinychen.com. udp 38 false 4096" - - 0 0.000418209s
Jan 14 14:29:33 coredns2 coredns: [ERROR] plugin/errors: 2 tinychen.com. A: read udp 127.0.0.1:51575->127.0.0.1:35353: read: connection refused
Jan 14 14:29:33 coredns2 coredns: [INFO] 10.31.53.2:41858 - 30773 "A IN tinychen.com. udp 38 false 4096" - - 0 0.000285244s
Jan 14 14:29:33 coredns2 coredns: [ERROR] plugin/errors: 2 tinychen.com. A: read udp 127.0.0.1:54027->127.0.0.1:35353: read: connection refused
Jan 14 14:29:34 coredns2 coredns: [INFO] 10.31.53.2:56657 - 31457 "A IN tinychen.com. udp 38 false 4096" - - 0 0.000280881s
Jan 14 14:29:34 coredns2 coredns: [ERROR] plugin/errors: 2 tinychen.com. A: read udp 127.0.0.1:59189->127.0.0.1:35353: read: connection refused
Jan 14 14:29:34 coredns2 coredns: [INFO] 10.31.53.2:48445 - 29130 "A IN tinychen.com. udp 38 false 4096" - - 0 0.000399419s
Jan 14 14:29:34 coredns2 coredns: [ERROR] plugin/errors: 2 tinychen.com. A: read udp 127.0.0.1:35171->127.0.0.1:35353: read: connection refused
Jan 14 14:29:35 coredns2 coredns: [INFO] 10.31.53.2:42290 - 310 "A IN tinychen.com. udp 38 false 4096" - - 0 0.000359849s
Jan 14 14:29:35 coredns2 coredns: [ERROR] plugin/errors: 2 tinychen.com. A: read udp 127.0.0.1:40662->127.0.0.1:35353: read: connection refused
Jan 14 14:29:36 coredns2 coredns: [INFO] 10.31.53.2:33545 - 28529 "A IN tinychen.com. udp 38 false 4096" - - 0 0.000456204s
Jan 14 14:29:36 coredns2 coredns: [ERROR] plugin/errors: 2 tinychen.com. A: read udp 127.0.0.1:37439->127.0.0.1:35353: read: connection refused
Jan 14 14:29:36 coredns2 coredns: [INFO] 10.31.53.2:53167 - 18372 "A IN tinychen.com. udp 38 false 4096" - - 0 0.000426039s
Jan 14 14:29:36 coredns2 coredns: [ERROR] plugin/errors: 2 tinychen.com. A: read udp 127.0.0.1:45881->127.0.0.1:35353: read: connection refused
Jan 14 14:29:37 coredns2 coredns: [INFO] 10.31.53.2:44851 - 52388 "A IN tinychen.com. udp 38 false 4096" - - 0 0.000264627s
Jan 14 14:29:37 coredns2 coredns: [ERROR] plugin/errors: 2 tinychen.com. A: read udp 127.0.0.1:58678->127.0.0.1:35353: read: connection refused
Jan 14 14:29:37 coredns2 coredns: [INFO] 10.31.53.2:59500 - 19169 "A IN tinychen.com. udp 38 false 4096" - - 0 0.000385832s
Jan 14 14:29:37 coredns2 coredns: [ERROR] plugin/errors: 2 tinychen.com. A: read udp 127.0.0.1:36355->127.0.0.1:35353: read: connection refused
Jan 14 14:29:38 coredns2 coredns: [INFO] 10.31.53.2:48272 - 35987 "A IN tinychen.com. udp 38 false 4096" - - 0 0.000359493s
Jan 14 14:29:38 coredns2 coredns: [ERROR] plugin/errors: 2 tinychen.com. A: read udp 127.0.0.1:52055->127.0.0.1:35353: read: connection refused
Jan 14 14:29:39 coredns2 coredns: [INFO] 10.31.53.2:55974 - 48529 "A IN tinychen.com. udp 38 false 4096" - - 0 0.00035554s
Jan 14 14:29:39 coredns2 coredns: [ERROR] plugin/errors: 2 tinychen.com. A: read udp 127.0.0.1:51253->127.0.0.1:35353: read: connection refused
Jan 14 14:29:39 coredns2 coredns: [INFO] 10.31.53.2:40670 - 43374 "A IN tinychen.com. udp 38 false 4096" - - 0 0.00024835s
Jan 14 14:29:39 coredns2 coredns: [ERROR] plugin/errors: 2 tinychen.com. A: read udp 127.0.0.1:41919->127.0.0.1:35353: read: connection refused
Jan 14 14:29:40 coredns2 coredns: [INFO] 10.31.53.2:42067 - 780 "A IN tinychen.com. udp 38 false 4096" - - 0 0.000257447s
Jan 14 14:29:40 coredns2 coredns: [ERROR] plugin/errors: 2 tinychen.com. A: read udp 127.0.0.1:49359->127.0.0.1:35353: read: connection refused
Jan 14 14:29:40 coredns2 coredns: [INFO] 10.31.53.2:56758 - 62500 "A IN tinychen.com. udp 38 false 4096" - - 0 0.000338204s
Jan 14 14:29:40 coredns2 coredns: [ERROR] plugin/errors: 2 tinychen.com. A: read udp 127.0.0.1:44658->127.0.0.1:35353: read: connection refused
Jan 14 14:29:41 coredns2 coredns: [INFO] 10.31.53.2:36149 - 32955 "A IN tinychen.com. udp 38 false 4096" - - 0 0.000400937s
Jan 14 14:29:41 coredns2 coredns: [ERROR] plugin/errors: 2 tinychen.com. A: read udp 127.0.0.1:33750->127.0.0.1:35353: read: connection refused
Jan 14 14:29:41 coredns2 coredns: [INFO] 10.31.53.2:60504 - 49021 "A IN tinychen.com. udp 38 false 4096" - - 0 0.000272807s
Jan 14 14:29:41 coredns2 coredns: [ERROR] plugin/errors: 2 tinychen.com. A: read udp 127.0.0.1:41003->127.0.0.1:35353: read: connection refused

# 合并错误日志之后,合并输出30s内符合条件的错误日志总数，并且日志级别变为我们自定义的[WARNING]
Jan 14 14:32:48 coredns2 coredns: [INFO] 10.31.53.2:60095 - 763 "A IN tinychen.com. udp 38 false 4096" - - 0 0.000389099s
Jan 14 14:33:10 coredns2 coredns: [INFO] 10.31.53.2:44256 - 21788 "A IN tinychen.com. udp 38 false 4096" - - 0 0.000556023s
Jan 14 14:33:11 coredns2 coredns: [INFO] 10.31.53.2:39654 - 18638 "A IN tinychen.com. udp 38 false 4096" - - 0 0.000394553s
Jan 14 14:33:11 coredns2 coredns: [INFO] 10.31.53.2:46340 - 23770 "A IN tinychen.com. udp 38 false 4096" - - 0 0.000405404s
Jan 14 14:33:12 coredns2 coredns: [INFO] 10.31.53.2:56168 - 59588 "A IN tinychen.com. udp 38 false 4096" - - 0 0.000351365s
Jan 14 14:33:13 coredns2 coredns: [INFO] 10.31.53.2:52337 - 28311 "A IN tinychen.com. udp 38 false 4096" - - 0 0.00033022s
Jan 14 14:33:14 coredns2 coredns: [INFO] 10.31.53.2:55564 - 52773 "A IN tinychen.com. udp 38 false 4096" - - 0 0.0002756s
Jan 14 14:33:14 coredns2 coredns: [INFO] 10.31.53.2:36233 - 21289 "A IN tinychen.com. udp 38 false 4096" - - 0 0.000307114s
Jan 14 14:33:15 coredns2 coredns: [INFO] 10.31.53.2:41822 - 41368 "A IN tinychen.com. udp 38 false 4096" - - 0 0.000379377s
Jan 14 14:33:18 coredns2 coredns: [WARNING] plugin/errors: 9 errors like '.* connection refused$' occurred in last 30s