目录
一、Keepalive的认识
1.Keepalive基础——VRRP
2.Keepalived工具介绍
2.1Keepalived介绍
2.2Keepalived架构
2.2.1用户空间核心组件
2.2.2WatchDog:监控进程(整个架构是否有问题)
二、安装Keepalived及相关配置文件详解
1.安装Keepalive
2.Keepalived配置详解
2.1相关文件
2.2配置组成
2.3全局配置
2.4配置虚拟路由器
三、实验操作
1.搭建实验环境
2.Keepalived服务器配置
2.1主Keepalived服务器配置
2.2从服务器配置
3.后端提供Web服务器配置
3.1Web1
3.2Web2
4.测试
5.主从切换
5.1抢占模式
5.1.1主服务器关闭
5.1.2备服务器代替主服务器
5.1.3客户端访问不受影响
5.2延迟抢占模式
5.2.1主配置
5.2.2从配置
5.2.3测试
5.3非抢占模式
5.3.1主配置
5.3.2从配置
6.单播/组播
6.1组播
6.1.1主配置
6.1.2从配置
6.1.3抓包测试
6.2单播
6.2.1主
6.2.2从
6.2.3抓包测试
7.通知脚本
7.1配置邮箱
7.2模拟故障
8.日志功能
四、脑裂——实现其他应用的高可用性VRRP Script
1.脑裂的定义
2.脑裂的原因
3.如何解决Keepalive脑裂问题
4.模拟脑裂
5.VRRP Script配置
5.1配置VRRP Script
5.1.1定义脚本
5.1.2调用脚本
5.2定义VRRP Script
6.实际操作
LVS部署的缺点:
LVS无健康性检查 无法检查后端真实服务器的健康性;
LVS容易发生单点故障
一、Keepalive的认识
1.Keepalive基础——VRRP
VRRP相关技术
- 虚拟路由器:Virtual Router 不是真实存在 ,虚构出来的
- 虚拟路由器标识:VRID(0-255),唯一标识虚拟路由器
- VIP:Virtual IP 192.168.241.11 路由1 路由2
- VMAC:Virutal MAC (00-00-5e-00-01-VRID) 虚拟MAC
- 物理路由器:
- master:主设备
- backup:备用设备
- priority:优先级
通告:是宣告自己的主权,不要妄想抢班夺权,不停的向外(心跳,优先级等;周期性)
工作方式:抢占式,非抢占式,延迟抢占模式
- 抢占式:主服务器宕机,过了一段时间修好了,再把主权抢过来
- 非抢占式:主服务器宕机,过了一段时间修好了,原来的主就作为备了
- 延迟抢占:主修好后,等待一定的时间(300s)后再次成为主
安全认证:如没有安全认证,不在集群中的keeplive服务器设置超高的优先级,会造成事故
- 无认证
- 简单字符认证:预共享密钥
- MD5
工作模式:
- 主/备:单虚拟路径器
- 主/主:主/备(虚拟路由器1),备/主(虚拟路由器2)
环境:有两台虚拟路由器,第一台虚拟路由器中服务器1为主,服务器2为备,那么虚拟IP1就飘在服务器1上,真正工作的只有服务器1;第二胎虚拟路由器中服务器2为主,服务器1为备,那么虚拟IP2就飘在服务器2上,真正工作的只有服务器2;
优点:提高了资源利用率;同样有备份功能;
缺点:虽然有备份冗余功能但是对机器的性能要求非常高,当其中一台出现故障,本来一台运行一个任务,现在所有业务全部压在了一台上,有十分大的危险。
Keepalive采用VRRP热备份协议,实现Linux服务器的多级热备功能;
VRRP(虚拟路由冗余协议)是针对路由器的一种备份解决方案;
- 由多台路由器组成一个热备组,通过共用的虚拟IP地址对外提供服务
- 每个热备组内同时只有一台主路由器提供服务,其他路由器处于冗余状态
- 若当前在线的路由器失效,则其他路由器会根据设置的优先级自动接替虚拟IP地址,继续提供服务
2.Keepalived工具介绍
Keepalived工具是专为LVS和HA设计的一款健康检查工具
- 支持故障自动切换(Failover)
- 支持节点健康状态检查(Health Checking)
判断LVS负载调度器、节点服务器的可用性,当master主机出现故障及时切换到backup节点保证业务正常,当master故障主机恢复后将其重新加入群集并且业务重新切换回master节点
官方网站:http://www.keepalived.org/
2.1Keepalived介绍
功能
- 基于vrrp协议完成地址流动
- 为vip地址所在的节点生成ipvs规则(在配置文件中预先定义)
- 为ipvs集群的各RS做健康状态检测
- 基于脚本调用接口完成脚本中定义的功能,进而影响集群事务,以此支持nginx、haproxy等服务
2.2Keepalived架构
官方文档: https://keepalived.org/doc/ http://keepalived.org/documentation.html
2.2.1用户空间核心组件
- vrrp stack:VIP消息通告 虚拟ip
- checkers:监测real server(简单来说 就是监控后端真实服务器的服务)是否存活
- system call:实现 vrrp 协议状态转换时调用脚本的功能
- SMTP:邮件组件(报警邮件)
- IPVS wrapper:生成IPVS规则(直接生成ipvsadm)
- Netlink Reflector:网络接口(将虚拟地址ip(vip)地址飘动)
2.2.2WatchDog:监控进程(整个架构是否有问题)
- 控制组件:提供keepalived.conf 的解析器,完成Keepalived配置
- IO复用器:针对网络目的而优化的自己的线程抽象
- 内存管理组件:为某些通用的内存管理功能(例如分配,重新分配,发布等)提供访问权限
keeplive可以配合ngnix等软件,反向代理
二、安装Keepalived及相关配置文件详解
1.安装Keepalive
[root@localhost ~]#yum info keepalived.x86_64
#查看yum安装源的Keepalive的详细信息
[root@localhost ~]#yum install keepalived.x86_64 -y
[root@localhost ~]#rpm -q keepalived
keepalived-1.3.5-19.el7.x86_64
2.Keepalived配置详解
2.1相关文件
- 软件包名:keepalived
- 主程序文件:/usr/sbin/keepalived
- 主配置文件:/etc/keepalived/keepalived.conf
- 配置文件示例:/usr/share/doc/keepalived/
- Unit File:/lib/systemd/system/keepalived.service
- Unit File的环境配置文件:
- /etc/sysconfig/keepalived CentOS
[root@localhost ~]#rpm -ql keepalived
/etc/keepalived
/etc/keepalived/keepalived.conf
/etc/sysconfig/keepalived
/usr/bin/genhash
/usr/lib/systemd/system/keepalived.service
/usr/libexec/keepalived
/usr/sbin/keepalived
/usr/share/doc/keepalived-1.3.5
/usr/share/doc/keepalived-1.3.5/AUTHOR
/usr/share/doc/keepalived-1.3.5/CONTRIBUTORS
/usr/share/doc/keepalived-1.3.5/COPYING
/usr/share/doc/keepalived-1.3.5/ChangeLog
/usr/share/doc/keepalived-1.3.5/NOTE_vrrp_vmac.txt
/usr/share/doc/keepalived-1.3.5/README
/usr/share/doc/keepalived-1.3.5/TODO
/usr/share/doc/keepalived-1.3.5/keepalived.conf.SYNOPSIS
/usr/share/doc/keepalived-1.3.5/samples
/usr/share/doc/keepalived-1.3.5/samples/keepalived.conf.HTTP_GET.port
/usr/share/doc/keepalived-1.3.5/samples/keepalived.conf.IPv6
/usr/share/doc/keepalived-1.3.5/samples/keepalived.conf.SMTP_CHECK
/usr/share/doc/keepalived-1.3.5/samples/keepalived.conf.SSL_GET
/usr/share/doc/keepalived-1.3.5/samples/keepalived.conf.fwmark
/usr/share/doc/keepalived-1.3.5/samples/keepalived.conf.inhibit
/usr/share/doc/keepalived-1.3.5/samples/keepalived.conf.misc_check
/usr/share/doc/keepalived-1.3.5/samples/keepalived.conf.misc_check_arg
/usr/share/doc/keepalived-1.3.5/samples/keepalived.conf.quorum
/usr/share/doc/keepalived-1.3.5/samples/keepalived.conf.sample
/usr/share/doc/keepalived-1.3.5/samples/keepalived.conf.status_code
/usr/share/doc/keepalived-1.3.5/samples/keepalived.conf.track_interface
/usr/share/doc/keepalived-1.3.5/samples/keepalived.conf.virtual_server_group
/usr/share/doc/keepalived-1.3.5/samples/keepalived.conf.virtualhost
/usr/share/doc/keepalived-1.3.5/samples/keepalived.conf.vrrp
/usr/share/doc/keepalived-1.3.5/samples/keepalived.conf.vrrp.localcheck
/usr/share/doc/keepalived-1.3.5/samples/keepalived.conf.vrrp.lvs_syncd
/usr/share/doc/keepalived-1.3.5/samples/keepalived.conf.vrrp.routes
/usr/share/doc/keepalived-1.3.5/samples/keepalived.conf.vrrp.rules
/usr/share/doc/keepalived-1.3.5/samples/keepalived.conf.vrrp.scripts
/usr/share/doc/keepalived-1.3.5/samples/keepalived.conf.vrrp.static_ipaddress
/usr/share/doc/keepalived-1.3.5/samples/keepalived.conf.vrrp.sync
/usr/share/doc/keepalived-1.3.5/samples/sample.misccheck.smbcheck.sh
/usr/share/man/man1/genhash.1.gz
/usr/share/man/man5/keepalived.conf.5.gz
/usr/share/man/man8/keepalived.8.gz
/usr/share/snmp/mibs/KEEPALIVED-MIB.txt
/usr/share/snmp/mibs/VRRP-MIB.txt
/usr/share/snmp/mibs/VRRPv3-MIB.txt
主配置文件为/etc/keepalived/keepalived.conf
2.2配置组成
/etc/keepalived/keepalived.conf 配置组成
-
GLOBAL CONFIGURATION
Global definitions(全局配置):定义邮件配置,route_id,vrrp配置,组播地址 等
-
VRRP CONFIGURATION
VRRP instance(s):定义vrrp协议中的每个vrrp虚拟路由器的规则,基本信息
-
LVS CONFIGURATION(lvs调度服务器的规则设置)
Virtual server group(s)
Virtual server(s):LVS集群的VS和RS
2.3全局配置
[root@localhost ~]#cd /etc/keepalived/
[root@localhost keepalived]#ls
keepalived.conf
[root@localhost keepalived]#pwd
/etc/keepalived
[root@localhost keepalived]#vim /etc/keepalived/keepalived.conf
global_defs {
notification_email {
acassen@firewall.loc
failover@firewall.loc
sysadmin@firewall.loc
}
#邮箱配置 Keepalived发生故障切换时可以通过这个模块的设置发送通知消息到目标邮箱
notification_email_from Alexandre.Cassen@firewall.loc
#发送邮件的地址
smtp_server 192.168.200.1
#邮件服务器地址 可以修改为127.0.0.1
smtp_connect_timeout 30
#邮件服务器连接超时等待时间 为30s
router_id LVS_DEVEL
#每个keepalived主机的唯一标识,建议使用当前主机名,但多节点重名不影响
vrrp_skip_check_adv_addr
#对所有通告报文都检查,会比较消耗性能,启用此配置后,如果收到的通告报文和上一个报文是同一个路由器,则跳过检查,默认值为全检查
vrrp_strict
#严格遵守VRRP协议,启用此项后以下状况将无法启动服务:1.无VIP地址 2.配置了单播邻居 3.在VRRP版本2中有IPv6地址,开启动此项并且没有配置vrrp_iptables时会自动开启iptables防火墙规则,默认导致VIP无法访问,建议不加此项配置。
vrrp_garp_interval 0
#gratuitous ARP messages 免费ARP报文发送延迟,0表示不延迟
vrrp_gna_interval 0
#unsolicited NA messages (不请自来)消息发送延迟
vrrp_mcast_group4 224.0.0.18
#指定组播IP地址范围:224.0.0.0到239.255.255.255,默认值:224.0.0.18
#默认keepalived主机之间利用多播相互通告消息,会造成网络拥塞,可以替换成单播,减少网络流量
注意:启用 vrrp_strict 时,不能启用单播
#在所有节点vrrp_instance语句块中设置对方主机的IP,建议设置为专用于对应心跳线网络的地址,而非使用业务网络
vrrp_iptables
#此项和vrrp_strict同时开启时,则不会添加防火墙规则,如果无配置vrrp_strict项,则无需启用此项配置
}
2.4配置虚拟路由器
vrrp_instance VI_1<STRING> {
#<STRING>为VRRP示例名 一般为业务名称 支持自定义
state MASTER
#当前节点在此虚拟路由器上的初始状态,状态为MASTER或者BACKUP 此处定义并不会影响主从关系
interface eth0
#绑定为当前虚拟路由器使用的物理接口,如:eth0,bond0,br0,可以和VIP不在一个网卡
virtual_router_id 51
#每个虚拟路由器惟一标识,范围:0-255,每个虚拟路由器此值必须唯一,否则服务无法启动,同属一个虚拟路由器的多个keepalived节点必须相同,务必要确认在同一网络中此值必须唯一
priority 100
#优先级 当前物理节点在此虚拟路由器的优先级,范围:1-254,值越大优先级越高,每个keepalived主机节点此值不同
advert_int 1
#VRRP通告时间间隔 默认1s 告诉从服务器我还活着
authentication {
#认证机制
auth_type PASS
#通过密码认证方式进行认证 AH为IPSEC认证(不推荐),PASS为简单密码(建议使用)
auth_pass 1111
#预共享密钥,仅前8位有效,同一个虚拟路由器的多个keepalived节点必须一样
}
#include /etc/keealived/conf.d/*.conf
#如果需要管理多台设备 要加入此项
virtual_ipaddress {
#虚拟IP地址 <IPADDR>/<MASK> brd <IPADDR> dev <STRING> scope <SCOPE> label <LABEL>
192.168.200.16
#指定VIP,不指定网卡,默认为,注意:不指定/prefix,默认为/32
192.168.200.17/24 dev eth1
#指定VIP的网卡,建议和interface指令指定的岗卡不在一个网卡
192.168.200.18/24 dev eth2 label eth2:1
#指定VIP的网卡label
}
track_interface {
#配置监控网络接口,一旦出现故障,则转为FAULT状态实现地址转移
eth0
eth1
}
}
#虚拟主机
virtual_server 192.168.200.100 443 {
#虚拟IP地址
delay_loop 6
#健康间隔为6s
lb_algo rr
#调度算法为rr 轮询
lb_kind NAT
#lvs模式为NAT 也可以设置为DR
persistence_timeout 50
#连接保持时间改为0 否则无法体现效果
protocol TCP
#采用的协议为TCP协议
real_server 192.168.201.100 443 {
#真实服务器地址
weight 1
#节点服务器权重
TCP_CHECK{
connect_port 80
#检查目标端口
connect_timeout 3
#连接超时
nb_get_retry 3
#重试次数
delay_before_retry 3
#重试间隔时间
}
}
三、实验操作
LVS + Keepalived 高可用群集
1.搭建实验环境
Centos 7-1作为主Keepalived服务器;Centos 7-2作为备Keepalived服务器;Centos 7-3作为提供Web1服务的服务器;Centos 7-4作为提供Web2服务的服务器;Centos 7-5作为客户机
[root@localhost keepalived]#systemctl stop firewalld
[root@localhost keepalived]#setenforce 0
setenforce: SELinux is disabled
[root@node2 ~]#systemctl stop firewalld
[root@node2 ~]#setenforce 0
[root@node3 ~]#systemctl stop firewalld
[root@node3 ~]#setenforce 0
[root@G ~]#systemctl stop firewalld
[root@G ~]#setenforce 0
[root@localhost ~]# systemctl stop firewalld
[root@localhost ~]# setenforce 0
setenforce: SELinux is disabled
2.Keepalived服务器配置
2.1主Keepalived服务器配置
[root@localhost keepalived]#rpm -q keepalived
keepalived-1.3.5-19.el7.x86_64
[root@localhost keepalived]#yum install ipvsadm.x86_64 -y
[root@localhost keepalived]#ls
keepalived.conf
[root@localhost keepalived]#cp keepalived.conf keepalived.conf.bak
#备份配置文件
[root@localhost keepalived]#vim keepalived.conf
[root@localhost keepalived]#systemctl start keepalived.service
[root@localhost keepalived]#systemctl status keepalived.service
● keepalived.service - LVS and VRRP High Availability Monitor
Loaded: loaded (/usr/lib/systemd/system/keepalived.service; disabled; vendor preset: disabled)
Active: active (running) since 四 2024-03-07 15:03:44 CST; 6s ago
Process: 7260 ExecStart=/usr/sbin/keepalived $KEEPALIVED_OPTIONS (code=exited, status=0/SUCCESS)
Main PID: 7261 (keepalived)
CGroup: /system.slice/keepalived.service
├─7261 /usr/sbin/keepalived -D
├─7262 /usr/sbin/keepalived -D
└─7263 /usr/sbin/keepalived -D
3月 07 15:03:48 localhost.localdomain Keepalived_vrrp[7263]: VRRP_Instanc...
3月 07 15:03:49 localhost.localdomain Keepalived_vrrp[7263]: VRRP_Instanc...
3月 07 15:03:49 localhost.localdomain Keepalived_vrrp[7263]: VRRP_Instanc...
3月 07 15:03:49 localhost.localdomain Keepalived_vrrp[7263]: VRRP_Instanc...
3月 07 15:03:49 localhost.localdomain Keepalived_vrrp[7263]: Sending grat...
3月 07 15:03:49 localhost.localdomain Keepalived_vrrp[7263]: VRRP_Instanc...
3月 07 15:03:49 localhost.localdomain Keepalived_vrrp[7263]: Sending grat...
3月 07 15:03:49 localhost.localdomain Keepalived_vrrp[7263]: Sending grat...
3月 07 15:03:49 localhost.localdomain Keepalived_vrrp[7263]: Sending grat...
3月 07 15:03:49 localhost.localdomain Keepalived_vrrp[7263]: Sending grat...
Hint: Some lines were ellipsized, use -l to show in full.
[root@localhost keepalived]#vim keepalived.conf
[root@localhost keepalived]#systemctl restart keepalived.service
[root@localhost keepalived]#ipvsadm -ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 127.0.0.1:80 rr
TCP 192.168.241.111:80 rr
-> 192.168.241.23:80 Route 1 0 0
-> 192.168.241.24:80 Route 1 0 0
2.2从服务器配置
[root@node2 ~]#yum install ipvsadm.x86_64 keepalived.86_64 -y
[root@node2 ~]#rpm -q ipvsadm
ipvsadm-1.27-8.el7.x86_64
[root@node2 ~]#rpm -q keepalived
keepalived-1.3.5-19.el7.x86_64
[root@node2 ~]#cd /etc/keepalived/
[root@node2 keepalived]#ls
keepalived.conf
[root@node2 keepalived]#cp keepalived.conf keepalived.conf.bak
[root@node2 keepalived]#ls
keepalived.conf keepalived.conf.bak
[root@localhost keepalived]#scp keepalived.conf 192.168.241.22:/etc/keepalived/
The authenticity of host '192.168.241.22 (192.168.241.22)' can't be established.
ECDSA key fingerprint is SHA256:CcASxxV4CvFA+6w68th3aaCYGbGB3UwaAK1xifsM/Pk.
ECDSA key fingerprint is MD5:d6:ee:2e:4d:f6:34:c5:14:0e:ef:99:8c:54:48:c6:be.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '192.168.241.22' (ECDSA) to the list of known hosts.
root@192.168.241.22's password:
keepalived.conf 100% 1182 3.6MB/s 00:00
[root@node2 keepalived]#vim keepalived.conf
[root@node2 keepalived]#systemctl start keepalived.service
[root@node2 keepalived]#ipvsadm-save > /etc/sysconfig/ipvsadm
[root@node2 keepalived]#ipvsadm -ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 192.168.241.111:80 rr
-> 192.168.241.23:80 Route 1 0 0
-> 192.168.241.24:80 Route 1 0 0
3.后端提供Web服务器配置
3.1Web1
[root@node3 ~]#rpm -q httpd
未安装软件包 httpd
[root@node3 ~]#yum install httpd -y
[root@node3 ~]#ifconfig lo:0 192.168.241.111 netmask 255.255.255.255
[root@node3 ~]#ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet 192.168.241.111/32 scope global lo:0
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 00:0c:29:6b:71:15 brd ff:ff:ff:ff:ff:ff
inet 192.168.241.23/24 brd 192.168.241.255 scope global ens33
valid_lft forever preferred_lft forever
inet6 fe80::f11e:5019:be57:47b8/64 scope link
valid_lft forever preferred_lft forever
3: virbr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN qlen 1000
link/ether 52:54:00:9d:e9:ac brd ff:ff:ff:ff:ff:ff
inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0
valid_lft forever preferred_lft forever
4: virbr0-nic: <BROADCAST,MULTICAST> mtu 1500 qdisc pfifo_fast master virbr0 state DOWN qlen 1000
link/ether 52:54:00:9d:e9:ac brd ff:ff:ff:ff:ff:ff
[root@node3 ~]#vim /etc/sysctl.conf
[root@node3 ~]#sysctl -p
net.ipv4.conf.all.arp_ignore = 1
net.ipv4.conf.all.arp_announce = 2
net.ipv4.conf.default.arp_ignore = 1
net.ipv4.conf.default.arp_announce = 2
net.ipv4.conf.lo.arp_ignore = 1
net.ipv4.conf.lo.arp_announce = 2
[root@node3 ~]#echo cxk > /var/www/html/index.html
[root@node3 ~]#cat /var/www/html/index.html
cxk
[root@node3 ~]#vim /etc/httpd/conf/httpd.conf
keepalive off
#因为Apache默认是长连接 所以要关闭长连接才可以看出效果
3.2Web2
[root@G ~]#rpm -q httpd
未安装软件包 httpd
[root@G ~]#yum install httpd -y
[root@G ~]#ifconfig lo:0 192.168.241.111 netmask 255.255.255.255
[root@G ~]#ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet 192.168.241.111/32 scope global lo:0
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 00:0c:29:8c:91:84 brd ff:ff:ff:ff:ff:ff
inet 192.168.241.24/24 brd 192.168.241.255 scope global ens33
valid_lft forever preferred_lft forever
inet6 fe80::871f:7f65:7279:5914/64 scope link
valid_lft forever preferred_lft forever
3: virbr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN qlen 1000
link/ether 52:54:00:d2:18:b8 brd ff:ff:ff:ff:ff:ff
inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0
valid_lft forever preferred_lft forever
4: virbr0-nic: <BROADCAST,MULTICAST> mtu 1500 qdisc pfifo_fast master virbr0 state DOWN qlen 1000
link/ether 52:54:00:d2:18:b8 brd ff:ff:ff:ff:ff:ff
[root@G ~]#vim /etc/sysctl.conf
[root@G ~]#sysctl -p
net.ipv4.conf.all.arp_ignore = 1
net.ipv4.conf.all.arp_announce = 2
net.ipv4.conf.default.arp_ignore = 1
net.ipv4.conf.default.arp_announce = 2
net.ipv4.conf.lo.arp_ignore = 1
net.ipv4.conf.lo.arp_announce = 2
[root@G ~]#echo wyb > /var/www/html/index.html
[root@G ~]#cat /var/www/html/index.html
wyb
[root@G ~]#vim /etc/httpd/conf/httpd.conf
keepalive off
#因为Apache默认是长连接 所以要关闭长连接才可以看出效果
4.测试
5.主从切换
5.1抢占模式
5.1.1主服务器关闭
[root@localhost keepalived]#systemctl stop keepalived.service
5.1.2备服务器代替主服务器
[root@node2 keepalived]#ipvsadm -ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 192.168.241.111:80 rr
-> 192.168.241.23:80 Route 1 0 4
-> 192.168.241.24:80 Route 1 0 4
5.1.3客户端访问不受影响
5.2延迟抢占模式
5.2.1主配置
[root@localhost keepalived]#vim keepalived.conf
[root@localhost keepalived]#systemctl restart keepalived.service
5.2.2从配置
[root@node2 keepalived]#vim keepalived.conf
[root@node2 keepalived]#systemctl restart keepalived.service
5.2.3测试
[root@node2 keepalived]#hostname -I
192.168.241.22 192.168.122.1
[root@localhost keepalived]#hostname -I
192.168.241.11 192.168.241.111 192.168.122.1
[root@localhost keepalived]#systemctl stop keepalived.service
[root@localhost keepalived]#hostname -I
192.168.241.11 192.168.122.1
[root@node2 keepalived]#hostname -I
192.168.241.22 192.168.241.111 192.168.122.1
[root@localhost keepalived]#systemctl start keepalived.service
30秒后
[root@localhost keepalived]#hostname -I
192.168.241.22 192.168.241.111 192.168.122.1
[root@node2 keepalived]#hostname -I
192.168.241.11 192.168.122.1
5.3非抢占模式
5.3.1主配置
[root@localhost keepalived]#vim keepalived.conf
[root@localhost keepalived]#systemctl restart keepalived.service
5.3.2从配置
[root@node2 keepalived]#vim keepalived.conf
[root@node2 keepalived]#systemctl restart keepalived.service
非抢占模式:主服务器宕机或者掉线的话,从服务器上线,如果后续主服务器再次恢复后,重新上线,那么就还是作为备服务器,不抢占当前的主服务器的Keepalive
6.单播/组播
6.1组播
6.1.1主配置
6.1.2从配置
6.1.3抓包测试
6.2单播
6.2.1主
6.2.2从
6.2.3抓包测试
7.通知脚本
当前节点成为主节点时触发的脚本
notify_master <STRING>|<QUOTED-STRING>
当前节点转为备节点时触发的脚本
notify_backup <STRING>|<QUOTED-STRING>
当前节点转为“失败”状态时触发的脚本
notify_fault <STRING>|<QUOTED-STRING>
通用格式的通知触发机制,一个脚本可完成以上三种状态的转换时的通知
notify <STRING>|<QUOTED-STRING>
当停止VRRP时触发的脚本
notify_stop <STRING>|<QUOTED-STRING>
7.1配置邮箱
[root@localhost opt]#vim /etc/mail.rc
set from=12345678@163.com
set smtp=smtp.163.com
set smtp-auth-user=12345678@163.com
set smtp-auth-password=
[root@localhost ~]#cd /opt
[root@localhost opt]#vim /etc/mail.rc
[root@localhost opt]#vim keepalive.sh
[root@localhost opt]#cat keepalive.sh
#!/bin/bash
contact='12345678@qq.com'
notify() {
mailsubject="$(hostname) to be $1, vip floating"
mailbody="$(date +'%F %T'): vrrp transition, $(hostname) changed to be $1"
echo "$mailbody" | mail -s "$mailsubject" $contact
}
case $1 in
master)
notify master
;;
backup)
notify backup
;;
fault)
notify fault
;;
*)
echo "Usage: $(basename $0) {master|backup|fault}"
exit 1
;;
esac
[root@localhost opt]#vim /etc/keepalived/keepalived.conf
[root@localhost opt]#chmod +x keepalived.sh
[root@localhost opt]#ll
总用量 4
-rwxr-xr-x 1 root root 392 3月 7 16:55 keepalive.sh
[root@localhost opt]#systemctl restart keepalived
7.2模拟故障
[root@localhost keepalived]#killall keepalived
8.日志功能
[root@localhost keepalived]#keepalived --help
Usage: keepalived [OPTION...]
-f, --use-file=FILE Use the specified configuration file
-P, --vrrp Only run with VRRP subsystem
-C, --check Only run with Health-checker subsystem
-l, --log-console Log messages to local console
-D, --log-detail Detailed log messages
-S, --log-facility=[0-7] Set syslog facility to LOG_LOCAL[0-7]
-X, --release-vips Drop VIP on transition from signal.
-V, --dont-release-vrrp Don't remove VRRP VIPs and VROUTEs on daemon stop
-I, --dont-release-ipvs Don't remove IPVS topology on daemon stop
-R, --dont-respawn Don't respawn child processes
-n, --dont-fork Don't fork the daemon process
-d, --dump-conf Dump the configuration data
-p, --pid=FILE Use specified pidfile for parent process
-r, --vrrp_pid=FILE Use specified pidfile for VRRP child process
-c, --checkers_pid=FILE Use specified pidfile for checkers child process
-a, --address-monitoring Report all address additions/deletions notified via netlink
-x, --snmp Enable SNMP subsystem
-A, --snmp-agent-socket=FILE Use the specified socket for master agent
-s, --namespace=NAME Run in network namespace NAME (overrides config)
-m, --core-dump Produce core dump if terminate abnormally
-M, --core-dump-pattern=PATN Also set /proc/sys/kernel/core_pattern to PATN (default 'core')
-i, --config_id id Skip any configuration lines beginning '@' that don't match id
-v, --version Display the version number
-h, --help Display this help message
[root@localhost opt]#vim /etc/sysconfig/keepalived
KEEPALIVED_OPTIONS="-D -S 6"
[root@localhost keepalived]#vim /etc/rsyslog.conf
[root@localhost keepalived]#systemctl restart rsyslog.service
[root@localhost keepalived]#ls /opt
keepalive.sh
[root@localhost keepalived]#systemctl restart keepalived.service
[root@localhost keepalived]#ls /opt
keepalive.sh log
[root@localhost keepalived]#ls /opt/log/keepalived.log
/opt/log/keepalived.log
[root@localhost keepalived]#ls /opt/log
keepalived.log
[root@localhost keepalived]#cat /opt/log/keepalived.log
四、脑裂——实现其他应用的高可用性VRRP Script
keepalived利用 VRRP Script 技术,可以调用外部的辅助脚本进行资源监控,并根据监控的结果实现优先动态调整,从而实现其它应用的高可用性功能
#参考配置文件
/usr/share/doc/keepalived/keepalived.conf.vrrp.localcheck
1.脑裂的定义
在高可用(HA)系统中,当联系2个节点的“心跳线”断开时,本来为一整体、动作协调的HA系统,就分裂成为两个独立的个体。由于相互失去了联系,都以为是对方出了故障。两个节点上的HA软件像“裂脑人”一样,争抢“共享资源”、争起“应用服务”,就会发生严重后果——或者共享资源被瓜分、两边“服务”都起不来了;或者两边“服务”都起来了,但同时读写“共享存储”,导致数据损坏(常见如数据库轮询着的联机日志出错)。
对付HA系统“裂脑”的对策,目前达成共识的的大概有以下几条:
- 添加冗余的心跳线,例如:双线条线(心跳线也HA),尽量减少“裂脑”发生几率;
- 启用磁盘锁。正在服务一方锁住共享磁盘,“裂脑”发生时,让对方完全“抢不走”共享磁盘资源。但使用锁磁盘也会有一个不小的问题,如果占用共享盘的一方不主动“解锁”,另一方就永远得不到共享磁盘。现实中假如服务节点突然死机或崩溃,就不可能执行解锁命令。后备节点也就接管不了共享资源和应用服务。于是有人在HA中设计了“智能”锁。即:正在服务的一方只在发现心跳线全部断开(察觉不到对端)时才启用磁盘锁。平时就不上锁了。
- 设置仲裁机制。例如设置参考IP(如网关IP),当心跳线完全断开时,2个节点都各自ping一下参考IP,不通则表明断点就出在本端。不仅“心跳”、还兼对外“服务”的本端网络链路断了,即使启动(或继续)应用服务也没有用了,那就主动放弃竞争,让能够ping通参考IP的一端去起服务。更保险一些,ping不通参考IP的一方干脆就自我重启,以彻底释放有可能还占用着的那些共享资源
2.脑裂的原因
- 高可用服务器对之间心跳线链路发生故障,导致无法正常通信
- 因心跳线坏了(包括断了,老化)
- 因网卡及相关驱动坏了,ip配置及冲突问题(网卡直连)
- 因心跳线间连接的设备故障(网卡及交换机)
- 因仲裁的机器出问题(采用仲裁的方案)
- 高可用服务器上开启了 iptables防火墙阻挡了心跳消息传输
- 高可用服务器上心跳网卡地址等信息配置不正确,导致发送心跳失败
- 其他服务配置不当等原因,如心跳方式不同,心跳广插冲突、软件Bug等
3.如何解决Keepalive脑裂问题
- 同时使用串行电缆和以太网电缆连接、同时使用两条心跳线路,这样一条线路断了,另外一条还是好的,依然能传送心跳消息
- 当检查脑裂时强行关闭一个心跳节点(这个功能需要特殊设备支持,如stonith、fence)相当于备节点接收不到心跳消息,通过单独的线路发送关机命令关闭主节点的电源
4.模拟脑裂
[root@node2 keepalived]#iptables -A INPUT -s 192.168.241.11 -j REJECT
[root@node2 keepalived]#ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 00:0c:29:3e:a0:08 brd ff:ff:ff:ff:ff:ff
inet 192.168.241.22/24 brd 192.168.241.255 scope global ens33
valid_lft forever preferred_lft forever
inet6 fe80::d9cd:6857:3bdc:7454/64 scope link
valid_lft forever preferred_lft forever
3: virbr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN qlen 1000
link/ether 52:54:00:fe:22:f2 brd ff:ff:ff:ff:ff:ff
inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0
valid_lft forever preferred_lft forever
4: virbr0-nic: <BROADCAST,MULTICAST> mtu 1500 qdisc pfifo_fast master virbr0 state DOWN qlen 1000
link/ether 52:54:00:fe:22:f2 brd ff:ff:ff:ff:ff:ff
[root@node2 keepalived]#ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 00:0c:29:3e:a0:08 brd ff:ff:ff:ff:ff:ff
inet 192.168.241.22/24 brd 192.168.241.255 scope global ens33
valid_lft forever preferred_lft forever
inet 192.168.241.111/32 scope global ens33
valid_lft forever preferred_lft forever
inet6 fe80::d9cd:6857:3bdc:7454/64 scope link
valid_lft forever preferred_lft forever
3: virbr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN qlen 1000
link/ether 52:54:00:fe:22:f2 brd ff:ff:ff:ff:ff:ff
inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0
valid_lft forever preferred_lft forever
4: virbr0-nic: <BROADCAST,MULTICAST> mtu 1500 qdisc pfifo_fast master virbr0 state DOWN qlen 1000
link/ether 52:54:00:fe:22:f2 brd ff:ff:ff:ff:ff:ff
[root@localhost keepalived]#ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether 00:0c:29:51:4b:b5 brd ff:ff:ff:ff:ff:ff
inet 192.168.241.11/24 brd 192.168.241.255 scope global ens33
valid_lft forever preferred_lft forever
inet 192.168.241.111/32 scope global ens33
valid_lft forever preferred_lft forever
inet6 fe80::de6f:32c8:5a64:a6b2/64 scope link
valid_lft forever preferred_lft forever
3: virbr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN qlen 1000
link/ether 52:54:00:53:c1:45 brd ff:ff:ff:ff:ff:ff
inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0
valid_lft forever preferred_lft forever
4: virbr0-nic: <BROADCAST,MULTICAST> mtu 1500 qdisc pfifo_fast master virbr0 state DOWN qlen 1000
link/ether 52:54:00:53:c1:45 brd ff:ff:ff:ff:ff:ff
但是不影响客户端访问
5.VRRP Script配置
5.1配置VRRP Script
5.1.1定义脚本
vrrp_script:自定义资源监控脚本,vrrp实例根据脚本返回值,公共定义,可被多个实例调用,定义在vrrp实例之外的独立配置块,一般放在global_defs设置块之后。通常此脚本用于监控指定应用的状态。一旦发现应用的状态异常,则触发对MASTER节点的权重减至低于SLAVE节点,从而实现 VIP 切换到 SLAVE 节点
vrrp_script <SCRIPT_NAME> {
script <STRING>|<QUOTED-STRING> #此脚本返回值为非0时,会触发下面OPTIONS执行
OPTIONS
}
5.1.2调用脚本
track_script:调用vrrp_script定义的脚本去监控资源,定义在VRRP实例之内,调用事先定义的vrrp_script
track_script {
SCRIPT_NAME_1
SCRIPT_NAME_2
}
5.2定义VRRP Script
vrrp_script <SCRIPT_NAME> { #定义一个检测脚本,在global_defs 之外配置
script <STRING>|<QUOTED-STRING> #shell命令或脚本路径(注意执行权限)
interval <INTEGER> #间隔时间,单位为秒,默认1秒
timeout <INTEGER> #超时时间
weight <INTEGER:-254..254> #默认为0,如果设置此值为负数,当上面脚本返回值为非0时,会将此值与本节点权重相加可以降低本节点权重,即表示fall. 如果是正数,当脚本返回值为0,会将此值与本节点权重相加可以提高本节点权重,即表示 rise.通常使用负值
fall <INTEGER> #执行脚本连续几次都失败,则转换为失败,建议设为2以上
rise <INTEGER> #执行脚本连续几次都成功,把服务器从失败标记为成功
user USERNAME [GROUPNAME] #执行监测脚本的用户或组
init_fail #设置默认标记为失败状态,监测成功之后再转换为成功状态
}
6.实际操作
[root@localhost keepalived]#systemctl stop ipvsadm.service
[root@localhost keepalived]#yum install epel-release.noarch -y
[root@localhost keepalived]#yum install nginx -y
[root@localhost keepalived]#systemctl start nginx
[root@localhost keepalived]#systemctl status nginx
[root@node2 keepalived]#systemctl stop ipvsadm.service
[root@node2 keepalived]#yum install epel-release.noarch -y
[root@node2 keepalived]#yum install nginx -y
[root@node2 keepalived]#systemctl start nginx
[root@node2 keepalived]#systemctl status nginx
[root@localhost keepalived]#vim /etc/nginx/nginx.conf
[root@localhost keepalived]#systemctl restart nginx
[root@localhost keepalived]#scp /etc/nginx/nginx.conf 192.168.241.22:/etc/nginx/nginx.conf
root@192.168.241.22's password:
nginx.conf 100% 2477 991.6KB/s 00:00
[root@localhost keepalived]#vim /etc/keepalived/nginx.sh
[root@localhost keepalived]#cat /etc/keepalived/nginx.sh
#!/bin/bash
killall -0 nginx
[root@localhost keepalived]#chmod +x /etc/keepalived/nginx.sh
[root@localhost keepalived]#ll /etc/keepalived/
总用量 12
-rw-r--r-- 1 root root 1316 3月 7 17:36 keepalived.conf
-rw-r--r-- 1 root root 3598 3月 7 14:57 keepalived.conf.bak
-rwxr-xr-x 1 root root 29 3月 7 18:43 nginx.sh
[root@localhost keepalived]#vim /etc/keepalived/keepalived.conf
[root@localhost keepalived]#scp /etc/keepalived/keepalived.conf 192.168.241.22:/etc/keepalived
root@192.168.241.22's password:
keepalived.conf 100% 1370 990.0KB/s 00:00
[root@localhost keepalived]#systemctl restart keepalived.service
[root@node2 keepalived]#systemctl restart keepalived.service
测试
如果将Centos7-1关机的话,Centos7-2将直接成为主
[root@localhost keepalived]#systemctl stop nginx