目录
一、高可用集群
1.1 集群的类型
1.2 实现高可用
1.3 VRRP:Virtual Router Redundancy Protocol
1.3.1 VRRP相关术语
1.5.2 VRRP 相关技术
二、部署KEEPALIVED
2.1 keepalived 简介
2.2 Keepalived 架构
2.3 Keepalived 环境准备
2.3.1 实验环境
2.4 Keepalived 相关文件
2.5 安装软件
2.6 KeepAlived 配置说明
2.6.1 配置虚拟路由
2.6.2 示例: keepalived配置文件中配置虚拟路由
2.6.2.1 抓包命令查看组播信息:
2.6.3 实现独立子配置文件
三.Keepalived 应用示例
3.1 实现master/slave的 Keepalived 单主架构
3.1.1 配置MASTER
3.1.2 配置BACKUP
3.1.3 抓包查看
3.2 抢占模式和非抢占模式
3.2.1 非抢占模式
3.2.2 抢占延迟模式
3.3vip单播模式
3.3.1 抓包命令查看: ka2上看不到,因为没有vip
3.4 keepalived实现邮件通知
3.5 实现keepalived的双主架构 master/master
3.6 实现IPVS的高可用性(DR模式)
3.6.1 应用层监测
3.6.2 TCP监测
3.6.3 实现单主的 LVS-DR 模式
3.7 vrrp Script 配置 脚本控制vip
3.7.1 利用脚本实现主从角色切换
3.8 keepalived-haproxy实现高可用
3.8.1 还原状态
3.8.2 实现haproxy高可用
3.8.3 测试
一、高可用集群
1.1 集群的类型
- LB:Load Balance 负载均衡
- LVS/HAProxy/nginx(http/upstream, stream/upstream)
- HA:High Availability 高可用集群 数据库、Redis
- SPoF: Single Point of Failure,解决单点故障
- HPC:High Performance Computing 高性能集群
1.2 实现高可用
- active/passive 主/备
- active/active 双主
- active --> HEARTBEAT --> passive
- active <--> HEARTBEAT <--> active
1.3 VRRP:Virtual Router Redundancy Protocol
- 物理层:路由器、三层交换机
- 软件层:keepalived
1.3.1 VRRP相关术语
- 虚拟路由器:Virtual Router
- 虚拟路由器标识:VRID(0-255),唯一标识虚拟路由器
- VIP:Virtual IP
- VMAC:Virutal MAC (00-00-5e-00-01-VRID)
- 物理路由器:
- master:主设备
- backup:备用设备
- priority:优先级
1.5.2 VRRP 相关技术
- 无认证
- 简单字符认证:预共享密钥
- MD5
- 主/备:单虚拟路由器
- 主/主:主/备(虚拟路由器1),备/主(虚拟路由器2)
二、部署KEEPALIVED
2.1 keepalived 简介
- 基于vrrp协议完成地址流动
- 为vip地址所在的节点生成ipvs规则(在配置文件中预先定义)
- 为ipvs集群的各RS做健康状态检测
- 基于脚本调用接口完成脚本中定义的功能,进而影响集群事务,以此支持nginx、haproxy等服务
2.2 Keepalived 架构
- 用户空间核心组件:
- 控制组件:提供keepalived.conf 的解析器,完成Keepalived配置
- IO复用器:针对网络目的而优化的自己的线程抽象
- 内存管理组件:为某些通用的内存管理功能(例如分配,重新分配,发布等)提供访问权限
2.3 Keepalived 环境准备
2.3.1 实验环境
主机名 | IP |
ka1 | 172.25.254.10/24 |
ka2 | 172.25.254.20/24 |
realserver1 | 172.25.254.110/24 |
realserver2 | 172.25.254.120/24 |
- 各节点时间必须同步:ntp, chrony
- 关闭防火墙及SELinux
- 各节点之间可通过主机名互相通信:非必须
- 建议使用/etc/hosts文件实现:非必须
- 各节点之间的root用户可以基于密钥认证的ssh服务完成互相通信:非必须
2.4 Keepalived 相关文件
- 软件包名:keepalived
- 主程序文件:/usr/sbin/keepalived
- 主配置文件:/etc/keepalived/keepalived.conf
- 配置文件示例:/usr/share/doc/keepalived/
- Unit File:/lib/systemd/system/keepalived.service
- Unit File的环境配置文件:/etc/sysconfig/keepalived
2.5 安装软件
[root@ka1 ~]# dnf install keepalived -y[root@ka1 ~]# systemctl start keepalived
2.6 KeepAlived 配置说明
配置文件:/etc/keepalived/keepalived.conf
- GLOBAL CONFIGURATION
- VRRP CONFIGURATION
- LVS CONFIGURATION
2.6.1 配置虚拟路由
配置master:
vim /etc/keepalived/keepalived.confglobal_defs {notification_email {2961953033@qq.com #keepalived 发生故障切换时邮件发送的目标邮箱,可以按行区分写多个}notification_email_from keepalived@ka1.exam.org #发邮件的地址smtp_server 127.0.0.1 #邮件服务器地址smtp_connect_timeout 30 #邮件服务器连接timeoutrouter_id ka1.exam.org #每个keepalived主机唯一标识,多节点 重名不影响vrrp_skip_check_adv_addr #对所有通告报文都检查,会比较消耗性能 启用此配置后,如果收到的通告报文和上 一个报文是同一个路由器,则跳过检查, 默认值为全检查#vrrp_strict #严格遵循vrrp协议#启用此项后以下状况将无法启动服务:1.无VIP地址2.配置了单播邻居3.在VRRP版本2中有IPv6地址#建议不加此项配置vrrp_garp_interval 0 #报文发送延迟,0表示不延迟vrrp_gna_interval 0 #消息发送延迟vrrp_mcast_group4 224.0.0.18 #指定组播IP地址范围:
}
配置slave:
vrrp_instance VI_1 {state MASTERinterface eth0 #绑定为当前虚拟路由器使用的物理接口,如:eth0,可以和 VIP不在一个网卡virtual_router_id 100 #每个虚拟路由器惟一标识,范围:0-255,每个虚拟路由器此 值必须唯一,否则服务无法启动同属一个虚拟路由器的多个keepalived节点必须相同务必要确认在同一网络中此值必须唯一priority 100 #当前物理节点在此虚拟路由器的优先级,范围:1-254值越大优先级越高,每个keepalived主机节点此值不同advert_int 1 #vrrp通告的时间间隔,默认1sauthentication { #认证机制auth_type PASS #PASS为简单密码(建议使用)auth_pass 1111 #预共享密钥,仅前8位有效同一个虚拟路由器的多个keepalived节点必须一样}virtual_ipaddress { #虚拟IP,生产环境可能指定上百个IP地址172.25.250.100/24 dev eth0 label eth0:1 #<IPADDR> dev <STRING> scope <SCOPE> label <LABEL>#指定VIP,不指定网卡,默认为eth0,注意:不指 定/prefix,默认32}
}
2.6.2 示例: keepalived配置文件中配置虚拟路由
ka1和ka2中安装软件包:yum install keepalived -y主配置文件:/etc/keepalived/keepalived.conf默认为抢占模式:
vim /etc/keepalived/keepalived.conf
配置全局参数:
global_defs {notification_email {2961953033@qq.com18883572121@163.com}notification_email_from keepalived@timingding.orgsmtp_server 127.0.0.1smtp_connect_timeout 30router_id ka1.timingding.orgvrrp_skip_check_adv_addrvrrp_strictvrrp_garp_interval 0vrrp_gna_interval 0vrrp_mcast_group4 224.0.0.18
}配置vip:
vrrp_instance VI_1 {state MASTERinterface eth0virtual_router_id 100priority 100advert_int 1nopreemptauthentication {auth_type PASSauth_pass 1111}virtual_ipaddress {172.25.254.100/24 dev eth0 label eth0:1}
}systemctl start keepalived.service
查看:
这里简便快捷一下:使用scp命令直接传到另一台ka上面
直接传到ka2上面:
scp /etc/keepalived/keepalived.conf root@172.25.254.20:/etc/keepalived/keepalived.conf
ka2上面的优先级设置的低一些:
ka2配置vip的地方要改优先级:vrrp_instance VI_1 {state BACKUP ------ 改为BACKUPinterface eth0virtual_router_id 100priority 80 ------ 改的比ka1低一些advert_int 1authentication {auth_type PASSauth_pass 1111}
2.6.2.1 抓包命令查看组播信息:
tcpdump -i eth0 -nn host 224.0.0.18
此时ping 172.25.254.100 不通。
命令查看:
iptables -nL
想ping通,得在全局配置里面加参数 vrrp_iptables,不想加就可以把vrrp_strict注释掉,要么都开,要么就注释掉
ka1和ka2都要加
global_defs {notification_email {2961953033@qq.com18883572121@163.com}notification_email_from keepalived@timingding.orgsmtp_server 127.0.0.1smtp_connect_timeout 30router_id ka1.timingding.orgvrrp_skip_check_adv_addrvrrp_strictvrrp_garp_interval 0vrrp_gna_interval 0vrrp_mcast_group4 224.0.0.18vrrp_iptables ------ 加这个参数
}此时就可以ping通172.25.254.100了。
2.6.3 实现独立子配置文件
vim /etc/keepalived/keepalived.conf
把虚拟路由注释掉:
#vrrp_instance VI_1 {
# state MASTER
# interface eth0
# virtual_router_id 100
# priority 100
# advert_int 1
# authentication {
# auth_type PASS
# auth_pass 1111
# }
# virtual_ipaddress {
# 172.25.254.100/24 dev eth0 label eth0:1
# }
#}
加上
include "/etc/keepalived/conf.d/*.conf"vim /etc/keepalived/conf.d/172.25.254.100.confvrrp_instance VI_1 {state MASTERinterface eth0virtual_router_id 100priority 100advert_int 1authentication {auth_type PASSauth_pass 1111}virtual_ipaddress {172.25.254.100/24 dev eth0 label eth0:1}
}systemctl restart keepalived.service
三.Keepalived 应用示例
3.1 实现master/slave的 Keepalived 单主架构
3.1.1 配置MASTER
[root@ka1 ~]# vim /etc/keepalived/keepalived.confglobal_defs {notification_email {2961953033@qq.com18883572121@163.com}notification_email_from keepalived@timingding.orgsmtp_server 127.0.0.1smtp_connect_timeout 30router_id ka1.timingding.orgvrrp_skip_check_adv_addr#vrrp_strict vrrp_garp_interval 0vrrp_gna_interval 0#vrrp_mcast_group4 224.0.0.18#vrrp_iptables
}vrrp_instance VI_1 {state MASTERinterface eth0virtual_router_id 100priority 100advert_int 1#preempt_delay 5sauthentication {auth_type PASSauth_pass 1111}virtual_ipaddress {172.25.254.100/24 dev eth0 label eth0:1}
}
3.1.2 配置BACKUP
[root@ka2 ~]# vim /etc/keepalived/keepalived.conf global_defs {notification_email {2961953033@qq.com18883572121@163.com}notification_email_from keepalived@ka2.timingding.orgsmtp_server 127.0.0.1smtp_connect_timeout 30router_id ka2.timingding.orgvrrp_skip_check_adv_addr#vrrp_strictvrrp_garp_interval 0vrrp_gna_interval 0vrrp_mcast_group 224.0.0.18#vrrp_iptables
}vrrp_instance VI_1 {state BACKUPinterface eth0virtual_router_id 100priority 80advert_int 1#preempt_delay 5sauthentication {auth_type PASSauth_pass 1111}virtual_ipaddress {172.25.254.100/24 dev eth0 label eth0:1}
}
3.1.3 抓包查看
tcpdump -i eth0 -nn host 224.0.0.18
3.2 抢占模式和非抢占模式
默认为抢占模式preempt,即当高优先级的主机恢复在线后,会抢占低先级的主机的 master 角色, 这样会使vip 在 KA 主机中来回漂移,造成网络抖动,建议设置为非抢占模式 nopreempt ,即高优先级主机恢复后,并不会抢占低优先级主机的 master 角色非抢占模块下,如果原主机 down 机 , VIP 迁移至的新主机 , 后续也发生 down 时 , 仍会将 VIP 迁移回原主机
master角色下线后,会转移到另一台主机后,master角色重新恢复之后,不会因为优先级抢占vip
3.2.1 非抢占模式
使用参数:nopreempt
ka1和ka2一样:
vim /etc/keepalived/keepalived.confvrrp_instance VI_1 {state BACKUP ---- 也得设置为BACKUP,两台都为备机interface eth0virtual_router_id 100 priority 100advert_int 1nopreempt ------- 加上这个参数authentication {auth_type PASSauth_pass 1111}virtual_ipaddress {172.25.254.100/24 dev eth0 label eth0:1}
}
3.2.2 抢占延迟模式
使用参数:preempt_delay
ka1主机上:ka2也一样
vim /etc/keepalived/keepalived.confvrrp_instance VI_1 {state BACKUPinterface eth0virtual_router_id 100priority 100advert_int 1preempt_delay 5s ---- 抢占延时为5s,一般设定一分钟,五分钟,这里演示。authentication {auth_type PASSauth_pass 1111}virtual_ipaddress {172.25.254.100/24 dev eth0 label eth0:1}
3.3vip单播模式
ka1主机上:ka2也一样
vim /etc/keepalived/keepalived.confvrrp_instance VI_1 {state BACKUPinterface eth0virtual_router_id 100priority 80advert_int 1authentication {auth_type PASSauth_pass 1111}virtual_ipaddress {172.25.254.100/24 dev eth0 label eth0:1}unicast_src_ip 172.25.254.10 ---- 指定谁到谁 ka2上相反unicast_peer {172.25.254.20}
3.3.1 抓包命令查看: ka2上看不到,因为没有vip
[root@ka1 ~]# tcpdump -i eth0 -nn src host 172.25.254.10 and dst 172.25.254.20
3.4 keepalived实现邮件通知
安装邮件发送工具:
dnf install mailx -yQQ邮箱配置:
vim /etc/mail.rcset from=2961953033@qq.com
set smtp=smtp.qq.com
set smtp-auth-user=2961953033@qq.com
set smtp-auth-password=gyzvbkvimprddecb ---- 去QQ邮箱上面登录弄授权码
set smtp-auth=login
set ssl-verify=ignore测试发送邮件:
echo test message |mail -s test 2961953033@qq.com写个脚本:
vim /etc/keepalived/mail.sh#!/bin/bash
mail_dst="2961953033@qq.com"
send_message()
{mail_sub="HOSTNAME to be $1 vip mopve"mail_msg="`date +%F\ %T`: vrrp move $HOSTNAME chage $1"echo $mail_msg | mail -s "$mail_sub" $mail_dst
}case $1 inmaster)send_message master;;backup)send_message backup;;fault)send_message fault;;*);;
esacchmod +x /etc/keepalived/mail.sh在ka1和ka2虚拟路由上面添加节点:
vim /etc/keepalived/keepalived.confvrrp_instance VI_1 {state BACKUPinterface eth0virtual_router_id 100priority 80advert_int 1#preempt_delay 5sauthentication {auth_type PASSauth_pass 1111}virtual_ipaddress {172.25.254.100/24 dev eth0 label eth0:1}unicast_src_ip 172.25.254.20unicast_peer {172.25.254.10}notify_master "/etc/keepalived/notify.sh master" ----- 添加这三行notify_backup "/etc/keepalived/notify.sh backup"notify_fault "/etc/keepalived/notify.sh fault"
}
3.5 实现keepalived的双主架构 master/master
master/slave的单主架构,同一时间只有一个Keepalived 对外提供服务,此主机繁忙,而另一台主机却 很空闲,利用率低下,可以使用master/master 的双主架构,解决此问题。master/master 的双主架构:即将两个或以上VIP分别运行在不同的 keepalived 服务器,以实现服务器并行提供 web 访问的目的,提高 服务器资源利用率
ka1和ka2上都加上,都有一个主备
vim /etc/keepalived/keepalived.confvrrp_instance VI_2 {state MASTWEinterface eth0virtual_router_id 200priority 80 ----- 优先级不能一样advert_int 1#preempt_delay 5sauthentication {auth_type PASSauth_pass 1111}virtual_ipaddress {172.25.254.200/24 dev eth0 label eth0:2}unicast_src_ip 172.25.254.10unicast_peer {172.25.254.20}
}
3.6 实现IPVS的高可用性(DR模式)
3.6.1 应用层监测
HTTP_GET|SSL_GET {url {path <URL_PATH> #-----------定义要监控的URLstatus_code <INT> #---------判断上述检测机制为健康状态的响应码,一般为 200}connect_timeout <INTEGER> #---------客户端请求的超时时长, 相当于haproxy的timeout servernb_get_retry <INT> #---------重试次数delay_before_retry <INT> #---------重试之前的延迟时长connect_ip <IP ADDRESS> #---------向当前RS哪个IP地址发起健康状态检测请求connect_port <PORT> #--------向当前RS的哪个PORT发起健康状态检测请求bindto <IP ADDRESS> #---------向当前RS发出健康状态检测请求时使用的源地址bind_port <PORT> #----------向当前RS发出健康状态检测请求时使用的源端口
}
3.6.2 TCP监测
TCP_CHECK {connect_ip <IP ADDRESS> #----------向当前RS的哪个IP地址发起健康状态检测请求connect_port <PORT> #---------向当前RS的哪个PORT发起健康状态检测请求bindto <IP ADDRESS> #--------发出健康状态检测请求时使用的源地址bind_port <PORT> #---------发出健康状态检测请求时使用的源端口connect_timeout <INTEGER> #------客户端请求的超时时长#-------等于haproxy的timeout server
}
3.6.3 实现单主的 LVS-DR 模式
永久设定vip:
vim /etc/sysconfig/network-scripts/ifcfg-lo 加上:
IPADDR1=172.25.254.100
NETMASK1=255.255.255.255
NETWORK=127.0.0.0这里我们给上临时的就行:
reslserver1和2上面给上vip(临时)
ip a a 172.25.254.100 dev lo解决响应问题:
查看arp:
sysctl -a | grep arpka1上面
vim /etc/sysctl.d/arp.conf
net.ipv4.conf.all.arp_ignore=1
net.ipv4.conf.all.arp_announce=2
net.ipv4.conf.lo.arp_announce=2
net.ipv4.conf.lo.arp_ignore=1用scp命令写到ka2上面:
scp /etc/sysctl.d/arp.conf root@172.25.254.110:/etc/sysctl.d/arp.confka1和ka2上面安装ipvsadm 用来查看策略:
yum install ipvsadm -yka1和ka2一样:
vim /etc/keepalived/keepalived.confvirtual_server 172.25.254.100 80 {delay_loop 6lb_algo wrrlb_kind DR#persistence_timeout 50protocol TCPreal_server 172.25.254.110 80 {weight 1HTTP_GET {url {path /status_code 200}connect_timeout 3nb_get_retry 2delay_before_retry 2}}real_server 172.25.254.120 80 {weight 1HTTP_GET {url {path /status_code 200}connect_timeout 3nb_get_retry 2delay_before_retry 2}}
}ssytemctl restart keepalived.service使用ipvsadm -Ln 查看策略是否写进去
3.7 vrrp Script 配置 脚本控制vip
3.7.1 利用脚本实现主从角色切换
示例:先用例子测试一下:
检测文件是否存在,存在为真(1),ka1就降低优先级,vip飘到ka2上面,不存在为假(0),vip就按兵不动
[root@ka1 ~]# vim /etc/keepalived/test.sh
#!/bin/bash
[ ! -f /mnt/ding ] ---- 取反chmod +x /etc/keepalived/test.sh ---- 给上可执行权限[root@ka1 ~]# vim /etc/keepalived/keepalived.confvrrp_script check_file {script "/etc/keepalived/test.sh" ------shell命令或脚本的路径interval 1 ------ 检测的间隔时间,默认为1sweight -30 ------ 默认为零,这里测试设置-30,当运行脚本检测到之后,为非0,就会与VIP的优先级相加,优先 降低,实现VIP飘移,为0时就VIP就按兵不动。fall 2 ------ 脚本连续几次都执行失败,则转换为失败,建议设置为2以上rise 2 ------ 执行脚本连续几次都成功,把服务器标记为成功timeout 2 ------ 设置的超时时间。
}vrrp_instance VI_1 {state MASTERinterface eth0virtual_router_id 100priority 100advert_int 1#preempt_delay 5sauthentication {auth_type PASSauth_pass 1111}virtual_ipaddress {172.25.254.100/24 dev eth0 label eth0:1}unicast_src_ip 172.25.254.10unicast_peer {172.25.254.20}track_script { ------ 调用这个script配置参数check_file}
}[root@ka1 ~]# systemctl restart keepalived.service sh /etc/keepalived/test.sh
echo $?0
此时/mnt底下没有ding这个文件,运行脚本输出0,VIP就按兵不动
此时在/mnt底下创建ding这个文件,输出为非0,执行脚本,此时vip飘到ka2上面
touch /mnt/ding
sh /etc/keepalived/test.sh
echo $?1
ka1上面:
ka2上面:
删除这个文件之后,VIP就自动飘回来了
rm -rf /mnt/ding
systemctl restart keepalived.serviceifconfig
3.8 keepalived-haproxy实现高可用
3.8.1 还原状态
两台ka1和ka2上安装haproxy
yum install haproxy -y把两台server机上面的VIP和arp响应给去掉:
[root@realserver1 ~]# ip a d 172.25.254.100/32 dev lo
[root@realserver2 ~]# ip a d 172.25.254.100/32 dev loarp响应:
[root@realserver1 ~]# vim /etc/sysctl.d/arp.conf
[root@realserver2 ~]# vim /etc/sysctl.d/arp.confnet.ipv4.conf.all.arp_ignore=0
net.ipv4.conf.all.arp_announce=0
net.ipv4.conf.lo.arp_announce=0
net.ipv4.conf.lo.arp_ignore=0sysctl --system ---- 查看ka1和ka2上都开启内核路由:
[root@ka1 ~]# vim /etc/sysctl.conf
[root@ka2 ~]# vim /etc/sysctl.conf et.ipv4.ip_nonlocal_bind=1 ---- 两台都开启内核路由sysctl -p ----- 刷新一下记住要把之前做的LVS给注释掉,lvs和haproxy不能同时使用!!!!
[root@ka1 ~]# vim /etc/haproxy/haproxy.cfg
[root@ka2 ~]# vim /etc/haproxy/haproxy.cfglisten webclusterbind 172.25.254.100:80mode httpbalance roundrobinserver web1 172.25.254.110:80 check inter 3 fall 2 rise 5server web1 172.25.254.120:80 check inter 3 fall 2 rise 5查看端口是否开启:
[root@ka1 ~]# netstat -antlupe | grep haproxy
tcp 0 0 172.25.254.100:80 0.0.0.0:* LISTEN 0 2636800 5709/haproxy
tcp 0 0 0.0.0.0:5000 0.0.0.0:* LISTEN 0 2636798 5709/haproxy
udp 0 0 0.0.0.0:35424 0.0.0.0:* 0 2636799 5708/haproxy
[root@ka1 ~]#
3.8.2 实现haproxy高可用
如何检测haproxy的存活状态:
killall -0 haproxy写入之前示例的那个脚本:
[root@ks2 ~]# vim /etc/keepalived/test.sh
#!/bin/bash
killall -0 haproxy[root@ka1 ~]# vim /etc/keepalived/keepalived.conf
[root@ka2~]# vim /etc/keepalived/keepalived.conf vrrp_script check_haproxy {script "/etc/keepalived/test.sh" interval 1 weight -30 fall 2 rise 2 timeout 2
}vrrp_instance VI_1 {state MASTERinterface eth0virtual_router_id 100priority 100advert_int 1#preempt_delay 5sauthentication {auth_type PASSauth_pass 1111}virtual_ipaddress {172.25.254.100/24 dev eth0 label eth0:1}unicast_src_ip 172.25.254.10unicast_peer {172.25.254.20}track_script { check_haproxy}
}systemctl restart keepalived.service
3.8.3 测试
复制一个ka1出来,写个死循环:
while true
>do
>curl 172.25.254.100;sleep 0.5
>done把ka1上的haproxy挂掉,这边不会受影响,一直循环,VIP会飘到ka2上面
把ka1上面的haproxy给挂掉:
[root@ka1 ~]# systemctl stop haproxy.service
此时ka1上面没有VIP了
查看ka2上面:
把ka1上面的haproxy重新启动,VIP又回来了
systemctl start haproxy.service死循环也不会受影响,因为ka1上面haproxy挂掉了,ka1上的VIP优先级降低,飘到ka2上面去了,ka2上面的haproxy还存活,也有VIP,所以死循环会一直能访问。最终不会影响架构的最终访问。
至此,keepalived-haproxy就实现了高可用