Nginx高可用部署
前言
目前是通过Nginx做Web服务器和负载均衡，如果Nginx服务宕掉的话，会导致所有服务都无法进行访问，需要对Nginx来做高可用，目前是通过keepalived的虚拟IP（VIP）的漂移来实现的，当一台Nginx服务器宕掉后VIP会自动漂到另一台服务器继续提供访问。
部署环境及版本
系统版本：CentOS Linux release 7.9.2009
Nginx版本：1.19.6
Keepalived版本：v1.3.5
准备
| 主机ip | 安装程序 |
|---|---|
| 192.168.102.212 | Nginx、Keepalived |
| 192.168.102.213 | Nginx、Keepalived |
1、关闭自带防火墙，用iptables,添加hosts文件解析
# 关闭SELinux
setenforce 0
sed -i 's/SELINUX=enforcing/SELINUX=disabled/g' /etc/selinux/config# 关闭Firewalld并禁止自启动
systemctl stop firewalld
systemctl disable firewalld# 安装iptables
yum install -y iptables-services# 启动iptables服务，并设置开机自启动（修改规则在/etc/sysconfig/iptables）
systemctl start iptables
systemctl enable iptables.service
2、时间同步
yum -y install ntp
systemctl enable ntpd
systemctl start ntpd
timedatectl set-timezone Asia/Shanghai
ntpdate -u time.nist.gov
ntpdate -u time.nist.gov
date
一、Nginx部署
1、基础环境部署脚本
vim Deployment_preparation.sh
#!/bin/bash
#部署前准备
yum update -y
yum install ntpdate -y
ntpdate time.windows.com
setenforce 0
sed -i 's/SELINUX=enforcing/SELINUX=disabled/g' /etc/selinux/config
systemctl stop firewalld.service
systemctl disable firewalld.service
firewall-cmd --state
systemctl list-unit-files | grep firewalld
yum install -y iptables-services
systemctl enable iptables.service
systemctl start iptables
systemctl restart iptables
2、Nginx部署脚本
#!/bin/bash
read -p "please input your Intranet address : " IP
package_PATH_="/server/tools"
yum install -y pcre pcre-devel
yum install -y zlib zlib-devel
yum install gcc-c++ -y
yum install -y openssl openssl-devel
yum install lua-devel -y
yum install -y gcc-c++ pcre pcre-devel zlib zlib-devel openssl openssl-devel wget unzip
if [ -d ${package_PATH_} ];thenecho dir ${package_PATH_} exist!
elsemkdir -p /server/tools
fi
cd $package_PATH_
if [ -e nginx-1.19.6.tar.gz ];thenecho "nginx-1.19.6.tar.gz is exist !"
elseecho "Please An installation package nginx-1.19.6.tar.gz is available"exit 24
fi
tar -zvxf nginx-1.19.6.tar.gz
useradd work
cd nginx-1.19.6
./configure --user=work --group=work --prefix=/data/nginx --with-http_v2_module --with-http_ssl_module --with-http_stub_status_module --with-http_gzip_static_module --with-pcre
make && make install
sed -i '2c user work work;' /data/nginx/conf/nginx.conf
sed -i '3c worker_processes auto;' /data/nginx/conf/nginx.conf
sed -i '9c pid logs/nginx.pid;' /data/nginx/conf/nginx.conf
sed -i '13c worker_connections 10240;' /data/nginx/conf/nginx.conf
sed -i '116c include /data/nginx/conf/vhost/*.conf;' /data/nginx/conf/nginx.conf
mkdir -pv /data/nginx/conf/vhost
cd /usr/lib/systemd/system
cat > nginx.service << EOF
[Unit]
Description=nginx
After=network.target[Service]
Type=forking
ExecStart=/data/nginx/sbin/nginx -c /data/nginx/conf/nginx.conf
ExecReload=/data/nginx/sbin/nginx -s reload
ExecStop=/data/nginx/sbin/nginx -s quit
PrivateTmp=true[Install]
WantedBy=multi-user.target
EOF
systemctl start nginx
systemctl status nginx
systemctl enable nginx
systemctl daemon-reload
echo "Cluster_Nginx is Deployment complete!"
二、Keepalived部署Nginx高可用
1、安装keepalived和killall
yum install psmisc keepalived -y
2、将原来配置备份，参考下面进行配置
Nginx的master配置
mv /etc/keepalived/keepalived.conf /etc/keepalived/keepalived.conf.bak
vim /etc/keepalived/keepalived.conf
! Configuration File for keepalived::
vrrp_script chk_nginx {
script "/usr/local/src/check_nginx_pid.sh"
interval 2 #脚本检测频率
weight -5 #脚本执行成功与否，权重怎么计算
fall 2 #如果连续两次检测失败，认为节点服务不可用
rise 1 #如果连续2次检查成功则认为节点正常
}vrrp_instance VI_1 {
state MASTER
interface ens160 #节点IP的网卡
virtual_router_id 200 #同一个instance相同
priority 212 # 优先级，数值越大，优先级越高
advert_int 1
authentication { #节点间的认证，所有的必须一致
auth_type PASS
auth_pass Hirain_ha_215
}
virtual_ipaddress { #VIP，自定的，和外网的IP要一个网段
192.168.102.99/24
}track_script { #指定前面脚本的名字
chk_nginx
}
}
Nginx的backup配置
mv /etc/keepalived/keepalived.conf /etc/keepalived/keepalived.conf.bak
vim /etc/keepalived/keepalived.conf
! Configuration File for keepalived#指定proxysql服务检测脚本
vrrp_script chk_nginx {
script "/usr/local/src/check_nginx_pid.sh"
interval 2 #脚本检测频率
weight -5 #脚本执行成功与否，权重怎么计算
fall 2 #如果连续两次检测失败，认为节点服务不可用
rise 1 #如果连续2次检查成功则认为节点正常
}vrrp_instance VI_1 {
state BACKUP
interface ens160 #节点IP的网卡
virtual_router_id 200 #同一个instance相同
priority 211 # 优先级，数值越大，优先级越高
advert_int 1
authentication { #节点间的认证，所有的必须一致
auth_type PASS
auth_pass Hirain_ha_215
}
virtual_ipaddress { #VIP，自定的，和外网的IP要一个网段
192.168.102.99/24
}track_script { #指定前面脚本的名字
chk_nginx
}
}
3、Nginx进程检测脚本
vim /usr/local/src/check_nginx_pid.sh
#!/bin/bash
A=`ps -C nginx --no-header |wc -l`
if [ $A -eq 0 ];thensystemctl start nginxsleep 5if [ `ps -C nginx --no-header |wc -l` -eq 0 ];thenkillall keepalivedfi
fi
脚本授权
chmod 755 /usr/local/src/check_nginx_pid.sh
4、启动keepalived并设置开机自启动
systemctl start keepalived
systemctl enable keepalived
三、测试
测试场景一
Nginx服务宕掉，测试Nginx是否自动重启
[root@mgr01 vhost]# ps axu|grep nginx
root 3817 0.0 0.0 46104 1160 ? Ss 10:10 0:00 nginx: master process /data/nginx/sbin/nginx -c /data/nginx/conf/nginx.conf
work 3818 0.0 0.1 50276 5612 ? S 10:10 0:00 nginx: worker process
work 3819 0.0 0.1 50276 5612 ? S 10:10 0:00 nginx: worker process
root 4143 0.0 0.0 112812 980 pts/0 S+ 10:12 0:00 grep --color=auto nginx
systemctl stop nginx
[root@mgr01 vhost]# ps axu|grep nginx
root 4302 0.0 0.0 115408 1444 ? S 10:13 0:00 /bin/bash /usr/local/src/check_nginx_pid.sh
root 4308 0.0 0.0 46104 1156 ? Ss 10:13 0:00 nginx: master process /data/nginx/sbin/nginx -c /data/nginx/conf/nginx.conf
work 4309 0.0 0.1 50276 5612 ? S 10:13 0:00 nginx: worker process
work 4310 0.0 0.1 50276 5612 ? S 10:13 0:00 nginx: worker process
root 4313 0.0 0.0 112812 980 pts/0 S+ 10:13 0:00 grep --color=auto nginx
[root@mgr01 vhost]# ps axu|grep nginx
root 4308 0.0 0.0 46104 1156 ? Ss 10:13 0:00 nginx: master process /data/nginx/sbin/nginx -c /data/nginx/conf/nginx.conf
work 4309 0.0 0.1 50276 5612 ? S 10:13 0:00 nginx: worker process
work 4310 0.0 0.1 50276 5612 ? S 10:13 0:00 nginx: worker process
root 4340 0.0 0.0 112812 980 pts/0 S+ 10:13 0:00 grep --color=auto nginx
测试结果：Nginx服务宕掉后，会自动重新启动
测试场景二
Nginx服务器宕掉后，或者是Nginx重启失败后，VIP是否会漂到另一台服务器
[root@mgr01 vhost]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00inet 127.0.0.1/8 scope host lovalid_lft forever preferred_lft foreverinet6 ::1/128 scope host valid_lft forever preferred_lft forever
2: ens160: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000link/ether 12:34:56:78:96:17 brd ff:ff:ff:ff:ff:ffinet 192.168.102.212/22 brd 192.168.103.255 scope global noprefixroute dynamic ens160valid_lft 123607sec preferred_lft 123607secinet 192.168.102.99/24 scope global ens160valid_lft forever preferred_lft foreverinet6 fe80::ef48:def0:4499:80ef/64 scope link tentative noprefixroute dadfailed valid_lft forever preferred_lft foreverinet6 fe80::37bf:4ad7:b805:198a/64 scope link tentative noprefixroute dadfailed valid_lft forever preferred_lft foreverinet6 fe80::30ad:fb50:af40:6b75/64 scope link tentative noprefixroute dadfailed valid_lft forever preferred_lft forever
[root@mgr01 vhost]# shutdown
[root@mgr02 src]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00inet 127.0.0.1/8 scope host lovalid_lft forever preferred_lft foreverinet6 ::1/128 scope host valid_lft forever preferred_lft forever
2: ens160: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000link/ether 12:34:56:78:96:18 brd ff:ff:ff:ff:ff:ffinet 192.168.102.213/22 brd 192.168.103.255 scope global noprefixroute ens160valid_lft forever preferred_lft foreverinet 192.168.102.99/24 scope global ens160valid_lft forever preferred_lft foreverinet6 fe80::ef48:def0:4499:80ef/64 scope link tentative noprefixroute dadfailed valid_lft forever preferred_lft foreverinet6 fe80::37bf:4ad7:b805:198a/64 scope link tentative noprefixroute dadfailed valid_lft forever preferred_lft foreverinet6 fe80::30ad:fb50:af40:6b75/64 scope link tentative noprefixroute dadfailed valid_lft forever preferred_lft forever
Nginx服务器宕掉后，或者是Nginx重启失败后，VIP会漂到另一台服务器
四、总结
Nginx+Keepalived,如果Nginx服务宕掉后，会先进行重启，重启失败后，会通过VIP的漂移到正常的Nginx服务器上，从而继续提供服务,恢复后，会漂到priority数值最大的Nginx服务上面。
