title: 18.Keepalived安装配置文档  
CreateTime: 2019-07-11 00:00:18  
UpdateTime: 2019-07-11 00:00:18  
CategoryName: web  
---

---
title: "18.Keepalived安装配置文档"
date: 2019-07-11T00:00:18+08:00
draft: false
tags: ["web"]
categories: ["web"]
author: "springrain"
---

**前提**
--------

### 1. 了解lo网卡与eth0网卡区别?

lo为本地网卡,但可配置多个IP,也就是后面配置的realServer.sh的配置即配置的lo网卡,为的是抑制ipv4的apr.如果不设置,即会出现,keepalived选它为realserver的时候,他会一直说,我不是这个IP啊.我不是这个IP啊.我怎么给你返回信息呢.

### 2. 了解LVS-DR的原理?

LVS有LVS-DR,LVS-NAT,LVS-TUN三种模式,DR是最常用,最好用的.keepalived配置的realserver就是与LVS的天生结合.具体可年查看下面的[12]链接了解下.

### 3. 常用工具

tcpdump监听tcp数据,以进一步调试问题所在

查看设置情况```ipvsadm -S -n```

查看负载转向情况```ipvsadm -L -n -c```

**场景**
--------

Keepalived即想处理nginx负载也想处理mysql负载,常规做法keepalived单独部署+nginx(lvs)+mysql(lvs),然后在keeplived配置80和3306端口的realServer.

另一种场景是keepalived只管理nginx且只处理高可用的情况,那样keepalived可以和nginx部署在一台,且不用设置LVS.

我们是第一种方式,但是为了节省系统资源,减少服务器数量LVS+NGINX+keepalived前端负载在同一台机器上(需要处理请求递归),LVS
MYSQL负载在另外两台服务器上.网络结构图如下:

![](/public/18/image1.png)

**下载安装ipvsadm**
-------------------
```shell
yum install ipvsadm

modprobe ip_vs

lsmod |grep ip_vs 
```

**下载安装keepalived**
----------------------
```shell
yum install popt-devel libnl libnl-devel libnfnetlink-devel

wget http://www.keepalived.org/software/keepalived-1.3.5.tar.gz

tar -zxvf keepalived-1.3.5.tar.gz

cd keepalived-1.3.5

./configure --prefix=/usr/local/keepalived
--with-kernel-dir=/usr/src/kernels/\`uname -r\`

make & make install
```
**配置**
--------

将配置文件拷贝到系统对应的目录下:
```shell

cp /usr/local/keepalived/etc/sysconfig/keepalived /etc/sysconfig/

mkdir /etc/keepalived

ln -s /usr/local/keepalived/sbin/keepalived /usr/sbin/

cp /usr/local/keepalived/etc/keepalived/keepalived.conf /etc/keepalived/

cp ./keepalived-1.3.5/keepalived/etc/init.d/keepalived /etc/init.d/

##再次强调:Keepalived的所有功能是配置keepalived.conf文件来实现的.

chkconfig keepalived on

service keepalived start

service keepalived stop

service keepalived restart

##注意:keepalive启动需要时间,稍等一会.如果出现failed: No route to host,如果你确定你的网络配置和DNS都设置正确的话,需要注释iptables的两行配置,最好是先关闭防火墙.

#-A INPUT -j REJECT --reject-with icmp-host-prohibited

#-A FORWARD -j REJECT --reject-with icmp-host-prohibited

#添加上一句

-A INPUT -p vrrp -j ACCEPT

```

**BACKUP配置**
--------------

注意修改:

router_id ###标识本节点,通常为hostname###

state ###备用节点设置为BACKUP###

priority ###备用节点必须比主节点优先级低###

**RealServer脚本**
------------------

LVS-DR模式中,后端真实服务器(RealServer)无需安装相关软件,只需要对VIP进行绑定和路由设置等一系列操作,可以理解为安装LVS的客户端.

这里整理为一个脚本:realserver,需要注册为服务,开机启动,顺序靠前.需要注意windows换行符和Linux不同,脚本容易报错可以使用以下命令去掉 \r

```shell
vi -b filename

:%s/r$//

:x
```
```realserver.sh``` 脚本如下:
```shell
#add for chkconfig
#chkconfig: 2345 70 30  ###234都是文本界面,5就是图形界面X,70启动顺序号,30系统关闭###
#description: RealServer's script  ###关于脚本的简短描述###
#processname: realserver       ###第一个进程名,后边设置自动时会用到###
#!/bin/bash
VIP=10.0.70.128 
##source /etc/rc.d/init.d/functions
case "$1" in
start)  
       ifconfig lo:0 $VIP netmask 255.255.255.255 broadcast $VIP  
       /sbin/route add -host $VIP dev lo:0  
       echo "1" >/proc/sys/net/ipv4/conf/lo/arp_ignore  
       echo "2" >/proc/sys/net/ipv4/conf/lo/arp_announce  
       echo "1" >/proc/sys/net/ipv4/conf/all/arp_ignore  
       echo "2" >/proc/sys/net/ipv4/conf/all/arp_announce  
       echo "realserver Start OK"  
       ;;  
stop)  
       ifconfig lo:0 down  
       route del $VIP >/dev/null 2>&1  
       echo "0" >/proc/sys/net/ipv4/conf/lo/arp_ignore  
       echo "0" >/proc/sys/net/ipv4/conf/lo/arp_announce  
       echo "0" >/proc/sys/net/ipv4/conf/all/arp_ignore  
       echo "0" >/proc/sys/net/ipv4/conf/all/arp_announce  
       echo "realserver Stoped"  
       ;;
       
restart)
        stop
        start  
        ;;   
        
       *)  
       echo "Usage: $0 {start|stop}"  
       exit 1  
esac  
exit 0  
```

**同一台服务器LVS+KEEPALIVED+NGINX的问题**
----------------------------------------

<font color="red">同一台做高可用即可,不建议做负载,需要制定复杂的防火墙策略

主要是因为:两台KeepAvlied都具有负载功能,同时具有realserver,会造成KeepAvlied1--KeepAvlied2--KeepAvlied1这样死循环.

需要防火墙iptable在所有的请求根据条件(如果是另一台keepalived
mac过来的就不走LVS),给iptables的mangle表做一个mark,keepalived只fwmark相匹配的才进LVS,否则就进真实的IP.
</font>

**在BACKUP10.0.70.32上执行**
```shell 
iptables -t mangle -I PREROUTING -d $VIP -p tcp -m tcp --dport 80 -m
mac ! --mac-source $33MAC -j MARK --set-mark 0x3
```
**在MASTER10.0.70.33上执行**
```shell 
iptables -t mangle -I PREROUTING -d $VIP -p tcp -m tcp --dport 80 -m
mac ! --mac-source $22MAC -j MARK --set-mark 0x4
```
**维护**
--------

```ipvsadm -ln --stats```

查看端口流量情况

查看设置情况

```ipvsadm -S -n```

查看负载转向情况

```ipvsadm -L -n -c```

**配置文件**
------------

### IP虚拟配置(主用)  
**[keepalived.conf](/public/18/master_keepalived.conf)**

```check_nginx.sh```脚本如下
```shell
#!/bin/bash
if [ "$(ps -ef | grep "nginx: master process"| grep -v grep )" == "" ]
then
 /usr/local/nginx/sbin/nginx
 sleep 5
 if [ "$(ps -ef | grep "nginx: master process"| grep -v grep )" == "" ]
 then
 killall keepalived
 fi
fi
```

### fwmark方式(NGINX_LVS_keepalivde同台时使用):  
**[keepalived.conf](/public/18/fwmark_keepalived.conf)**

**参考资料**
------------


[1] http://www.linuxvirtualserver.org/zh/lvs1.html

[2] http://www.linuxvirtualserver.org/zh/lvs2.html

[3] http://www.linuxvirtualserver.org/zh/lvs3.html

[4] http://www.linuxvirtualserver.org/zh/lvs4.html

[5] http://blog.csdn.net/m582445672/article/details/7670015

[6] http://blog.csdn.net/xyang81/article/details/52554398

[7] https://www.cnblogs.com/edisonchou/p/4281978.html

[8] http://www.linuxde.net/2013/04/13381.html

[9] http://www.linuxde.net/2012/05/10652.html

[10] http://blog.csdn.net/nimasike/article/details/53911363

[11] http://blog.csdn.net/nimasike/article/details/53911363

[12] http://blog.csdn.net/pi9nc/article/details/23380589