前言

keepalived想必运维都不陌生，其核心是通过VRRP协议来进行实现的。目前在网络上流行的使用方案大部分是keepalived+lvs对VIP进行高可用以及后端RS负载均衡。但在公有云例如阿里云上，keepalived的使用是有限制的：

  • 不能使用广播，只能使用单播
  • VPC不支持VIP，这个需要去后台提工单开启，有数量限制（好像是一个账号5个，每台ecs上只能存在一个VIP）

今天来点不一样的，使用keepalived+openapi实现服务的高可用，避免云上ecs服务器出现宕机时影响服务，尤其是自建slb的小伙伴，此处不再使用VIP~而是EIP或ENI ( •̀ ω •́ )✧

keepalived+openapi对EIP的高可用

涉及EIP的操作接口

master状态切换脚本

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
# 此处是简单示例，try..except..finally未增加，读者可以自行增加超时捕获逻辑
[root@hk01 ~]# cat >/server/scripts/eip_change.py<<'EOF'
#!/usr/bin/env python3
# -*- coding: utf-8 -*-
import sys
import time
from typing import List

from alibabacloud_vpc20160428.client import Client as Vpc20160428Client
from alibabacloud_tea_openapi import models as open_api_models
from alibabacloud_vpc20160428 import models as vpc_20160428_models


class Sample:
    def __init__(self):
        pass

    @staticmethod
    def create_client(
        access_key_id: str,
        access_key_secret: str,
    ) -> Vpc20160428Client:
        """
        使用AK&SK初始化账号Client
        @param access_key_id:
        @param access_key_secret:
        @return: Client
        @throws Exception
        """
        config = open_api_models.Config(
            access_key_id="xxxxx",
            access_key_secret="xxxxx"
        )
        # 访问的域名,使用内网vpc地址
        config.endpoint = f'vpc-vpc.cn-guangzhou.aliyuncs.com'
        return Vpc20160428Client(config)

    @staticmethod
    def main(eip_id:str,local_ecs_id:str) -> None:

        client = Sample.create_client('accessKeyId', 'accessKeySecret')
        describe_eip_addresses_request = vpc_20160428_models.DescribeEipAddressesRequest(
            region_id='cn-guangzhou',
            allocation_id=eip_id,
        )
        
        rs = client.describe_eip_addresses(describe_eip_addresses_request).to_map()

        eip_status = rs["body"]["EipAddresses"]["EipAddress"][0]["Status"]
        if eip_status == "InUse":
            ecs_id = rs["body"]["EipAddresses"]["EipAddress"][0]["InstanceId"]

            if ecs_id == local_ecs_id:
                print("=======EIP is binding=======",eip_id,ecs_id)
            else:
                print("=======Unbinding EIP=======",eip_id,ecs_id)
                unassociate_eip_address_request = vpc_20160428_models.UnassociateEipAddressRequest(
                    allocation_id=eip_id
                )
                
                client.unassociate_eip_address(unassociate_eip_address_request)
                time.sleep(5)

                print("=======Binding EIP=======",eip_id,local_ecs_id)
                associate_eip_address_request = vpc_20160428_models.AssociateEipAddressRequest(
                    region_id='cn-guangzhou',
                    allocation_id=eip_id,
                    instance_id=local_ecs_id
                )
                
                client.associate_eip_address(associate_eip_address_request)

        else:
            print("=======Binding EIP=======",eip_id,local_ecs_id)
            associate_eip_address_request = vpc_20160428_models.AssociateEipAddressRequest(
                region_id='cn-guangzhou',
                allocation_id=eip_id,
                instance_id=local_ecs_id
            )
            
            client.associate_eip_address(associate_eip_address_request)

if __name__ == '__main__':
    # print(sys.argv[1:])
    eip_id = sys.argv[1]
    local_ecs_id = sys.argv[2]
    Sample.main(eip_id,local_ecs_id)
EOF

[root@hk01 ~]# chmod +x /server/scripts/eip_change.py

部署并测试脚本

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
# 安装环境，master节点
[root@hk01 ~]# pip3 install --upgrade pip
[root@hk01 ~]# pip3 install -i https://pypi.tuna.tsinghua.edu.cn/simple alibabacloud_vpc20160428
# 安装环境，backup节点
[root@hk02 ~]# pip3 install --upgrade pip
[root@hk02 ~]# pip3 install -i https://pypi.tuna.tsinghua.edu.cn/simple alibabacloud_vpc20160428

# 测试eip切换脚本,自行将上文的脚本放在master和backup节点上
# eip未绑定使用
[root@hk01 ~]# python3 /server/scripts/eip_change.py eip-7xvt6qbyo2m4g1yftp8fs i-7xvfffujb8qdlzzbvxe6
/usr/local/lib/python3.6/site-packages/alibabacloud_openapi_util/client.py:8: CryptographyDeprecationWarning: Python 3.6 is no longer supported by the Python core team. Therefore, support for it is deprecated in cryptography and will be removed in a future release.
  from cryptography.hazmat.backends import default_backend

=======Binding EIP======= eip-7xvt6xxxxx1yftp8fs i-7xvffxxxxxlzzbvxe6


# eip已绑定且刚好是本机
[root@hk01 ~]# python3 /server/scripts/eip_change.py eip-7xvt6xxxxx1yftp8fs i-7xvffxxxxxlzzbvxe6
/usr/local/lib/python3.6/site-packages/alibabacloud_openapi_util/client.py:8: CryptographyDeprecationWarning: Python 3.6 is no longer supported by the Python core team. Therefore, support for it is deprecated in cryptography and will be removed in a future release.
  from cryptography.hazmat.backends import default_backend

=======EIP is binding======= eip-7xvt6xxxxx1yftp8fs i-7xvffxxxxxlzzbvxe6

# eip已绑定，但非本机
[root@hk01 ~]# python3 /server/scripts/eip_change.py eip-7xvt6qbyo2m4g1yftp8fs i-7xve2wh48aopzwpb9y1g
/usr/local/lib/python3.6/site-packages/alibabacloud_openapi_util/client.py:8: CryptographyDeprecationWarning: Python 3.6 is no longer supported by the Python core team. Therefore, support for it is deprecated in cryptography and will be removed in a future release.
  from cryptography.hazmat.backends import default_backend
=======Unbinding EIP======= eip-7xvt6xxxxx1yftp8fs i-7xvffxxxxxlzzbvxe6
=======Binding EIP======= eip-7xvt6xxxxx1yftp8fs i-7xve2wxxxxxwpb9y1g

master节点keepalived配置

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
[root@hk01 ~]# cat >/etc/keepalived/keepalived.conf<<'EOF'
! Configuration File for keepalived
global_defs {
    router_id hk01
}
vrrp_script check_web {
    script "/server/scripts/check_web.sh"
    interval 2
    weight 2
}
vrrp_instance VI_1 {
    state MASTER
    interface eth0
    virtual_router_id 51
    priority 100
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    # 配置单播源地址，即本机地址
    unicast_src_ip 172.16.3.225
    # 配置单播目的地址，即对端备机地址
    unicast_peer {
        172.16.3.222
    }
    track_script {
        check_web
    }
    # 此处使用master的ecsid
    notify_master "/server/scripts/eip_change.py eip-7xvt6qbxxxxxxxxftp8fs i-7xvffxxxxxxxxlzzbvxe6"
}
EOF

backup节点keepalived配置

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
[root@hk02 ~]# cat >/etc/keepalived/keepalived.conf<<'EOF'
! Configuration File for keepalived
global_defs {
    router_id hk02
}
vrrp_script check_web {
    script "/server/scripts/check_web.sh"
    interval 2
    weight 2
}
vrrp_instance VI_1 {
    state MASTER
    interface eth0
    virtual_router_id 51
    priority 50
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    # 配置单播源地址，即本机地址
    unicast_src_ip 172.16.3.222
    # 配置单播目的地址，即对端备机地址
    unicast_peer {
        172.16.3.225
    }
    track_script {
        check_web
    }
    # 此处使用backup的ecsid
    notify_master "/server/scripts/eip_change.py eip-7xvt6qbxxxxxxxxftp8fs i-7xve2wxxxxxwpb9y1g"
}
EOF

故障切换测试

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
# 1.准备一个客户机,配置hosts解析
[root@test ~]# echo 8.xx.xx.122 eiplb.xadocker.cn >> /etc/hosts
[root@test ~]# curl eiplb.xadocker.cn --connect-timeout 1
hk01
[root@test ~]# for i in `seq 100`;do curl eiplb.xadocker.cn --connect-timeout 1 ; sleep 1;done

# 2.停止master上的nginx服务后可以看到终端输出，有一段有损时段，而后切换为备机服务
[root@iZ7xvfffujb8qdlzzbvxe7Z ~]# for i in `seq 100`;do curl eiplb.xadocker.cn --connect-timeout 1 ; sleep 1;done
hk01
hk01
hk01
hk01
hk01
hk01
hk01
curl: (7) Failed connect to eiplb.xadocker.cn:80; Connection refused
curl: (7) Failed connect to eiplb.xadocker.cn:80; Connection refused
curl: (7) Failed connect to eiplb.xadocker.cn:80; Connection refused
curl: (7) Failed connect to eiplb.xadocker.cn:80; Connection refused
curl: (7) Failed connect to eiplb.xadocker.cn:80; Connection refused
curl: (28) Connection timed out after 1000 milliseconds
curl: (28) Connection timed out after 1000 milliseconds
hk02
hk02
hk02
hk02
hk02
hk02
hk02
hk02
hk02

keepalived+openapi对弹性网卡做高可用

上文是对eip进行故障切换，通常可以用在自建服务公网入口上，此处则是使用弹性网卡来实现一个内网服务入口的高可用。一台ecs可以绑定多个辅助弹性网卡，一个辅助弹性网卡又可以有多个私网ip，格局瞬间打开~，当然辅助弹性网卡有些限制，比如支持的ecs实例规格以及热插拔形式，可参阅说明文档

弹性网卡ENI涉及的接口

master状态切换脚本

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
[root@hk01 ~]# cat >/server/scripts/eni_change.py<<'EOF'
#!/usr/bin/env python3
# -*- coding: utf-8 -*-
# This file is auto-generated, don't edit it. Thanks.
import sys
import time
from typing import List

from alibabacloud_ecs20140526.client import Client as Ecs20140526Client
from alibabacloud_tea_openapi import models as open_api_models
from alibabacloud_ecs20140526 import models as ecs_20140526_models


class Sample:
    def __init__(self):
        pass

    @staticmethod
    def create_client(
        access_key_id: str,
        access_key_secret: str,
    ) -> Ecs20140526Client:
        """
        使用AK&SK初始化账号Client
        @param access_key_id:
        @param access_key_secret:
        @return: Client
        @throws Exception
        """
        config = open_api_models.Config(
            access_key_id="xxx",
            access_key_secret="xxx"
        )
        # 访问的域名,使用私网ecs openapi地址
        config.endpoint = f'ecs-vpc.cn-guangzhou.aliyuncs.com'
        return Ecs20140526Client(config)

    @staticmethod
    def main(netif_id,local_ecs_id) -> None:
        client = Sample.create_client('accessKeyId', 'accessKeySecret')
        describe_network_interfaces_request = ecs_20140526_models.DescribeNetworkInterfacesRequest(
            region_id='cn-guangzhou',
            type='Secondary',
            network_interface_id=[
                netif_id
            ]
        )

        rs = client.describe_network_interfaces(describe_network_interfaces_request).to_map()
        netif_status = rs["body"]["NetworkInterfaceSets"]["NetworkInterfaceSet"][0]["Status"]

        if netif_status == "InUse":
            netif_bind_ecs_id = rs["body"]["NetworkInterfaceSets"]["NetworkInterfaceSet"][0]["InstanceId"]
            if netif_bind_ecs_id == local_ecs_id:
                print("======Elastic Network Interface is binding======",netif_id,netif_bind_ecs_id,local_ecs_id)
            else:
                print("======Elastic Network Interface unbinding======", netif_id, netif_bind_ecs_id)
                detach_network_interface_request = ecs_20140526_models.DetachNetworkInterfaceRequest(
                    region_id='cn-guangzhou',
                    network_interface_id=netif_id,
                    instance_id=netif_bind_ecs_id
                )
              
                client.detach_network_interface(detach_network_interface_request)
                time.sleep(5)

                print("======Elastic Network Interface binding======", netif_id, local_ecs_id)
                attach_network_interface_request = ecs_20140526_models.AttachNetworkInterfaceRequest(
                    region_id='cn-guangzhou',
                    network_interface_id=netif_id,
                    instance_id=local_ecs_id
                )
          
                client.attach_network_interface(attach_network_interface_request)
        else:
            print("======Elastic Network Interface binding======", netif_id, local_ecs_id)
            attach_network_interface_request = ecs_20140526_models.AttachNetworkInterfaceRequest(
                region_id='cn-guangzhou',
                network_interface_id=netif_id,
                instance_id=local_ecs_id
            )
            
            client.attach_network_interface(attach_network_interface_request)

if __name__ == '__main__':
    netif_id = sys.argv[1]
    local_ecs_id = sys.argv[2]
    Sample.main(netif_id,local_ecs_id)
EOF

其余配置略，可以参考上文keepalived配置，master和backup需要安装以下SDK

1
[root@hk01 ~]# pip3 install alibabacloud_ecs20140526==2.1.1

配置自动识别网卡及私网地址，master和backup均需要安装，参考文档

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
 [root@hk01 ~]# wget https://image-offline.oss-cn-hangzhou.aliyuncs.com/multi-nic-util/multi-nic-util-0.6.tgz && tar -zxvf multi-nic-util-0.6.tgz && cd multi-nic-util-0.6 && bash install.sh
[root@hk01 ~]# systemctl restart eni.service
# 可以看到eth1已获取到私网ip
[root@hk01 ~]# ifconfig
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 172.16.3.228  netmask 255.255.248.0  broadcast 172.16.7.255
        ether 00:16:3e:01:38:c7  txqueuelen 1000  (Ethernet)
        RX packets 371938  bytes 107542312 (102.5 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 310805  bytes 36083525 (34.4 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

eth1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 172.16.3.226  netmask 255.255.248.0  broadcast 172.16.7.255
        ether 00:16:3e:01:79:c3  txqueuelen 1000  (Ethernet)
        RX packets 1  bytes 590 (590.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 2  bytes 384 (384.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        loop  txqueuelen 1000  (Local Loopback)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

测试故障切换结果

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
[root@test ~]# echo 172.16.3.226 pipslb.xadocker.cn >>/etc/hosts
[root@test ~]# for i in `seq 100`;do curl pipslb.xadocker.cn --connect-timeout 1 ;sleep 1;done
hk01
hk01
hk01
hk01
hk01
hk01
hk01
hk01
hk01
hk01
curl: (28) Connection timed out after 1001 milliseconds
curl: (28) Connection timed out after 1000 milliseconds
curl: (28) Connection timed out after 1000 milliseconds
curl: (28) Connection timed out after 1000 milliseconds
curl: (28) Connection timed out after 1000 milliseconds
hk02
hk02
hk02
hk02
hk02
hk02
hk02
hk02

总结

通过本文可以了解eip或eni通过keepalived进行高可用的故障切换，组合的方式有很多，读者可以自行尝试，例如单eni多私网ip，多eni，eni绑定eip等的组合方案。。。。故障切换导致服务不可用的时间要看openapi了，博主测试了下从解绑到绑定估计3s-5s的间隔，所以两个操作之间的时间自行把控或捕获异常且重试。最后还有一点需要开发配合，那就是服务需要具备重连机制！！！