前言

最近遇到个场景，k8s集群外网入口是ingress，该ingress外有个SLB，根据阿里云SLB带宽限制上限5Gbps，此处则存在带宽瓶颈，虽然业务可能达不到这么高的带宽，但是如果遇到流量攻击，那流量就大了去了，而且其攻击成本似乎不高（几十元就可以持续几分钟达到10G+），本篇就以此展开描述下ingress的实践方案ngress使用Daemonset+HostNetwork模式

ingress的部署方方式可以有多种

  • Deployment + LoadBalancer： 适合云厂商
  • Deployment + NodePort：适用于宿主机的ip相对固定，因为需要NodePort所以多了一层nat，请求量很大的时候可能会存在影响
  • DaemonSet + HostNetwork：直接使用宿主机端口，性能较NodePort好，但是一个node只能部署一个，适合高并发场景使用

为了保证入口流量的性能，我们会单独准备多个节点，并且这些节点用来专门跑ingress，不会存在其他业务pod，方式则是使用k8s的labels和选择器来实现

使用Daemonset+HostNetwork方式部署ingress

为ingress节点设置label

1
2
3
4
# 为ingress节点配置labels
kubectl  label node node-lb1 ingress-lb=slb
kubectl  label node node-lb2 ingress-lb=slb
kubectl  label node node-lb3 ingress-lb=slb

修改ingress为Daemonset+Hostnetwork

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
cat ingress.yml
.....
---
# Source: ingress-nginx/templates/controller-deployment.yaml
apiVersion: apps/v1
# 设置DaemonSet资源方式
kind: DaemonSet
metadata:
  labels:
    helm.sh/chart: ingress-nginx-2.1.0
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/version: 0.32.0
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/component: controller
  name: ingress-nginx-controller
  namespace: ingress-nginx
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: ingress-nginx
      app.kubernetes.io/instance: ingress-nginx
      app.kubernetes.io/component: controller
  revisionHistoryLimit: 10
  minReadySeconds: 0
  template:
    metadata:
      labels:
        app.kubernetes.io/name: ingress-nginx
        app.kubernetes.io/instance: ingress-nginx
        app.kubernetes.io/component: controller
    spec:

      # 设置HostNetwork
      hostNetwork: true

      # 配置节点标签选择器
      nodeSelector:
        ingress-lb: slb
      dnsPolicy: ClusterFirst
      containers:
        - name: controller
          image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.32.0
          imagePullPolicy: IfNotPresent
          lifecycle:
            preStop:
              exec:
                command:
                  - /wait-shutdown
          args:
            - /nginx-ingress-controller
            - --publish-service=ingress-nginx/ingress-nginx-controller
            - --election-id=ingress-controller-leader
            - --ingress-class=nginx
            - --configmap=ingress-nginx/ingress-nginx-controller
            - --validating-webhook=:8443
            - --validating-webhook-certificate=/usr/local/certificates/cert
            - --validating-webhook-key=/usr/local/certificates/key
          securityContext:
            capabilities:
              drop:
                - ALL
              add:
                - NET_BIND_SERVICE
            runAsUser: 101
            allowPrivilegeEscalation: true
          env:
            - name: POD_NAME
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
            - name: POD_NAMESPACE
              valueFrom:
                fieldRef:
                  fieldPath: metadata.namespace
          livenessProbe:
            httpGet:
              path: /healthz
              port: 10254
              scheme: HTTP
            initialDelaySeconds: 10
            periodSeconds: 10
            timeoutSeconds: 1
            successThreshold: 1
            failureThreshold: 3
          readinessProbe:
            httpGet:
              path: /healthz
              port: 10254
              scheme: HTTP
            initialDelaySeconds: 10
            periodSeconds: 10
            timeoutSeconds: 1
            successThreshold: 1
            failureThreshold: 3
          ports:
            - name: http
              containerPort: 80
              protocol: TCP

              # 设置hostPort
              hostPort: 80
            - name: https
              containerPort: 443
              protocol: TCP

              # 设置hostPort
              hostPort: 443
            - name: webhook
              containerPort: 8443

              # 设置hostPort
              hostPort: 8443
              protocol: TCP
          volumeMounts:
            - name: webhook-cert
              mountPath: /usr/local/certificates/
              readOnly: true
          resources:
            requests:
              cpu: 100m
              memory: 90Mi
      serviceAccountName: ingress-nginx
      terminationGracePeriodSeconds: 300
      volumes:
        - name: webhook-cert
          secret:
            secretName: ingress-nginx-admission
---
......

# 可以看到此时我的ingress在特定的节点上
[root@node1 ingress-install]# kubectl get pods -n ingress-nginx -o wide
NAME                                   READY   STATUS      RESTARTS   AGE    IP             NODE    NOMINATED NODE   READINESS GATES
ingress-nginx-admission-create-fzsmp   0/1     Completed   0          112m   10.100.104.7   node2   <none>           <none>
ingress-nginx-admission-patch-w8bbx    0/1     Completed   1          112m   10.100.104.6   node2   <none>           <none>
ingress-nginx-controller-nsmvs         1/1     Running     0          112m   172.16.3.239   node-lb1   <none>           <none>
ingress-nginx-controller-xrdvr         1/1     Running     0          112m   172.16.3.237   node-lb2   <none>           <none>
ingress-nginx-controller-zk869         1/1     Running     0          112m   172.16.3.238   node-lb3   <none>           <none>

使用terraform将该三个ingress节点绑定至SLB

博主穷，遂用私网SLB代替。。。。

声明资源配置文件

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
[root@node-nfs slb-bind]# cat terraform.tf
terraform {
  required_providers {
    alicloud = {
      source = "aliyun/alicloud"
      #source  = "local-registry/aliyun/alicloud"
      version = "1.166.0"
    }
  }
}
provider "alicloud" {
  access_key = "xxx"
  secret_key = "xxx"
  region     = "cn-guangzhou"
}

data "alicloud_vpcs" "vpcs_ds" {
  cidr_block = "172.16.0.0/12"
  status     = "Available"
  name_regex = "^game-cluster-xxx"
}

output "first_vpc_id" {
  value = "${data.alicloud_vpcs.vpcs_ds.vpcs.0.id}"
}

data "alicloud_vswitches" "slb_vsws" {
  vpc_id = "${data.alicloud_vpcs.vpcs_ds.vpcs.0.id}"
}

output "slb_vswid" {
  value = "${data.alicloud_vswitches.slb_vsws.vswitches.0.id}"
}

data "alicloud_instances" "instances_ds" {
  name_regex = "^game-cluster-xxx-ingress"
  status     = "Running"
}

output "instance_ids" {
  value = "${data.alicloud_instances.instances_ds.ids}"
}

data "alicloud_slbs" "slbs_ds" {
  name_regex = "auto_named_slb"
}

output "first_slb_id" {
  value = data.alicloud_slbs.slbs_ds.slbs[0].id
}

resource "alicloud_slb_server_group" "ingress_80_group" {
  load_balancer_id = data.alicloud_slbs.slbs_ds.slbs[0].id
  name             = "ingress_80"

  servers {
    server_ids = data.alicloud_instances.instances_ds.ids
    port       = 80
    weight     = 100
  }
}

resource "alicloud_slb_listener" "ingress_80" {
  load_balancer_id          = data.alicloud_slbs.slbs_ds.slbs[0].id
  backend_port              = 80
  frontend_port             = 80
  protocol                  = "tcp"
  bandwidth                 = -1
  established_timeout = 900
  scheduler = "wrr"
  server_group_id = alicloud_slb_server_group.ingress_80_group.id
}

resource "alicloud_slb_server_group" "ingress_443_group" {
  load_balancer_id = data.alicloud_slbs.slbs_ds.slbs[0].id
  name             = "ingress_443"

  servers {
    server_ids = data.alicloud_instances.instances_ds.ids
    port       = 443
    weight     = 100
  }
}

resource "alicloud_slb_listener" "ingress_443" {
  load_balancer_id          = data.alicloud_slbs.slbs_ds.slbs[0].id
  backend_port              = 443
  frontend_port             = 443
  protocol                  = "tcp"
  bandwidth                 = -1
  established_timeout = 900
  scheduler = "wrr"
  server_group_id = alicloud_slb_server_group.ingress_443_group.id
}

创建绑定

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
[root@node-nfs slb-bind]# terraform apply

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  # alicloud_slb_listener.ingress_443 will be created
  + resource "alicloud_slb_listener" "ingress_443" {
      + acl_status                = "off"
      + backend_port              = 443
      + bandwidth                 = -1
      + established_timeout       = 900
      + frontend_port             = 443
      + health_check_connect_port = (known after apply)
      + health_check_http_code    = (known after apply)
      + health_check_interval     = 2
      + health_check_method       = (known after apply)
      + health_check_timeout      = 5
      + health_check_type         = "tcp"
      + healthy_threshold         = 3
      + id                        = (known after apply)
      + listener_forward          = (known after apply)
      + load_balancer_id          = "lb-xxxxcmirpz2qtv0cx24vgk"
      + persistence_timeout       = 0
      + protocol                  = "tcp"
      + scheduler                 = "wrr"
      + server_certificate_id     = (known after apply)
      + server_group_id           = (known after apply)
      + ssl_certificate_id        = (known after apply)
      + unhealthy_threshold       = 3

      + x_forwarded_for {
          + retrive_client_ip = (known after apply)
          + retrive_slb_id    = (known after apply)
          + retrive_slb_ip    = (known after apply)
          + retrive_slb_proto = (known after apply)
        }
    }

  # alicloud_slb_listener.ingress_80 will be created
  + resource "alicloud_slb_listener" "ingress_80" {
      + acl_status                = "off"
      + backend_port              = 80
      + bandwidth                 = -1
      + established_timeout       = 900
      + frontend_port             = 80
      + health_check_connect_port = (known after apply)
      + health_check_http_code    = (known after apply)
      + health_check_interval     = 2
      + health_check_method       = (known after apply)
      + health_check_timeout      = 5
      + health_check_type         = "tcp"
      + healthy_threshold         = 3
      + id                        = (known after apply)
      + listener_forward          = (known after apply)
      + load_balancer_id          = "lb-xxxxcmirpz2qtv0cx24vgk"
      + persistence_timeout       = 0
      + protocol                  = "tcp"
      + scheduler                 = "wrr"
      + server_certificate_id     = (known after apply)
      + server_group_id           = (known after apply)
      + ssl_certificate_id        = (known after apply)
      + unhealthy_threshold       = 3

      + x_forwarded_for {
          + retrive_client_ip = (known after apply)
          + retrive_slb_id    = (known after apply)
          + retrive_slb_ip    = (known after apply)
          + retrive_slb_proto = (known after apply)
        }
    }

  # alicloud_slb_server_group.ingress_443_group will be created
  + resource "alicloud_slb_server_group" "ingress_443_group" {
      + delete_protection_validation = false
      + id                           = (known after apply)
      + load_balancer_id             = "lb-xxxxcmirpz2qtv0cx24vgk"
      + name                         = "ingress_443"

      + servers {
          + port       = 443
          + server_ids = [
              + "i-xxxx5juwtrji2t4le8fy8",
              + "i-xxxx5juwtrji2t4le8fy9",
              + "i-xxxx5juwtrji2t4le8fy7",
            ]
          + type       = "ecs"
          + weight     = 100
        }
    }

  # alicloud_slb_server_group.ingress_80_group will be created
  + resource "alicloud_slb_server_group" "ingress_80_group" {
      + delete_protection_validation = false
      + id                           = (known after apply)
      + load_balancer_id             = "lb-xxxxcmirpz2qtv0cx24vgk"
      + name                         = "ingress_80"

      + servers {
          + port       = 80
          + server_ids = [
              + "i-xxxx5juwtrji2t4le8fy8",
              + "i-xxxx5juwtrji2t4le8fy9",
              + "i-xxxx5juwtrji2t4le8fy7",
            ]
          + type       = "ecs"
          + weight     = 100
        }
    }

Plan: 4 to add, 0 to change, 0 to destroy.

Changes to Outputs:
  + first_slb_id = "lb-xxxxcmirpz2qtv0cx24vgk"
  + first_vpc_id = "vpc-xxxx8xhxni9b2fx43v7t79"
  + instance_ids = [
      + "i-xxxx5juwtrji2t4le8fy8",
      + "i-xxxx5juwtrji2t4le8fy9",
      + "i-xxxx5juwtrji2t4le8fy7",
    ]
  + slb_vswid    = "vsw-xxxx9teaxnewvrrbbteona"
╷
│ Warning: "servers": [DEPRECATED] Field 'servers' has been deprecated from provider version 1.163.0 and it will be removed in the future version. Please use the new resource 'alicloud_slb_server_group_server_attachment'.
│
│   with alicloud_slb_server_group.ingress_80_group,
│   on terraform.tf line 54, in resource "alicloud_slb_server_group" "ingress_80_group":
│   54: resource "alicloud_slb_server_group" "ingress_80_group" {
│
│ (and 3 more similar warnings elsewhere)
╵

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

alicloud_slb_server_group.ingress_443_group: Creating...
alicloud_slb_server_group.ingress_80_group: Creating...
alicloud_slb_server_group.ingress_443_group: Creation complete after 1s [id=rsp-xxxxkrrjwm0ftd]
alicloud_slb_server_group.ingress_80_group: Creation complete after 1s [id=rsp-xxxxnn6i95o342]
alicloud_slb_listener.ingress_443: Creating...
alicloud_slb_listener.ingress_80: Creating...
alicloud_slb_listener.ingress_443: Creation complete after 1s [id=lb-xxxxcmirpz2qtv0cx24vgk:tcp:443]
alicloud_slb_listener.ingress_80: Creation complete after 2s [id=lb-xxxxcmirpz2qtv0cx24vgk:tcp:80]
╷
│ Warning: "servers": [DEPRECATED] Field 'servers' has been deprecated from provider version 1.163.0 and it will be removed in the future version. Please use the new resource 'alicloud_slb_server_group_server_attachment'.
│
│   with alicloud_slb_server_group.ingress_80_group,
│   on terraform.tf line 54, in resource "alicloud_slb_server_group" "ingress_80_group":
│   54: resource "alicloud_slb_server_group" "ingress_80_group" {
│
│ (and one more similar warning elsewhere)
╵

Apply complete! Resources: 4 added, 0 changed, 0 destroyed.

Outputs:

first_slb_id = "lb-xxxxcmirpz2qtv0cx24vgk"
first_vpc_id = "vpc-xxxx8xhxni9b2fx43v7t79"
instance_ids = tolist([
  "i-xxxx5juwtrji2t4le8fy8",
  "i-xxxx5juwtrji2t4le8fy9",
  "i-xxxx5juwtrji2t4le8fy7",
])
slb_vswid = "vsw-xxxx9teaxnewvrrbbteona"

使用SLB私网地址测试访问

1
2
3
4
5
6
7
8
9
10
11
[root@node1 ingress-install]# for i in `seq 10`;do curl -H "HOST:bussinis.xadocker.cn"  http://172.16.3.240/index.html;done
bussinis-nginx-6d867fdb88-xhnt9 10.100.104.10
bussinis-nginx-6d867fdb88-2vr4d 10.100.139.3
bussinis-nginx-6d867fdb88-hdhhn 10.100.104.11
bussinis-nginx-6d867fdb88-xzdzm 10.100.135.3
bussinis-nginx-6d867fdb88-fcjb5 10.100.139.2
bussinis-nginx-6d867fdb88-gvhch 10.100.33.130
bussinis-nginx-6d867fdb88-2vr4d 10.100.139.3
bussinis-nginx-6d867fdb88-xhnt9 10.100.104.10
bussinis-nginx-6d867fdb88-hdhhn 10.100.104.11
bussinis-nginx-6d867fdb88-hdhhn 10.100.104.11

ingress使用多个SLB

单SLB带宽有上限，那么可以给ingress绑定多个SLB，其实只需要将新的SLB上绑定ingress节点就行，因为ECS是可以加入多个SLB的

使用terrform新建SLB和监听绑定

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
[root@node-nfs slb-new]# cat terraform.tf
terraform {
  required_providers {
    alicloud = {
      source = "aliyun/alicloud"
      #source  = "local-registry/aliyun/alicloud"
      version = "1.166.0"
    }
  }
}
provider "alicloud" {
  access_key = "xxx"
  secret_key = "xxx"
  region     = "cn-guangzhou"
}

data "alicloud_vpcs" "vpcs_ds" {
  cidr_block = "172.16.0.0/12"
  status     = "Available"
  name_regex = "^game-cluster-xxx"
}

output "first_vpc_id" {
  value = "${data.alicloud_vpcs.vpcs_ds.vpcs.0.id}"
}

data "alicloud_vswitches" "slb_vsws" {
  vpc_id = "${data.alicloud_vpcs.vpcs_ds.vpcs.0.id}"
}


data "alicloud_instances" "instances_ds" {
  name_regex = "^launch-advisor-20220524"
  status     = "Running"
}

output "instance_ids" {
  value = "${data.alicloud_instances.instances_ds.ids}"
}

output "slb_vswid" {
  value = "${data.alicloud_vswitches.slb_vsws.vswitches.0.id}"
}

resource "alicloud_slb" "instance" {
  load_balancer_name = "game_cluster_api"
  address_type = "intranet"
  specification = "slb.s1.small"
  vswitch_id           = "${data.alicloud_vswitches.slb_vsws.vswitches.0.id}"
  internet_charge_type = "PayByTraffic"
  instance_charge_type = "PrePaid"
  period = 1
}

resource "alicloud_slb_server_group" "ingress_80_group" {
  load_balancer_id = alicloud_slb.instance.id
  name             = "ingress_80"

  servers {
    server_ids = data.alicloud_instances.instances_ds.ids
    port       = 80
    weight     = 100
  }
}

resource "alicloud_slb_listener" "ingress_80" {
  load_balancer_id          = alicloud_slb.instance.id
  backend_port              = 80
  frontend_port             = 80
  protocol                  = "tcp"
  bandwidth                 = -1
  established_timeout = 900
  scheduler = "wrr"
  server_group_id = alicloud_slb_server_group.ingress_80_group.id
}

resource "alicloud_slb_server_group" "ingress_443_group" {
  load_balancer_id = alicloud_slb.instance.id
  name             = "ingress_443"

  servers {
    server_ids = data.alicloud_instances.instances_ds.ids
    port       = 443
    weight     = 100
  }
}

resource "alicloud_slb_listener" "ingress_443" {
  load_balancer_id          = alicloud_slb.instance.id
  backend_port              = 443
  frontend_port             = 443
  protocol                  = "tcp"
  bandwidth                 = -1
  established_timeout = 900
  scheduler = "wrr"
  server_group_id = alicloud_slb_server_group.ingress_443_group.id
}

创建资源

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
[root@node-nfs slb-new]# terraform apply

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  # alicloud_slb.instance will be created
  + resource "alicloud_slb" "instance" {
      + address                        = (known after apply)
      + address_type                   = "intranet"
      + id                             = (known after apply)
      + instance_charge_type           = "PrePaid"
      + internet                       = (known after apply)
      + load_balancer_name             = "game_cluster_api"
      + load_balancer_spec             = (known after apply)
      + master_zone_id                 = (known after apply)
      + modification_protection_status = (known after apply)
      + name                           = (known after apply)
      + payment_type                   = (known after apply)
      + period                         = 1
      + resource_group_id              = (known after apply)
      + slave_zone_id                  = (known after apply)
      + specification                  = "slb.s1.small"
      + status                         = (known after apply)
      + vswitch_id                     = "vsw-xxxx9teaxnewvrrbbteona"
    }

  # alicloud_slb_listener.ingress_443 will be created
  + resource "alicloud_slb_listener" "ingress_443" {
      + acl_status                = "off"
      + backend_port              = 443
      + bandwidth                 = -1
      + established_timeout       = 900
      + frontend_port             = 443
      + health_check_connect_port = (known after apply)
      + health_check_http_code    = (known after apply)
      + health_check_interval     = 2
      + health_check_method       = (known after apply)
      + health_check_timeout      = 5
      + health_check_type         = "tcp"
      + healthy_threshold         = 3
      + id                        = (known after apply)
      + listener_forward          = (known after apply)
      + load_balancer_id          = (known after apply)
      + persistence_timeout       = 0
      + protocol                  = "tcp"
      + scheduler                 = "wrr"
      + server_certificate_id     = (known after apply)
      + server_group_id           = (known after apply)
      + ssl_certificate_id        = (known after apply)
      + unhealthy_threshold       = 3

      + x_forwarded_for {
          + retrive_client_ip = (known after apply)
          + retrive_slb_id    = (known after apply)
          + retrive_slb_ip    = (known after apply)
          + retrive_slb_proto = (known after apply)
        }
    }

  # alicloud_slb_listener.ingress_80 will be created
  + resource "alicloud_slb_listener" "ingress_80" {
      + acl_status                = "off"
      + backend_port              = 80
      + bandwidth                 = -1
      + established_timeout       = 900
      + frontend_port             = 80
      + health_check_connect_port = (known after apply)
      + health_check_http_code    = (known after apply)
      + health_check_interval     = 2
      + health_check_method       = (known after apply)
      + health_check_timeout      = 5
      + health_check_type         = "tcp"
      + healthy_threshold         = 3
      + id                        = (known after apply)
      + listener_forward          = (known after apply)
      + load_balancer_id          = (known after apply)
      + persistence_timeout       = 0
      + protocol                  = "tcp"
      + scheduler                 = "wrr"
      + server_certificate_id     = (known after apply)
      + server_group_id           = (known after apply)
      + ssl_certificate_id        = (known after apply)
      + unhealthy_threshold       = 3

      + x_forwarded_for {
          + retrive_client_ip = (known after apply)
          + retrive_slb_id    = (known after apply)
          + retrive_slb_ip    = (known after apply)
          + retrive_slb_proto = (known after apply)
        }
    }

  # alicloud_slb_server_group.ingress_443_group will be created
  + resource "alicloud_slb_server_group" "ingress_443_group" {
      + delete_protection_validation = false
      + id                           = (known after apply)
      + load_balancer_id             = (known after apply)
      + name                         = "ingress_443"

      + servers {
          + port       = 443
          + server_ids = [
              + "i-xxxx5juwtrji2t4le8fy8",
              + "i-xxxx5juwtrji2t4le8fy9",
              + "i-xxxx5juwtrji2t4le8fy7",
            ]
          + type       = "ecs"
          + weight     = 100
        }
    }

  # alicloud_slb_server_group.ingress_80_group will be created
  + resource "alicloud_slb_server_group" "ingress_80_group" {
      + delete_protection_validation = false
      + id                           = (known after apply)
      + load_balancer_id             = (known after apply)
      + name                         = "ingress_80"

      + servers {
          + port       = 80
          + server_ids = [
              + "i-xxxx5juwtrji2t4le8fy8",
              + "i-xxxx5juwtrji2t4le8fy9",
              + "i-xxxx5juwtrji2t4le8fy7",
            ]
          + type       = "ecs"
          + weight     = 100
        }
    }

Plan: 5 to add, 0 to change, 0 to destroy.

Changes to Outputs:
  + first_vpc_id = "vpc-xxxx8xhxni9b2fx43v7t79"
  + instance_ids = [
      + "i-xxxx5juwtrji2t4le8fy8",
      + "i-xxxx5juwtrji2t4le8fy9",
      + "i-xxxx5juwtrji2t4le8fy7",
    ]
  + slb_vswid    = "vsw-xxxx9teaxnewvrrbbteona"
╷
│ Warning: "specification": [DEPRECATED] Field 'specification' has been deprecated from provider version 1.123.1. New field 'load_balancer_spec' instead
│
│   with alicloud_slb.instance,
│   on terraform.tf line 44, in resource "alicloud_slb" "instance":
│   44: resource "alicloud_slb" "instance" {
│
│ (and one more similar warning elsewhere)
╵
╷
│ Warning: "instance_charge_type": [DEPRECATED] Field 'instance_charge_type' has been deprecated from provider version 1.124. Use 'payment_type' replaces it.
│
│   with alicloud_slb.instance,
│   on terraform.tf line 44, in resource "alicloud_slb" "instance":
│   44: resource "alicloud_slb" "instance" {
│
│ (and one more similar warning elsewhere)
╵
╷
│ Warning: "servers": [DEPRECATED] Field 'servers' has been deprecated from provider version 1.163.0 and it will be removed in the future version. Please use the new resource 'alicloud_slb_server_group_server_attachment'.
│
│   with alicloud_slb_server_group.ingress_80_group,
│   on terraform.tf line 54, in resource "alicloud_slb_server_group" "ingress_80_group":
│   54: resource "alicloud_slb_server_group" "ingress_80_group" {
│
│ (and 3 more similar warnings elsewhere)
╵

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

alicloud_slb.instance: Creating...
alicloud_slb.instance: Still creating... [10s elapsed]
alicloud_slb.instance: Still creating... [20s elapsed]
alicloud_slb.instance: Still creating... [30s elapsed]
alicloud_slb.instance: Still creating... [40s elapsed]
alicloud_slb.instance: Still creating... [50s elapsed]
alicloud_slb.instance: Creation complete after 56s [id=lb-xxxx5lws55aqwho34mnp4d]
alicloud_slb_server_group.ingress_80_group: Creating...
alicloud_slb_server_group.ingress_443_group: Creating...
alicloud_slb_server_group.ingress_443_group: Creation complete after 0s [id=rsp-xxxxgqtutbglw7]
alicloud_slb_listener.ingress_443: Creating...
alicloud_slb_server_group.ingress_80_group: Creation complete after 0s [id=rsp-xxxxwnbmp683r8]
alicloud_slb_listener.ingress_80: Creating...
alicloud_slb_listener.ingress_80: Creation complete after 1s [id=lb-xxxx5lws55aqwho34mnp4d:tcp:80]
alicloud_slb_listener.ingress_443: Creation complete after 1s [id=lb-xxxx5lws55aqwho34mnp4d:tcp:443]
╷
│ Warning: "specification": [DEPRECATED] Field 'specification' has been deprecated from provider version 1.123.1. New field 'load_balancer_spec' instead
│
│   with alicloud_slb.instance,
│   on terraform.tf line 44, in resource "alicloud_slb" "instance":
│   44: resource "alicloud_slb" "instance" {
│
╵
╷
│ Warning: "instance_charge_type": [DEPRECATED] Field 'instance_charge_type' has been deprecated from provider version 1.124. Use 'payment_type' replaces it.
│
│   with alicloud_slb.instance,
│   on terraform.tf line 44, in resource "alicloud_slb" "instance":
│   44: resource "alicloud_slb" "instance" {
│
╵
╷
│ Warning: "servers": [DEPRECATED] Field 'servers' has been deprecated from provider version 1.163.0 and it will be removed in the future version. Please use the new resource 'alicloud_slb_server_group_server_attachment'.
│
│   with alicloud_slb_server_group.ingress_80_group,
│   on terraform.tf line 54, in resource "alicloud_slb_server_group" "ingress_80_group":
│   54: resource "alicloud_slb_server_group" "ingress_80_group" {
│
│ (and one more similar warning elsewhere)
╵

Apply complete! Resources: 5 added, 0 changed, 0 destroyed.

Outputs:

first_vpc_id = "vpc-xxxx8xhxni9b2fx43v7t79"
instance_ids = tolist([
  "i-xxxx5juwtrji2t4le8fy8",
  "i-xxxx5juwtrji2t4le8fy9",
  "i-xxxx5juwtrji2t4le8fy7",
])
slb_vswid = "vsw-xxxx9teaxnewvrrbbteona"

测试新SLB

1
2
3
4
5
6
7
8
9
10
11
[root@node1 ingress-install]# for i in `seq 10`;do curl -H "HOST:bussinis.xadocker.cn"  http://172.16.3.241/index.html;done
bussinis-nginx-6d867fdb88-f9kt5 10.100.135.4
bussinis-nginx-6d867fdb88-f9kt5 10.100.135.4
bussinis-nginx-6d867fdb88-fcjb5 10.100.139.2
bussinis-nginx-6d867fdb88-65gv7 10.100.3.66
bussinis-nginx-6d867fdb88-xzdzm 10.100.135.3
bussinis-nginx-6d867fdb88-gvhch 10.100.33.130
bussinis-nginx-6d867fdb88-xzdzm 10.100.135.3
bussinis-nginx-6d867fdb88-2vr4d 10.100.139.3
bussinis-nginx-6d867fdb88-xhnt9 10.100.104.10
bussinis-nginx-6d867fdb88-hdhhn 10.100.104.11