redis cluster简介

redi再3.0前只支持单例模式，3.0之后才支持集群功能。该集群模式每个master节点都是不同的数据，集群模式采用虚拟slots分槽，所有的键根据哈希函数映射到0~16383整数槽内，计算公式：Slot=CRC16(key)&16384。每个节点负责维护一部分槽以及槽所映射的键值数据。如下3分片节点A/B/C:

  • Node A contains hash slots from 0 to 5500.
  • Node B contains hash slots from 5501 to 11000.
  • Node C contains hash slots from 11001 to 16383.

集群模式中的高可用

单例模式下的高可用可以用sentinel来做自动投票选举，而集群模式下不需要sentinel，而是依赖集群中的master角色节点。当集群某分片下slave发现自己的master为FAIL状态时，便开始尝试发起选举：

  1. 某分片slave发现自己的master为FAIL状态
  2. 将自己记录的集群currentEpoch（选举轮次标记）加1，并广播给集群中其他节点
  3. 集群内节点收到该信息后，只有为master的才会响应，判断请求的合法性并响应结果
  4. 该分片slave收集各master的返回结果，收到超半数master的同意后编成新master
  5. 广播Pong消息通知集群内所有节点，每个节点会更新最新的角色信息map配置

集群模式下客户端的访问方式

集群模式下，没有代理提供给客户端，客户端可以通过任意一个节点IP+PORT的方式来调用集群，也可以通过多个节点IP+PORT方式调用：

  • 客户端所请求数据刚好在请求的分片节点slots上时，该分片节点直接返回数据
  • 客户端所请求数据不在请求的分片节点slots上时，该分片节点会返回响应跳转重定位（moved+目标节点），告诉客户端去这个节点请求数据。有点类似nginx上配置了302跳转。业务中调用应当减少跳转重定位，因为这种情况下要多出一次请求链接，客户端中可以将slots的映射关系缓存到本地，并与变化时更新

集群部署

此处博主用单台服务器上部署多个实例来模拟：3主6从（3分片集群，每个分片一个master，2个slave）

配置ruby和依赖

安装ruby

1
[root@blog ~]# yum -y install ruby ruby-devel rubygems

上面安装ruby后会因版本出问题，需要重新装一个合适的版本

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
[root@blog ~]# curl -sSL https://rvm.io/pkuczynski.asc  gpg2 --import -
gpg: key 39499BDB: public key "Piotr Kuczynski <piotr.kuczynski@gmail.com>" imported
gpg: Total number processed: 1
gpg:               imported: 1  (RSA: 1)
[root@blog ~]# curl -L get.rvm.io  bash -s stable
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   194  100   194    0     0    200      0 --:--:-- --:--:-- --:--:--   200
100 24535  100 24535    0     0  11264      0  0:00:02  0:00:02 --:--:-- 20377
Downloading https://github.com/rvm/rvm/archive/1.29.12.tar.gz
Downloading https://github.com/rvm/rvm/releases/download/1.29.12/1.29.12.tar.gz.asc
gpg: Signature made Sat 16 Jan 2021 02:46:22 AM CST using RSA key ID 39499BDB
gpg: Good signature from "Piotr Kuczynski <piotr.kuczynski@gmail.com>"
gpg: WARNING: This key is not certified with a trusted signature!
gpg:          There is no indication that the signature belongs to the owner.
Primary key fingerprint: 7D2B AF1C F37B 13E2 069D  6956 105B D0E7 3949 9BDB
GPG verified '/usr/local/rvm/archives/rvm-1.29.12.tgz'
Creating group 'rvm'
Installing RVM to /usr/local/rvm/
Installation of RVM in /usr/local/rvm/ is almost complete:

  * First you need to add all users that will be using rvm to 'rvm' group,
    and logout - login again, anyone using rvm will be operating with `umask u=rwx,g=rwx,o=rx`.

  * To start using RVM you need to run `source /etc/profile.d/rvm.sh`
    in all your open shell windows, in rare cases you need to reopen all shell windows.
  * Please do NOT forget to add your users to the rvm group.
     The installer no longer auto-adds root or users to the rvm group. Admins must do this.
     Also, please note that group memberships are ONLY evaluated at login time.
     This means that users must log out then back in before group membership takes effect!
Thanks for installing RVM 🙏
Please consider donating to our open collective to help us maintain RVM.

👉  Donate: https://opencollective.com/rvm/donate

调整ruby版本

1
2
3
4
5
6
7
8
9
10
# 使用rvm查看ruby可用版本
[root@blog ~]# rvm list known

# 使用rvm安装ruby指定版本
[root@blog ~]# rvm install 2.5.8

# 使用rvm移除先前版本
[root@blog ~]# rvm remove 2.0.0
ruby-2.0.0-p648 - #already gone
Using /usr/local/rvm/gems/ruby-2.5.8

ruby安装redis依赖

1
[root@blog ~]# gem install redis

准备节点配置文件

创建数据和配置存放目录

1
[root@blog ~]# mkdir /data/700{0..8}

准备每个实例的redis配置文件

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
cat >/data/7000/redis.conf<<EOF
port 7000
daemonize yes
pidfile /data/7000/redis.pid
loglevel notice
logfile "/data/7000/redis.log"
dbfilename dump.rdb
dir /data/7000
protected-mode no
cluster-enabled yes
cluster-config-file nodes.conf
cluster-node-timeout 5000
appendonly yes
EOF

cat > /data/7001/redis.conf<<EOF
port 7001
daemonize yes
pidfile /data/7001/redis.pid
loglevel notice
logfile "/data/7001/redis.log"
dbfilename dump.rdb
dir /data/7001
protected-mode no
cluster-enabled yes
cluster-config-file nodes.conf
cluster-node-timeout 5000
appendonly yes
EOF

cat > /data/7002/redis.conf<<EOF
port 7002
daemonize yes
pidfile /data/7002/redis.pid
loglevel notice
logfile "/data/7002/redis.log"
dbfilename dump.rdb
dir /data/7002
protected-mode no
cluster-enabled yes
cluster-config-file nodes.conf
cluster-node-timeout 5000
appendonly yes
EOF

cat > /data/7003/redis.conf<<EOF
port 7003
daemonize yes
pidfile /data/7003/redis.pid
loglevel notice
logfile "/data/7003/redis.log"
dbfilename dump.rdb
dir /data/7003
protected-mode no
cluster-enabled yes
cluster-config-file nodes.conf
cluster-node-timeout 5000
appendonly yes
EOF

cat > /data/7004/redis.conf<<EOF
port 7004
daemonize yes
pidfile /data/7004/redis.pid
loglevel notice
logfile "/data/7004/redis.log"
dbfilename dump.rdb
dir /data/7004
protected-mode no
cluster-enabled yes
cluster-config-file nodes.conf
cluster-node-timeout 5000
appendonly yes
EOF

cat > /data/7005/redis.conf<<EOF
port 7005
daemonize yes
pidfile /data/7005/redis.pid
loglevel notice
logfile "/data/7005/redis.log"
dbfilename dump.rdb
dir /data/7005
protected-mode no
cluster-enabled yes
cluster-config-file nodes.conf
cluster-node-timeout 5000
appendonly yes
EOF

cat > /data/7006/redis.conf<<EOF
port 7006
daemonize yes
pidfile /data/7006/redis.pid
loglevel notice
logfile "/data/7006/redis.log"
dbfilename dump.rdb
dir /data/7006
protected-mode no
cluster-enabled yes
cluster-config-file nodes.conf
cluster-node-timeout 5000
appendonly yes
EOF

cat > /data/7007/redis.conf<<EOF
port 7007
daemonize yes
pidfile /data/7007/redis.pid
loglevel notice
logfile "/data/7006/redis.log"
dbfilename dump.rdb
dir /data/7007
protected-mode no
cluster-enabled yes
cluster-config-file nodes.conf
cluster-node-timeout 5000
appendonly yes
EOF

cat > /data/7008/redis.conf<<EOF
port 7008
daemonize yes
pidfile /data/7008/redis.pid
loglevel notice
logfile "/data/7008/redis.log"
dbfilename dump.rdb
dir /data/7008
protected-mode no
cluster-enabled yes
cluster-config-file nodes.conf
cluster-node-timeout 5000
appendonly yes
EOF

启动所有实例

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
# 启动所有实例
[root@blog ~]# redis-server /data/7000/redis.conf 
[root@blog ~]# redis-server /data/7001/redis.conf 
[root@blog ~]# redis-server /data/7002/redis.conf 
[root@blog ~]# redis-server /data/7003/redis.conf 
[root@blog ~]# redis-server /data/7004/redis.conf 
[root@blog ~]# redis-server /data/7005/redis.conf
[root@blog ~]# redis-server /data/7006/redis.conf
[root@blog ~]# redis-server /data/7007/redis.conf
[root@blog ~]# redis-server /data/7008/redis.conf

# 查看实例允许装状态
[root@blog ~]# ps -ef grep redis
root       3315      1  0 23:15 ?        00:00:00 redis-server *:7000 [cluster]
root       3317      1  0 23:15 ?        00:00:00 redis-server *:7001 [cluster]
root       3323      1  0 23:15 ?        00:00:00 redis-server *:7002 [cluster]
####### 略

使用redis-trib.rb初始化集群

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
# 注意这里--replicas 2，代表每个分片有2个slave
[root@blog ~]# redis-trib.rb create --replicas 2 127.0.0.1:7000 127.0.0.1:7001 127.0.0.1:7002 127.0.0.1:7003 127.0.0.1:7004 127.0.0.1:7005 127.0.0.1:7006 127.0.0.1:7007 127.0.0.1:7008
>>> Creating cluster
>>> Performing hash slots allocation on 9 nodes...
Using 3 masters:
127.0.0.1:7000
127.0.0.1:7001
127.0.0.1:7002
Adding replica 127.0.0.1:7003 to 127.0.0.1:7000
Adding replica 127.0.0.1:7004 to 127.0.0.1:7000
Adding replica 127.0.0.1:7005 to 127.0.0.1:7001
Adding replica 127.0.0.1:7006 to 127.0.0.1:7001
Adding replica 127.0.0.1:7007 to 127.0.0.1:7002
Adding replica 127.0.0.1:7008 to 127.0.0.1:7002
M: 661ee09e9b2b01bc0630b20ab2a9aed5e9eab8ea 127.0.0.1:7000
   slots:0-5460 (5461 slots) master
M: b7fbfbcdb17e31246e89b69f4e0261fbc417d3b1 127.0.0.1:7001
   slots:5461-10922 (5462 slots) master
M: e00fe07d6394f5b8e3dad2619a4e998394ca13ad 127.0.0.1:7002
   slots:10923-16383 (5461 slots) master
S: 4147647eae7ad8818729ae669f203138db3b3c96 127.0.0.1:7003
   replicates 661ee09e9b2b01bc0630b20ab2a9aed5e9eab8ea
S: 9c45df73cd2ecdc0186e73aae89a5773ef3971b8 127.0.0.1:7004
   replicates 661ee09e9b2b01bc0630b20ab2a9aed5e9eab8ea
S: a9fc1c5bc41e3b066ff2dc9e9f902de432969f34 127.0.0.1:7005
   replicates b7fbfbcdb17e31246e89b69f4e0261fbc417d3b1
S: e1c7aee712563b1fc9e1977d54d60ce171c89abf 127.0.0.1:7006
   replicates b7fbfbcdb17e31246e89b69f4e0261fbc417d3b1
S: 1961aba797f12a5f58ca727ba6cd0f427cca00c3 127.0.0.1:7007
   replicates e00fe07d6394f5b8e3dad2619a4e998394ca13ad
S: dba0b21206eb18aac23c74471c32fd2360a43a38 127.0.0.1:7008
   replicates e00fe07d6394f5b8e3dad2619a4e998394ca13ad
Can I set the above configuration? (type 'yes' to accept): yes
>>> Nodes configuration updated
>>> Assign a different config epoch to each node
>>> Sending CLUSTER MEET messages to join the cluster
Waiting for the cluster to join..
>>> Performing Cluster Check (using node 127.0.0.1:7000)
M: 661ee09e9b2b01bc0630b20ab2a9aed5e9eab8ea 127.0.0.1:7000
   slots:0-5460 (5461 slots) master
   2 additional replica(s)
S: dba0b21206eb18aac23c74471c32fd2360a43a38 127.0.0.1:7008
   slots: (0 slots) slave
   replicates e00fe07d6394f5b8e3dad2619a4e998394ca13ad
S: a9fc1c5bc41e3b066ff2dc9e9f902de432969f34 127.0.0.1:7005
   slots: (0 slots) slave
   replicates b7fbfbcdb17e31246e89b69f4e0261fbc417d3b1
S: 4147647eae7ad8818729ae669f203138db3b3c96 127.0.0.1:7003
   slots: (0 slots) slave
   replicates 661ee09e9b2b01bc0630b20ab2a9aed5e9eab8ea
S: 9c45df73cd2ecdc0186e73aae89a5773ef3971b8 127.0.0.1:7004
   slots: (0 slots) slave
   replicates 661ee09e9b2b01bc0630b20ab2a9aed5e9eab8ea
M: e00fe07d6394f5b8e3dad2619a4e998394ca13ad 127.0.0.1:7002
   slots:10923-16383 (5461 slots) master
   2 additional replica(s)
S: e1c7aee712563b1fc9e1977d54d60ce171c89abf 127.0.0.1:7006
   slots: (0 slots) slave
   replicates b7fbfbcdb17e31246e89b69f4e0261fbc417d3b1
S: 1961aba797f12a5f58ca727ba6cd0f427cca00c3 127.0.0.1:7007
   slots: (0 slots) slave
   replicates e00fe07d6394f5b8e3dad2619a4e998394ca13ad
M: b7fbfbcdb17e31246e89b69f4e0261fbc417d3b1 127.0.0.1:7001
   slots:5461-10922 (5462 slots) master
   2 additional replica(s)
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.

查看集群状态

查看节点信息

1
2
3
4
5
6
7
8
9
10
11
12
[root@blog ~]# redis-cli -p 7000 cluster info
cluster_state:ok
cluster_slots_assigned:16384
cluster_slots_ok:16384
cluster_slots_pfail:0
cluster_slots_fail:0
cluster_known_nodes:9
cluster_size:3
cluster_current_epoch:9
cluster_my_epoch:1
cluster_stats_messages_sent:1240
cluster_stats_messages_received:1240

master节点状态

1
2
3
4
[root@blog ~]# redis-cli -p 7000 cluster nodes  grep master
661ee09e9b2b01bc0630b20ab2a9aed5e9eab8ea 127.0.0.1:7000 myself,master - 0 0 1 connected 0-5460
e00fe07d6394f5b8e3dad2619a4e998394ca13ad 127.0.0.1:7002 master - 0 1675077063725 3 connected 10923-16383
b7fbfbcdb17e31246e89b69f4e0261fbc417d3b1 127.0.0.1:7001 master - 0 1675077062215 2 connected 5461-10922

slave节点状态

1
2
3
4
5
6
7
[root@blog ~]# redis-cli -p 7000 cluster nodes  grep slave
dba0b21206eb18aac23c74471c32fd2360a43a38 127.0.0.1:7008 slave e00fe07d6394f5b8e3dad2619a4e998394ca13ad 0 1675077081853 9 connected
a9fc1c5bc41e3b066ff2dc9e9f902de432969f34 127.0.0.1:7005 slave b7fbfbcdb17e31246e89b69f4e0261fbc417d3b1 0 1675077081350 6 connected
4147647eae7ad8818729ae669f203138db3b3c96 127.0.0.1:7003 slave 661ee09e9b2b01bc0630b20ab2a9aed5e9eab8ea 0 1675077082860 4 connected
9c45df73cd2ecdc0186e73aae89a5773ef3971b8 127.0.0.1:7004 slave 661ee09e9b2b01bc0630b20ab2a9aed5e9eab8ea 0 1675077081854 5 connected
e1c7aee712563b1fc9e1977d54d60ce171c89abf 127.0.0.1:7006 slave b7fbfbcdb17e31246e89b69f4e0261fbc417d3b1 0 1675077081854 7 connected
1961aba797f12a5f58ca727ba6cd0f427cca00c3 127.0.0.1:7007 slave e00fe07d6394f5b8e3dad2619a4e998394ca13ad 0 1675077082860 8 connected

集群完整性检测

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
# 只要存在一个slots未分配，则会出错
[root@blog ~]# redis-trib.rb check 127.0.0.1:7000
>>> Performing Cluster Check (using node 127.0.0.1:7000)
M: 661ee09e9b2b01bc0630b20ab2a9aed5e9eab8ea 127.0.0.1:7000
   slots:0-5460 (5461 slots) master
   2 additional replica(s)
S: dba0b21206eb18aac23c74471c32fd2360a43a38 127.0.0.1:7008
   slots: (0 slots) slave
   replicates e00fe07d6394f5b8e3dad2619a4e998394ca13ad
S: a9fc1c5bc41e3b066ff2dc9e9f902de432969f34 127.0.0.1:7005
   slots: (0 slots) slave
   replicates b7fbfbcdb17e31246e89b69f4e0261fbc417d3b1
S: 4147647eae7ad8818729ae669f203138db3b3c96 127.0.0.1:7003
   slots: (0 slots) slave
   replicates 661ee09e9b2b01bc0630b20ab2a9aed5e9eab8ea
S: 9c45df73cd2ecdc0186e73aae89a5773ef3971b8 127.0.0.1:7004
   slots: (0 slots) slave
   replicates 661ee09e9b2b01bc0630b20ab2a9aed5e9eab8ea
M: e00fe07d6394f5b8e3dad2619a4e998394ca13ad 127.0.0.1:7002
   slots:10923-16383 (5461 slots) master
   2 additional replica(s)
S: e1c7aee712563b1fc9e1977d54d60ce171c89abf 127.0.0.1:7006
   slots: (0 slots) slave
   replicates b7fbfbcdb17e31246e89b69f4e0261fbc417d3b1
S: 1961aba797f12a5f58ca727ba6cd0f427cca00c3 127.0.0.1:7007
   slots: (0 slots) slave
   replicates e00fe07d6394f5b8e3dad2619a4e998394ca13ad
M: b7fbfbcdb17e31246e89b69f4e0261fbc417d3b1 127.0.0.1:7001
   slots:5461-10922 (5462 slots) master
   2 additional replica(s)
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.

redis-cli测试使用

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
# 向集群7000这个实例写入数据，返回了moved
[root@blog ~]# redis-cli -p 7000 set name 'xadocker'
(error) MOVED 5798 127.0.0.1:7001
[root@blog ~]# redis-cli -p 7000 get name
(error) MOVED 5798 127.0.0.1:7001

# 该数据并未自动写入7001实例，说明节点并不会将请求代理到真正的节点上写入
# 而是客户端自行重新再发起请求
[root@blog ~]# redis-cli -p 7001 get name
(nil)

# 客户端只能再次发起请求到7001写入数据
[root@blog ~]# redis-cli -p 7001 set name 'xadocker'
OK
[root@blog ~]# redis-cli -p 7001 get name
"xadocker"

# redis-cli使用-c参数开启跟随move和ask
#  -c                 Enable cluster mode (follow -ASK and -MOVED redirections).
[root@blog ~]# redis-cli -p 7000 set name 'xadocker'
(error) MOVED 5798 127.0.0.1:7001
[root@blog ~]# redis-cli -p 7000 -c set name 'xadocker'
OK
[root@blog ~]# redis-cli -p 7000 -c get name
"xadocker"