Gitlab该如何升级？

各位用Gitlab的小伙伴们，你们的Gitlab升级了吗？还是一直未升级过呢？又或者是已经很久没升级了？仔细想想博主第一升级gitlab的时候是2019年中之时，当时的需求是将7.x升级12.x。当时公司项目不是很多，人员和用户组也是很少，所以便没有进行升级，而是进行仓库迁移，用户权限全部新建。而且我们当时还有gitea这个仓库，也一并迁移仓库到了gitlab。当时迁移就是用gitlab控制台中的仓库导入功能即可，没啥技术含量~

近期博主又遇到升级需求，特此记录下。此时就不是gitlab导入远程仓库这么简单了，因为这种方式只适用于项目工程少的场景。目前的场景项目工程和用户、组都很庞大，只能一步一步走升级流程了。目前的版本依旧是7.x，目标升级到15.x，且运行环境从ecs转为k8s中。。。

官方升级文档：https://docs.gitlab.com/ee/update/，从官方的升级建议来看，博主可能需要升级20+个版本的`(*>﹏<*)′，而且官方镜像只有8.x开始

7.x -> 8.0.5 -> 8.12.0 -> 8.17.7 -> 9.5.10 -> 10.8.7 -> 11.11.8 -> 12.0.12 -> 12.1.17 -> 12.10.14 -> 13.0.14 -> 13.1.11 -> 13.8.8 -> 13.12.15 -> 14.0.12 -> 14.3.6 -> 14.9.5 -> 14.10.Z -> 15.0.Z -> 15.4.0 -> latest 15.Y.Z

因为不能跳大版本升级，所以我们按官方上述的版本升级来制定下升级和平台迁移计划

模拟场景

terraform创建基础层资源

模拟上图需要准备以下基础资源，读者模拟测试可以自行缩减，此处只是为了最大复原场景

两台ecs（原服务器、新服务器）
一个k8s集群
- 一个内网slb
- 三台ecs(master,ingress node/app node)
- 一个nas

terraform工程文件demo

xadocker@xadocker-virtual-machine:~/workdir/datadir/terraform/gitlab-upgrade$ cat providers.tf
terraform {
  backend "oss" {
    profile = "terraform"
    bucket  = "iac-tf-oss-backend"
    prefix  = "gitlab-upgrade/env_dev/"
    access_key = "xxxxxxxxx"
    secret_key = "xxxxxxxxx"
    region     = "cn-guangzhou"
  }

  required_providers {
    alicloud = {
      #source = "aliyun/alicloud"
      source = "local-registry/aliyun/alicloud"
      version = "1.166.0"
    }
  }
}
 
provider "alicloud" {
  access_key = "xxxxxxxxxddddd"
  secret_key = "xxxxxxxxxddddd"
  region = "cn-guangzhou"
}

xadocker@xadocker-virtual-machine:~/workdir/datadir/terraform/gitlab-upgrade$ cat terraform.tf 
resource "alicloud_vpc" "main" {
  # VPC名称
  vpc_name = "terraform-vpc-test"
  # VPC地址块
  cidr_block = "10.1.0.0/21"
}
 
resource "alicloud_vswitch" "main" {
  # VPC ID
  vpc_id            = alicloud_vpc.main.id
  # 交换机地址块
  cidr_block        = "10.1.0.0/24"
  # 可用区
  zone_id           = "cn-guangzhou-a"
  # 资源依赖,会优先创建该依赖资源
  depends_on = [alicloud_vpc.main]
}
 
resource "alicloud_security_group" "default" {
  name = "tf-security-group"
  vpc_id = alicloud_vpc.main.id
}
 
resource "alicloud_security_group_rule" "allow_all_tcp" {
  type              = "ingress"
  ip_protocol       = "tcp"
  nic_type          = "intranet"
  policy            = "accept"
  port_range        = "1/65535"
  priority          = 1
  security_group_id = alicloud_security_group.default.id
  cidr_ip           = "0.0.0.0/0"
}

# ecs.c7.large 2c4g 0.407
# ecs.g7a.large 2c8g 0.44
# ecs.c7.xlarge 4c8g 0.81
# centos_7_6_x64_20G_alibase_20211130.vhd

resource "alicloud_instance" "gitlab-instance" {
  availability_zone = "cn-guangzhou-a"
  security_groups   = alicloud_security_group.default.*.id
  instance_type              = "ecs.c7.large"
  system_disk_category       = "cloud_essd"
  system_disk_size           = 40
  system_disk_performance_level = "PL0"
  image_id                   = "centos_7_6_x64_20G_alibase_20211130.vhd"
  instance_name              = "gitlab_node_${count.index}"
  vswitch_id                 = alicloud_vswitch.main.id
  internet_max_bandwidth_out = 10
  internet_charge_type       = "PayByTraffic"
  instance_charge_type       = "PostPaid"
  password                   = "1qaz@WSXddddddddddd"
  count                      = 2
}

resource "alicloud_alidns_record" "record" {
  domain_name = "xadocker.cn"
  rr          = "gitlab"
  type        = "A"
  value       = alicloud_instance.gitlab-instance.0.public_ip
  remark      = "Test new alidns record."
  status      = "ENABLE"
}

resource "alicloud_instance" "k8s-instance-master" {
  availability_zone = "cn-guangzhou-a"
  security_groups   = alicloud_security_group.default.*.id
  instance_type              = "ecs.c7.large"
  system_disk_category       = "cloud_essd"
  system_disk_size           = 40
  system_disk_performance_level = "PL0"
  image_id                   = "centos_7_6_x64_20G_alibase_20211130.vhd"
  instance_name              = "k8s_master_${count.index}"
  vswitch_id                 = alicloud_vswitch.main.id
  internet_max_bandwidth_out = 10
  internet_charge_type       = "PayByTraffic"
  instance_charge_type       = "PostPaid"
  password                   = "1qaz@WSX@XAD"
  count                      = 1
}

resource "alicloud_instance" "k8s-instance-node-ingress" {
  availability_zone = "cn-guangzhou-a"
  security_groups   = alicloud_security_group.default.*.id
  instance_type              = "ecs.c7.large"
  system_disk_category       = "cloud_essd"
  system_disk_size           = 40
  system_disk_performance_level = "PL0"
  image_id                   = "centos_7_6_x64_20G_alibase_20211130.vhd"
  instance_name              = "k8s_node_ingress_${count.index}"
  vswitch_id                 = alicloud_vswitch.main.id
  internet_max_bandwidth_out = 10
  internet_charge_type       = "PayByTraffic"
  instance_charge_type       = "PostPaid"
  password                   = "1qaz@WSX@XAD"
  count                      = 2
}

resource "alicloud_instance" "k8s-instance-node-app" {
  availability_zone = "cn-guangzhou-a"
  security_groups   = alicloud_security_group.default.*.id
  instance_type              = "ecs.c7.xlarge"
  system_disk_category       = "cloud_essd"
  system_disk_size           = 40
  system_disk_performance_level = "PL0"
  image_id                   = "centos_7_6_x64_20G_alibase_20211130.vhd"
  instance_name              = "k8s_node_app_${count.index}"
  vswitch_id                 = alicloud_vswitch.main.id
  internet_max_bandwidth_out = 10
  internet_charge_type       = "PayByTraffic"
  instance_charge_type       = "PostPaid"
  password                   = "1qaz@WSX@XAD"
  count                      = 1
}

resource "alicloud_slb" "ingress" {
  load_balancer_name    = "k8s-cluster-paas-ingress"
  load_balancer_spec    = "slb.s1.small"
  address_type          = "intranet"
  internet_charge_type  = "PayByTraffic"
  instance_charge_type  = "PostPaid"
  vswitch_id            = alicloud_vswitch.main.id
  tags = {
    k8s-cluster = "paas"
    compoment   = "ingress"
  }
}

resource "alicloud_slb_listener" "gitlab-tcp-443" {
  load_balancer_id          = alicloud_slb.ingress.id
  backend_port              = 443
  bandwidth                 = -1
  frontend_port             = 443
  protocol                  = "tcp"
  request_timeout = 180
  idle_timeout    = 30
}

resource "alicloud_slb_listener" "gitlab-tcp-2222" {
  load_balancer_id          = alicloud_slb.ingress.id
  backend_port              = 2222
  bandwidth                 = -1
  frontend_port             = 2222
  protocol                  = "tcp"
  request_timeout = 180
  idle_timeout    = 30
}

resource "alicloud_slb_backend_server" "default" {
  load_balancer_id = alicloud_slb.ingress.id
  count     = length(alicloud_instance.k8s-instance-node-ingress)

  backend_servers {
    server_id = alicloud_instance.k8s-instance-node-ingress["${count.index}"].id
    weight    = 100
  }
}

使用rpm包安装gitlab7.x

两台ecs都安装gitlab7.x

# 安装依赖
yum install -y curl policycoreutils-python openssh-server postfix git 
systemctl enable sshd
systemctl start sshd
systemctl enable postfix
systemctl start postfix

# 下载安装包
wget https://mirrors.tuna.tsinghua.edu.cn/gitlab-ce/yum/el7/gitlab-ce-7.14.3-ce.1.el7.x86_64.rpm
rpm -ivh gitlab-ce-7.14.3-ce.1.el7.x86_64.rpm

# 编辑配置文件,该配置文件需要证书，请自行准备并放置好
egrep -v config/gitlab.rb '^$^#' 
external_url 'https://gitlab.xadocker.cn' # default: http://hostname
nginx['enable'] = true
nginx['ssl_certificate'] = "/etc/gitlab/ssl/gitlab.xadocker.cn.crt"
nginx['ssl_certificate_key'] = "/etc/gitlab/ssl/gitlab.xadocker.cn.key"

# 初始化gitlab各组件
gitlab-ctl reconfigure

# 查看服务状态
gitlab-ctl status

第一台giilab服务器控制台远程导入样例仓库数据

快速搭建k8s集群

初始化master

新增两个节点

安装并配置ingress

安装nfs-subpath插件

部署gitlab 8.x statefulset

开始升级

gitlab节点1备份并迁移数据至gitlab节点2

# 创建备份数据

gitlab-backup create

# 查看备份文件
ll /var/opt/gitlab/backups


# 备份配置文件,若有ssl证书，也复制过去
mkdir /var/opt/gitlab/backups/conf/
\cp /etc/gitlab/gitlab.rb /var/opt/gitlab/backups/conf/
\cp /etc/gitlab/gitlab-secrets.json  /var/opt/gitlab/backups/conf/
\cp /var/opt/gitlab/nginx/conf/nginx.conf  /var/opt/gitlab/backups/conf/
tar -zvcf gitlab.conf.tar.gz /var/opt/gitlab/backups/conf/

# 备份数据再/var/opt/gitlab/backups，将数据复制到新服务器上
scp 1665213562_gitlab_backup.tar gitlab.conf.tar.gz root@xxxxxxxx:/xxxx

gitlab节点2通过备份数据还原

# 通过备份数据还原时需要停止以下两个服务
gitlab-ctl stop unicorn
gitlab-ctl stop sidekiq

# 查看当前服务状态
gitlab-ctl status

#开始通过备份数据还原
[root@localhost backups]# ll
total 10240
-rw-------. 1 git git 10485760 Oct  8 15:35 1665213562_gitlab_backup.tar
[root@localhost backups]# gitlab-rake gitlab:backup:restore BACKUP=1665213562

# 还原配置文件
tar -zvxf gitlab.conf.tar.gz
cp /var/opt/gitlab/backups/conf/gitlab.rb /etc/gitlab/gitlab.rb
cp /var/opt/gitlab/backups/conf/gitlab-secrets.json /etc/gitlab/gitlab-secrets.json
cp /var/opt/gitlab/backups/conf/nginx.conf /var/opt/gitlab/nginx/conf/nginx.conf 

# 重新启动所有服务，完成备份恢复
[root@localhost backups]# gitlab-ctl restart
ok: run: logrotate: (pid 11847) 0s
ok: run: nginx: (pid 11854) 1s
ok: run: postgresql: (pid 11862) 0s
ok: run: redis: (pid 11870) 1s
ok: run: sidekiq: (pid 11874) 0s
ok: run: unicorn: (pid 11878) 0s

# 修改gitlab.xadocker.cn的dns解析，略

gitlab节点2用rpm包原地升级至8.x

# 先备份数据
gitlab-rake gitlab:backup:create

# 备份配置文件，若有ssl记得备份
mkdir /var/opt/gitlab/backups/conf
\cp /etc/gitlab/gitlab.rb /var/opt/gitlab/backups/conf/
\cp /etc/gitlab/gitlab-secrets.json  /var/opt/gitlab/backups/conf/
\cp /var/opt/gitlab/nginx/conf/nginx.conf  /var/opt/gitlab/backups/conf/

# 停止服务，只保留postgresql
gitlab-ctl stop
gitlab-ctl start postgresql
gitlab-ctl status

# 卸载旧版gitlab
rpm -e gitlab-ce

# 安装新版gitlab
wget https://mirrors.tuna.tsinghua.edu.cn/gitlab-ce/yum/el7/gitlab-ce-8.0.5-ce.0.el7.x86_64.rpm
rpm -Uvh gitlab-ce-8.0.5-ce.0.el7.x86_64.rpm

# 初始化各组件，此时初始化会自动将7.x的数据升级到8.0
gitlab-ctl reconfigure

# 查看此时版本信息
[root@localhost ~]# gitlab-rake gitlab:env:info
System information
System:
Current User:   git
Using RVM:      no
Ruby Version:   2.3.5p376
Gem Version:    2.6.6
Bundler Version:1.13.7
Rake Version:   12.0.0
Redis Version:  3.2.5
Git Version:    2.13.5
Sidekiq Version:5.0.4
Go Version:     unknown

GitLab information
Version:        9.5.9
Revision:       19893b0
Directory:      /opt/gitlab/embedded/service/gitlab-rails
DB Adapter:     postgresql
URL:            http://192.168.109.144
HTTP Clone URL: http://192.168.109.144/some-group/some-project.git
SSH Clone URL:  git@192.168.109.144:some-group/some-project.git
Using LDAP:     no
Using Omniauth: no

GitLab Shell
Version:        5.8.0
Repository storage paths:
- default:      /var/opt/gitlab/git-data/repositories
Hooks:          /opt/gitlab/embedded/service/gitlab-shell/hooks
Git:            /opt/gitlab/embedded/bin/git

gitlab节点2备份并迁移数据至k8s集群的gitlab pod中

# 先备份数据
gitlab-rake gitlab:backup:create

# 将数据迁移到gitlab pod中
# 将nas挂载到gitlab node2的mnt上
mount -t nfs xxxxxx /mnt
scp 1665213562_gitlab_backup.tar /mnt/gitlab/data/gitlab

gitlab pod通过备份数据还原

# 进入pod中将数据还原
kubectl exec -it 

# 通过备份数据还原时需要停止以下两个服务
gitlab-ctl stop unicorn
gitlab-ctl stop sidekiq

# 查看当前服务状态
gitlab-ctl status

# 开始还原数据
gitlab-rake gitlab:backup:restore BACKUP=1665213562

# 重新启动服务
# gitlab-ctl restart

gitlab pod替换镜像版本来升级

14.x及以上版本升级时出现以下问题

StandardError: An error has occurred, all later migrations canceled:

Expected batched background migration for the given configuration to be marked as 'finished', but it is 'active':
  {:job_class_name=>"CopyColumnUsingBackgroundMigrationJob", :table_name=>"push_event_payloads", :column_name=>"event_id", :job_arguments=>[["event_id"], ["event_id_convert_to_bigint"]]}

是因为后台任务未执行完成，需要等任务执行完成才可以进行升级：