gitlab挂掉,重启后依然失败

  • Choerodon平台版本: 0.13.0

  • 遇到问题的执行步骤:
    平台运行使用了一段时间没出现问题,然后按照开发文档开发了一个demo项目,在持续集成maven打包的时候出错了,然后gitlab runner里面重试了几次,最后gitlab就挂掉了


    重启过gitlab,启动失败

  • 文档地址:

  • 环境信息(如:节点信息):

  • 报错日志:

  • 原因分析:

  • 疑问:

Hi,参考下这个


删除了,重启gitlab,还是失败,报错信息一样

kubectl describe pod gitlab-85d587c458-nmm28 -n c7n-system

kubectl logs gitlab-85d587c458-nmm28 -n c7n-system

请问一下还有没有别的方法可以尝试一下?

重启下nfs服务试试

重启nfs服务,然后重启gitlab服务后,gitlab启动正常了

请问下,这个可能是什么原因导致gitlab服务down掉了呢?

可能是nfs服务内部出现了问题

gitlab再次down掉,还是一样的报错,然后我看了一下nfs服务的日志,如下,还请帮忙看看是否有问题
I0218 02:20:07.235199 1 main.go:63] Provisioner choerodon.io/nfs-provisioner specified
I0218 02:20:07.235374 1 main.go:87] Setting up NFS server!
I0218 02:20:07.830997 1 server.go:144] starting RLIMIT_NOFILE rlimit.Cur 1048576, rlimit.Max 1048576
I0218 02:20:07.831045 1 server.go:155] ending RLIMIT_NOFILE rlimit.Cur 1048576, rlimit.Max 1048576
I0218 02:20:07.849741 1 server.go:129] Running NFS server!
I0218 02:20:12.915751 1 leaderelection.go:185] attempting to acquire leader lease kube-system/choerodon.io-nfs-provisioner…
I0218 02:20:30.328591 1 leaderelection.go:194] successfully acquired lease kube-system/choerodon.io-nfs-provisioner
I0218 02:20:30.328891 1 controller.go:631] Starting provisioner controller choerodon.io/nfs-provisioner_nfs-provisioner-65bdd5f955-mxwzz_b8d1df5d-3323-11e9-891c-0a580ae940a9!
I0218 02:20:30.336526 1 event.go:221] Event(v1.ObjectReference{Kind:“Endpoints”, Namespace:“kube-system”, Name:“choerodon.io-nfs-provisioner”, UID:“295d24bf-1507-11e9-9a59-4ccc6aaa26b9”, APIVersion:“v1”, ResourceVersion:“4601498”, FieldPath:""}): type: ‘Normal’ reason: ‘LeaderElection’ nfs-provisioner-65bdd5f955-mxwzz_b8d1df5d-3323-11e9-891c-0a580ae940a9 became leader
I0218 02:20:30.429065 1 controller.go:680] Started provisioner controller choerodon.io/nfs-provisioner_nfs-provisioner-65bdd5f955-mxwzz_b8d1df5d-3323-11e9-891c-0a580ae940a9!
_E0218 02:22:16.449022 1 main.go:97] NFS server Exited Unexpectedly with err: ganesha.nfsd failed with error: signal: segmentation fault, output: _
I0218 02:22:17.449294 1 server.go:129] Running NFS server!
_E0219 03:41:22.892929 1 main.go:97] NFS server Exited Unexpectedly with err: ganesha.nfsd failed with error: signal: segmentation fault, output: _
I0219 03:41:23.893255 1 server.go:129] Running NFS server!

重启nfs后,再重启gitlab,gitlab恢复正常, 重启nfs服务的日志如下,
I0219 05:37:31.504708 1 main.go:63] Provisioner choerodon.io/nfs-provisioner specified
I0219 05:37:31.504786 1 main.go:87] Setting up NFS server!
I0219 05:37:31.644002 1 server.go:144] starting RLIMIT_NOFILE rlimit.Cur 1048576, rlimit.Max 1048576
I0219 05:37:31.644017 1 server.go:155] ending RLIMIT_NOFILE rlimit.Cur 1048576, rlimit.Max 1048576
I0219 05:37:31.644503 1 server.go:129] Running NFS server!
I0219 05:37:36.680246 1 leaderelection.go:185] attempting to acquire leader lease kube-system/choerodon.io-nfs-provisioner…
I0219 05:37:54.081985 1 leaderelection.go:194] successfully acquired lease kube-system/choerodon.io-nfs-provisioner
I0219 05:37:54.082232 1 controller.go:631] Starting provisioner controller choerodon.io/nfs-provisioner_nfs-provisioner-65bdd5f955-qbqnt_76aac593-3408-11e9-9604-0a580ae940ac!
I0219 05:37:54.082208 1 event.go:221] Event(v1.ObjectReference{Kind:“Endpoints”, Namespace:“kube-system”, Name:“choerodon.io-nfs-provisioner”, UID:“295d24bf-1507-11e9-9a59-4ccc6aaa26b9”, APIVersion:“v1”, ResourceVersion:“4803561”, FieldPath:""}): type: ‘Normal’ reason: ‘LeaderElection’ nfs-provisioner-65bdd5f955-qbqnt_76aac593-3408-11e9-9604-0a580ae940ac became leader
I0219 05:37:54.182572 1 controller.go:680] Started provisioner controller choerodon.io/nfs-provisioner_nfs-provisioner-65bdd5f955-qbqnt_76aac593-3408-11e9-9604-0a580ae940ac!
_E0219 05:39:23.493498 1 main.go:97] NFS server Exited Unexpectedly with err: ganesha.nfsd failed with error: signal: segmentation fault, output: _
I0219 05:39:24.493682 1 server.go:129] Running NFS server!