安装时gitlat pod 状态 为 Unhealthy

Choerodon Installation management
#1

  • Choerodon平台版本: 0.18.0

  • 遇到问题的执行步骤:
    问题1： gitlat pod 状态 为 Unhealthy ，提示Readiness 探针检测失败。
    Events:
    Type Reason Age From Message

    Warning Unhealthy 17s (x362 over 1h) kubelet, sv-c7ndev02 Readiness probe failed: HTTP probe failed with statuscode: 502

[root@sv-c7ndev01 ~]# kubectl get po -n c7n-system -o wide | grep git
gitlab-5bcb86768d-gvrq7 0/1 Running 0 1h 10.233.66.20 sv-c7ndev02
gitlab-client-cbpb8 0/1 Error 0 51m 10.233.67.24 sv-c7ndev04
gitlab-client-gfnzb 0/1 Error 0 51m 10.233.66.22 sv-c7ndev02
gitlab-postgresql-postgresql-0 1/1 Running 0 1h 10.233.65.46 sv-c7ndev03
gitlab-redis-ffbc5cbbd-hw6ng 1/1 Running 0 1h 10.233.66.19 sv-c7ndev02
gitlab-user-identities-nqjg6 0/1 Completed 0 48m 10.233.64.42 sv-c7ndev01

安装git-client的步骤：
helm install c7n/mysql-client
–set env.MYSQL_HOST=c7n-mysql.c7n-system.svc
–set env.MYSQL_PORT=3306
–set env.MYSQL_USER=root
–set env.MYSQL_PASS=password
–set env.SQL_SCRIPT="
INSERT INTO iam_service.oauth_client (
name,organization_id,resource_ids,secret,scope,
authorized_grant_types,web_server_redirect_uri,
access_token_validity,refresh_token_validity,
additional_information,auto_approve,object_version_number,
created_by,creation_date,last_updated_by,last_update_date)
VALUES(‘gitlab’,1,‘default’,‘secret’,‘default’,
‘password,implicit,client_credentials,authorization_code,refresh_token’,
‘http://c7nd01-gitlab.jnpc.com.cn’,3600,3600,’{}’,‘default’,1,0,NOW(),0,NOW());"
–version 0.1.0
–name gitlab-client
–namespace c7n-system

问题2：
gitlab-client pod 状态为 error
[root@sv-c7ndev01 ~]# kubectl get po -n c7n-system -o wide | grep git
gitlab-5bcb86768d-gvrq7 0/1 Running 0 1h 10.233.66.20 sv-c7ndev02
gitlab-client-cbpb8 0/1 Error 0 51m 10.233.67.24 sv-c7ndev04
gitlab-client-gfnzb 0/1 Error 0 51m 10.233.66.22 sv-c7ndev02

但查看日志却没错误
Events:
Type Reason Age From Message

Normal Scheduled 53m default-scheduler Successfully assigned gitlab-client-cbpb8 to sv-c7ndev04
Normal SuccessfulMountVolume 53m kubelet, sv-c7ndev04 MountVolume.SetUp succeeded for volume “default-token-t9922”
Normal Pulling 53m kubelet, sv-c7ndev04 pulling image “registry.cn-hangzhou.aliyuncs.com/choerodon-tools/mysql-client:10.2.15-r0”
Normal Pulled 53m kubelet, sv-c7ndev04 Successfully pulled image “registry.cn-hangzhou.aliyuncs.com/choerodon-tools/mysql-client:10.2.15-r0”
Normal Created 53m kubelet, sv-c7ndev04 Created container
Normal Started 53m kubelet, sv-c7ndev04 Started container
Normal Pulled 53m kubelet, sv-c7ndev04 Container image “registry.cn-hangzhou.aliyuncs.com/choerodon-tools/mysql-client:10.2.15-r0” already present on machine
Normal Created 53m kubelet, sv-c7ndev04 Created container
Normal Started 53m kubelet, sv-c7ndev04 Started container

  • 文档地址:

  • 环境信息(如:节点信息):

  • 报错日志:

  • 原因分析:

    提出您分析问题的过程,以便我们能更准确的找到问题所在

  • 疑问:

    提出您对于遇到和解决该问题时的疑问

#2

现在的问题是gitlab 在执行完下面的升级后，状态部位ready。

helm upgrade gitlab c7n/gitlab \
    -f <(helm get values gitlab) \
    --set env.config.CHOERODON_OMNIAUTH_ENABLED=true \
    --set env.config.OMNIAUTH_AUTO_SIGN_IN_WITH_PROVIDER=oauth2_generic \
    --set env.config.OMNIAUTH_BLOCK_AUTO_CREATED_USERS=false \
    --set env.config.CHOERODON_API_URL=http://c7nd01-api.jnpc.com.cn \
    --set env.config.CHOERODON_CLIENT_ID=gitlab \
    --set env.config.CHOERODON_CLIENT_ID=secret \
    --version 0.5.0 \
    --namespace c7n-system





==> /var/log/gitlab/unicorn/unicorn_stdout.log <==
bundler: failed to load command: unicorn (/opt/gitlab/embedded/service/gitlab-rails/vendor/bundle/ruby/2.5.0/bin/unicorn)

==> /var/log/gitlab/unicorn/current <==
2019-08-28_04:16:03.04477 master failed to start, check stderr log for details
2019-08-28_04:16:04.04944 failed to start a new unicorn master
2019-08-28_04:16:04.06837 starting new unicorn master

==> /var/log/gitlab/unicorn/unicorn_stderr.log <==
I, [2019-08-28T12:16:04.824377 #1340]  INFO -- : Refreshing Gem list

==> /var/log/gitlab/gitlab-workhorse/current <==
2019-08-28_04:16:07.22175 time="2019-08-28T12:16:07+08:00" level=error msg=error correlation_id=nIS1nxfe4E3 error="badgateway: failed after 0s: dial unix /var/opt/gitlab/gitlab-rails/sockets/gitlab.socket: connect: connection refused" method=GET uri=/help
2019-08-28_04:16:07.22177 10.233.66.24:80 127.0.0.1:0 - - [2019/08/28:12:16:07 +0800] "GET /help HTTP/1.1" 502 2940 "" "kube-probe/1.10" 0.003

==> /var/log/gitlab/nginx/gitlab_access.log <==
10.233.66.1 - - [28/Aug/2019:12:16:07 +0800] "GET /help HTTP/1.1" 502 2940 "" "kube-probe/1.10"
^C
#3

该问题已解决。

*kubectl edit deployment gitlab -n c7n-system*

*将*

*image: dockerhub.azk8s.cn/gitlab/gitlab-ce:11.11.7-ce.0*

*改为*

*image: registry.cn-shanghai.aliyuncs.com/choerodon/gitlab-ce:11.11.7-ce.0*