集群创建成功，但是日志有错误, 环境创建失败

[ldhai]

• Choerodon平台版本：0.13.0

• 运行环境：自主搭建

• 问题描述：

  集群管理->创建集群 成功，环境流水线-> 创建环境 失败
  

  

• 执行的操作：

• 报错信息(请尽量使用代码块的形式展现)：
  [root@node4 ~]# kubectl logs choerodon-cluster-agent-devops-test-688c7d5cc9-k74jn -n choerodon
  I0218 15:18:57.782614 8 agent.go:123] KubeClient init success.
  I0218 15:18:57.782647 8 agent.go:125] Starting connect to tiller…
  I0218 15:18:57.782849 8 agent.go:127] Tiller connect success
  I0218 15:18:57.782856 8 agent.go:287] check k8s role binding…
  I0218 15:18:57.854666 8 agent.go:293] k8s role binding succeed.
  I0218 15:18:57.854715 8 agent.go:208] kubectl /usr/local/bin/kubectl
  I0218 15:18:57.854758 8 client.go:79] Started agent
  I0218 15:18:59.143071 8 sync.go:178] kubectl apply -f - , took 1.288311903s, err: , output: customresourcedefinition “c7nhelmreleases.choerodon.io” configured
  I0218 15:18:59.143153 8 worker.go:143] get command: cluster:1/init_agent
  I0218 15:18:59.143171 8 worker.go:143] get command: inter:inter/upgrade_cluster
  I0218 15:18:59.143352 8 controller.go:108] Starting controllers for envs []
  I0218 15:18:59.143393 8 client.go:173] send response key cluster:1, type init_agent_succeed
  I0218 15:18:59.143725 8 controller_utils.go:1019] Waiting for caches to sync for endpoint controller
  I0218 15:18:59.243855 8 client.go:173] send response key env:choerodon-test, type resource_sync
  I0218 15:18:59.244037 8 client.go:173] send response key env:choerodon-test, type resource_sync
  I0218 15:18:59.244067 8 client.go:173] send response key env:choerodon-test, type resource_sync
  I0218 15:18:59.244161 8 client.go:173] send response key env:choerodon-test, type resource_sync
  I0218 15:18:59.246993 8 client.go:173] send response key env:choerodon-test, type resource_sync
  I0218 15:18:59.336614 8 upgrade.go:24] cluster agent upgrade: {“token”:“817a809a-9952-4d5a-8705-92403cabfee8”,“platformCode”:“df6ab661”}
  I0218 15:18:59.336635 8 client.go:173] send response key inter:inter, type cert_manager_info
  I0218 15:18:59.336668 8 client.go:173] send response key inter:inter, type upgrade
  I0218 15:18:59.340444 8 worker.go:143] get command: cluster:1.release:choerodon-cert-manager/helm_install_release
  I0218 15:18:59.343772 8 controller_utils.go:1026] Caches are synced for endpoint controller
  I0218 15:18:59.343926 8 client.go:173] send response key env:choerodon-test, type resource_sync
  I0218 15:18:59.344032 8 client.go:173] send response key env:choerodon-test, type resource_sync
  E0218 15:19:04.623939 8 error.go:19] load chart: find chart: chart “cert-manager” version “0.1.0” not found in https://openchart.choerodon.com.cn/choerodon/infra/ repository
  I0218 15:19:04.623985 8 client.go:173] send response key cluster:1.release:choerodon-cert-manager.commit:, type helm_release_install_failed
  I0218 15:19:05.243808 8 client.go:173] send response key none:none, type namespace_update
  I0218 15:20:04.596839 8 worker.go:143] get command: cluster:1/create_env
  I0218 15:20:04.609087 8 controller.go:108] Starting controllers for envs [choerodon-test ht-dev]
  I0218 15:20:04.609464 8 controller_utils.go:1019] Waiting for caches to sync for endpoint controller
  I0218 15:20:04.709641 8 client.go:173] send response key env:choerodon-test, type resource_sync
  I0218 15:20:04.709680 8 client.go:173] send response key env:choerodon-test, type resource_sync
  I0218 15:20:04.709701 8 client.go:173] send response key env:ht-dev, type resource_sync
  I0218 15:20:04.709717 8 client.go:173] send response key env:ht-dev, type resource_sync
  I0218 15:20:04.809782 8 client.go:173] send response key env:choerodon-test, type resource_sync
  I0218 15:20:04.809907 8 client.go:173] send response key env:ht-dev, type resource_sync
  I0218 15:20:04.809956 8 client.go:173] send response key env:choerodon-test, type resource_sync
  I0218 15:20:04.809997 8 client.go:173] send response key env:choerodon-test, type resource_sync
  I0218 15:20:04.810003 8 controller_utils.go:1026] Caches are synced for endpoint controller
  I0218 15:20:04.810034 8 client.go:173] send response key env:ht-dev, type resource_sync
  I0218 15:20:04.810073 8 client.go:173] send response key env:ht-dev, type resource_sync
  I0218 15:20:04.810112 8 client.go:173] send response key env:choerodon-test, type resource_sync
  I0218 15:20:04.810157 8 client.go:173] send response key env:ht-dev, type resource_sync
  I0218 15:20:04.812693 8 client.go:173] send response key env:choerodon-test, type resource_sync
  I0218 15:20:04.812785 8 client.go:173] send response key env:ht-dev, type resource_sync
  I0218 15:20:10.712702 8 client.go:173] send response key none:none, type namespace_update
  I0218 15:20:14.609498 8 client.go:173] send response key cluster:1, type env_create_succeed
  E0218 15:20:14.609554 8 daemon.go:94] ht-dev do sync: git repo not ready
  E0218 15:20:14.789818 8 repo.go:245] env: ht-dev repo new: git clone --mirror: fatal: Could not read from remote repository.
  E0218 15:20:24.914749 8 repo.go:245] env: ht-dev repo new: git clone --mirror: fatal: Could not read from remote repository.
  E0218 15:20:35.018206 8 repo.go:245] env: ht-dev repo new: git clone --mirror: fatal: Could not read from remote repository.

• 建议：

[Lano]

第一个错误是申请证书的额外组件安装失败，可能的原因是网络不通，该组件非必须组件，但若安装失败则平台的证书申请功能无法使用，该功能是需要外部网络的。
第二个错误是远程gitops库异常，麻烦检查下该项目对应的Ops库是否正常创建，同时检查下该项目的saga事件是否正常运行，是否有失败的实例

第一个错误不会阻碍坏境的正常创建，环境异常的原因是由于第二个错误引起的

[ldhai]

网络是通的，服务器能正常访问外部网络

gitops库异常？项目对应的ops库是否创建正常，saga事件是否正常如何查看？
我看到gitlab中项目代码已经正常创建

[ldhai]

请问一下，关于第二个错误具体该如何检查？
ops库是否正常创建，saga事件是否正茬运行我该如何查看呢？

[Lano]

查看失败的实例中是否有创建项目相关的错误

[Lano]

如果都成功，在gitlab里会有一个“组织-项目-ops”的group生成，这里会记录你每个环境部署的资源，我们采用的gitops进行环境的部署管理的

[ldhai]

没有看到关于创建项目相关的错误

你看看gitlab里面group生成有没有问题

[Lano]

devops-create-env的错误是什么？麻烦截图看下报错信息

[ldhai]

异常信息如下：

java.lang.RuntimeException: com.netflix.client.ClientException: Load balancer does not have available server for client: gitlab-service
_ at org.springframework.cloud.netflix.feign.ribbon.LoadBalancerFeignClient.execute(LoadBalancerFeignClient.java:71)_
_ at feign.SynchronousMethodHandler.executeAndDecode(SynchronousMethodHandler.java:97)_
_ at feign.SynchronousMethodHandler.invoke(SynchronousMethodHandler.java:76)_
_ at feign.ReflectiveFeign$FeignInvocationHandler.invoke(ReflectiveFeign.java:103)_
_ at com.sun.proxy.$Proxy199.getProjectByName(Unknown Source)_
_ at io.choerodon.devops.infra.persistence.impl.GitlabRepositoryImpl.getProjectByName(GitlabRepositoryImpl.java:221)_
_ at io.choerodon.devops.app.service.impl.DevopsEnvironmentServiceImpl.handleCreateEnvSaga(DevopsEnvironmentServiceImpl.java:471)_
_ at io.choerodon.devops.app.service.impl.DevopsEnvironmentServiceImpl$$FastClassBySpringCGLIB$$2e44f241.invoke()_
_ at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)_
_ at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:667)_
_ at io.choerodon.devops.app.service.impl.DevopsEnvironmentServiceImpl$$EnhancerBySpringCGLIB$$798a6b42.handleCreateEnvSaga()_
_ at io.choerodon.devops.api.eventhandler.DevopsSagaHandler.devopsCreateEnv(DevopsSagaHandler.java:84)_
_ at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)_
_ at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)_
_ at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)_
_ at java.lang.reflect.Method.invoke(Method.java:498)_
_ at io.choerodon.asgard.saga.consumer.SagaConsumer.invoke(SagaConsumer.java:87)_
_ at io.choerodon.asgard.saga.consumer.SagaConsumer.lambda$null$0(SagaConsumer.java:59)_
_ at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1590)_
_ at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)_
_ at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)_
_ at java.lang.Thread.run(Thread.java:748)_
Caused by: com.netflix.client.ClientException: Load balancer does not have available server for client: gitlab-service
_ at com.netflix.loadbalancer.LoadBalancerContext.getServerFromLoadBalancer(LoadBalancerContext.java:483)_
_ at com.netflix.loadbalancer.reactive.LoadBalancerCommand$1.call(LoadBalancerCommand.java:184)_
_ at com.netflix.loadbalancer.reactive.LoadBalancerCommand$1.call(LoadBalancerCommand.java:180)_
_ at rx.Observable.unsafeSubscribe(Observable.java:10211)_
_ at rx.internal.operators.OnSubscribeConcatMap.call(OnSubscribeConcatMap.java:94)_
_ at rx.internal.operators.OnSubscribeConcatMap.call(OnSubscribeConcatMap.java:42)_
_ at rx.Observable.unsafeSubscribe(Observable.java:10211)_
_ at rx.internal.operators.OperatorRetryWithPredicate$SourceSubscriber$1.call(OperatorRetryWithPredicate.java:127)_
_ at rx.internal.schedulers.TrampolineScheduler$InnerCurrentThreadScheduler.enqueue(TrampolineScheduler.java:73)_
_ at rx.internal.schedulers.TrampolineScheduler$InnerCurrentThreadScheduler.schedule(TrampolineScheduler.java:52)_
_ at rx.internal.operators.OperatorRetryWithPredicate$SourceSubscriber.onNext(OperatorRetryWithPredicate.java:79)_
_ at rx.internal.operators.OperatorRetryWithPredicate$SourceSubscriber.onNext(OperatorRetryWithPredicate.java:45)_
_ at rx.internal.util.ScalarSynchronousObservable$WeakSingleProducer.request(ScalarSynchronousObservable.java:276)_
_ at rx.Subscriber.setProducer(Subscriber.java:209)_
_ at rx.internal.util.ScalarSynchronousObservable$JustOnSubscribe.call(ScalarSynchronousObservable.java:138)_
_ at rx.internal.util.ScalarSynchronousObservable$JustOnSubscribe.call(ScalarSynchronousObservable.java:129)_
_ at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:48)_
_ at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30)_
_ at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:48)_
_ at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30)_
_ at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:48)_
_ at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30)_
_ at rx.Observable.subscribe(Observable.java:10307)_
_ at rx.Observable.subscribe(Observable.java:10274)_
_ at rx.observables.BlockingObservable.blockForSingle(BlockingObservable.java:445)_
_ at rx.observables.BlockingObservable.single(BlockingObservable.java:342)_
_ at com.netflix.client.AbstractLoadBalancerAwareClient.executeWithLoadBalancer(AbstractLoadBalancerAwareClient.java:117)_
_ at org.springframework.cloud.netflix.feign.ribbon.LoadBalancerFeignClient.execute(LoadBalancerFeignClient.java:63)_
_ … 21 more_

[ldhai]

gitlab服务down掉的时候，gitlab-service服务也挂了， gitlab-service忘了重启了，
现在好了，把以前创建失败的环境都停用删除了，重新创建环境后，正常了， 谢谢你！

[Lano]

不客气