创建应用模板失败

Choerodon User Guide-Continuous Delivery
#1

  • Choerodon平台版本: 0.10.0

  • 遇到问题的执行步骤:
    创建应用模板，输入信息后点创建按钮，报错

  • 文档地址:

  • 环境信息(如:节点信息):

  • 报错日志:
    feign.FeignException: status 500 reading GitlabServiceClient#queryGroupByName(String,Integer); content: {“failed”:true,“code”:“Service Temporarily Unavailable”,“message”:“Service Temporarily Unavailable”}

  • 原因分析:

    提出您分析问题的过程,以便我们能更准确的找到问题所在

  • 疑问:
    报这个错，是不是漏了一些步骤没有做？

#2

请查看一下Gitlab-Service pod的日志，看看里面报的什么错误

#3

[root@node1 ~]# kubectl get po -n c7n-system
NAME READY STATUS RESTARTS AGE
agile-service-769f844466-dkmmx 1/1 Running 0 16h
api-gateway-5794765f44-5d6nm 1/1 Running 0 16h
asgard-service-65879b99d6-sd8ll 1/1 Running 0 16h
c7n-mysql-64746b49cf-m729n 1/1 Running 0 16h
c7n-redis-5cf967b4b7-rw99m 1/1 Running 0 16h
chartmuseum-chartmuseum-864cb9cc5d-fvqdz 1/1 Running 0 16h
choerodon-front-57c87d7d9b-97x8x 1/1 Running 0 16h
config-server-6f788b6cd6-c25fm 1/1 Running 0 16h
devops-service-5fbc549864-zp98h 1/1 Running 0 16h
dnsmasq-5f5777c999-77ht9 1/1 Running 5 126d
file-service-558b8686c4-gfpnc 1/1 Running 0 16h
gateway-helper-76d679cb56-wsghf 1/1 Running 0 16h
gitlab-85596c9b4f-27b5s 0/1 CrashLoopBackOff 162 16h
gitlab-8d98f8cb5-5gvc8 0/1 Running 0 16h
gitlab-mysql-7f9df57f66-xdgft 1/1 Running 0 16h
gitlab-redis-6c6cd6c7f-tv8ww 1/1 Running 0 16h
gitlab-service-69696cc586-rzrhj 1/1 Running 0 16h
harbor-harbor-adminserver-848698c4cb-tlzkg 1/1 Running 3 16h
harbor-harbor-database-0 1/1 Running 0 16h
harbor-harbor-jobservice-8789bb4b-rcfv9 1/1 Running 155 16h
harbor-harbor-registry-786bc87b47-2xp85 1/1 Running 0 16h
harbor-harbor-ui-59df9446f8-rvbxg 0/1 Error 203 16h
harbor-redis-master-0 0/1 CrashLoopBackOff 200 16h
iam-service-595b85f6b9-2m4kg 1/1 Running 0 16h
kafka-0 1/1 Running 0 16h
manager-service-696694d76c-p978t 1/1 Running 0 16h
minio-c4c85b847-zddw6 1/1 Running 0 16h
nfs-client-provisioner-fd656b8b8-rsrks 1/1 Running 7 127d
notify-service-54db7976b4-vt5fl 1/1 Running 0 16h
oauth-server-844cc8bd79-qwhv2 1/1 Running 0 16h
register-server-df64c645c-5hkfw 1/1 Running 0 16h
test-manager-service-788778bfc6-qhwqh 1/1 Running 0 16h
wiki-service-d76c4c666-77sgz 1/1 Running 0 16h
xwiki-67856f8786-f44qp 1/1 Running 0 16h
zookeeper-0 1/1 Running 0 16h
[root@node1 ~]#
[root@node1 ~]#
[root@node1 ~]#
[root@node1 ~]#
[root@node1 ~]#
[root@node1 ~]#
[root@node1 ~]#
[root@node1 ~]# kubectl logs gitlab-service
Error from server (NotFound): pods “gitlab-service” not found
[root@node1 ~]# kubectl logs gitlab-service-69696cc586-rzrhj
Error from server (NotFound): pods “gitlab-service-69696cc586-rzrhj” not found

按照pod name看日志，报错了

#4

kubectl logs gitlab-service-69696cc586-rzrhj -n c7n-system

#5

2019-02-21 11:04:21.288 INFO [gitlab-service,152aed032e4c211d,99b4438278aa00d2,true] 1 — [ XNIO-3 task-33] i.c.r.h.ControllerExceptionHandler : exception info io.choerodon.core.exception.FeignException: Service Temporarily Unavailable
at io.choerodon.gitlab.infra.common.client.Gitlab4jClient.createGitLabApi(Gitlab4jClient.java:36)
at io.choerodon.gitlab.infra.common.client.Gitlab4jClient.getGitLabApi(Gitlab4jClient.java:69)
at io.choerodon.gitlab.infra.common.client.Gitlab4jClient.getGitLabApi(Gitlab4jClient.java:53)
at io.choerodon.gitlab.app.service.impl.GroupServiceImpl.queryGroupByName(GroupServiceImpl.java:93)
at io.choerodon.gitlab.api.controller.v1.GroupsController.queryGroupByName(GroupsController.java:217)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205)
at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:133)
at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:97)
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:827)
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:738)
at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85)
at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:967)
at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:901)
at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970)
at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:861)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:687)
at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
at io.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.java:74)
at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:129)
at org.springframework.boot.web.filter.ApplicationContextHeaderFilter.doFilterInternal(ApplicationContextHeaderFilter.java:55)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)
at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)
at io.choerodon.resource.filter.JwtTokenFilter.doFilter(JwtTokenFilter.java:101)
at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)
at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)
at org.springframework.boot.actuate.trace.WebRequestTraceFilter.doFilterInternal(WebRequestTraceFilter.java:111)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)
at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)
at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:208)
at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:177)
at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:347)
at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:263)
at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)
at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)
at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:99)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)
at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)
at org.springframework.web.filter.HttpPutFormContentFilter.doFilterInternal(HttpPutFormContentFilter.java:109)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)
at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)
at org.springframework.web.filter.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:93)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)
at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)
at org.springframework.cloud.sleuth.instrument.web.TraceFilter.doFilter(TraceFilter.java:186)
at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)
at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)
at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:197)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)
at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)
at org.springframework.boot.actuate.autoconfigure.MetricsFilter.doFilterInternal(MetricsFilter.java:106)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)
at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)
at io.micrometer.spring.web.servlet.WebMvcMetricsFilter.doFilterInternal(WebMvcMetricsFilter.java:106)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)
at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)
at io.undertow.servlet.handlers.FilterHandler.handleRequest(FilterHandler.java:84)
at io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62)
at io.undertow.servlet.handlers.ServletChain$1.handleRequest(ServletChain.java:64)
at io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36)
at io.undertow.servlet.handlers.security.SSLInformationAssociationHandler.handleRequest(SSLInformationAssociationHandler.java:132)
at io.undertow.servlet.handlers.security.ServletAuthenticationCallHandler.handleRequest(ServletAuthenticationCallHandler.java:57)
at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
at io.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:46)
at io.undertow.servlet.handlers.security.ServletConfidentialityConstraintHandler.handleRequest(ServletConfidentialityConstraintHandler.java:64)
at io.undertow.security.handlers.AuthenticationMechanismsHandler.handleRequest(AuthenticationMechanismsHandler.java:60)
at io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:77)
at io.undertow.security.handlers.AbstractSecurityContextAssociationHandler.handleRequest(AbstractSecurityContextAssociationHandler.java:43)
at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:292)
at io.undertow.servlet.handlers.ServletInitialHandler.access$100(ServletInitialHandler.java:81)
at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:138)
at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:135)
at io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48)
at io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43)
at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:272)
at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81)
at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:104)
at io.undertow.server.Connectors.executeRootHandler(Connectors.java:336)
at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:830)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.gitlab4j.api.GitLabApiException: Service Temporarily Unavailable
at org.gitlab4j.api.AbstractApi.validate(AbstractApi.java:330)
at org.gitlab4j.api.AbstractApi.get(AbstractApi.java:88)
at org.gitlab4j.api.UserApi.getUser(UserApi.java:231)
at org.gitlab4j.api.GitLabApi.setSudoAsId(GitLabApi.java:441)
at io.choerodon.gitlab.infra.common.client.Gitlab4jClient.createGitLabApi(Gitlab4jClient.java:32)
… 95 more

#6

你gitlab是正常的吗？

#7

访问 http://gitlab.example.choerodon.io/ ，报： 503 Service Temporarily Unavailable，好像自己就挂了，要怎样重开呢？

#8

去集群看下gitlab日志

#9

Running handlers:
There was an error running gitlab-ctl reconfigure:

service[gitaly] (gitaly::enable line 227) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received ‘1’
---- Begin output of /opt/gitlab/embedded/bin/chpst -u root /opt/gitlab/embedded/bin/sv restart /opt/gitlab/service/gitaly ----
STDOUT: timeout: run: /opt/gitlab/service/gitaly: (pid 593) 36s, got TERM
STDERR:
---- End output of /opt/gitlab/embedded/bin/chpst -u root /opt/gitlab/embedded/bin/sv restart /opt/gitlab/service/gitaly ----
Ran /opt/gitlab/embedded/bin/chpst -u root /opt/gitlab/embedded/bin/sv restart /opt/gitlab/service/gitaly returned 1

Running handlers complete
Chef Client failed. 149 resources updated in 01 minutes 06 seconds

这是输出的最后一段

#10

手动重启下gitlab试过吗？

#11

没有，应该怎么做？

#12 
kubectl  delete  po   gitlab-xxx (gitlab pod名)    -n   xxxxx（命名空间名）
#13

delete了会自动生成吗？

#14

嗯 会重新新建一个pod

#15

gitlab-85596c9b4f-27b5s
gitlab-8d98f8cb5-5gvc8
gitlab有两个pod

#16

重启过了，还是不行

==> /var/log/gitlab/nginx/gitlab_access.log <==
10.233.64.1 - - [21/Feb/2019:11:43:02 +0800] “GET /help HTTP/1.1” 502 2916 “” “kube-probe/1.9”

==> /var/log/gitlab/nginx/gitlab_error.log <==
2019/02/21 11:43:02 [error] 557#0: *35 connect() to unix:/var/opt/gitlab/gitlab-workhorse/socket failed (111: Connection refused) while connecting to upstream, client: 10.233.64.1, server: gitlab.example.choerodon.io, request: “GET /help HTTP/1.1”, upstream: “http://unix:/var/opt/gitlab/gitlab-workhorse/socket:/help”, host: “10.233.64.8:80”

==> /var/log/gitlab/nginx/gitlab_access.log <==
10.233.64.1 - - [21/Feb/2019:11:43:12 +0800] “GET /help HTTP/1.1” 502 2916 “” “kube-probe/1.9”

==> /var/log/gitlab/nginx/gitlab_error.log <==
2019/02/21 11:43:12 [error] 558#0: *37 connect() to unix:/var/opt/gitlab/gitlab-workhorse/socket failed (111: Connection refused) while connecting to upstream, client: 10.233.64.1, server: gitlab.example.choerodon.io, request: “GET /help HTTP/1.1”, upstream: “http://unix:/var/opt/gitlab/gitlab-workhorse/socket:/help”, host: “10.233.64.8:80”

#17

我也看了一下harbor的地址，也不行。。

#18

有反馈吗？

#19

已关联到topic Gitlab和Harbor报错