gitlab-service 一直重启容器,无法正常部署应用

  • Choerodon平台版本:0.15.0

  • 运行环境:自主搭建

  • 问题描述:

    pod:gitlab-service容器总是不断停止后重启,导致平台上部署应用时无响应,等容器状态恢复后可继续使用

  • pod状态:

    gitlab-service-975b87497-95k8j 1/1 Running 0 16h 172.30.116.6 172.16.10.59
    gitlab-service-975b87497-95k8j 0/1 Running 0 16h 172.30.116.6 172.16.10.59
    gitlab-service-975b87497-95k8j 1/1 Running 0 16h 172.30.116.6 172.16.10.59
    gitlab-service-975b87497-95k8j 0/1 Running 0 16h 172.30.116.6 172.16.10.59
    gitlab-service-975b87497-95k8j 1/1 Running 0 16h 172.30.116.6 172.16.10.59

  • pod日志(error信息)

    2019-09-03 21:44:50.852 ERROR 1 — [ XNIO-2 task-38] io.undertow.request : UT005023: Exception handling request to /health
    org.springframework.web.util.NestedServletException: Request processing failed; nested exception is io.choerodon.core.exception.CommonException: org.gitlab4j.api.GitLabApiException: java.net.ConnectException: Connection refused (Connection refused)
    at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:982) ~[spring-webmvc-4.3.18.RELEASE.jar!/:4.3.18.RELEASE]
    at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:861) ~[spring-webmvc-4.3.18.RELEASE.jar!/:4.3.18.RELEASE]
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:687) ~[javax.servlet-api-3.1.0.jar!/:3.1.0]
    at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846) ~[spring-webmvc-4.3.18.RELEASE.jar!/:4.3.18.RELEASE]
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) ~[javax.servlet-api-3.1.0.jar!/:3.1.0]
    at io.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.java:74) ~[undertow-servlet-1.4.25.Final.jar!/:1.4.25.Final]
    at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:129) ~[undertow-servlet-1.4.25.Final.jar!/:1.4.25.Final]
    at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:208) ~[spring-security-web-4.2.7.RELEASE.jar!/:4.2.7.RELEASE]
    at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:177) ~[spring-security-web-4.2.7.RELEASE.jar!/:4.2.7.RELEASE]
    at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) ~[undertow-servlet-1.4.25.Final.jar!/:1.4.25.Final]
    at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) ~[undertow-servlet-1.4.25.Final.jar!/:1.4.25.Final]
    at io.undertow.servlet.handlers.FilterHandler.handleRequest(FilterHandler.java:84) ~[undertow-servlet-1.4.25.Final.jar!/:1.4.25.Final]
    at io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62) ~[undertow-servlet-1.4.25.Final.jar!/:1.4.25.Final]
    at io.undertow.servlet.handlers.ServletChain$1.handleRequest(ServletChain.java:64) ~[undertow-servlet-1.4.25.Final.jar!/:1.4.25.Final]
    at io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36) ~[undertow-servlet-1.4.25.Final.jar!/:1.4.25.Final]
    at io.undertow.servlet.handlers.security.SSLInformationAssociationHandler.handleRequest(SSLInformationAssociationHandler.java:132) ~[undertow-servlet-1.4.25.Final.jar!/:1.4.25.Final]
    at io.undertow.servlet.handlers.security.ServletAuthenticationCallHandler.handleRequest(ServletAuthenticationCallHandler.java:57) ~[undertow-servlet-1.4.25.Final.jar!/:1.4.25.Final]
    at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) ~[undertow-core-1.4.25.Final.jar!/:1.4.25.Final]
    at io.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:46) ~[undertow-core-1.4.25.Final.jar!/:1.4.25.Final]
    at io.undertow.servlet.handlers.security.ServletConfidentialityConstraintHandler.handleRequest(ServletConfidentialityConstraintHandler.java:64) ~[undertow-servlet-1.4.25.Final.jar!/:1.4.25.Final]
    at io.undertow.security.handlers.AuthenticationMechanismsHandler.handleRequest(AuthenticationMechanismsHandler.java:60) ~[undertow-core-1.4.25.Final.jar!/:1.4.25.Final]
    at io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:77) ~[undertow-servlet-1.4.25.Final.jar!/:1.4.25.Final]
    at io.undertow.security.handlers.AbstractSecurityContextAssociationHandler.handleRequest(AbstractSecurityContextAssociationHandler.java:43) ~[undertow-core-1.4.25.Final.jar!/:1.4.25.Final]
    at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) ~[undertow-core-1.4.25.Final.jar!/:1.4.25.Final]
    at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) ~[undertow-core-1.4.25.Final.jar!/:1.4.25.Final]
    at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:292) ~[undertow-servlet-1.4.25.Final.jar!/:1.4.25.Final]
    at io.undertow.servlet.handlers.ServletInitialHandler.access$100(ServletInitialHandler.java:81) ~[undertow-servlet-1.4.25.Final.jar!/:1.4.25.Final]
    at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:138) ~[undertow-servlet-1.4.25.Final.jar!/:1.4.25.Final]
    at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:135) ~[undertow-servlet-1.4.25.Final.jar!/:1.4.25.Final]
    at io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48) [undertow-servlet-1.4.25.Final.jar!/:1.4.25.Final]
    at io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43) [undertow-servlet-1.4.25.Final.jar!/:1.4.25.Final]
    at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:272) ~[undertow-servlet-1.4.25.Final.jar!/:1.4.25.Final]
    at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81) ~[undertow-servlet-1.4.25.Final.jar!/:1.4.25.Final]
    at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:104) [undertow-servlet-1.4.25.Final.jar!/:1.4.25.Final]
    at io.undertow.server.Connectors.executeRootHandler(Connectors.java:336) [undertow-core-1.4.25.Final.jar!/:1.4.25.Final]
    at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:830) [undertow-core-1.4.25.Final.jar!/:1.4.25.Final]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_181]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_181]
    at java.lang.Thread.run(Thread.java:748) [na:1.8.0_181]
    Caused by: io.choerodon.core.exception.CommonException: org.gitlab4j.api.GitLabApiException: java.net.ConnectException: Connection refused (Connection refused)
    at io.choerodon.gitlab.domain.config.GitlabHealthy.health(GitlabHealthy.java:33) ~[classes!/:0.15.0]
    at org.springframework.boot.actuate.health.CompositeHealthIndicator.health(CompositeHealthIndicator.java:68) ~[spring-boot-actuator-1.5.14.RELEASE.jar!/:1.5.14.RELEASE]
    at org.springframework.boot.actuate.endpoint.HealthEndpoint.invoke(HealthEndpoint.java:86) ~[spring-boot-actuator-1.5.14.RELEASE.jar!/:1.5.14.RELEASE]
    at org.springframework.boot.actuate.endpoint.mvc.HealthMvcEndpoint.getCurrentHealth(HealthMvcEndpoint.java:178) ~[spring-boot-actuator-1.5.14.RELEASE.jar!/:1.5.14.RELEASE]
    at org.springframework.boot.actuate.endpoint.mvc.HealthMvcEndpoint.getHealth(HealthMvcEndpoint.java:167) ~[spring-boot-actuator-1.5.14.RELEASE.jar!/:1.5.14.RELEASE]
    at org.springframework.boot.actuate.endpoint.mvc.HealthMvcEndpoint.invoke(HealthMvcEndpoint.java:144) ~[spring-boot-actuator-1.5.14.RELEASE.jar!/:1.5.14.RELEASE]
    at sun.reflect.GeneratedMethodAccessor250.invoke(Unknown Source) ~[na:na]
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_181]
    at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_181]
    at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205) ~[spring-web-4.3.18.RELEASE.jar!/:4.3.18.RELEASE]
    at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:133) ~[spring-web-4.3.18.RELEASE.jar!/:4.3.18.RELEASE]
    at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:97) ~[spring-webmvc-4.3.18.RELEASE.jar!/:4.3.18.RELEASE]
    at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:827) ~[spring-webmvc-4.3.18.RELEASE.jar!/:4.3.18.RELEASE]
    at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:738) ~[spring-webmvc-4.3.18.RELEASE.jar!/:4.3.18.RELEASE]
    at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85) ~[spring-webmvc-4.3.18.RELEASE.jar!/:4.3.18.RELEASE]
    at org.springframework.boot.actuate.autoconfigure.EndpointWebMvcChildContextConfiguration$CompositeHandlerAdapter.handle(EndpointWebMvcChildContextConfiguration.java:286) ~[spring-boot-actuator-1.5.14.RELEASE.jar!/:1.5.14.RELEASE]
    at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:967) ~[spring-webmvc-4.3.18.RELEASE.jar!/:4.3.18.RELEASE]
    at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:901) ~[spring-webmvc-4.3.18.RELEASE.jar!/:4.3.18.RELEASE]
    at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970) ~[spring-webmvc-4.3.18.RELEASE.jar!/:4.3.18.RELEASE]
    … 38 common frames omitted
    Caused by: org.gitlab4j.api.GitLabApiException: java.net.ConnectException: Connection refused (Connection refused)
    at org.gitlab4j.api.AbstractApi.handle(AbstractApi.java:352) ~[choerodon-gitlab4j-api-0.9.3.RELEASE.jar!/:0.9.3.RELEASE]
    at org.gitlab4j.api.AbstractApi.get(AbstractApi.java:90) ~[choerodon-gitlab4j-api-0.9.3.RELEASE.jar!/:0.9.3.RELEASE]
    at org.gitlab4j.api.UserApi.getCurrentUser(UserApi.java:452) ~[choerodon-gitlab4j-api-0.9.3.RELEASE.jar!/:0.9.3.RELEASE]
    at io.choerodon.gitlab.domain.config.GitlabHealthy.health(GitlabHealthy.java:26) ~[classes!/:0.15.0]
    … 56 common frames omitted
    Caused by: javax.ws.rs.ProcessingException: java.net.ConnectException: Connection refused (Connection refused)
    at org.glassfish.jersey.client.internal.HttpUrlConnector.apply(HttpUrlConnector.java:287) ~[jersey-client-2.25.1.jar!/:na]
    at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:252) ~[jersey-client-2.25.1.jar!/:na]
    at org.glassfish.jersey.client.JerseyInvocation$1.call(JerseyInvocation.java:684) ~[jersey-client-2.25.1.jar!/:na]
    at org.glassfish.jersey.client.JerseyInvocation$1.call(JerseyInvocation.java:681) ~[jersey-client-2.25.1.jar!/:na]
    at org.glassfish.jersey.internal.Errors.process(Errors.java:315) ~[jersey-common-2.25.1.jar!/:na]
    at org.glassfish.jersey.internal.Errors.process(Errors.java:297) ~[jersey-common-2.25.1.jar!/:na]
    at org.glassfish.jersey.internal.Errors.process(Errors.java:228) ~[jersey-common-2.25.1.jar!/:na]
    at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:444) ~[jersey-common-2.25.1.jar!/:na]
    at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:681) ~[jersey-client-2.25.1.jar!/:na]
    at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:411) ~[jersey-client-2.25.1.jar!/:na]
    at org.glassfish.jersey.client.JerseyInvocation$Builder.get(JerseyInvocation.java:311) ~[jersey-client-2.25.1.jar!/:na]
    at org.gitlab4j.api.GitLabApiClient.get(GitLabApiClient.java:322) ~[choerodon-gitlab4j-api-0.9.3.RELEASE.jar!/:0.9.3.RELEASE]
    at org.gitlab4j.api.GitLabApiClient.get(GitLabApiClient.java:310) ~[choerodon-gitlab4j-api-0.9.3.RELEASE.jar!/:0.9.3.RELEASE]
    at org.gitlab4j.api.AbstractApi.get(AbstractApi.java:88) ~[choerodon-gitlab4j-api-0.9.3.RELEASE.jar!/:0.9.3.RELEASE]
    … 58 common frames omitted
    Caused by: java.net.ConnectException: Connection refused (Connection refused)
    at java.net.PlainSocketImpl.socketConnect(Native Method) ~[na:1.8.0_181]
    at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) ~[na:1.8.0_181]
    at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) ~[na:1.8.0_181]
    at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) ~[na:1.8.0_181]
    at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[na:1.8.0_181]
    at java.net.Socket.connect(Socket.java:589) ~[na:1.8.0_181]
    at sun.net.NetworkClient.doConnect(NetworkClient.java:175) ~[na:1.8.0_181]
    at sun.net.www.http.HttpClient.openServer(HttpClient.java:463) ~[na:1.8.0_181]
    at sun.net.www.http.HttpClient.openServer(HttpClient.java:558) ~[na:1.8.0_181]
    at sun.net.www.http.HttpClient.(HttpClient.java:242) ~[na:1.8.0_181]
    at sun.net.www.http.HttpClient.New(HttpClient.java:339) ~[na:1.8.0_181]
    at sun.net.www.http.HttpClient.New(HttpClient.java:357) ~[na:1.8.0_181]
    at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:1220) ~[na:1.8.0_181]
    at sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1156) ~[na:1.8.0_181]
    at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1050) ~[na:1.8.0_181]
    at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:984) ~[na:1.8.0_181]
    at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1564) ~[na:1.8.0_181]
    at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1492) ~[na:1.8.0_181]
    at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:480) ~[na:1.8.0_181]
    at org.glassfish.jersey.client.internal.HttpUrlConnector._apply(HttpUrlConnector.java:399) ~[jersey-client-2.25.1.jar!/:na]
    at org.glassfish.jersey.client.internal.HttpUrlConnector.apply(HttpUrlConnector.java:285) ~[jersey-client-2.25.1.jar!/:na]
    … 71 common frames omitted

  • 正常情况下的健康状态

    root@gitlab-service-975b87497-95k8j:/# curl -s http://localhost:8071/health
    {“status”:“UP”,“gitlabHealthy”:{“status”:“UP”},“discoveryComposite”:{“description”:“Spring Cloud Eureka Discovery Client”,“status”:“UP”,“discoveryClient”:{“description”:“Spring Cloud Eureka Discovery Client”,“status”:“UP”,“services”:[“state-machine-service”,“notify-service”,“asgard-service”,“manager-service”,“config-server”,“file-service”,“devops-service”,“issue-service”,“gitlab-service”,“agile-service”,“go-register-server”,“api-gateway”,“iam-service”,“gateway-helper”,“oauth-server”]},“eureka”:{“description”:“Remote status from Eureka server”,“status”:“UNKNOWN”,“applications”:{“iam-service”:1,“go-register-server”:1,“agile-service”:1,“asgard-service”:1,“state-machine-service”:1,“notify-service”:1,“file-service”:1,“oauth-server”:1,“gitlab-service”:1,“config-server”:1,“issue-service”:1,“manager-service”:1,“api-gateway”:1,“gateway-helper”:1,“devops-service”:1}}},“diskSpace”:{“status”:“UP”,“total”:63315644416,“free”:14981246976,“threshold”:10485760},“refreshScope”:{“status”:“UP”},“configServer”:{“status”:“UP”,“propertySources”:[“gitlab-service-default-null”]},“hystrix”:{“status”:“UP”}}

  • 异常时候的健康状态

    root@gitlab-service-975b87497-95k8j:/# curl -s http://localhost:8071/health
    {“timestamp”:“2019-09-04 12:26:49”,“status”:500,“error”:“Internal Server Error”,“exception”:“io.choerodon.core.exception.CommonException”,“message”:“Request processing failed; nested exception is io.choerodon.core.exception.CommonException: org.gitlab4j.api.GitLabApiException: java.net.ConnectException: Connection refused (Connection refused)”,“path”:"/health"}

所有的pod昨天按照官网给出的部署启动方式,按顺序重新部署,但gitlab-service问题依旧,请问是什么原因呢,应如何解决

  1. gitlab-service会定时的执行健康检查,健康检查的内容是调取gitlab的api

  2. 根据您给出的异常信息是连接gitlab拒绝了,导致gitlab-service健康检查状态是异常的

3.你可以检查下gitlab的日志,看gitlab是否异常,或者检查gitlab-service pod所在节点和gitlab pod所在节点网络是否正常