-
Choerodon平台版本:0.15.0
-
运行环境:自主搭建
-
问题描述:
pod:gitlab-service容器总是不断停止后重启,导致平台上部署应用时无响应,等容器状态恢复后可继续使用
-
pod状态:
gitlab-service-975b87497-95k8j 1/1 Running 0 16h 172.30.116.6 172.16.10.59
gitlab-service-975b87497-95k8j 0/1 Running 0 16h 172.30.116.6 172.16.10.59
gitlab-service-975b87497-95k8j 1/1 Running 0 16h 172.30.116.6 172.16.10.59
gitlab-service-975b87497-95k8j 0/1 Running 0 16h 172.30.116.6 172.16.10.59
gitlab-service-975b87497-95k8j 1/1 Running 0 16h 172.30.116.6 172.16.10.59 -
pod日志(error信息)
2019-09-03 21:44:50.852 ERROR 1 — [ XNIO-2 task-38] io.undertow.request : UT005023: Exception handling request to /health
org.springframework.web.util.NestedServletException: Request processing failed; nested exception is io.choerodon.core.exception.CommonException: org.gitlab4j.api.GitLabApiException: java.net.ConnectException: Connection refused (Connection refused)
at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:982) ~[spring-webmvc-4.3.18.RELEASE.jar!/:4.3.18.RELEASE]
at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:861) ~[spring-webmvc-4.3.18.RELEASE.jar!/:4.3.18.RELEASE]
at javax.servlet.http.HttpServlet.service(HttpServlet.java:687) ~[javax.servlet-api-3.1.0.jar!/:3.1.0]
at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846) ~[spring-webmvc-4.3.18.RELEASE.jar!/:4.3.18.RELEASE]
at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) ~[javax.servlet-api-3.1.0.jar!/:3.1.0]
at io.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.java:74) ~[undertow-servlet-1.4.25.Final.jar!/:1.4.25.Final]
at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:129) ~[undertow-servlet-1.4.25.Final.jar!/:1.4.25.Final]
at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:208) ~[spring-security-web-4.2.7.RELEASE.jar!/:4.2.7.RELEASE]
at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:177) ~[spring-security-web-4.2.7.RELEASE.jar!/:4.2.7.RELEASE]
at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) ~[undertow-servlet-1.4.25.Final.jar!/:1.4.25.Final]
at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) ~[undertow-servlet-1.4.25.Final.jar!/:1.4.25.Final]
at io.undertow.servlet.handlers.FilterHandler.handleRequest(FilterHandler.java:84) ~[undertow-servlet-1.4.25.Final.jar!/:1.4.25.Final]
at io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62) ~[undertow-servlet-1.4.25.Final.jar!/:1.4.25.Final]
at io.undertow.servlet.handlers.ServletChain$1.handleRequest(ServletChain.java:64) ~[undertow-servlet-1.4.25.Final.jar!/:1.4.25.Final]
at io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36) ~[undertow-servlet-1.4.25.Final.jar!/:1.4.25.Final]
at io.undertow.servlet.handlers.security.SSLInformationAssociationHandler.handleRequest(SSLInformationAssociationHandler.java:132) ~[undertow-servlet-1.4.25.Final.jar!/:1.4.25.Final]
at io.undertow.servlet.handlers.security.ServletAuthenticationCallHandler.handleRequest(ServletAuthenticationCallHandler.java:57) ~[undertow-servlet-1.4.25.Final.jar!/:1.4.25.Final]
at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) ~[undertow-core-1.4.25.Final.jar!/:1.4.25.Final]
at io.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:46) ~[undertow-core-1.4.25.Final.jar!/:1.4.25.Final]
at io.undertow.servlet.handlers.security.ServletConfidentialityConstraintHandler.handleRequest(ServletConfidentialityConstraintHandler.java:64) ~[undertow-servlet-1.4.25.Final.jar!/:1.4.25.Final]
at io.undertow.security.handlers.AuthenticationMechanismsHandler.handleRequest(AuthenticationMechanismsHandler.java:60) ~[undertow-core-1.4.25.Final.jar!/:1.4.25.Final]
at io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:77) ~[undertow-servlet-1.4.25.Final.jar!/:1.4.25.Final]
at io.undertow.security.handlers.AbstractSecurityContextAssociationHandler.handleRequest(AbstractSecurityContextAssociationHandler.java:43) ~[undertow-core-1.4.25.Final.jar!/:1.4.25.Final]
at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) ~[undertow-core-1.4.25.Final.jar!/:1.4.25.Final]
at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) ~[undertow-core-1.4.25.Final.jar!/:1.4.25.Final]
at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:292) ~[undertow-servlet-1.4.25.Final.jar!/:1.4.25.Final]
at io.undertow.servlet.handlers.ServletInitialHandler.access$100(ServletInitialHandler.java:81) ~[undertow-servlet-1.4.25.Final.jar!/:1.4.25.Final]
at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:138) ~[undertow-servlet-1.4.25.Final.jar!/:1.4.25.Final]
at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:135) ~[undertow-servlet-1.4.25.Final.jar!/:1.4.25.Final]
at io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48) [undertow-servlet-1.4.25.Final.jar!/:1.4.25.Final]
at io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43) [undertow-servlet-1.4.25.Final.jar!/:1.4.25.Final]
at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:272) ~[undertow-servlet-1.4.25.Final.jar!/:1.4.25.Final]
at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81) ~[undertow-servlet-1.4.25.Final.jar!/:1.4.25.Final]
at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:104) [undertow-servlet-1.4.25.Final.jar!/:1.4.25.Final]
at io.undertow.server.Connectors.executeRootHandler(Connectors.java:336) [undertow-core-1.4.25.Final.jar!/:1.4.25.Final]
at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:830) [undertow-core-1.4.25.Final.jar!/:1.4.25.Final]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_181]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_181]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_181]
Caused by: io.choerodon.core.exception.CommonException: org.gitlab4j.api.GitLabApiException: java.net.ConnectException: Connection refused (Connection refused)
at io.choerodon.gitlab.domain.config.GitlabHealthy.health(GitlabHealthy.java:33) ~[classes!/:0.15.0]
at org.springframework.boot.actuate.health.CompositeHealthIndicator.health(CompositeHealthIndicator.java:68) ~[spring-boot-actuator-1.5.14.RELEASE.jar!/:1.5.14.RELEASE]
at org.springframework.boot.actuate.endpoint.HealthEndpoint.invoke(HealthEndpoint.java:86) ~[spring-boot-actuator-1.5.14.RELEASE.jar!/:1.5.14.RELEASE]
at org.springframework.boot.actuate.endpoint.mvc.HealthMvcEndpoint.getCurrentHealth(HealthMvcEndpoint.java:178) ~[spring-boot-actuator-1.5.14.RELEASE.jar!/:1.5.14.RELEASE]
at org.springframework.boot.actuate.endpoint.mvc.HealthMvcEndpoint.getHealth(HealthMvcEndpoint.java:167) ~[spring-boot-actuator-1.5.14.RELEASE.jar!/:1.5.14.RELEASE]
at org.springframework.boot.actuate.endpoint.mvc.HealthMvcEndpoint.invoke(HealthMvcEndpoint.java:144) ~[spring-boot-actuator-1.5.14.RELEASE.jar!/:1.5.14.RELEASE]
at sun.reflect.GeneratedMethodAccessor250.invoke(Unknown Source) ~[na:na]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_181]
at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_181]
at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205) ~[spring-web-4.3.18.RELEASE.jar!/:4.3.18.RELEASE]
at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:133) ~[spring-web-4.3.18.RELEASE.jar!/:4.3.18.RELEASE]
at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:97) ~[spring-webmvc-4.3.18.RELEASE.jar!/:4.3.18.RELEASE]
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:827) ~[spring-webmvc-4.3.18.RELEASE.jar!/:4.3.18.RELEASE]
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:738) ~[spring-webmvc-4.3.18.RELEASE.jar!/:4.3.18.RELEASE]
at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85) ~[spring-webmvc-4.3.18.RELEASE.jar!/:4.3.18.RELEASE]
at org.springframework.boot.actuate.autoconfigure.EndpointWebMvcChildContextConfiguration$CompositeHandlerAdapter.handle(EndpointWebMvcChildContextConfiguration.java:286) ~[spring-boot-actuator-1.5.14.RELEASE.jar!/:1.5.14.RELEASE]
at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:967) ~[spring-webmvc-4.3.18.RELEASE.jar!/:4.3.18.RELEASE]
at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:901) ~[spring-webmvc-4.3.18.RELEASE.jar!/:4.3.18.RELEASE]
at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970) ~[spring-webmvc-4.3.18.RELEASE.jar!/:4.3.18.RELEASE]
… 38 common frames omitted
Caused by: org.gitlab4j.api.GitLabApiException: java.net.ConnectException: Connection refused (Connection refused)
at org.gitlab4j.api.AbstractApi.handle(AbstractApi.java:352) ~[choerodon-gitlab4j-api-0.9.3.RELEASE.jar!/:0.9.3.RELEASE]
at org.gitlab4j.api.AbstractApi.get(AbstractApi.java:90) ~[choerodon-gitlab4j-api-0.9.3.RELEASE.jar!/:0.9.3.RELEASE]
at org.gitlab4j.api.UserApi.getCurrentUser(UserApi.java:452) ~[choerodon-gitlab4j-api-0.9.3.RELEASE.jar!/:0.9.3.RELEASE]
at io.choerodon.gitlab.domain.config.GitlabHealthy.health(GitlabHealthy.java:26) ~[classes!/:0.15.0]
… 56 common frames omitted
Caused by: javax.ws.rs.ProcessingException: java.net.ConnectException: Connection refused (Connection refused)
at org.glassfish.jersey.client.internal.HttpUrlConnector.apply(HttpUrlConnector.java:287) ~[jersey-client-2.25.1.jar!/:na]
at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:252) ~[jersey-client-2.25.1.jar!/:na]
at org.glassfish.jersey.client.JerseyInvocation$1.call(JerseyInvocation.java:684) ~[jersey-client-2.25.1.jar!/:na]
at org.glassfish.jersey.client.JerseyInvocation$1.call(JerseyInvocation.java:681) ~[jersey-client-2.25.1.jar!/:na]
at org.glassfish.jersey.internal.Errors.process(Errors.java:315) ~[jersey-common-2.25.1.jar!/:na]
at org.glassfish.jersey.internal.Errors.process(Errors.java:297) ~[jersey-common-2.25.1.jar!/:na]
at org.glassfish.jersey.internal.Errors.process(Errors.java:228) ~[jersey-common-2.25.1.jar!/:na]
at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:444) ~[jersey-common-2.25.1.jar!/:na]
at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:681) ~[jersey-client-2.25.1.jar!/:na]
at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:411) ~[jersey-client-2.25.1.jar!/:na]
at org.glassfish.jersey.client.JerseyInvocation$Builder.get(JerseyInvocation.java:311) ~[jersey-client-2.25.1.jar!/:na]
at org.gitlab4j.api.GitLabApiClient.get(GitLabApiClient.java:322) ~[choerodon-gitlab4j-api-0.9.3.RELEASE.jar!/:0.9.3.RELEASE]
at org.gitlab4j.api.GitLabApiClient.get(GitLabApiClient.java:310) ~[choerodon-gitlab4j-api-0.9.3.RELEASE.jar!/:0.9.3.RELEASE]
at org.gitlab4j.api.AbstractApi.get(AbstractApi.java:88) ~[choerodon-gitlab4j-api-0.9.3.RELEASE.jar!/:0.9.3.RELEASE]
… 58 common frames omitted
Caused by: java.net.ConnectException: Connection refused (Connection refused)
at java.net.PlainSocketImpl.socketConnect(Native Method) ~[na:1.8.0_181]
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) ~[na:1.8.0_181]
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) ~[na:1.8.0_181]
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) ~[na:1.8.0_181]
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[na:1.8.0_181]
at java.net.Socket.connect(Socket.java:589) ~[na:1.8.0_181]
at sun.net.NetworkClient.doConnect(NetworkClient.java:175) ~[na:1.8.0_181]
at sun.net.www.http.HttpClient.openServer(HttpClient.java:463) ~[na:1.8.0_181]
at sun.net.www.http.HttpClient.openServer(HttpClient.java:558) ~[na:1.8.0_181]
at sun.net.www.http.HttpClient.(HttpClient.java:242) ~[na:1.8.0_181]
at sun.net.www.http.HttpClient.New(HttpClient.java:339) ~[na:1.8.0_181]
at sun.net.www.http.HttpClient.New(HttpClient.java:357) ~[na:1.8.0_181]
at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:1220) ~[na:1.8.0_181]
at sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1156) ~[na:1.8.0_181]
at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1050) ~[na:1.8.0_181]
at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:984) ~[na:1.8.0_181]
at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1564) ~[na:1.8.0_181]
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1492) ~[na:1.8.0_181]
at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:480) ~[na:1.8.0_181]
at org.glassfish.jersey.client.internal.HttpUrlConnector._apply(HttpUrlConnector.java:399) ~[jersey-client-2.25.1.jar!/:na]
at org.glassfish.jersey.client.internal.HttpUrlConnector.apply(HttpUrlConnector.java:285) ~[jersey-client-2.25.1.jar!/:na]
… 71 common frames omitted -
正常情况下的健康状态
root@gitlab-service-975b87497-95k8j:/# curl -s http://localhost:8071/health
{“status”:“UP”,“gitlabHealthy”:{“status”:“UP”},“discoveryComposite”:{“description”:“Spring Cloud Eureka Discovery Client”,“status”:“UP”,“discoveryClient”:{“description”:“Spring Cloud Eureka Discovery Client”,“status”:“UP”,“services”:[“state-machine-service”,“notify-service”,“asgard-service”,“manager-service”,“config-server”,“file-service”,“devops-service”,“issue-service”,“gitlab-service”,“agile-service”,“go-register-server”,“api-gateway”,“iam-service”,“gateway-helper”,“oauth-server”]},“eureka”:{“description”:“Remote status from Eureka server”,“status”:“UNKNOWN”,“applications”:{“iam-service”:1,“go-register-server”:1,“agile-service”:1,“asgard-service”:1,“state-machine-service”:1,“notify-service”:1,“file-service”:1,“oauth-server”:1,“gitlab-service”:1,“config-server”:1,“issue-service”:1,“manager-service”:1,“api-gateway”:1,“gateway-helper”:1,“devops-service”:1}}},“diskSpace”:{“status”:“UP”,“total”:63315644416,“free”:14981246976,“threshold”:10485760},“refreshScope”:{“status”:“UP”},“configServer”:{“status”:“UP”,“propertySources”:[“gitlab-service-default-null”]},“hystrix”:{“status”:“UP”}} -
异常时候的健康状态
root@gitlab-service-975b87497-95k8j:/# curl -s http://localhost:8071/health
{“timestamp”:“2019-09-04 12:26:49”,“status”:500,“error”:“Internal Server Error”,“exception”:“io.choerodon.core.exception.CommonException”,“message”:“Request processing failed; nested exception is io.choerodon.core.exception.CommonException: org.gitlab4j.api.GitLabApiException: java.net.ConnectException: Connection refused (Connection refused)”,“path”:"/health"}
所有的pod昨天按照官网给出的部署启动方式,按顺序重新部署,但gitlab-service问题依旧,请问是什么原因呢,应如何解决