添加集群失败,devops-service容器报异常


#1
  • Choerodon平台版本:0.17.0

  • 运行环境:自主搭建

  • 问题描述:

    参考了https://forum.choerodon.io/t/topic/1761这个帖子,问题不太一样,agent报错一样,但是dns、ping都正常。后面查看了devops-service容器的日志,发现下面报错

  • 执行的操作:
    已经搭建完毕,添加集群失败。

  • 报错信息(请尽量使用代码块或系统截图的形式展现):

2019-06-09 18:36:49.118 WARN 7 --- [nio-8060-exec-8] .m.m.a.ExceptionHandlerExceptionResolver : Resolved [io.choerodon.core.exception.CommonException: error.cluster.delete]

2019-06-09 18:36:50.384 DEBUG 7 --- [l-1 housekeeper] com.zaxxer.hikari.pool.HikariPool : HikariPool-1 - Pool stats (total=15, active=0, idle=15, waiting=0)

2019-06-09 18:37:12.822 ERROR 7 --- [nio-8060-exec-2] i.c.websocket.process.ProcessManager : process msg error 

java.lang.NullPointerException: null

at io.choerodon.devops.app.service.impl.DeployMsgHandlerServiceImpl.commandNotSend(DeployMsgHandlerServiceImpl.java:738)

at io.choerodon.devops.app.service.impl.DeployMsgHandlerServiceImpl$$FastClassBySpringCGLIB$$b2587643.invoke(<generated>)

at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)

at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:684)

at io.choerodon.devops.app.service.impl.DeployMsgHandlerServiceImpl$$EnhancerBySpringCGLIB$$be840434.commandNotSend(<generated>)

at io.choerodon.devops.api.eventhandler.SocketMessageHandler.process(SocketMessageHandler.java:133)

at io.choerodon.websocket.process.ProcessManager.processInter(ProcessManager.java:88)

at io.choerodon.websocket.process.ProcessManager.process(ProcessManager.java:48)

at io.choerodon.websocket.listener.SimpleMsgListener.onMsg(SimpleMsgListener.java:18)

at io.choerodon.websocket.listener.MsgListenerDecorator.onMsg(MsgListenerDecorator.java:14)

at io.choerodon.websocket.listener.AgentCommandListener.onMsg(AgentCommandListener.java:16)

at io.choerodon.websocket.helper.CommandSender.sendMsg(CommandSender.java:21)

at io.choerodon.devops.domain.service.impl.DeployServiceImpl.deleteEnv(DeployServiceImpl.java:286)

at io.choerodon.devops.app.service.impl.DevopsEnvironmentServiceImpl.deleteDeactivatedEnvironment(DevopsEnvironmentServiceImpl.java:724)

at io.choerodon.devops.app.service.impl.DevopsEnvironmentServiceImpl$$FastClassBySpringCGLIB$$2e44f241.invoke(<generated>)

at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)

at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:746)

at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)

at org.springframework.transaction.interceptor.TransactionInterceptor$$Lambda$897.0000000054039A50.proceedWithInvocation(Unknown Source)

at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:294)

at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:98)

at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185)

at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:688)

at io.choerodon.devops.app.service.impl.DevopsEnvironmentServiceImpl$$EnhancerBySpringCGLIB$$34bfe6ee.deleteDeactivatedEnvironment(<generated>)

at io.choerodon.devops.api.controller.v1.DevopsEnvironmentController.deleteDeactivatedEnvironment(DevopsEnvironmentController.java:347)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:498)

at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:209)

at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:136)

at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:102)

at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:891)

at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:797)

at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87)

at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:991)

at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:925)

at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:974)

at org.springframework.web.servlet.FrameworkServlet.doDelete(FrameworkServlet.java:899)

at javax.servlet.http.HttpServlet.service(HttpServlet.java:667)

at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:851)

at javax.servlet.http.HttpServlet.service(HttpServlet.java:742)

at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)

at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)

at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)

at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)

at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)

at io.choerodon.resource.filter.JwtTokenFilter.doFilter(JwtTokenFilter.java:90)

at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)

at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)

at org.springframework.boot.actuate.web.trace.servlet.HttpTraceFilter.doFilterInternal(HttpTraceFilter.java:90)

at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)

at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)

at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)

at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:209)

at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:178)

at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:357)

at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:270)

at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)

at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)

at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:99)

at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)

at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)

at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)

at org.springframework.web.filter.HttpPutFormContentFilter.doFilterInternal(HttpPutFormContentFilter.java:109)

at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)

at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)

at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)

at org.springframework.web.filter.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:93)

at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)

at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)

at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)

at org.springframework.boot.actuate.metrics.web.servlet.WebMvcMetricsFilter.filterAndRecordMetrics(WebMvcMetricsFilter.java:155)

at org.springframework.boot.actuate.metrics.web.servlet.WebMvcMetricsFilter.filterAndRecordMetrics(WebMvcMetricsFilter.java:123)

at org.springframework.boot.actuate.metrics.web.servlet.WebMvcMetricsFilter.doFilterInternal(WebMvcMetricsFilter.java:108)

at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)

at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)

at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)

at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:200)

at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)

at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)

at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)

at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:198)

at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96)

at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:493)

at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:140)

at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:81)

at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87)

at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:342)

at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:800)

at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)

at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:806)

at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1498)

at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)

at java.lang.Thread.run(Thread.java:813)
  • 建议:

    提出您认为不合理的地方,帮助我们优化用户操作


#2

这个异常的原因是agent集群和devops-service之间的websocket未连接成功,然后去删除环境,这是集群未连接所导致的,不是集群未连接的原因, agent里面的日志连接ws握手失败,一般devops-service里面也会有bad handshake的报错日志,可以找下报错日志,例如:


#3

你好,访问devops.xxxx 显示的是 default backend - 404 是否是devops-service ingress的问题,我的是一键部署的,然后通过分步-> 持续交付部署-> 部署devops-service->验证部署,这条命令验证成功。

kubectl edit ingress -n c7n-system
配置里没有关于devops-service的配置
c7n-slaver下有devops.xxxx
其他c7n-slaver下的域名都正常

[root@node1 ~]# kubectl get ingress -n c7n-system
NAME                      HOSTS                                                                             ADDRESS   PORTS     AGE
api-gateway               api.xxx.com                                                                         80        20h
c7n-slaverchecker         gitlab.xxx.com,minio.xxx.com,harbor.xxx.com + 6 more...             80        20h
chartmuseum-chartmuseum   chart.xxx.com                                                                       80        20h
choerodon-front           c7n.xxx.com                                                                         80        20h
gitlab                    gitlab.xxx.com                                                                      80        20h
harbor-harbor-ingress     harbor.xxx.com                                                                      80, 443   20h
minio-ing                 minio.xxx.com                                                                       80        20h
notify-service            notify.xxx.com                                                                      80        20h
xwiki                     wiki.xxx.com                                                                        80        20h

看样子应该是缺少了devops-service的ingress 接下来应该怎么操作呢?


#4

你好,目前一键部署脚本中,版本设置有误, 请执行下面的命令升级指令

helm upgrade devops-service c7n/devops-service \
    --set service.enabled=true \
    --set ingress.enabled=true \
    -f <(helm get values devops-service) \
    --version 0.17.2

#5

感谢,已解决


执行有以下错误

[root@node1 ~]# helm upgrade devops-service c7n/devops-service \
>     --set service.enabled=true \
>     --set ingress.enabled=true \
>     -f <(helm get values devops-service) \
>     --version 0.17.2
UPGRADE FAILED
ROLLING BACK
Error: jobs.batch "devops-service-init-db" already exists
Error: UPGRADE FAILED: jobs.batch "devops-service-init-db" already exists

参考了更新手册 执行了

kubectl delete job --all -n c7n-system

再次执行上面指令 发现一直卡着 后面出现

Error: timed out waiting for the condition
Error: UPGRADE FAILED: timed out waiting for the condition

看了pod ImagePullBackOff状态

c7n-system    devops-service-init-config-rpdnv                 0/1       ImagePullBackOff   0          2m
c7n-system    devops-service-init-db-jgx5j                     0/1       ImagePullBackOff   0          2m