集群重启了， cd 发布一直执行中

lisen2023 · 2020 年1 月 9 日 05:53

Choerodon平台版本：0.19.0
运行环境：自主搭建
问题描述：

lisen2023 · 2020 年1 月 9 日 05:55

2020-01-09 13:49:06.563  INFO 8 --- [pool-5-thread-1] i.c.w.a.s.i.ProcessInstanceServiceImpl   : 部署CD流程:1415  开始
2020-01-09 13:49:06.665  INFO 8 --- [pool-5-thread-1] i.c.w.i.delegate.DevopsDeployDelegate    : ServiceTask:subProcess0-auto.1415.1422.1429.false 开始
2020-01-09 13:49:06.744  INFO 8 --- [pool-5-thread-1] c.n.u.concurrent.ShutdownEnabledTimer    : Shutdown hook installed for: NFLoadBalancer-PingTimer-devops-service
2020-01-09 13:49:06.745  INFO 8 --- [pool-5-thread-1] c.netflix.loadbalancer.BaseLoadBalancer  : Client: devops-service instantiated a LoadBalancer: DynamicServerListLoadBalancer:{NFLoadBalancer:name=devops-service,current list of Servers=[],Load balancer stats=Zone stats: {},Server stats: []}ServerList:null
2020-01-09 13:49:06.747  INFO 8 --- [pool-5-thread-1] c.n.l.DynamicServerListLoadBalancer      : Using serverListUpdater PollingServerListUpdater
2020-01-09 13:49:06.751  INFO 8 --- [pool-5-thread-1] c.netflix.config.ChainedDynamicProperty  : Flipping property: devops-service.ribbon.ActiveConnectionsLimit to use NEXT property: niws.loadbalancer.availabilityFilteringRule.activeConnectionsLimit = 2147483647
2020-01-09 13:49:06.755  INFO 8 --- [pool-5-thread-1] c.n.l.DynamicServerListLoadBalancer      : DynamicServerListLoadBalancer for client devops-service initialized: DynamicServerListLoadBalancer:{NFLoadBalancer:name=devops-service,current list of Servers=[10.88.59.43:8060],Load balancer stats=Zone stats: {defaultzone=[Zone:defaultzone;	Instance count:1;	Active connections count: 0;	Circuit breaker tripped count: 0;	Active connections per server: 0.0;]
},Server stats: [[Server:10.88.59.43:8060;	Zone:defaultZone;	Total Requests:0;	Successive connection failure:0;	Total blackout seconds:0;	Last connection made:Thu Jan 01 08:00:00 CST 1970;	First connection made: Thu Jan 01 08:00:00 CST 1970;	Active Connections:0;	total failure count in last (1000) msecs:0;	average resp time:0.0;	90 percentile resp time:0.0;	95 percentile resp time:0.0;	min resp time:0.0;	max resp time:0.0;	stddev resp time:0.0]
]}ServerList:org.springframework.cloud.netflix.ribbon.eureka.DomainExtractingServerList@9dd030b5
2020-01-09 13:49:07.017 DEBUG 8 --- [l-1 housekeeper] com.zaxxer.hikari.pool.HikariPool        : HikariPool-1 - Pool stats (total=10, active=1, idle=9, waiting=0)
2020-01-09 13:49:07.749  INFO 8 --- [erListUpdater-1] c.netflix.config.ChainedDynamicProperty  : Flipping property: devops-service.ribbon.ActiveConnectionsLimit to use NEXT property: niws.loadbalancer.availabilityFilteringRule.activeConnectionsLimit = 2147483647
2020-01-09 13:49:09.895  INFO 8 --- [      Thread-25] i.c.w.i.delegate.DevopsDeployDelegate    : running

lisen2023 · 2020 年1 月 9 日 05:56

2020-01-09 13:48:58.239 DEBUG 7 --- [l-1 housekeeper] com.zaxxer.hikari.pool.HikariPool        : HikariPool-1 - Pool stats (total=15, active=0, idle=15, waiting=0)
2020-01-09 13:49:04.514  INFO 7 --- [readScheduler-1] s.c.a.AnnotationConfigApplicationContext : Refreshing SpringClientFactory-workflow-service: startup date [Thu Jan 09 13:49:04 CST 2020]; parent: org.springframework.boot.web.servlet.context.AnnotationConfigServletWebServerApplicationContext@da6b1f70
2020-01-09 13:49:04.545  INFO 7 --- [readScheduler-1] f.a.AutowiredAnnotationBeanPostProcessor : JSR-330 'javax.inject.Inject' annotation found and supported for autowiring
2020-01-09 13:49:04.566  INFO 7 --- [readScheduler-1] c.n.u.concurrent.ShutdownEnabledTimer    : Shutdown hook installed for: NFLoadBalancer-PingTimer-workflow-service
2020-01-09 13:49:04.567  INFO 7 --- [readScheduler-1] c.netflix.loadbalancer.BaseLoadBalancer  : Client: workflow-service instantiated a LoadBalancer: DynamicServerListLoadBalancer:{NFLoadBalancer:name=workflow-service,current list of Servers=[],Load balancer stats=Zone stats: {},Server stats: []}ServerList:null
2020-01-09 13:49:04.571  INFO 7 --- [readScheduler-1] c.n.l.DynamicServerListLoadBalancer      : Using serverListUpdater PollingServerListUpdater
2020-01-09 13:49:04.574  INFO 7 --- [readScheduler-1] c.netflix.config.ChainedDynamicProperty  : Flipping property: workflow-service.ribbon.ActiveConnectionsLimit to use NEXT property: niws.loadbalancer.availabilityFilteringRule.activeConnectionsLimit = 2147483647
2020-01-09 13:49:04.576  INFO 7 --- [readScheduler-1] c.n.l.DynamicServerListLoadBalancer      : DynamicServerListLoadBalancer for client workflow-service initialized: DynamicServerListLoadBalancer:{NFLoadBalancer:name=workflow-service,current list of Servers=[10.88.59.2:8065],Load balancer stats=Zone stats: {defaultzone=[Zone:defaultzone;	Instance count:1;	Active connections count: 0;	Circuit breaker tripped count: 0;	Active connections per server: 0.0;]
},Server stats: [[Server:10.88.59.2:8065;	Zone:defaultZone;	Total Requests:0;	Successive connection failure:0;	Total blackout seconds:0;	Last connection made:Thu Jan 01 08:00:00 CST 1970;	First connection made: Thu Jan 01 08:00:00 CST 1970;	Active Connections:0;	total failure count in last (1000) msecs:0;	average resp time:0.0;	90 percentile resp time:0.0;	95 percentile resp time:0.0;	min resp time:0.0;	max resp time:0.0;	stddev resp time:0.0]
]}ServerList:org.springframework.cloud.netflix.ribbon.eureka.DomainExtractingServerList@a8a5d0a4
2020-01-09 13:49:05.572  INFO 7 --- [erListUpdater-0] c.netflix.config.ChainedDynamicProperty  : Flipping property: workflow-service.ribbon.ActiveConnectionsLimit to use NEXT property: niws.loadbalancer.availabilityFilteringRule.activeConnectionsLimit = 2147483647
2020-01-09 13:49:06.786  INFO 7 --- [nio-8060-exec-8] i.c.d.a.s.impl.PipelineServiceImpl       : autoDeploy:stageRecordId: 1422 taskId: 1429
2020-01-09 13:49:28.240 DEBUG 7 --- [l-1 housekeeper] com.zaxxer.hikari.pool.HikariPool        : HikariPool-1 - Pool stats (total=15, active=0, idle=15, waiting=0)
2020-01-09 13:49:42.295  INFO 7 --- [nio-8060-exec-2] i.c.d.a.s.i.AgentMsgHandlerServiceImpl   : sync command status!
2020-01-09 13:49:42.324  INFO 7 --- [nio-8060-exec-4] i.c.d.a.s.i.AgentMsgHandlerServiceImpl   : sync command status!
2020-01-09 13:49:42.344  INFO 7 --- [nio-8060-exec-4] i.c.d.a.s.i.AgentMsgHandlerServiceImpl   : sync command status!
2020-01-09 13:49:58.240 DEBUG 7 --- [l-1 housekeeper] com.zaxxer.hikari.pool.HikariPool        : HikariPool-1 - Pool stats

ZmfCn · 2020 年1 月 9 日 05:59

你好,流水线失败的,手动部署可以用的吗？

lisen2023 · 2020 年1 月 9 日 06:00

手工也不行

lisen2023 · 2020 年1 月 9 日 06:01

能跟我说下这流水线 cicd 大概流程服务调用啥的吗？

ZmfCn · 2020 年1 月 9 日 06:02

手动部署也不行,GitOps日志是什么情况呢?

lisen2023 · 2020 年1 月 9 日 06:04

lisen2023 · 2020 年1 月 9 日 06:05

不一致重试了貌似也不行

ZmfCn · 2020 年1 月 9 日 06:06

这个界面持续多久了?这种第一步第二步不一致是devops-service还没解析完GitOps配置库 ,如果这个情况挺久了去平台层看看事务实例里失败或者运行中的实例, 列表用gitops关键字过滤

lisen2023 · 2020 年1 月 9 日 06:08

2020-01-09 14:05:27.897  INFO 7 --- [nio-8060-exec-1] i.c.d.a.s.i.AppServiceVersionServiceImpl : autoDeploy: versionId:1494, version:2020.1.9-140237-dev pipelineId:50
2020-01-09 14:05:28.254 DEBUG 7 --- [l-1 housekeeper] com.zaxxer.hikari.pool.HikariPool        : HikariPool-1 - Pool stats (total=15, active=0, idle=15, waiting=0)
2020-01-09 14:05:29.113  INFO 7 --- [nio-8060-exec-7] i.c.d.a.s.impl.DevopsGitServiceImpl      : ````````````````````````````` f52d707b094ba3e448da8108747492f3c79f658e
2020-01-09 14:05:29.132  INFO 7 --- [nio-8060-exec-7] i.c.d.a.s.impl.DevopsGitServiceImpl      : update devopsCommit successfully: f52d707b094ba3e448da8108747492f3c79f658e
2020-01-09 14:05:29.353  INFO 7 --- [io-8060-exec-10] i.c.d.a.s.impl.PipelineServiceImpl       : autoDeploy:stageRecordId: 1423 taskId: 1430
2020-01-09 14:05:42.296  INFO 7 --- [nio-8060-exec-2] i.c.d.a.s.i.AgentMsgHandlerServiceImpl   : sync command status!
2020-01-09 14:05:42.324  INFO 7 --- [nio-8060-exec-5] i.c.d.a.s.i.AgentMsgHandlerServiceImpl   : sync command status!
2020-01-09 14:05:42.337  INFO 7 --- [nio-8060-exec-5] i.c.d.a.s.i.AgentMsgHandlerServiceImpl   : sync command status!
2020-01-09 14:05:58.255 DEBUG 7 --- [l-1 housekeeper] com.zaxxer.hikari.pool.HikariPool        : HikariPool-1 - Pool stats (total=15, active=0, idle=15, waiting=0)
2020-01-09 14:05:59.287  WARN 7 --- [nio-8060-exec-5] com.zaxxer.hikari.pool.PoolBase          : HikariPool-1 - Failed to validate connection com.mysql.jdbc.JDBC4Connection@94c7de87 (No operations allowed after connection closed.)
2020-01-09 14:05:59.288 DEBUG 7 --- [nnection closer] com.zaxxer.hikari.pool.PoolBase          : HikariPool-1 - Closing connection com.mysql.jdbc.JDBC4Connection@94c7de87: (connection is dead)
2020-01-09 14:05:59.289 DEBUG 7 --- [nnection closer] com.zaxxer.hikari.pool.PoolBase          : HikariPool-1 - Closing connection com.mysql.jdbc.JDBC4Connection@94c7de87 failed

com.mysql.jdbc.exceptions.jdbc4.MySQLNonTransientConnectionException: No operations allowed after connection closed.
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
	at com.mysql.jdbc.Util.handleNewInstance(Util.java:425)
	at com.mysql.jdbc.Util.getInstance(Util.java:408)
	at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:919)
	at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:898)
	at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:887)
	at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:861)
	at com.mysql.jdbc.ConnectionImpl.throwConnectionClosedException(ConnectionImpl.java:1192)
	at com.mysql.jdbc.ConnectionImpl.checkClosed(ConnectionImpl.java:1187)
	at com.mysql.jdbc.ConnectionImpl.setNetworkTimeout(ConnectionImpl.java:5494)
	at com.zaxxer.hikari.pool.PoolBase.setNetworkTimeout(PoolBase.java:541)
	at com.zaxxer.hikari.pool.PoolBase.quietlyCloseConnection(PoolBase.java:129)
	at com.zaxxer.hikari.pool.HikariPool.lambda$closeConnection$1(HikariPool.java:434)
	at com.zaxxer.hikari.pool.HikariPool$$Lambda$1027.00000000C0143AC0.run(Unknown Source)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:813)