manager-service初始化job导致mysql的pod挂了

  • Choerodon平台版本:

helm manager-service版本:0.20.0

  • 遇到问题的执行步骤:

mysql安装无异常,数据库创建job也正常执行完成,此时安装部署manager-service服务时,数据库的pod挂了。具体报错日志如下。

[root@centos7 c7nctl-0.19.0]# kubectl logs -f c7n-mysql-f6d79f9c6-hp4qj --namespace=c7n-system

2020-01-06T06:58:40.305489Z 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).

2020-01-06T06:58:41.357040Z 0 [Note] mysqld (mysqld 5.7.23) starting as process 1 …

2020-01-06T06:58:41.371879Z 0 [Note] InnoDB: PUNCH HOLE support available

2020-01-06T06:58:41.371935Z 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins

2020-01-06T06:58:41.371943Z 0 [Note] InnoDB: Uses event mutexes

2020-01-06T06:58:41.371950Z 0 [Note] InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier

2020-01-06T06:58:41.371955Z 0 [Note] InnoDB: Compressed tables use zlib 1.2.3

2020-01-06T06:58:41.371960Z 0 [Note] InnoDB: Using Linux native AIO

2020-01-06T06:58:41.372861Z 0 [Note] InnoDB: Number of pools: 1

2020-01-06T06:58:41.373121Z 0 [Note] InnoDB: Using CPU crc32 instructions

2020-01-06T06:58:41.377023Z 0 [Note] InnoDB: Initializing buffer pool, total size = 128M, instances = 1, chunk size = 128M

2020-01-06T06:58:41.393734Z 0 [Note] InnoDB: Completed initialization of buffer pool

2020-01-06T06:58:41.398174Z 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority().

2020-01-06T06:58:41.457901Z 0 [Note] InnoDB: Highest supported file format is Barracuda.

2020-01-06T06:58:41.464344Z 0 [Note] InnoDB: Log scan progressed past the checkpoint lsn 12359079

2020-01-06T06:58:41.464386Z 0 [Note] InnoDB: Doing recovery: scanned up to log sequence number 12359107

2020-01-06T06:58:41.464407Z 0 [Note] InnoDB: Database was not shutdown normally!

2020-01-06T06:58:41.464413Z 0 [Note] InnoDB: Starting crash recovery.

2020-01-06T06:58:41.705499Z 0 [Note] InnoDB: Removed temporary tablespace data file: “ibtmp1”

2020-01-06T06:58:41.705582Z 0 [Note] InnoDB: Creating shared tablespace for temporary tables

2020-01-06T06:58:41.709758Z 0 [Note] InnoDB: Setting file ‘./ibtmp1’ size to 12 MB. Physically writing the file full; Please wait …

2020-01-06T06:58:41.768575Z 0 [Note] InnoDB: File ‘./ibtmp1’ size is now 12 MB.

2020-01-06T06:58:41.773645Z 0 [Note] InnoDB: 96 redo rollback segment(s) found. 96 redo rollback segment(s) are active.

2020-01-06T06:58:41.773682Z 0 [Note] InnoDB: 32 non-redo rollback segment(s) are active.

2020-01-06T06:58:41.775065Z 0 [Note] InnoDB: Waiting for purge to start

2020-01-06T06:58:41.860436Z 0 [Warning] InnoDB: Retry attempts for reading partial data failed.

2020-01-06T06:58:41.860495Z 0 [ERROR] InnoDB: Tried to read 16384 bytes at offset 0, but was only able to read 0

2020-01-06T06:58:41.860511Z 0 [ERROR] InnoDB: Operating system error number 5 in a file operation.

2020-01-06T06:58:41.860550Z 0 [ERROR] InnoDB: Error number 5 means ‘Input/output error’

2020-01-06T06:58:41.860560Z 0 [Note] InnoDB: Some operating system error numbers are described at http://dev.mysql.com/doc/refman/5.7/en/operating-system-error-codes.html

2020-01-06T06:58:41.860571Z 0 [ERROR] InnoDB: File (unknown): ‘read’ returned OS error 105. Cannot continue operation

2020-01-06T06:58:41.860584Z 0 [ERROR] InnoDB: Cannot continue operation.

节点状态及其相关信息:

[root@centos7 c7nctl-0.19.0]# kubectl describe pod c7n-mysql-f6d79f9c6-hp4qj --namespace=c7n-system
Name: c7n-mysql-f6d79f9c6-hp4qj
Namespace: c7n-system
Priority: 0
Node: 10.6.134.55/10.6.134.55
Start Time: Wed, 25 Dec 2019 17:09:48 +0800
Labels: choerodon.io/infra=mysql
choerodon.io/release=c7n-mysql
pod-template-hash=f6d79f9c6
Annotations: choerodon.io/metrics-group: mysql
choerodon.io/metrics-path: /metrics
Status: Running
IP: 10.244.3.7
Controlled By: ReplicaSet/c7n-mysql-f6d79f9c6
Containers:
c7n-mysql:
Container ID: docker://45c5fcd6c76cc5cbc742d8ee2d1fc7ca75db4dc939b073a830c0a5022a4fa8d0
Image: mysql:5.7.23
Image ID: docker-pullable://mysql@sha256:953b53af26805d82eca95f28df6ae82e8e15cd1e587b4c5cd06a78be80e84050
Port: 3306/TCP
Host Port: 0/TCP
State: Waiting
Reason: CrashLoopBackOff
Last State: Terminated
Reason: Error
Exit Code: 3
Started: Mon, 06 Jan 2020 14:58:39 +0800
Finished: Mon, 06 Jan 2020 14:58:47 +0800
Ready: False
Restart Count: 7
Liveness: tcp-socket :3306 delay=60s timeout=5s period=10s #success=1 #failure=3
Environment:
MYSQL_ROOT_PASSWORD: Z26b@GrXk_K5
TZ: Asia/Shanghai
Mounts:
/etc/mysql/conf.d/my.cnf from config-volume (rw,path=“my.cnf”)
/var/lib/mysql from mysql (rw)
/var/run/secrets/kubernetes.io/serviceaccount from default-token-t6nhv (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
mysql:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: c7n-mysql-pvc
ReadOnly: false
config-volume:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: c7n-mysql-cm
Optional: false
default-token-t6nhv:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-t6nhv
Optional: false
QoS Class: BestEffort
Node-Selectors:
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events:
Type Reason Age From Message


Normal Created 14m (x5 over 11d) kubelet, 10.6.134.55 Created container c7n-mysql
Normal Started 14m (x5 over 11d) kubelet, 10.6.134.55 Started container c7n-mysql
Normal Pulled 14m (x4 over 16m) kubelet, 10.6.134.55 Container image “mysql:5.7.23” already present on machine
Warning BackOff 80s (x75 over 16m) kubelet, 10.6.134.55 Back-off restarting failed container

[root@centos7 c7nctl-0.19.0]# kubectl get pods --namespace=c7n-system
NAME READY STATUS RESTARTS AGE
c7n-mysql-f6d79f9c6-hp4qj 0/1 CrashLoopBackOff 10 11d

请排查你的磁盘是否正常,谢谢

不是很懂,这个是云主机,有磁盘损坏的可能性吗?

排查了,磁盘没有任何问题