c7n-mysql 挂了

Evan · 2019 年2 月 11 日 03:26

Choerodon平台版本：0.12
运行环境：自主搭建

[root@node1 ~]# kubectl NAME agile-service-56f89c897d-9bqdk api-gateway-f4559b988-c8ftq asgard-service-684b44fbf7-8xffm c7n-mysql-64746b49cf-sfws8 c7n-redis-85887b987c-dghqx chartmuseum-chartmuseum-864cb9cc5d-65kn9 choerodon-front-6d9677bb54-fz6pp config-server-5666bbcd7c-44hj5 devops-service-6c6fb8d6c7-jdrtm file-service-78544fd6f5-qlj8r gateway-helper-6479586b55-lwtdd gitlab-56f95dc68f-qcjdm gitlab-mysql-7f9df57f66-pw6s9 gitlab-redis-68d6788d7c-8htmf gitlab-service-7d85c869cc-rjr89 harbor-harbor-adminserver-58db79cd46-xndqv harbor-harbor-database-0 harbor-harbor-jobservice-8458b65b4d-gfb7m harbor-harbor-registry-5c8774b645-8gpcs harbor-harbor-ui-59df9446f8-4jcfb harbor-redis-master-0 iam-service-64c4474558-ftmqx issue-service-78d6dc5b6d-bhdnx manager-service-675d8dbf48-l9h4j minio-c4c85b847-rtkt2 notify-service-6b5676dc7f-jm8pr oauth-server-8565b6cb5d-fjhtl register-server-56f9f76b85-8fq72 state-machine-service-84c46dfb54-tlkk6 test-manager-service-5f666d56d-tkdw5 wiki-service-b7b57c46b-45v78 xwiki-78994554dc-zpf9n get po -n c7n-system
READY STATUS RESTARTS AGE
0/1 Running 0 34d
1/1 Running 0 34d
0/1 Running 0 34d
0/1 CrashLoopBackOff 1690 38d
1/1 Running 0 38d
1/1 Running 0 38d
1/1 Running 0 32d
1/1 Running 0 37d
0/1 Running 0 34d
1/1 Running 0 34d
1/1 Running 0 34d
1/1 Running 0 10m
1/1 Running 34 37d
1/1 Running 0 37d
1/1 Running 0 34d
1/1 Running 1 38d
1/1 Running 0 38d
1/1 Running 23 38d
1/1 Running 0 38d
1/1 Running 3 38d
1/1 Running 0 38d
0/1 Running 0 26d
0/1 Running 0 34d
0/1 Running 0 34d
1/1 Running 0 38d
0/1 Running 0 34d
0/1 Running 0 34d
1/1 Running 0 37d
0/1 Running 0 34d
1/1 Running 0 33d
0/1 Running 0 33d
1/1 Running 3 33d

删除
[root@node1 ~]# kubectl delete pod c7n-mysql-64746b49cf-sfws8 -n c7n-system
pod “c7n-mysql-64746b49cf-sfws8” deleted

[root@node1 ~]# kubectl get po -n c7n-system
NAME READY STATUS RESTARTS AGE
agile-service-56f89c897d-9bqdk 1/1 Running 0 34d
api-gateway-f4559b988-c8ftq 1/1 Running 0 34d
asgard-service-684b44fbf7-8xffm 0/1 Running 0 34d
c7n-mysql-64746b49cf-kwlwj 1/1 Running 2 3m
c7n-redis-85887b987c-dghqx 1/1 Running 0 38d
chartmuseum-chartmuseum-864cb9cc5d-65kn9 1/1 Running 0 38d
choerodon-front-6d9677bb54-fz6pp 1/1 Running 0 32d
config-server-5666bbcd7c-44hj5 1/1 Running 0 37d
devops-service-6c6fb8d6c7-jdrtm 1/1 Running 0 34d
file-service-78544fd6f5-qlj8r 1/1 Running 0 34d
gateway-helper-6479586b55-lwtdd 1/1 Running 0 34d
gitlab-56f95dc68f-qcjdm 1/1 Running 0 16m
gitlab-mysql-7f9df57f66-pw6s9 1/1 Running 34 37d
gitlab-redis-68d6788d7c-8htmf 1/1 Running 0 37d
gitlab-service-7d85c869cc-rjr89 1/1 Running 0 34d
harbor-harbor-adminserver-58db79cd46-xndqv 1/1 Running 1 38d
harbor-harbor-database-0 1/1 Running 0 38d
harbor-harbor-jobservice-8458b65b4d-gfb7m 1/1 Running 23 38d
harbor-harbor-registry-5c8774b645-8gpcs 1/1 Running 0 38d
harbor-harbor-ui-59df9446f8-4jcfb 1/1 Running 3 38d
harbor-redis-master-0 1/1 Running 0 38d
iam-service-64c4474558-ftmqx 1/1 Running 0 26d
issue-service-78d6dc5b6d-bhdnx 1/1 Running 0 34d
manager-service-675d8dbf48-l9h4j 1/1 Running 0 34d
minio-c4c85b847-rtkt2 1/1 Running 0 38d
notify-service-6b5676dc7f-jm8pr 1/1 Running 0 34d
oauth-server-8565b6cb5d-fjhtl 1/1 Running 0 34d
register-server-56f9f76b85-8fq72 1/1 Running 0 37d
state-machine-service-84c46dfb54-tlkk6 1/1 Running 0 34d
test-manager-service-5f666d56d-tkdw5 1/1 Running 0 33d
wiki-service-b7b57c46b-45v78 1/1 Running 0 33d
xwiki-78994554dc-zpf9n 1/1 Running 3 33d

5分钟后

[root@node1 ~]# kubectl get po -n c7n-system
NAME READY STATUS RESTARTS AGE
agile-service-56f89c897d-9bqdk 0/1 Running 0 34d
api-gateway-f4559b988-c8ftq 1/1 Running 0 34d
asgard-service-684b44fbf7-8xffm 0/1 Running 0 34d
c7n-mysql-64746b49cf-kwlwj 0/1 CrashLoopBackOff 10 41m
c7n-redis-85887b987c-dghqx 1/1 Running 0 38d
chartmuseum-chartmuseum-864cb9cc5d-65kn9 1/1 Running 0 38d
choerodon-front-6d9677bb54-fz6pp 1/1 Running 0 32d
config-server-5666bbcd7c-44hj5 1/1 Running 0 37d
devops-service-6c6fb8d6c7-jdrtm 0/1 Running 0 34d
file-service-78544fd6f5-qlj8r 1/1 Running 0 34d
gateway-helper-6479586b55-lwtdd 1/1 Running 0 34d
gitlab-56f95dc68f-qcjdm 1/1 Running 0 54m
gitlab-mysql-7f9df57f66-pw6s9 1/1 Running 34 37d
gitlab-redis-68d6788d7c-8htmf 1/1 Running 0 37d
gitlab-service-7d85c869cc-rjr89 1/1 Running 0 34d
harbor-harbor-adminserver-58db79cd46-xndqv 1/1 Running 1 38d
harbor-harbor-database-0 1/1 Running 0 38d
harbor-harbor-jobservice-8458b65b4d-gfb7m 1/1 Running 23 38d
harbor-harbor-registry-5c8774b645-8gpcs 1/1 Running 0 38d
harbor-harbor-ui-59df9446f8-4jcfb 1/1 Running 3 38d
harbor-redis-master-0 1/1 Running 0 38d
iam-service-64c4474558-ftmqx 0/1 Running 0 27d
issue-service-78d6dc5b6d-bhdnx 0/1 Running 0 34d
manager-service-675d8dbf48-l9h4j 0/1 Running 0 34d
minio-c4c85b847-rtkt2 1/1 Running 0 38d
notify-service-6b5676dc7f-jm8pr 0/1 Running 0 34d
oauth-server-8565b6cb5d-fjhtl 0/1 Running 0 34d
register-server-56f9f76b85-8fq72 1/1 Running 0 37d
state-machine-service-84c46dfb54-tlkk6 0/1 Running 0 34d
test-manager-service-5f666d56d-tkdw5 1/1 Running 0 33d
wiki-service-b7b57c46b-45v78 0/1 Running 0 33d
xwiki-78994554dc-zpf9n 1/1 Running 3 33d

日志和描述信息
[root@node1 ~]# kubectl logs c7n-mysql-64746b49cf-kwlwj -n c7n-system
2019-02-11T02:56:40.080984Z 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
2019-02-11T02:56:40.085163Z 0 [Note] mysqld (mysqld 5.7.23) starting as process 1 …
2019-02-11T02:56:40.092132Z 0 [Note] InnoDB: PUNCH HOLE support available
2019-02-11T02:56:40.092157Z 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2019-02-11T02:56:40.092161Z 0 [Note] InnoDB: Uses event mutexes
2019-02-11T02:56:40.092164Z 0 [Note] InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier
2019-02-11T02:56:40.092166Z 0 [Note] InnoDB: Compressed tables use zlib 1.2.3
2019-02-11T02:56:40.092169Z 0 [Note] InnoDB: Using Linux native AIO
2019-02-11T02:56:40.092650Z 0 [Note] InnoDB: Number of pools: 1
2019-02-11T02:56:40.092794Z 0 [Note] InnoDB: Using CPU crc32 instructions
2019-02-11T02:56:40.094658Z 0 [Note] InnoDB: Initializing buffer pool, total size = 128M, instances = 1, chunk size = 128M
2019-02-11T02:56:40.103821Z 0 [Note] InnoDB: Completed initialization of buffer pool
2019-02-11T02:56:40.106325Z 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority().
2019-02-11T02:56:40.136906Z 0 [Note] InnoDB: Highest supported file format is Barracuda.
2019-02-11T02:56:40.140296Z 0 [Note] InnoDB: Log scan progressed past the checkpoint lsn 1985460331
2019-02-11T02:56:40.140320Z 0 [Note] InnoDB: Doing recovery: scanned up to log sequence number 1985460340
2019-02-11T02:56:40.140326Z 0 [Note] InnoDB: Database was not shutdown normally!
2019-02-11T02:56:40.140331Z 0 [Note] InnoDB: Starting crash recovery.
2019-02-11T02:56:40.748433Z 0 [Note] InnoDB: Removed temporary tablespace data file: “ibtmp1”
2019-02-11T02:56:40.748481Z 0 [Note] InnoDB: Creating shared tablespace for temporary tables
2019-02-11T02:56:40.749421Z 0 [Note] InnoDB: Setting file ‘./ibtmp1’ size to 12 MB. Physically writing the file full; Please wait …
2019-02-11T02:56:40.920311Z 0 [Note] InnoDB: File ‘./ibtmp1’ size is now 12 MB.
2019-02-11T02:56:40.923346Z 0 [Note] InnoDB: 96 redo rollback segment(s) found. 96 redo rollback segment(s) are active.
2019-02-11T02:56:40.923374Z 0 [Note] InnoDB: 32 non-redo rollback segment(s) are active.
2019-02-11T02:56:40.924758Z 0 [Note] InnoDB: 5.7.23 started; log sequence number 1985460340
2019-02-11T02:56:40.925645Z 0 [Note] Plugin ‘FEDERATED’ is disabled.
2019-02-11T02:56:40.926345Z 0 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool
2019-02-11T02:56:40.953309Z 0 [Note] InnoDB: Buffer pool(s) load completed at 190211 10:56:40
2019-02-11T02:56:40.958437Z 0 [Note] Found ca.pem, server-cert.pem and server-key.pem in data directory. Trying to enable SSL support using them.
2019-02-11T02:56:40.961664Z 0 [Warning] CA certificate ca.pem is self signed.
2019-02-11T02:56:40.965433Z 0 [Note] Server hostname (bind-address): ‘*’; port: 3306
2019-02-11T02:56:40.965563Z 0 [Note] IPv6 is available.
2019-02-11T02:56:40.965580Z 0 [Note] - ‘::’ resolves to ‘::’;
2019-02-11T02:56:40.965606Z 0 [Note] Server socket created on IP: ‘::’.
2019-02-11T02:56:40.972351Z 0 [Warning] Insecure configuration for --pid-file: Location ‘/var/run/mysqld’ in the path is accessible to all OS users. Consider choosing a different directory.
2019-02-11T02:56:40.987944Z 0 [Warning] ‘user’ entry ‘root@localhost’ ignored in --skip-name-resolve mode.
2019-02-11T02:56:40.988004Z 0 [Warning] ‘user’ entry ‘mysql.session@localhost’ ignored in --skip-name-resolve mode.
2019-02-11T02:56:40.988017Z 0 [Warning] ‘user’ entry ‘mysql.sys@localhost’ ignored in --skip-name-resolve mode.
2019-02-11T02:56:40.988230Z 0 [Warning] ‘db’ entry ‘performance_schema mysql.session@localhost’ ignored in --skip-name-resolve mode.
2019-02-11T02:56:40.988251Z 0 [Warning] ‘db’ entry ‘sys mysql.sys@localhost’ ignored in --skip-name-resolve mode.
2019-02-11T02:56:40.988498Z 0 [Warning] ‘proxies_priv’ entry ‘@ root@localhost’ ignored in --skip-name-resolve mode.
2019-02-11T02:56:41.013421Z 0 [Warning] ‘tables_priv’ entry ‘user mysql.session@localhost’ ignored in --skip-name-resolve mode.
2019-02-11T02:56:41.013444Z 0 [Warning] ‘tables_priv’ entry ‘sys_config mysql.sys@localhost’ ignored in --skip-name-resolve mode.
2019-02-11T02:56:41.136718Z 0 [Note] Event Scheduler: Loaded 0 events
2019-02-11T02:56:41.137018Z 0 [Note] mysqld: ready for connections.
Version: ‘5.7.23’ socket: ‘/var/run/mysqld/mysqld.sock’ port: 3306 MySQL Community Server (GPL)

[root@node1 ~]# kubectl describe pod c7n-mysql-64746b49cf-kwlwj -n c7n-system
Name: c7n-mysql-64746b49cf-kwlwj
Namespace: c7n-system
Node: node1/192.168.177.157
Start Time: Mon, 11 Feb 2019 10:38:48 +0800
Labels: choerodon.io/infra=mysql
choerodon.io/release=c7n-mysql
pod-template-hash=2030260579
Annotations: choerodon.io/metrics-group=mysql
choerodon.io/metrics-path=/metrics
kubernetes.io/created-by={“kind”:“SerializedReference”,“apiVersion”:“v1”,“reference”:{“kind”:“ReplicaSet”,“namespace”:“c7n-system”,“name”:“c7n-mysql-64746b49cf”,“uid”:"d7c14113-0f38-11e9-8f76-02ec013b…
Status: Running
IP: 10.233.64.48
Created By: ReplicaSet/c7n-mysql-64746b49cf
Controlled By: ReplicaSet/c7n-mysql-64746b49cf
Containers:
c7n-mysql:
Container ID: docker://62935bdfafe806362bbfe58d0f98fd7bdeee3e16d32579671a7e74f57a181623
Image: mysql:5.7.23
Image ID: docker-pullable://mysql@sha256:953b53af26805d82eca95f28df6ae82e8e15cd1e587b4c5cd06a78be80e84050
Port: 3306/TCP
State: Waiting
Reason: CrashLoopBackOff
Last State: Terminated
Reason: Error
Exit Code: 2
Started: Mon, 11 Feb 2019 10:50:27 +0800
Finished: Mon, 11 Feb 2019 10:51:29 +0800
Ready: False
Restart Count: 6
Liveness: tcp-socket :3306 delay=60s timeout=5s period=10s #success=1 #failure=3
Environment:
MYSQL_ROOT_PASSWORD: password
TZ: Asia/Shanghai
Mounts:
/etc/mysql/conf.d/my.cnf from config-volume (rw)
/var/lib/mysql from mysql (rw)
/var/run/secrets/kubernetes.io/serviceaccount from default-token-ptkjf (ro)
Conditions:
Type Status
Initialized True
Ready False
PodScheduled True
Volumes:
mysql:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: c7n-mysql-pvc
ReadOnly: false
config-volume:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: c7n-mysql-cm
Optional: false
default-token-ptkjf:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-ptkjf
Optional: false
QoS Class: BestEffort
Node-Selectors:
Tolerations:
Events:
Type Reason Age From Message

Normal Scheduled 17m default-scheduler Successfully assigned c7n-mysql-64746b49cf-kwlwj to node1
Normal SuccessfulMountVolume 17m kubelet, node1 MountVolume.SetUp succeeded for volume “config-volume”
Normal SuccessfulMountVolume 17m kubelet, node1 MountVolume.SetUp succeeded for volume “default-token-ptkjf”
Normal SuccessfulMountVolume 17m kubelet, node1 MountVolume.SetUp succeeded for volume “pvc-d36f5b89-0f38-11e9-b4d2-0285346f8fdd”
Normal Pulled 13m (x4 over 16m) kubelet, node1 Container image “mysql:5.7.23” already present on machine
Normal Created 13m (x4 over 16m) kubelet, node1 Created container
Normal Started 13m (x4 over 16m) kubelet, node1 Started container
Warning BackOff 12m (x5 over 14m) kubelet, node1 Back-off restarting failed container
Warning FailedSync 1m (x42 over 14m) kubelet, node1 Error syncing pod

vinkdong · 2019 年2 月 11 日 08:39

看下报错日志

Evan · 2019 年2 月 12 日 02:33

Warning FailedSync 1m (x42 over 14m) kubelet, node1 Error syncing pod

mysql的日志里没有报错啊，我看到的报错就是上面这句

Evan · 2019 年2 月 19 日 08:45

2019-02-19T02:03:03.628583Z 0 [ERROR] InnoDB: Cannot allocate 18446744073709549680 bytes of memory after 60 retries over 60 seconds. OS error: Cannot allocate memory (12). Check if you should increase the swap file or ulimits of your operating system. Note that on most 32-bit computers the process memory space is limited to 2 GB or 4 GB.
2019-02-19 10:03:03 0x7fb93c819700 InnoDB: Assertion failure in thread 140433560803072 in file ut0ut.cc line 957
InnoDB: Failing assertion: !m_fatal
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.7/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
02:03:03 UTC - mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
Attempting to collect some information that could help diagnose the problem.
As this is a crash and something is definitely wrong, the information
collection process might fail.

key_buffer_size=8388608
read_buffer_size=131072
max_used_connections=2
max_threads=500
thread_count=1
connection_count=1
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 206879 K bytes of memory
Hope that’s ok; if not, decrease some variables in the equation.

Thread pointer: 0x7fb934000900
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong…
stack_bottom = 7fb93c818cb8 thread_stack 0x40000
mysqld(my_print_stacktrace+0x2c)[0x7fb95a8bba6c]
mysqld(handle_fatal_signal+0x479)[0x7fb95a1e7709]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x110c0)[0x7fb9597630c0]
/lib/x86_64-linux-gnu/libc.so.6(gsignal+0xcf)[0x7fb957eeffff]
/lib/x86_64-linux-gnu/libc.so.6(abort+0x16a)[0x7fb957ef142a]
mysqld(+0x628c7b)[0x7fb95a1bdc7b]
mysqld(_ZN2ib14fatal_or_errorD2Ev+0x2b8)[0x7fb95aa8baf8]
mysqld(_Z26mem_heap_create_block_funcP16mem_block_info_tmm+0x405)[0x7fb95a96ac85]
mysqld(_Z18mem_heap_add_blockP16mem_block_info_tm+0x76)[0x7fb95a96b026]
mysqld(_Z12mem_heap_dupP16mem_block_info_tPKvm+0x54)[0x7fb95a96b1b4]
mysqld(+0xebd079)[0x7fb95aa52079]
mysqld(_Z9trx_purgemmb+0x599)[0x7fb95aa527f9]
mysqld(srv_purge_coordinator_thread+0xac5)[0x7fb95aa2a3c5]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x7494)[0x7fb959759494]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x3f)[0x7fb957fa5acf]

Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (0): Connection ID (thread ID): 0
Status: NOT_KILLED

The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.