-
Choerodon平台版本: 0.10.0
-
遇到问题的执行步骤:ZooKeeper安装后,因为磁盘空间不足,master节点宕过一次。恢复后,zookeeper一个pod就一直卡在ContainerCreating
-
文档地址:
-
环境信息(如:节点信息):
master01 Ready master 5d v1.8.5
master02 Ready master 5d v1.8.5
master03 Ready master 5d v1.8.5
worker01 Ready 5d v1.8.5
worker04 Ready 5d v1.8.5 -
报错日志:
Type Reason Age From Message
Warning FailedMount 47m (x1204 over 2d) kubelet, worker01 Unable to mount volumes for pod “zookeeper-0_c7n-system(ef8ea10f-e6e7-11e8-b079-00505688fc77)”: timeout expired waiting for volumes to attach/mount for pod “c7n
-system”/“zookeeper-0”. list of unattached/unmounted volumes=[zookeeper] Warning FailedMount 32m (x508 over 1d) kubelet, worker01 (combined from similar events): MountVolume.SetUp failed for volume “pvc-4afee96b-e3da-11e8-8499-00505688fc77” : mount failed: exit status 32
Mounting command: systemd-run
Mounting arguments: --description=Kubernetes transient mount for /var/lib/kubelet/pods/ef8ea10f-e6e7-11e8-b079-00505688fc77/volumes/kubernetes.io~nfs/pvc-4afee96b-e3da-11e8-8499-00505688fc77 --scope – mount -t nfs -o vers=4
.1 10.233.13.16:/export/pvc-4afee96b-e3da-11e8-8499-00505688fc77 /var/lib/kubelet/pods/ef8ea10f-e6e7-11e8-b079-00505688fc77/volumes/kubernetes.io~nfs/pvc-4afee96b-e3da-11e8-8499-00505688fc77Output: Running scope as unit run-63310.scope.
mount.nfs: Connection timed out
Warning FailedSync 22m (x1273 over 2d) kubelet, worker01 Error syncing pod -
原因分析:
提出您分析问题的过程,以便我们能更准确的找到问题所在
-
疑问:
提出您对于遇到和解决该问题时的疑问
你对磁盘进行了什么操作吗
没有,把eviction-hard nodefs.available 调低了
麻烦 看下 nfs-provisioner 的日志
I1112 08:37:27.480310 1 main.go:63] Provisioner choerodon.io/nfs-provisioner specified
I1112 08:37:27.480386 1 main.go:87] Setting up NFS server!
I1112 08:37:27.582087 1 server.go:144] starting RLIMIT_NOFILE rlimit.Cur 65536, rlimit.Max 65536
I1112 08:37:27.582106 1 server.go:155] ending RLIMIT_NOFILE rlimit.Cur 1048576, rlimit.Max 1048576
I1112 08:37:27.593118 1 server.go:129] Running NFS server!
I1112 08:37:32.603135 1 leaderelection.go:185] attempting to acquire leader lease c7n-system/choerodon.io-nfs-provisioner…
E1112 08:38:04.991459 1 leaderelection.go:268] Failed to update lock: etcdserver: request timed out
E1112 08:38:23.644565 1 leaderelection.go:268] Failed to update lock: etcdserver: request timed out
E1112 08:38:40.807130 1 leaderelection.go:268] Failed to update lock: etcdserver: request timed out
E1112 08:38:58.188010 1 leaderelection.go:268] Failed to update lock: etcdserver: request timed out
E1112 08:39:15.425725 1 leaderelection.go:268] Failed to update lock: etcdserver: request timed out
E1112 08:39:33.153305 1 leaderelection.go:268] Failed to update lock: etcdserver: request timed out
E1112 08:39:51.395279 1 leaderelection.go:268] Failed to update lock: etcdserver: request timed out
E1112 08:40:10.353147 1 leaderelection.go:268] Failed to update lock: etcdserver: request timed out
E1112 08:40:27.872515 1 leaderelection.go:268] Failed to update lock: etcdserver: request timed out
E1112 08:40:45.791352 1 leaderelection.go:268] Failed to update lock: etcdserver: request timed out
I1112 08:40:53.019790 1 leaderelection.go:194] successfully acquired lease c7n-system/choerodon.io-nfs-provisioner
I1112 08:40:53.020335 1 controller.go:631] Starting provisioner controller choerodon.io/nfs-provisioner_nfs-provisioner-7984dc5957-xwvs5_32a4ab50-e656-11e8-8935-0a580ae9441d!
I1112 08:40:53.020345 1 event.go:221] Event(v1.ObjectReference{Kind:“Endpoints”, Namespace:“c7n-system”, Name:“choerodon.io-nfs-provisioner”, UID:“bd49dad2-e3d6-11e8-8499-00505688fc77”, APIVersion:“v1”, ResourceVersion
:“441227”, FieldPath:""}): type: ‘Normal’ reason: ‘LeaderElection’ nfs-provisioner-7984dc5957-xwvs5_32a4ab50-e656-11e8-8935-0a580ae9441d became leaderI1112 08:40:53.120503 1 controller.go:680] Started provisioner controller choerodon.io/nfs-provisioner_nfs-provisioner-7984dc5957-xwvs5_32a4ab50-e656-11e8-8935-0a580ae9441d!
hi 你尝试把自己该pod删除再看下 如果问题仍然存在 可以私信我们远程查看一下这个问题