zookeeper卡在ContainerCreating, mount failed: exit status 32

  • Choerodon平台版本: 0.10.0

  • 遇到问题的执行步骤:ZooKeeper安装后,因为磁盘空间不足,master节点宕过一次。恢复后,zookeeper一个pod就一直卡在ContainerCreating

  • 文档地址:

  • 环境信息(如:节点信息):
    master01 Ready master 5d v1.8.5
    master02 Ready master 5d v1.8.5
    master03 Ready master 5d v1.8.5
    worker01 Ready 5d v1.8.5
    worker04 Ready 5d v1.8.5

  • 报错日志:
    Type Reason Age From Message


    Warning FailedMount 47m (x1204 over 2d) kubelet, worker01 Unable to mount volumes for pod “zookeeper-0_c7n-system(ef8ea10f-e6e7-11e8-b079-00505688fc77)”: timeout expired waiting for volumes to attach/mount for pod “c7n
    -system”/“zookeeper-0”. list of unattached/unmounted volumes=[zookeeper] Warning FailedMount 32m (x508 over 1d) kubelet, worker01 (combined from similar events): MountVolume.SetUp failed for volume “pvc-4afee96b-e3da-11e8-8499-00505688fc77” : mount failed: exit status 32
    Mounting command: systemd-run
    Mounting arguments: --description=Kubernetes transient mount for /var/lib/kubelet/pods/ef8ea10f-e6e7-11e8-b079-00505688fc77/volumes/kubernetes.io~nfs/pvc-4afee96b-e3da-11e8-8499-00505688fc77 --scope – mount -t nfs -o vers=4
    .1 10.233.13.16:/export/pvc-4afee96b-e3da-11e8-8499-00505688fc77 /var/lib/kubelet/pods/ef8ea10f-e6e7-11e8-b079-00505688fc77/volumes/kubernetes.io~nfs/pvc-4afee96b-e3da-11e8-8499-00505688fc77Output: Running scope as unit run-63310.scope.
    mount.nfs: Connection timed out
    Warning FailedSync 22m (x1273 over 2d) kubelet, worker01 Error syncing pod

  • 原因分析:

    提出您分析问题的过程,以便我们能更准确的找到问题所在

  • 疑问:

    提出您对于遇到和解决该问题时的疑问

你对磁盘进行了什么操作吗

没有,把eviction-hard nodefs.available 调低了

麻烦 看下 nfs-provisioner 的日志

I1112 08:37:27.480310 1 main.go:63] Provisioner choerodon.io/nfs-provisioner specified
I1112 08:37:27.480386 1 main.go:87] Setting up NFS server!
I1112 08:37:27.582087 1 server.go:144] starting RLIMIT_NOFILE rlimit.Cur 65536, rlimit.Max 65536
I1112 08:37:27.582106 1 server.go:155] ending RLIMIT_NOFILE rlimit.Cur 1048576, rlimit.Max 1048576
I1112 08:37:27.593118 1 server.go:129] Running NFS server!
I1112 08:37:32.603135 1 leaderelection.go:185] attempting to acquire leader lease c7n-system/choerodon.io-nfs-provisioner…
E1112 08:38:04.991459 1 leaderelection.go:268] Failed to update lock: etcdserver: request timed out
E1112 08:38:23.644565 1 leaderelection.go:268] Failed to update lock: etcdserver: request timed out
E1112 08:38:40.807130 1 leaderelection.go:268] Failed to update lock: etcdserver: request timed out
E1112 08:38:58.188010 1 leaderelection.go:268] Failed to update lock: etcdserver: request timed out
E1112 08:39:15.425725 1 leaderelection.go:268] Failed to update lock: etcdserver: request timed out
E1112 08:39:33.153305 1 leaderelection.go:268] Failed to update lock: etcdserver: request timed out
E1112 08:39:51.395279 1 leaderelection.go:268] Failed to update lock: etcdserver: request timed out
E1112 08:40:10.353147 1 leaderelection.go:268] Failed to update lock: etcdserver: request timed out
E1112 08:40:27.872515 1 leaderelection.go:268] Failed to update lock: etcdserver: request timed out
E1112 08:40:45.791352 1 leaderelection.go:268] Failed to update lock: etcdserver: request timed out
I1112 08:40:53.019790 1 leaderelection.go:194] successfully acquired lease c7n-system/choerodon.io-nfs-provisioner
I1112 08:40:53.020335 1 controller.go:631] Starting provisioner controller choerodon.io/nfs-provisioner_nfs-provisioner-7984dc5957-xwvs5_32a4ab50-e656-11e8-8935-0a580ae9441d!
I1112 08:40:53.020345 1 event.go:221] Event(v1.ObjectReference{Kind:“Endpoints”, Namespace:“c7n-system”, Name:“choerodon.io-nfs-provisioner”, UID:“bd49dad2-e3d6-11e8-8499-00505688fc77”, APIVersion:“v1”, ResourceVersion
:“441227”, FieldPath:""}): type: ‘Normal’ reason: ‘LeaderElection’ nfs-provisioner-7984dc5957-xwvs5_32a4ab50-e656-11e8-8935-0a580ae9441d became leaderI1112 08:40:53.120503 1 controller.go:680] Started provisioner controller choerodon.io/nfs-provisioner_nfs-provisioner-7984dc5957-xwvs5_32a4ab50-e656-11e8-8935-0a580ae9441d!

hi 你尝试把自己该pod删除再看下 如果问题仍然存在 可以私信我们远程查看一下这个问题