不同的PV可以使用相同的StorageClass，它们是一对多的关系。
PV可以设置节点亲和性。比如下图，local-storage-class-waitforfirstconsumer-pv-ubuntuc只能在节点ubuntuc上；local-storage-class-waitforfirstconsumer-pv-ubuntud只能在节点ubuntud上。
如果我们使用《研发工程师玩转Kubernetes——PVC使用storageClassName选择PV》一文中的“立即绑定”型的StorageClass。
# local_storage_class_immediate.yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:name: local-storage-class-immediate
provisioner: kubernetes.io/no-provisioner
volumeBindingMode: Immediate
则在创建PVC时，立即会选择一个PV。这样这个PVC可能绑定到节点ubuntuc对应的PV上，也可能绑定到节点ubuntud对应的PV上。现在我们假定它绑定到ubuntuc对应的PV——local-storage-class-immediate-pv-ubuntuc上。
假如PVC的使用者——Pod在调度时，清单文件要求它只能在ubuntud上使用。而其PVC却在ubuntuc上，则会调度失败。
立即绑定导致Pod调度失败的案例
StorageClass
注意volumeBindingMode是Immediate，即PVC创建时立即绑定PV。
# local_storage_class_immediate.yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:name: local-storage-class-immediate
provisioner: kubernetes.io/no-provisioner
volumeBindingMode: Immediate
PersistentVolume
ubuntuc
# local_storage_class_immediate_pv_ubuntuc.yaml
apiVersion: v1
kind: PersistentVolume
metadata:name: local-storage-class-immediate-pv-ubuntuc
spec:capacity:storage: 1MivolumeMode: FilesystemaccessModes:- ReadWriteOncepersistentVolumeReclaimPolicy: DeletestorageClassName: local-storage-class-immediatelocal:path: /tmpnodeAffinity:required:nodeSelectorTerms:- matchExpressions:- key: kubernetes.io/hostnameoperator: Invalues:- ubuntuc
ubuntud
# local_storage_class_immediate_pv_ubuntud.yaml
apiVersion: v1
kind: PersistentVolume
metadata:name: local-storage-class-immediate-pv-ubuntud
spec:capacity:storage: 1MivolumeMode: FilesystemaccessModes:- ReadWriteOncepersistentVolumeReclaimPolicy: DeletestorageClassName: local-storage-class-immediatelocal:path: /tmpnodeAffinity:required:nodeSelectorTerms:- matchExpressions:- key: kubernetes.io/hostnameoperator: Invalues:- ubuntud
PersistentVolumeClaim
# local_storage_class_immediate_pvc_600k.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:name: local-storage-class-immediate-pvc-600k
spec:resources:requests:storage: 600KiaccessModes:- ReadWriteOncestorageClassName: local-storage-class-immediate
创建完上述组件，我们查看下PVC的状态。
kubectl describe persistentvolumeclaims local-storage-class-immediate-pvc-600k
Name: local-storage-class-immediate-pvc-600k
Namespace: default
StorageClass: local-storage-class-immediate
Status: Bound
Volume: local-storage-class-immediate-pv-ubuntuc
Labels: <none>
Annotations: pv.kubernetes.io/bind-completed: yespv.kubernetes.io/bound-by-controller: yes
Finalizers: [kubernetes.io/pvc-protection]
Capacity: 1Mi
Access Modes: RWO
VolumeMode: Filesystem
Used By: <none>
Events: <none>
可以看到这个PVC处于绑定状态。
Deployment
# local_deployment_immediate.yaml
apiVersion: apps/v1
kind: Deployment
metadata:name: local-pv-app-immediate-deployment
spec:replicas: 1selector:matchLabels:app: local-pv-app-immediatetemplate:metadata:labels:app: local-pv-app-immediatespec:containers:- name: local-pv-app-immediateimage: busyboxcommand: ["/bin/sh", "-c", "if [ -f /tempdir/lockfile ] && ! { set -C; 2>/dev/null >/tempdir/lockfile; }; then tail -f /tempdir/lockfile; else exec 3>/tempdir/lockfile; if [ -n \"$POD_NAME\" ]; then name=$POD_NAME; else name=\"unknown\"; fi; while true; do echo \"this is $name.$name write something to lockfile\"; echo \"$name write something to lockfile\" >&3; sleep 1; done; fi"] volumeMounts:- name: local-pvc-volumemountPath: /tempdirenv:- name: POD_NAMEvalueFrom:fieldRef:fieldPath: metadata.namevolumes:- name: local-pvc-volumepersistentVolumeClaim:claimName: local-storage-class-immediate-pvc-600kaffinity:nodeAffinity:requiredDuringSchedulingIgnoredDuringExecution:nodeSelectorTerms:- matchExpressions:- key: kubernetes.io/hostnameoperator: Invalues:- ubuntud
这个清单要求Pod使用local-storage-class-immediate-pvc-600k这个PVC，但是要求自己只能被部署在节点ubuntud上。
错误表现
kubectl describe pod local-pv-app-immediate-deployment-6dd57d98f5-s5vpz
Name: local-pv-app-immediate-deployment-6dd57d98f5-s5vpz
Namespace: default
Priority: 0
Service Account: default
Node: <none>
Labels: app=local-pv-app-immediatepod-template-hash=6dd57d98f5
Annotations: <none>
Status: Pending
IP:
IPs: <none>
Controlled By: ReplicaSet/local-pv-app-immediate-deployment-6dd57d98f5
Containers:local-pv-app-immediate:Image: busyboxPort: <none>Host Port: <none>Command:/bin/sh-cif [ -f /tempdir/lockfile ] && ! { set -C; 2>/dev/null >/tempdir/lockfile; }; then tail -f /tempdir/lockfile; else exec 3>/tempdir/lockfile; if [ -n "$POD_NAME" ]; then name=$POD_NAME; else name="unknown"; fi; while true; do echo "this is $name.$name write something to lockfile"; echo "$name write something to lockfile" >&3; sleep 1; done; fiEnvironment:POD_NAME: local-pv-app-immediate-deployment-6dd57d98f5-s5vpz (v1:metadata.name)Mounts:/tempdir from local-pvc-volume (rw)/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-r48fn (ro)
Conditions:Type StatusPodScheduled False
Volumes:local-pvc-volume:Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)ClaimName: local-storage-class-immediate-pvc-600kReadOnly: falsekube-api-access-r48fn:Type: Projected (a volume that contains injected data from multiple sources)TokenExpirationSeconds: 3607ConfigMapName: kube-root-ca.crtConfigMapOptional: <nil>DownwardAPI: true
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300snode.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:Type Reason Age From Message---- ------ ---- ---- -------Warning FailedScheduling 8s default-scheduler 0/5 nodes are available: 1 node(s) didn't match Pod's node affinity/selector. preemption: 0/5 nodes are available: 1 Preemption is not helpful for scheduling, 4 No preemption victims found for incoming pod..
可以看到Pod调度失败，处于Pending状态。
延迟绑定导致Pod调度成功的案例
StorageClass
注意volumeBindingMode是WaitForFirstConsumer，即PVC创建时不绑定PV。而在PVC被使用（Pod被调度到）时绑定PV。
# local_storage_class_waitforfirstconsumer.yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:name: local-storage-class-waitforfirstconsumer
provisioner: kubernetes.io/no-provisioner
volumeBindingMode: WaitForFirstConsumer
PersistentVolume
和之前的PV设置相似，核心就是storageClassName不同，使用了延迟绑定的StorageClass。
ubuntuc
# local_storage_class_waitforfirstconsumer_pv_ubuntuc.yaml
apiVersion: v1
kind: PersistentVolume
metadata:name: local-storage-class-waitforfirstconsumer-pv-ubuntuc
spec:capacity:storage: 1MivolumeMode: FilesystemaccessModes:- ReadWriteOncepersistentVolumeReclaimPolicy: DeletestorageClassName: local-storage-class-waitforfirstconsumerlocal:path: /tmpnodeAffinity:required:nodeSelectorTerms:- matchExpressions:- key: kubernetes.io/hostnameoperator: Invalues:- ubuntuc
ubuntud
# local_storage_class_waitforfirstconsumer_pv_ubuntud.yaml
apiVersion: v1
kind: PersistentVolume
metadata:name: local-storage-class-waitforfirstconsumer-pv-ubuntud
spec:capacity:storage: 1MivolumeMode: FilesystemaccessModes:- ReadWriteOncepersistentVolumeReclaimPolicy: DeletestorageClassName: local-storage-class-waitforfirstconsumerlocal:path: /tmpnodeAffinity:required:nodeSelectorTerms:- matchExpressions:- key: kubernetes.io/hostnameoperator: Invalues:- ubuntud
PersistentVolumeClaim
和之前例子的区别就是storageClassName选择了延迟绑定的local-storage-class-waitforfirstconsumer。
# local_storage_class_waitforfirstconsumer_pvc_600k.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:name: local-storage-class-waitforfirstconsumer-pvc-600k
spec:resources:requests:storage: 600KiaccessModes:- ReadWriteOncestorageClassName: local-storage-class-waitforfirstconsumer
我们先创建上述组件，然后观察PVC的状态。
kubectl describe persistentvolumeclaims local-storage-class-waitforfirstconsumer-pvc-600k
Name: local-storage-class-waitforfirstconsumer-pvc-600k
Namespace: default
StorageClass: local-storage-class-waitforfirstconsumer
Status: Pending
Volume:
Labels: <none>
Annotations: <none>
Finalizers: [kubernetes.io/pvc-protection]
Capacity:
Access Modes:
VolumeMode: Filesystem
Used By: <none>
Events:Type Reason Age From Message---- ------ ---- ---- -------Normal WaitForFirstConsumer 2s (x2 over 11s) persistentvolume-controller waiting for first consumer to be created before binding
可以看到这次PVC没有立即绑定，而是处于Pending状态，且原因是等待第一个使用者触发绑定。
Deployment
# local_deployment_waitforfirstconsumer.yaml
apiVersion: apps/v1
kind: Deployment
metadata:name: local-pv-app-waitforfirstconsumer-deployment
spec:replicas: 1selector:matchLabels:app: local-pv-app-waitforfirstconsumertemplate:metadata:labels:app: local-pv-app-waitforfirstconsumerspec:containers:- name: local-pv-app-waitforfirstconsumerimage: busyboxcommand: ["/bin/sh", "-c", "if [ -f /tempdir/lockfile ] && ! { set -C; 2>/dev/null >/tempdir/lockfile; }; then tail -f /tempdir/lockfile; else exec 3>/tempdir/lockfile; if [ -n \"$POD_NAME\" ]; then name=$POD_NAME; else name=\"unknown\"; fi; while true; do echo \"this is $name.$name write something to lockfile\"; echo \"$name write something to lockfile\" >&3; sleep 1; done; fi"] volumeMounts:- name: local-pvc-volumemountPath: /tempdirenv:- name: POD_NAMEvalueFrom:fieldRef:fieldPath: metadata.namevolumes:- name: local-pvc-volumepersistentVolumeClaim:claimName: local-storage-class-waitforfirstconsumer-pvc-600kaffinity:nodeAffinity:requiredDuringSchedulingIgnoredDuringExecution:nodeSelectorTerms:- matchExpressions:- key: kubernetes.io/hostnameoperator: Invalues:- ubuntud
创建完我们查看Pod的状态。
kubectl describe pod local-pv-app-waitforfirstconsumer-deployment-84449895c4-x7ddx
Name: local-pv-app-waitforfirstconsumer-deployment-84449895c4-x7ddx
Namespace: default
Priority: 0
Service Account: default
Node: ubuntud/172.22.244.197
Start Time: Wed, 09 Aug 2023 17:07:01 +0000
Labels: app=local-pv-app-waitforfirstconsumerpod-template-hash=84449895c4
Annotations: cni.projectcalico.org/containerID: cb10dba20771f872b242bc6284eb9d790565b7f2c1a2fbb096ff1581a73d4de5cni.projectcalico.org/podIP: 10.1.202.206/32cni.projectcalico.org/podIPs: 10.1.202.206/32
Status: Running
IP: 10.1.202.206
IPs:IP: 10.1.202.206
Controlled By: ReplicaSet/local-pv-app-waitforfirstconsumer-deployment-84449895c4
Containers:local-pv-app-waitforfirstconsumer:Container ID: containerd://3fda11a2670236dc37409dd1fd6c5efae36d48bbcf1ce71266f72bd7b0b55b98Image: busyboxImage ID: docker.io/library/busybox@sha256:3fbc632167424a6d997e74f52b878d7cc478225cffac6bc977eedfe51c7f4e79Port: <none>Host Port: <none>Command:/bin/sh-cif [ -f /tempdir/lockfile ] && ! { set -C; 2>/dev/null >/tempdir/lockfile; }; then tail -f /tempdir/lockfile; else exec 3>/tempdir/lockfile; if [ -n "$POD_NAME" ]; then name=$POD_NAME; else name="unknown"; fi; while true; do echo "this is $name.$name write something to lockfile"; echo "$name write something to lockfile" >&3; sleep 1; done; fiState: RunningStarted: Wed, 09 Aug 2023 17:07:04 +0000Ready: TrueRestart Count: 0Environment:POD_NAME: local-pv-app-waitforfirstconsumer-deployment-84449895c4-x7ddx (v1:metadata.name)Mounts:/tempdir from local-pvc-volume (rw)/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-52426 (ro)
Conditions:Type StatusInitialized True Ready True ContainersReady True PodScheduled True
Volumes:local-pvc-volume:Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)ClaimName: local-storage-class-waitforfirstconsumer-pvc-600kReadOnly: falsekube-api-access-52426:Type: Projected (a volume that contains injected data from multiple sources)TokenExpirationSeconds: 3607ConfigMapName: kube-root-ca.crtConfigMapOptional: <nil>DownwardAPI: true
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300snode.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:Type Reason Age From Message---- ------ ---- ---- -------Normal Scheduled 8s default-scheduler Successfully assigned default/local-pv-app-waitforfirstconsumer-deployment-84449895c4-x7ddx to ubuntudNormal Pulling 8s kubelet Pulling image "busybox"Normal Pulled 5s kubelet Successfully pulled image "busybox" in 2.266071612s (2.266078813s including waiting)Normal Created 5s kubelet Created container local-pv-app-waitforfirstconsumerNormal Started 5s kubelet Started container local-pv-app-waitforfirstconsumer
可以看到Pod按清单要求被成功调度到ubuntud上。
kubectl describe persistentvolumeclaims local-storage-class-waitforfirstconsumer-pvc-600k
Name: local-storage-class-waitforfirstconsumer-pvc-600k
Namespace: default
StorageClass: local-storage-class-waitforfirstconsumer
Status: Bound
Volume: local-storage-class-waitforfirstconsumer-pv-ubuntud
Labels: <none>
Annotations: pv.kubernetes.io/bind-completed: yespv.kubernetes.io/bound-by-controller: yes
Finalizers: [kubernetes.io/pvc-protection]
Capacity: 1Mi
Access Modes: RWO
VolumeMode: Filesystem
Used By: local-pv-app-waitforfirstconsumer-deployment-84449895c4-x7ddx
Events:Type Reason Age From Message---- ------ ---- ---- -------Normal WaitForFirstConsumer 2m23s (x16 over 6m2s) persistentvolume-controller waiting for first consumer to be created before binding
这个Pod使用的PVC也被分配到ubuntud上。
参考资料
- https://www.qikqiak.com/k8strain/storage/local/
