Run the following commands to troubleshoot VAST CSI Driver:
-
Display all pods:
kubectl get pods --all-namespaces -o wide
-
Check logs from the controller.
Typically, there are four containers:
csi-provisioner
,csi-attacher
,csi-resizer
, andcsi-vast-plugin
. Check all of them.Run the following command to display the most important information:
kubectl logs csi-vast-controller-0 --namespace vast-csi -c csi-vast-plugin | less
-
Check logs from the node.
There are probably two containers:
csi-node-driver-registrar
andcsi-vast-plugin
. Check both.Run the following command to display the most important information:
kubectl logs csi-vast-node-<NODE_ID> --namespace vast-csi -c csi-vast-plugin
-
Check to see if there are any system-level events, such as errors.
Run the following command to list events sorted by timestamp:
kubectl get events --sort-by=.metadata.creationTimestamp
-
Display persistent volume claims:
kubectl get pvc
The output is similar to the following:
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE shared-claim Bound pvc-41e3cd02-1a80-4dfb-bb50-6d272a9e649e 1Mi RWO managed-nfs-storage 12d
-
Display persistent volumes:
kubectl get pv
The output is similar to the following:
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE pvc-41e3cd02-1a80-4dfb-bb50-6d272a9e649e 1Mi RWO Delete Bound default/shared-claim managed-nfs-storage 12d
-
Display the defined storage classes:
kubectl get sc
The output is similar to the following:
NAME PROVISIONER AGE managed-nfs-storage nfs-client 13d
If none of the above helps, check the underlying Kubernetes logs.
The location of Kubernetes logs depends on your operating system. On CentOS, the typical location is /var/log/messages
.
These logs often contain more detailed information about the container startup process and provide an indication of errors.
VAST CSI Driver contacts VAST Management Service (VMS) to create directories and quotas on behalf of the Kubernetes cluster. So checking VMS logs can help troubleshoot your environment.
VMS logs can be found in /vast/vman/vms/log
on the VMS node.
Useful information is often found in vms.log and vapi.log. For example, the vms.log can provide the following information (the mount for VAST CSI Driver is /keystest/k8s):
grep k8s vms.log [2022-01-05 17:18:47,313] INFO [services.py:382/430] AUDIT - VMS - Manager: admin (10.61.203.63) POST /api/quotas/ with body {'create_dir': True, 'name': 'csi:nfs-vol-claim:nfs-vol-claim:pvc-0f5d878e-4fa4-4c91-a915-7a5c', 'path': '/keystest/k8s/pvc-0f5d878e-4fa4-4c91-a915-7a5c9fcdfc2c', 'hard_limit': 1048576} [2022-01-05 17:18:47,562] INFO [services.py:837/430] Creating Quota with {'create_dir': True, 'name': 'csi:nfs-vol-claim:nfs-vol-claim:pvc-0f5d878e-4fa4-4c91-a915-7a5c', 'path': '/keystest/k8s/pvc-0f5d878e-4fa4-4c91-a915-7a5c9fcdfc2c', 'hard_limit': 1048576} [2022-01-05 17:18:47,573] INFO [services.py:843/430] Creating directory path: /keystest/k8s/pvc-0f5d878e-4fa4-4c91-a915-7a5c9fcdfc2c [2022-01-05 17:18:47,578] INFO [services.py:852/430] Created directory path: /keystest/k8s/pvc-0f5d878e-4fa4-4c91-a915-7a5c9fcdfc2c
If a PVC does not mount with a timeout, a likely outcome is the Kubernetes worker node is missing the NFS client.
Run the kubectl logs csi-vast-node-<NODE_ID> --namespace vast-csi -c csi-vast-plugin
command to check the node logs.
The output can be similar to the following:
Warning FailedMount 50s kubelet, k8s-a-node03 Unable to attach or mount volumes: unmounted volumes=[data], unattached volumes=[mariadb-credentials default-token-zpkp5 data config]: timed out waiting for the condition Warning FailedMount 37s kubelet, k8s-a-node03 MountVolume.SetUp failed for volume "pvc-4ebe1936-3613-4c37-957c-4d21e623aa98" : kubernetes.io/csi: mounter.SetupAt failed: rpc error: code = DeadlineExceeded desc = context deadline exceeded
To resolve the problem, install the NFS client and test mounts locally on the client, then retry via Kubernetes.
sudo yum -y install nfs-utils
Kubernetes nodes can be on the VAST data network and not on the VAST management network. In this case, the return path from VMS to the Kubernetes node will fail.
Ensure that you have valid routes on the correct network.
On the Kubernetes node, run the following command and verify that the correct interface is used:
ip route get <vms_ip>
Run this command on the VMS node:
ip route get <k8s_node_ip>
Comments
0 comments
Article is closed for comments.