Articles in this section

Troubleshooting VAST CSI Driver

Irina Malysheva
  • Updated
Follow

Steps to Troubleshoot VAST CSI Driver

Run Basic Troubleshooting Commands

Run the following commands to troubleshoot VAST CSI Driver:

  • Display all pods:

    kubectl get pods --all-namespaces -o wide

  • Check logs from the controller.

    Typically, there are four containers: csi-provisioner, csi-attacher, csi-resizer, and csi-vast-plugin. Check all of them.

    Run the following command to display the most important information:

    kubectl logs csi-vast-controller-0 --namespace vast-csi -c csi-vast-plugin | less

  • Check logs from the node.

    There are probably two containers: csi-node-driver-registrar and csi-vast-plugin . Check both.

    Run the following command to display the most important information:

    kubectl logs csi-vast-node-<NODE_ID> --namespace vast-csi -c csi-vast-plugin

  • Check to see if there are any system-level events, such as errors.

    Run the following command to list events sorted by timestamp:

    kubectl get events --sort-by=.metadata.creationTimestamp

  • Display persistent volume claims:

    kubectl get pvc

    The output is similar to the following:

    NAME         STATUS VOLUME                                   CAPACITY ACCESS MODES STORAGECLASS        AGE
shared-claim Bound  pvc-41e3cd02-1a80-4dfb-bb50-6d272a9e649e 1Mi      RWO          managed-nfs-storage 12d

  • Display persistent volumes:

    kubectl get pv

    The output is similar to the following:

    NAME                                     CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM                STORAGECLASS        REASON AGE
pvc-41e3cd02-1a80-4dfb-bb50-6d272a9e649e 1Mi      RWO          Delete         Bound  default/shared-claim managed-nfs-storage        12d

  • Display the defined storage classes:

    kubectl get sc

    The output is similar to the following:

    NAME                PROVISIONER AGE
managed-nfs-storage nfs-client  13d

Note

To get more detailed information, use the describe option on each command.

Check Kubernetes Logs

If none of the above helps, check the underlying Kubernetes logs.

The location of Kubernetes logs depends on your operating system. On CentOS, the typical location is /var/log/messages.

These logs often contain more detailed information about the container startup process and provide an indication of errors.

Check VMS Logs

VAST CSI Driver contacts VAST Management Service (VMS) to create directories and quotas on behalf of the Kubernetes cluster. So checking VMS logs can help troubleshoot your environment.

VMS logs can be found in /vast/vman/vms/log on the VMS node.

Useful information is often found in vms.log and vapi.log. For example, the vms.log can provide the following information (the mount for VAST CSI Driver is /keystest/k8s):

grep k8s vms.log
[2022-01-05 17:18:47,313] INFO [services.py:382/430] AUDIT - VMS - Manager: admin (10.61.203.63) POST /api/quotas/ with body {'create_dir': True, 'name': 'csi:nfs-vol-claim:nfs-vol-claim:pvc-0f5d878e-4fa4-4c91-a915-7a5c', 'path': '/keystest/k8s/pvc-0f5d878e-4fa4-4c91-a915-7a5c9fcdfc2c', 'hard_limit': 1048576}
[2022-01-05 17:18:47,562] INFO [services.py:837/430] Creating Quota with {'create_dir': True, 'name': 'csi:nfs-vol-claim:nfs-vol-claim:pvc-0f5d878e-4fa4-4c91-a915-7a5c', 'path': '/keystest/k8s/pvc-0f5d878e-4fa4-4c91-a915-7a5c9fcdfc2c', 'hard_limit': 1048576}
[2022-01-05 17:18:47,573] INFO [services.py:843/430] Creating directory path: /keystest/k8s/pvc-0f5d878e-4fa4-4c91-a915-7a5c9fcdfc2c
[2022-01-05 17:18:47,578] INFO [services.py:852/430] Created directory path: /keystest/k8s/pvc-0f5d878e-4fa4-4c91-a915-7a5c9fcdfc2c

Resolving PVC Timeouts

PVC Timeout due to Missing NFS Client on Kubernetes Worker Node

If a PVC does not mount with a timeout, a likely outcome is the Kubernetes worker node is missing the NFS client.

Run the kubectl logs csi-vast-node-<NODE_ID> --namespace vast-csi -c csi-vast-plugin command to check the node logs.

The output can be similar to the following:

Warning  FailedMount             50s        kubelet, k8s-a-node03    Unable to attach or mount volumes: unmounted volumes=[data], unattached volumes=[mariadb-credentials default-token-zpkp5 data config]: timed out waiting for the condition 
Warning  FailedMount             37s        kubelet, k8s-a-node03    MountVolume.SetUp failed for volume "pvc-4ebe1936-3613-4c37-957c-4d21e623aa98" : kubernetes.io/csi: mounter.SetupAt failed: rpc error: code = DeadlineExceeded desc = context deadline exceeded

To resolve the problem, install the NFS client and test mounts locally on the client, then retry via Kubernetes.

To install the NFS client:

sudo yum -y install nfs-utils

PVC Timeout due to No Communication with VMS

Kubernetes nodes can be on the VAST data network and not on the VAST management network. In this case, the return path from VMS to the Kubernetes node will fail.

Ensure that you have valid routes on the correct network.

On the Kubernetes node, run the following command and verify that the correct interface is used:

ip route get <vms_ip>

Run this command on the VMS node:

ip route get <k8s_node_ip>

Related articles

Comments

0 comments

Article is closed for comments.