Configuring Linux Server Machines as NFSoRDMA Clients

Pre-requisites

NFS over RDMA (NFSoRDMA) requires:

 

Software Packages Needed

Two software packages need to be installed on the Linux server.

1. Mellanox OFED driver (aka MOFED)

2. Vast NFSoRDMA driver

 

Where to Get the Software Packages ?

1. Send the following outputs from the Linux server to Vast

   • Linux distribution name and version. For example

     cat /etc/redhat-release

   • Linux kernel version as seen in. 

     uname -a

   • Mellanox OFED version (only if your Linux server already has MOFED already installed)

     $ ofed_info -s
     $ modinfo rpcrdma

2. Vast will provide Vast NFSoRDMA driver specific tied to the Linux kernel version of your Linux server.

   • For example

     mlnx-nfsrdma-vast_1.0_30_gbfeb8d0-OFED.4.7.1.0.0.1_kernel_3.10.0_1062.4.1.el7.x86_64.rpm

3. If a matching MOFED driver is not already installed on your Linux server, then download the matching version of Mellanox OFED driver from Mellanox website.

   • The version number is the "OFED" string embeded in Vast RPM file name (as highlighted in blue)
     In this example, you should download Mellanox OFED.4.7.1.0.0.1.

Software Installation

1. First, to install Mellanox OFED driver

   • Please follow Mellanox documentation for how to install Mellanox OFED driver.

2. Make sure the drivers were reloaded successfully before proceeding, as being instructed at the end of the MOFED installation process. 

   $ sudo /etc/init.d/openibd restart

3. Validate the current MOFED version to verify that MOFED is indeed installed:

   $ ofed_info -s
   MLNX_OFED_LINUX-4.7-1.0.0.1:
   

4. Run this command to list the RPMs and all the libraries that were installed. The command output should be as shown here:

   $ ofed_rpm_info
   ar_mgr 1.0 0.45.MLNX20190923.g5aec6dc
   cc_mgr 1.0 0.44.MLNX20190923.g5aec6dc
   dapl 2.1.10mlnx OFED.3.4.2.1.0
   dump_pr 1.0 0.40.MLNX20190923.g5aec6dc
   fabric-collector 1.1.0.MLNX20170103.89bb2aa 0.1
   gpio-mlxbf 1.0 0.g6d44a8a
   hcoll 4.4.2938 1
   i2c-mlx 1.0 0.gab579c6
   ibacm 41mlnx1 OFED.4.3.3.0.0
   ibdump 5.0.0 3
   ibsim 0.7mlnx1 0.11.g85c342b
   ibutils 1.5.7.1 0.12.gdcaeae2
   ibutils2 2.1.1 0.110.MLNX20190922.gd4efc48
   infiniband-diags 5.4.0.MLNX20190908.5f40e4f 0.1
   iser 4.7 OFED.4.7.1.0.0.1.g1c4bf42
   isert 4.7 OFED.4.7.1.0.0.1.g1c4bf42
   kernel-mft 4.13.0 102
   knem 1.1.3.90mlnx1 OFED.4.6.2.0.8.1.g8d875ee
   libdisni 1.9 OFED.4.5.0.0.6
   libibcm 41mlnx1 OFED.4.1.0.1.0
   libibmad 5.4.0.MLNX20190423.1d917ae 0.1
   libibumad 43.1.1.MLNX20190905.1080879 0.1
   libibverbs 41mlnx1 OFED.4.7.0.0.2
   libmlx4 41mlnx1 OFED.4.5.0.0.3
   libmlx5 41mlnx1 OFED.4.7.0.3.3
   libpka 1.0 1.g6cc68a2
   librdmacm 41mlnx1 OFED.4.7.0.3.3
   librxe 41mlnx1 OFED.4.4.2.4.6
   libvma 8.9.4 1
   mlnx-en 4.7 1.0.0.0.g1c4bf42
   mlnx-ethtool 5.1 1
   mlnx-iproute2 5.2.0 1
   mlnx-nfsrdma 4.7 OFED.4.7.1.0.0.1.g1c4bf42
   mlnx-nvme 4.7 OFED.4.7.1.0.0.1.g1c4bf42
   mlnx-ofa_kernel 4.7 OFED.4.7.1.0.0.1.g1c4bf42
   mlnx-rdma-rxe 4.7 OFED.4.7.1.0.0.1.g1c4bf42
   mlx-bootctl 1.1 0.g5b90483
   mlx-l3cache 0.1 1.gebb0728
   mlx-pmc 1.1 0.g1141c2e
   mlx-trio 0.1 1.g9d13513
   mlxbf-livefish 1.0 0.gec08328
   mpi-selector 1.0.3 1
   mpitests 3.2.20 e1a0676
   mstflint 4.13.0 1.41.g4e8819c
   multiperf 3.0 0.13.gcdaa426
   mxm 3.7.3112 1
   nvme-snap 2.0.1 2.mlnx
   ofed-docs 4.7 OFED.4.7.1.0.0
   openmpi 4.0.2rc3 1
   opensm 5.5.0.MLNX20190923.1c78385 0.1
   openvswitch 2.11.0 1
   perftest 4.4 0.8.g7af08be
   pka-mlxbf 1.0 0.gd3455da
   qperf 0.4.9 9.46101
   qperf 0.4.9 9
   rdma-core 47mlnx1 1
   rshim 1.8 0.g463f780
   sharp 2.0.0.MLNX20190922.a9ebf22 1
   sockperf 3.6 0.git737d1e3e5576
   srp 4.7 OFED.4.7.1.0.0.1.g1c4bf42
   srptools 41mlnx1 5
   tmfifo 1.3 0.g6aa30c7
   ucx 1.7.0 1
   ofed-scripts 4.7 OFED.4.7.1.0.0

    

5. Install the correct VAST Data RPM file:

   $ sudo rpm -vih mlnx-nfsrdma-vast_1.0_30_gbfeb8d0-OFED.4.7.1.0.0.1_kernel_3.10.0_1062.4.1.el7.x86_64.rpm

    

6. Verify that:

   • the rpcrdma.ko file can be found as shown here under /extra/mlnx-nfsrdma/ under a kernel version-specific path like in below example

   •  srcversion is the VAST Data driver version (A6E1294087824479F5BF44C):

     {file name points to this path under kernel path}
     $ modinfo rpcrdma
     filename: /lib/modules/3.10.0-1062.4.1.el7.x86_64/extra/mlnx-nfsrdma/rpcrdma.ko
     alias: xprtrdma
     alias: svcrdma
     license: Dual BSD/GPL
     description: RPC/RDMA Transport
     author: Open Grid Computing and Network Appliance, Inc.
     retpoline: Y
     rhelversion: 7.7
     srcversion: A6E1294087824479F5BF44C
     depends: ib_core,sunrpc,mlx_compat,rdma_cm
     vermagic: 3.10.0-1062.4.1.el7.x86_64 SMP mod_unload modversions
     

      

 

Preparing NFS Client for Mouting NFS

Ensure that the host can mount NFS exports. You can do this by installing the nfs-common or nfs-utils package, depending on your Linux distribution.

Mounting NFS Exports over RDMA

When mounting an export on an NFSoRDMA client, specify the following options in the mount call:

• rdma or proto=rdma

• port=20049

• vers=3 (VAST Cluster supports NVSv3 and not NFSv4.)

Your complete mount command looks like below:

# mount -t nfs -o vers=3,proto=rdma,port=20049 <VIP>:<exportpath> <dir>

in which:

• <VIP> is one of the CNode VIPs,

  Note

  If you are working with CNode groups, you'll need to specify a VIP for one of the CNodes in the relevant group.

• <exportpath> is the path to the export you are mounting, and

• <dir> is the directory onto which you are mounting the export.

 

For example, if you're mounting an export called mydir onto the directory /mnt/mydir and one of your CNode VIPs is 192.0.2.22, you could use::

# mount -t nfs -o vers=3,proto=rdma,port=20049 192.0.2.22:/mydir /mnt/mydir/

 

 

Notes:

This document assumes the followings about your Linux server as an example.

• Linux Distribution: CentOS 7.7

• Kernel: 3.10.0-1062.4.1.el7.x86_64

• MOFED Version: 4.7-1.0.0.1