Notice
NFSoRDMA is supported from VAST Cluster 1.2.2.
Pre-requisites
NFS over RDMA (NFSoRDMA) requires:
-
An RDMA-enabled network adapter on the client host
-
Connection to VAST Cluster over one the following transport infrastructures:
-
RDMA over Converged Ethernet (RoCE)
-
InfiniBand (IB)
-
Software Packages Needed
Two software packages need to be installed on the Linux server.
-
Mellanox OFED driver (aka MOFED)
-
Vast NFSoRDMA driver
Where to Get the Software Packages ?
-
Send the following outputs from the Linux server to Vast
-
Linux distribution name and version. For example
cat /etc/redhat-release
-
Linux kernel version as seen in.
uname -a
-
Mellanox OFED version (only if your Linux server already has MOFED already installed)
$ ofed_info -s
$ modinfo rpcrdma
-
-
Vast will provide Vast NFSoRDMA driver specific tied to the Linux kernel version of your Linux server.
-
For example
mlnx-nfsrdma-vast_1.0_30_gbfeb8d0-OFED.4.7.1.0.0.1_kernel_3.10.0_1062.4.1.el7.x86_64.rpm
-
-
If a matching MOFED driver is not already installed on your Linux server, then download the matching version of Mellanox OFED driver from Mellanox website.
-
The version number is the "OFED" string embeded in Vast RPM file name (as highlighted in blue)
In this example, you should download Mellanox OFED.4.7.1.0.0.1.
-
Software Installation
-
First, to install Mellanox OFED driver
-
Please follow Mellanox documentation for how to install Mellanox OFED driver.
-
-
Make sure the drivers were reloaded successfully before proceeding, as being instructed at the end of the MOFED installation process.
$ sudo /etc/init.d/openibd restart
-
Validate the current MOFED version to verify that MOFED is indeed installed:
$ ofed_info -s MLNX_OFED_LINUX-4.7-1.0.0.1:
-
Run this command to list the RPMs and all the libraries that were installed. The command output should be as shown here:
$ ofed_rpm_info ar_mgr 1.0 0.45.MLNX20190923.g5aec6dc cc_mgr 1.0 0.44.MLNX20190923.g5aec6dc dapl 2.1.10mlnx OFED.3.4.2.1.0 dump_pr 1.0 0.40.MLNX20190923.g5aec6dc fabric-collector 1.1.0.MLNX20170103.89bb2aa 0.1 gpio-mlxbf 1.0 0.g6d44a8a hcoll 4.4.2938 1 i2c-mlx 1.0 0.gab579c6 ibacm 41mlnx1 OFED.4.3.3.0.0 ibdump 5.0.0 3 ibsim 0.7mlnx1 0.11.g85c342b ibutils 1.5.7.1 0.12.gdcaeae2 ibutils2 2.1.1 0.110.MLNX20190922.gd4efc48 infiniband-diags 5.4.0.MLNX20190908.5f40e4f 0.1 iser 4.7 OFED.4.7.1.0.0.1.g1c4bf42 isert 4.7 OFED.4.7.1.0.0.1.g1c4bf42 kernel-mft 4.13.0 102 knem 1.1.3.90mlnx1 OFED.4.6.2.0.8.1.g8d875ee libdisni 1.9 OFED.4.5.0.0.6 libibcm 41mlnx1 OFED.4.1.0.1.0 libibmad 5.4.0.MLNX20190423.1d917ae 0.1 libibumad 43.1.1.MLNX20190905.1080879 0.1 libibverbs 41mlnx1 OFED.4.7.0.0.2 libmlx4 41mlnx1 OFED.4.5.0.0.3 libmlx5 41mlnx1 OFED.4.7.0.3.3 libpka 1.0 1.g6cc68a2 librdmacm 41mlnx1 OFED.4.7.0.3.3 librxe 41mlnx1 OFED.4.4.2.4.6 libvma 8.9.4 1 mlnx-en 4.7 1.0.0.0.g1c4bf42 mlnx-ethtool 5.1 1 mlnx-iproute2 5.2.0 1 mlnx-nfsrdma 4.7 OFED.4.7.1.0.0.1.g1c4bf42 mlnx-nvme 4.7 OFED.4.7.1.0.0.1.g1c4bf42 mlnx-ofa_kernel 4.7 OFED.4.7.1.0.0.1.g1c4bf42 mlnx-rdma-rxe 4.7 OFED.4.7.1.0.0.1.g1c4bf42 mlx-bootctl 1.1 0.g5b90483 mlx-l3cache 0.1 1.gebb0728 mlx-pmc 1.1 0.g1141c2e mlx-trio 0.1 1.g9d13513 mlxbf-livefish 1.0 0.gec08328 mpi-selector 1.0.3 1 mpitests 3.2.20 e1a0676 mstflint 4.13.0 1.41.g4e8819c multiperf 3.0 0.13.gcdaa426 mxm 3.7.3112 1 nvme-snap 2.0.1 2.mlnx ofed-docs 4.7 OFED.4.7.1.0.0 openmpi 4.0.2rc3 1 opensm 5.5.0.MLNX20190923.1c78385 0.1 openvswitch 2.11.0 1 perftest 4.4 0.8.g7af08be pka-mlxbf 1.0 0.gd3455da qperf 0.4.9 9.46101 qperf 0.4.9 9 rdma-core 47mlnx1 1 rshim 1.8 0.g463f780 sharp 2.0.0.MLNX20190922.a9ebf22 1 sockperf 3.6 0.git737d1e3e5576 srp 4.7 OFED.4.7.1.0.0.1.g1c4bf42 srptools 41mlnx1 5 tmfifo 1.3 0.g6aa30c7 ucx 1.7.0 1 ofed-scripts 4.7 OFED.4.7.1.0.0
- Install the correct VAST Data RPM file:
$ sudo rpm -vih mlnx-nfsrdma-vast_1.0_30_gbfeb8d0-OFED.4.7.1.0.0.1_kernel_3.10.0_1062.4.1.el7.x86_64.rpm
-
Verify that:
-
the rpcrdma.ko file can be found as shown here under /extra/mlnx-nfsrdma/ under a kernel version-specific path like in below example
-
srcversion is the VAST Data driver version (A6E1294087824479F5BF44C):
{file name points to this path under kernel path} $ modinfo rpcrdma filename: /lib/modules/3.10.0-1062.4.1.el7.x86_64/extra/mlnx-nfsrdma/rpcrdma.ko alias: xprtrdma alias: svcrdma license: Dual BSD/GPL description: RPC/RDMA Transport author: Open Grid Computing and Network Appliance, Inc. retpoline: Y rhelversion: 7.7 srcversion: A6E1294087824479F5BF44C depends: ib_core,sunrpc,mlx_compat,rdma_cm vermagic: 3.10.0-1062.4.1.el7.x86_64 SMP mod_unload modversions
-
Preparing NFS Client for Mouting NFS
Ensure that the host can mount NFS exports. You can do this by installing thenfs-common
or nfs-utils
package, depending on your Linux distribution.Mounting NFS Exports over RDMA
When mounting an export on an NFSoRDMA client, specify the following options in the mount call:
-
rdma
orproto=rdma
-
port=20049
-
vers=3
(VAST Cluster supports NVSv3 and not NFSv4.)
Your complete mount command looks like below:
# mount -t nfs -o vers=3,proto=rdma,port=20049 <VIP>:<exportpath> <dir>
in which:
-
<VIP>
is one of the CNode VIPs,Note
If you are working with CNode groups, you'll need to specify a VIP for one of the CNodes in the relevant group.
-
<exportpath>
is the path to the export you are mounting, and -
<dir>
is the directory onto which you are mounting the export.
For example, if you're mounting an export called mydir onto the directory /mnt/mydir and one of your CNode VIPs is 192.0.2.22, you could use::
# mount -t nfs -o vers=3,proto=rdma,port=20049 192.0.2.22:/mydir /mnt/mydir/
Notes:
This document assumes the followings about your Linux server as an example.
-
Linux Distribution: CentOS 7.7
-
Kernel: 3.10.0-1062.4.1.el7.x86_64
-
MOFED Version: 4.7-1.0.0.1
Comments
0 comments
Please sign in to leave a comment.