Articles in this section

Configuring Linux Server Machines as NFSoRDMA Clients

Sara Levy
  • Updated
Follow
Pre-requisites

NFS over RDMA (NFSoRDMA) requires:

  • An RDMA-enabled network adapter on the client host

  • Connection to VAST Cluster over one the following transport infrastructures:

    • RDMA over Converged Ethernet (RoCE)

    • InfiniBand (IB)

Configuration

To enable NFSoRDMA on each client host:

  1. Make sure MOFED is installed. Then read Resolving RDMA Support Issue with Mellanox OFED and follow the instructions to install the relevant RPM file if needed.

  2. Ensure that the host can mount NFS exports. You can do this by installing the nfs-common or nfs-utils package, depending on your Linux distribution.

Mounting NFS Exports

When mounting an export on an NFSoRDMA client, specify the following options in the mount call:

  • rdma or proto=rdma

  • port=20049

  • vers=3 (VAST Cluster supports NVSv3 and not NFSv4.)

Your complete mount call will look like this:

# mount -t nfs -o vers=3,proto=rdma,port=20049 <VIP>:<exportpath> <dir>

in which:

  • <VIP> is one of the CNode VIPs,

    Note

    If you are working with CNode groups, you'll need to specify a VIP for one of the CNodes in the relevant group.

  • <exportpath> is the path to the export you are mounting, and

  • <dir> is the directory onto which you are mounting the export.

For example, if you're mounting an export called mydir onto the directory /mnt/mydir and one of your CNode VIPs is 192.0.2.22, you could use::

# mount -t nfs -o vers=3,proto=rdma,port=20049 192.0.2.22:/mydir /mnt/mydir/
Example: NFSoRDMA Client on CentOS

In this example, we'll set up an NFSoRDMA client on CentOS 7.7 running kernel 3.10, on which MOFED 4.7-1.0.0.1 is already installed along with all packages required by Mellanox for the distribution.

  • Linux Distribution: CentOS 7.7

  • Kernel: 3.10.0-1062.4.1.el7.x86_64

  • MOFED Version: 4.7-1.0.0.1

Tip

For a new OS\MOFED install, make sure the drivers were reloaded successfully before proceeding, as written at the end of the MOFED installation process.

$ sudo /etc/init.d/openibd restart

Note

In this example, all commands are preceded with sudo to assume superuser privileges. If you are logged in as the super user, miss out sudo from the commands.

  1. Validate the current MOFED version to verify that MOFED is indeed installed:

    $ ofed_info -s
MLNX_OFED_LINUX-4.7-1.0.0.1:

  2. Run this command to list the RPMs and all the libraries that were installed. The command output should be as shown here:

    $ ofed_rpm_info
ar_mgr 1.0 0.45.MLNX20190923.g5aec6dc
cc_mgr 1.0 0.44.MLNX20190923.g5aec6dc
dapl 2.1.10mlnx OFED.3.4.2.1.0
dump_pr 1.0 0.40.MLNX20190923.g5aec6dc
fabric-collector 1.1.0.MLNX20170103.89bb2aa 0.1
gpio-mlxbf 1.0 0.g6d44a8a
hcoll 4.4.2938 1
i2c-mlx 1.0 0.gab579c6
ibacm 41mlnx1 OFED.4.3.3.0.0
ibdump 5.0.0 3
ibsim 0.7mlnx1 0.11.g85c342b
ibutils 1.5.7.1 0.12.gdcaeae2
ibutils2 2.1.1 0.110.MLNX20190922.gd4efc48
infiniband-diags 5.4.0.MLNX20190908.5f40e4f 0.1
iser 4.7 OFED.4.7.1.0.0.1.g1c4bf42
isert 4.7 OFED.4.7.1.0.0.1.g1c4bf42
kernel-mft 4.13.0 102
knem 1.1.3.90mlnx1 OFED.4.6.2.0.8.1.g8d875ee
libdisni 1.9 OFED.4.5.0.0.6
libibcm 41mlnx1 OFED.4.1.0.1.0
libibmad 5.4.0.MLNX20190423.1d917ae 0.1
libibumad 43.1.1.MLNX20190905.1080879 0.1
libibverbs 41mlnx1 OFED.4.7.0.0.2
libmlx4 41mlnx1 OFED.4.5.0.0.3
libmlx5 41mlnx1 OFED.4.7.0.3.3
libpka 1.0 1.g6cc68a2
librdmacm 41mlnx1 OFED.4.7.0.3.3
librxe 41mlnx1 OFED.4.4.2.4.6
libvma 8.9.4 1
mlnx-en 4.7 1.0.0.0.g1c4bf42
mlnx-ethtool 5.1 1
mlnx-iproute2 5.2.0 1
mlnx-nfsrdma 4.7 OFED.4.7.1.0.0.1.g1c4bf42
mlnx-nvme 4.7 OFED.4.7.1.0.0.1.g1c4bf42
mlnx-ofa_kernel 4.7 OFED.4.7.1.0.0.1.g1c4bf42
mlnx-rdma-rxe 4.7 OFED.4.7.1.0.0.1.g1c4bf42
mlx-bootctl 1.1 0.g5b90483
mlx-l3cache 0.1 1.gebb0728
mlx-pmc 1.1 0.g1141c2e
mlx-trio 0.1 1.g9d13513
mlxbf-livefish 1.0 0.gec08328
mpi-selector 1.0.3 1
mpitests 3.2.20 e1a0676
mstflint 4.13.0 1.41.g4e8819c
multiperf 3.0 0.13.gcdaa426
mxm 3.7.3112 1
nvme-snap 2.0.1 2.mlnx
ofed-docs 4.7 OFED.4.7.1.0.0
openmpi 4.0.2rc3 1
opensm 5.5.0.MLNX20190923.1c78385 0.1
openvswitch 2.11.0 1
perftest 4.4 0.8.g7af08be
pka-mlxbf 1.0 0.gd3455da
qperf 0.4.9 9.46101
qperf 0.4.9 9
rdma-core 47mlnx1 1
rshim 1.8 0.g463f780
sharp 2.0.0.MLNX20190922.a9ebf22 1
sockperf 3.6 0.git737d1e3e5576
srp 4.7 OFED.4.7.1.0.0.1.g1c4bf42
srptools 41mlnx1 5
tmfifo 1.3 0.g6aa30c7
ucx 1.7.0 1
ofed-scripts 4.7 OFED.4.7.1.0.0

  3. Install the correct VAST Data RPM file:

    $ sudo rpm -vih mlnx-nfsrdma-vast_1.0_30_gbfeb8d0-OFED.4.7.1.0.0.1_kernel_3.10.0_1062.4.1.el7.x86_64.rpm

  4. Verify that the rpcrdma.ko file can be found as shown here under /extra/mlnx-nfsrdma/ under a kernel version-specific path like this one and that srcversion is the VAST Data driver version (A6E1294087824479F5BF44C):

    {file name points to this path under kernel path}
$ modinfo rpcrdma
filename: /lib/modules/3.10.0-1062.4.1.el7.x86_64/extra/mlnx-nfsrdma/rpcrdma.ko
alias: xprtrdma
alias: svcrdma
license: Dual BSD/GPL
description: RPC/RDMA Transport
author: Open Grid Computing and Network Appliance, Inc.
retpoline: Y
rhelversion: 7.7
srcversion: A6E1294087824479F5BF44C
depends: ib_core,sunrpc,mlx_compat,rdma_cm
vermagic: 3.10.0-1062.4.1.el7.x86_64 SMP mod_unload modversions

Related articles

Comments

0 comments

Article is closed for comments.