Articles in this section

Configuring Linux Server Machines as NFSoRDMA Clients

Sara Levy
  • Updated
Follow

Notice

NFSoRDMA is supported from VAST Cluster 1.2.2.

Pre-requisites

NFS over RDMA (NFSoRDMA) requires:

  • An RDMA-enabled network adapter on the client host

  • Connection to VAST Cluster over one the following transport infrastructures:

    • RDMA over Converged Ethernet (RoCE)

    • InfiniBand (IB)

Configurations

There are two ways to enable NFSoRDMA on each client host:

  • Using Mellanox OFED (MOFED) (recommended). For this option, make sure MOFED is installed. Then read Resolving RDMA Support Issue with Mellanox OFED and follow the instructions to install the relevant RPM file if needed.

    or

  • Using RDMA inbox drivers. For this option, make sure the following configurations are all done:

    • The RDMA driver for the network adapter is installed and enabled.

    • Install the packages required for RDMA operations. This varies with Linux distribution.

    • Enable the RDMA service (Red Hat/CentOS systems) or set the RDMA drivers to load on boot.

In both cases, you'll need to ensure that the host can mount NFS exports. You can do this by installing the nfs-common or nfs-utils package, depending on your Linux distribution.

Mounting NFS Exports

When mounting an export on an NFSoRDMA client, specify the following options in the mount call:

  • rdma or proto=rdma

  • port=20049

  • vers=3 (VAST Cluster supports NVSv3 and not NFSv4.)

Your complete mount call will look like this:

# mount -t nfs -o vers=3,proto=rdma,port=20049 <VIP>:<exportpath> <dir>

in which:

  • <VIP> is one of the CNode VIPs,

    Note

    If you are working with CNode groups, you'll need to specify a VIP for one of the CNodes in the relevant group.

  • <exportpath> is the path to the export you are mounting, and

  • <dir> is the directory onto which you are mounting the export.

For example, if you're mounting an export called mydir onto the directory /mnt/mydir and one of your CNode VIPs is 192.0.2.22, you could use::

# mount -t nfs -o vers=3,proto=rdma,port=20049 192.0.2.22:/mydir /mnt/mydir/

Example 1: NFSoRDMA Client on CentOS with Mellanox OFED

In this example, we'll set up an NFSoRDMA client on CentOS 7.7 running kernel 3.10, on which MOFED 4.7-1.0.0.1 is already installed along with all packages required by Mellanox for the distribution.

  • Linux Distribution: CentOS 7.7

  • Kernel: 3.10.0-1062.4.1.el7.x86_64

  • MOFED Version: 4.7-1.0.0.1

Tip

For a new OS\MOFED install, make sure the drivers were reloaded successfully before proceeding, as written at the end of the MOFED installation process.

$ sudo /etc/init.d/openibd restart

Note

In this example, all commands are preceded with sudo to assume superuser privileges. If you are logged in as the super user, miss out sudo from the commands.

  1. Validate the current MOFED version to verify that MOFED is indeed installed:

    $ ofed_info -s
MLNX_OFED_LINUX-4.7-1.0.0.1:

  2. Run this command to list the RPMs and all the libraries that were installed. The command output should be as shown here:

    $ ofed_rpm_info
ar_mgr 1.0 0.45.MLNX20190923.g5aec6dc
cc_mgr 1.0 0.44.MLNX20190923.g5aec6dc
dapl 2.1.10mlnx OFED.3.4.2.1.0
dump_pr 1.0 0.40.MLNX20190923.g5aec6dc
fabric-collector 1.1.0.MLNX20170103.89bb2aa 0.1
gpio-mlxbf 1.0 0.g6d44a8a
hcoll 4.4.2938 1
i2c-mlx 1.0 0.gab579c6
ibacm 41mlnx1 OFED.4.3.3.0.0
ibdump 5.0.0 3
ibsim 0.7mlnx1 0.11.g85c342b
ibutils 1.5.7.1 0.12.gdcaeae2
ibutils2 2.1.1 0.110.MLNX20190922.gd4efc48
infiniband-diags 5.4.0.MLNX20190908.5f40e4f 0.1
iser 4.7 OFED.4.7.1.0.0.1.g1c4bf42
isert 4.7 OFED.4.7.1.0.0.1.g1c4bf42
kernel-mft 4.13.0 102
knem 1.1.3.90mlnx1 OFED.4.6.2.0.8.1.g8d875ee
libdisni 1.9 OFED.4.5.0.0.6
libibcm 41mlnx1 OFED.4.1.0.1.0
libibmad 5.4.0.MLNX20190423.1d917ae 0.1
libibumad 43.1.1.MLNX20190905.1080879 0.1
libibverbs 41mlnx1 OFED.4.7.0.0.2
libmlx4 41mlnx1 OFED.4.5.0.0.3
libmlx5 41mlnx1 OFED.4.7.0.3.3
libpka 1.0 1.g6cc68a2
librdmacm 41mlnx1 OFED.4.7.0.3.3
librxe 41mlnx1 OFED.4.4.2.4.6
libvma 8.9.4 1
mlnx-en 4.7 1.0.0.0.g1c4bf42
mlnx-ethtool 5.1 1
mlnx-iproute2 5.2.0 1
mlnx-nfsrdma 4.7 OFED.4.7.1.0.0.1.g1c4bf42
mlnx-nvme 4.7 OFED.4.7.1.0.0.1.g1c4bf42
mlnx-ofa_kernel 4.7 OFED.4.7.1.0.0.1.g1c4bf42
mlnx-rdma-rxe 4.7 OFED.4.7.1.0.0.1.g1c4bf42
mlx-bootctl 1.1 0.g5b90483
mlx-l3cache 0.1 1.gebb0728
mlx-pmc 1.1 0.g1141c2e
mlx-trio 0.1 1.g9d13513
mlxbf-livefish 1.0 0.gec08328
mpi-selector 1.0.3 1
mpitests 3.2.20 e1a0676
mstflint 4.13.0 1.41.g4e8819c
multiperf 3.0 0.13.gcdaa426
mxm 3.7.3112 1
nvme-snap 2.0.1 2.mlnx
ofed-docs 4.7 OFED.4.7.1.0.0
openmpi 4.0.2rc3 1
opensm 5.5.0.MLNX20190923.1c78385 0.1
openvswitch 2.11.0 1
perftest 4.4 0.8.g7af08be
pka-mlxbf 1.0 0.gd3455da
qperf 0.4.9 9.46101
qperf 0.4.9 9
rdma-core 47mlnx1 1
rshim 1.8 0.g463f780
sharp 2.0.0.MLNX20190922.a9ebf22 1
sockperf 3.6 0.git737d1e3e5576
srp 4.7 OFED.4.7.1.0.0.1.g1c4bf42
srptools 41mlnx1 5
tmfifo 1.3 0.g6aa30c7
ucx 1.7.0 1
ofed-scripts 4.7 OFED.4.7.1.0.0

  3. Install the correct VAST Data RPM file:

    $ sudo rpm -vih mlnx-nfsrdma-vast_1.0_30_gbfeb8d0-OFED.4.7.1.0.0.1_kernel_3.10.0_1062.4.1.el7.x86_64.rpm

  4. Verify that the rpcrdma.ko file can be found as shown here under /extra/mlnx-nfsrdma/ under a kernel version-specific path like this one and that srcversion is the VAST Data driver version (A6E1294087824479F5BF44C):

    {file name points to this path under kernel path}
$ modinfo rpcrdma
filename: /lib/modules/3.10.0-1062.4.1.el7.x86_64/extra/mlnx-nfsrdma/rpcrdma.ko
alias: xprtrdma
alias: svcrdma
license: Dual BSD/GPL
description: RPC/RDMA Transport
author: Open Grid Computing and Network Appliance, Inc.
retpoline: Y
rhelversion: 7.7
srcversion: A6E1294087824479F5BF44C
depends: ib_core,sunrpc,mlx_compat,rdma_cm
vermagic: 3.10.0-1062.4.1.el7.x86_64 SMP mod_unload modversions

Example 2: NFSoRDMA Client on Ubuntu, No OFED, RoCE

Pmake clear that its a choice of how to work - this is if you decide to work with inbox drivers for some reason}

In this example, we will enable NFSoRDMA on Ubuntu 16.04 without Mellanox OFED, using a Mellanox ConnectX-4 network adapter over an Ethernet link (RoCE).

  • Linux Distribution: Ubuntu 16.04

  • Kernel: 4.15

  • Mellanox OFED Version: Not installed

  • Network Adapter: Mellanox ConnectX-4

  • Transport link: Ethernet

Note

In this example, all commands are preceded with sudo to assume superuser privileges. If you are logged in as the super user, miss out sudo from the commands.

  1. Run the following command to install the packages needed:

    $ sudo apt-get install libmlx4-1 infiniband-diags ibutils ibverbs-utils rdmacm-utils perftest nfs-common

    The above command installs:

    • libmlx4-1, the package that contains drivers for the Mellanox ConnectX-4 adapter.

    • Packages containing device drivers needed for RDMA operations: infiniband-diags, ibutils, ibverbs-utils, rdmacm-utils, perftest

    • The NFS Common application to enable the host to mount NFS exports as an NFS client.

  2. Run the following command to add the RDMA device driver for the network adapter to be loaded when the host boots:

    $ echo mlx5_ib >> /etc/modules

  3. Since the default link protocol for the ConnectX-4 adapter is Infiniband, we need to change the link mode to Ethernet. Create the RDMA configuration file directory and then create and edit the mlx4.conf file like this:

    $ sudo mkdir -p /etc/rdma 

$ sudo sh -c "lspci | grep -i mellanox | awk '{print \$1 \" eth\"}' > /etc/rdma/mlx4.conf"$

  4. Reboot the server for the mode change to take effect. This will also load the adapter device driver.

  5. Enable the relevant network interface, and configure it's IP, netmask and MTU:

    $ sudo ip link set dev ens785f0 up
$ sudo ip addr add 198.51.100.10/24 dev ens785f0
$ sudo ip link set dev ens785f0 mtu 9216

  6. Now we're ready to mount an NFS export. Assuming the mount point was not yet created, run this command to create the mount point:

    $ sudo mkdir -p /mnt/mydir

  7. Mount the export (in this example, our export is called "/mydir") onto the mount point. For the NFS server IP, we need to use any of the VAST Cluster CNode VIPs. We'll use 192.0.2.11.

    $ sudo mount -v -o vers=3,rdma,port=20049 192.0.2.11:/mydir /mnt/mydir/

Related articles

Comments

0 comments

Please sign in to leave a comment.