Articles in this section

Prepare a Scheme File

Sara Levy
  • Updated
Follow

This document is for support use. It may not be complete or final. Please send feedback to sara@vastdata.com. 

Installation via the VAST Web UI enables you to choose from a few predefined scheme files or to upload a scheme file that you customized for the installation. After the scheme is loaded, the VAST Web UI enables you to change the values for the parameters that you included in the scheme file. It does not enable you to add any parameters that were not included in the scheme file.

This article is intended to guide you in selecting or creating a scheme file that you can use with the installer to set up the cluster.

How to Prepare the Scheme File

  1. Identify which of the scheme examples below is the closest to the type of installation that you are doing. Copy and paste the sample file content from that example into your preferred editor.

  2. Customize the content with the help of the customization notes from your chosen example and the full format and parameters description in Scheme File Format and Parameters.

  3. Save the file with a .yaml extension.

 

Scheme Examples

Identify the example below that closest matches the installation you're doing. Use the example content as a boilerplate.

Example 1: Single Switch Pair, 16 port switches, Ethernet Only, Out of Band Management

This type of installation has:

  • One pair of 16 port Ethernet switches

  • Client data network, VAST internal network and management network all running on Ethernet

  • Management out of band

  • One NIC per CNode, with both interfaces used for both VAST internal and client data traffic.

Example Scheme Content

This example is identical to the content of the 2x16_switch_eth_only_mgmt_outbound scheme file. If this needs no customization for the installation, you can select this one in the installer without creating and uploading a custom scheme file.

switch_conf:
  num_of_switches: 2
  num_of_ports: 16
  roles:
    - spine
  spine:
    isl_ports:
    ipl_ports: [13,14]
    external_ports: [15,16]
    cnode_ports: [7/2,8/2,9/2,10/2,11/2,12/2]
    dnode_ports: [1,2,3,4,5,6]
host_conf:
  mode: eth
  mgmt: outband
  cnode:
    ext_interface: eno1 # mgmt interface
    technician_interface: eno2
    port_mode: eth
    eth_mtu: 9216
    internal_interfaces:
    external_interfaces:
    internal_virtual_interfaces:
  dnode:
    ext_interface: ens6  # mgmt interface
    technician_interface: ens4
    port_mode: eth
    eth_mtu: 9216
    internal_interfaces:
    external_interfaces:
    internal_virtual_interfaces:
Customization Notes

  • Interface names vary depending on the model of CNode you are using. Therefore, you may need to change ext_interface and technician_interface.

    Tip

    You may need to SSH to one of the CNodes and DNodes and run ip link to verify or identify interface names.

  • If the MTU on the client data network is greater than 9000, you'll need to set eth-mtu the CNodes and DNodes to the MTU of the client data network.

  • Notice that internal_interfaces, external_interfaces and internal_virtual_interfaces are unspecified. These interface assignments are inferred automatically. When there is one NIC per CNode the inference is expected to be correct.

  • VAST internal data is isolated from the client data network using a VLAN. This VLAN is 69, by default. In case of a conflicting use of the default VLAN (69), add the data-vlan parameter to the CNode and DNode configurations. Specify a different VLAN that is not already used on the client network.

 

Example 2: Mixed Mode with Ethernet for VAST Internal Network, IB Client Data Network

This type of installation has:

  • Client data network running on Infiniband, VAST internal network running on Ethernet.

  • Two NICs per CNode one connected to the client data network and one connected to the VAST internal network.

Example Scheme Content

In this example, management is out of band and there is one pair of 16 port switches. Since there is only one switch pair, there is only one switch role (spine).

This example is similar to the content of the predefined scheme 2X16_switch_mixed_mgmt_outbound.yaml. We added ib-mode to set a non-default mode for the Infiniband ports and ib-mtu to set the MTU of the client data network. We also set port-mode so that it configures one NIC in Ethernet mode for the VAST internal network and the other NIC in Infiniband mode for the client data network, as needed.

switch_conf:
  num_of_switches: 2
  num_of_ports: 16
  roles:
    - spine
  spine:
    isl_ports:
    ipl_ports: [13,14]
    external_ports: [15,16]
    cnode_ports: [7/2,8/2,9/2,10/2,11/2,12/2]
    dnode_ports: [1,2,3,4,5,6]
host_conf:
  mode: mixed
  mgmt: outband
  cnode:
    ext_interface: eno1 # mgmt interface
    technician_interface: eno2
    port_mode: internal=eth,external=ib
    eth_mtu: 9216
    ib_mode: datagram
    ib_mtu: 2048
    internal_interfaces: ens785f0,ens785f1
    external_interfaces: ib0,ib1
    internal_virtual_interfaces: ens785f2,ens785f3
  dnode:
    ext_interface: ens6  # mgmt interface
    technician_interface: ens4
    port_mode: eth
    eth_mtu: 9216
Customization Notes

  • The interface names in the example may not be correct for the CNode model that you're installing.

    You will probably need to identify the names of interfaces on the CNodes and DNodes. You can do this by SSHing to one of the CNodes or DNodes and running the ip link command on the OS to identify interface names.

  • The default IB mode is connected. Customers often require datagram, as specified in the example.

  • When specifying port mode, note that you need to set port mode differently for each NIC. The order is according to PCI enumeration. In the example, we have internal=eth,external=ib. If the NICs are ordered the other way round, switch it to external=ib,internal=eth

  • DNode port mode should be set to the transport protocol of the VAST internal network (usually Ethernet for this type of installation), as in the example.

 

Example 3: Dual Switch Pair

In a dual switch pair Ethernet installation, there are four 32 port switches, with one pair configured in the spine role for connectivity to the client data network, and the other pair connected only to the VAST internal network.

Both spine and leaf roles are declared in the scheme file and the port assignments for each role are specified as shown below.

Example Scheme Content

In this example, the networking is assumed Ethernet only and management is out of band.

switch_conf:
  roles:
    - spine
    - leaf
  num_of_switches: 4
  num_of_ports: 32
  spine:
    isl_ports: [23,24,25,26,27,28,29,30]
    ipl_ports: [31,32]
    external_ports: [15,16,17,18,19,20,21,22]
    cnode_ports: [9/2,10/2,11/2,12/2,13/2,14/2]
    dnode_ports: [1,2,3,4,5,6,7,8]
  leaf:
    isl_ports: [23,24,25,26,27,28,29,30]
    ipl_ports: [31,32]
    external_ports:
    cnode_ports: [9/2,10/2,11/2,12/2,13/2,14/2,15/2,16/2,17/2,18/2]
    dnode_ports: [1,2,3,4,5,6,7,8]
host_conf:
  mode: eth
  mgmt: outband
  cnode:
    ext_interface: eno1 # mgmt interface
    technician_interface: eno2
    port_mode: eth
    eth_mtu: 9216
    internal_interfaces:
    external_interfaces:
  dnode:
    ext_interface: ens6 # mgmt interface
    technician_interface: ens4
    port_mode: eth
    eth_mtu: 9216
    internal_interfaces:
 

Example 4: IB Only, Inband Management

This type of installation has:

  • Unmanaged IB switches, negating the need for switch configuration

  • Client data network and VAST internal network both running on Infiniband,

  • Management inband

  • One NIC per CNode

Example Scheme Content
host_conf:
  mode: ib
  mgmt: inband
  cnode:
    ext_interface: eno1 # mgmt interface
    technician_interface: eno2
    port_mode: ib    
    ib_mode: datagram
    ib_mtu: 2044
    internal_interfaces:
    external_interfaces:
  dnode:
    ext_interface: ens6 # mgmt interface
    technician_interface: ens4
    port_mode: ib
    ib_mode: datagram
    ib_mtu: 2044
    internal_interfaces:
Customization Notes

  • By default, VAST uses the following subnets: 172.16.1.x, 172.16.2.x, 172.16.3.x. In an IB configuration, there can be an IP address collision with these subnets. In such cases, add the template parameter and specify an alternate template for the subnets.

  • The interface assignments will be inferred automatically.

 
 

Scheme File Format and Parameters

The scheme file is a follows the following format:

<section>:
  <parameter>: <value>
  <parameter>: <value>
  ...
<section>:
  <parameter>: <value>
  <parameter>: <value>
  ...

Two sections can be included: switch_conf for switch configurations (not needed if, for example, the cluster will use unmanaged Infiniband switches) and host_conf needed for every installation for host configuration.

Each section should contain any of the parameters detailed in the following tables that are relevant to the installation. (Additional configurations, such as IP addresses, gateways etc. are provided through the later screens in the installer and not through the scheme file.)

Switch Configuration Parameters

The installer runs the switch_conf.py script on all switches

Section

Parameter

Description and usage notes

Value format, possible values

Example

switch_conf

 

Provides part of the switch configuration. Include it for all installations that include Ethernet data switches provided by VAST Data.

 

 
 

num_of_switches

The number of Ethernet data switches included

Integer, even number

num_of_switches: 2

 

num_of_ports

The number of ports per switch.

16 or 32

num_of_ports: 16

 

roles

Enables two different port configurations to be applied to different switches. The second role is used for larger scale Ethernet topologies.

First declare one of the following:

  • Spine only:

       - spine

  • Spine and leaf:

        - spine
    - leaf

Then include a sub section per included role:

spine:
    ...
leaf:
   ...
 
 roles:
  - spine
spine: 
  ...

spine, leaf

isl_ports

Ports assigned to connections between switches for large scale Ethernet topologies. Leave blank for spine-only topologies.

Comma separated list of port numbers 1-32, enclosed in square brackets []

isl_ports: [23,24,25, 26,27,28,29,30]

 

ipl_ports

Ports assigned for connection between the two switches in each pair. Specify two ports.

Comma separated list of port numbers 1-32, enclosed in square brackets []

ipl_ports: [13,14]

 

external_ports

Ports assigned for uplinks to client data network.

In dual pair installations, these are entered only for the spine role. There is no connection from the leaf switch pairs to the client data network.

Comma separated list of port numbers 1-32, enclosed in square brackets []

external_ports: [15,16]

 

cnode_ports

Ports assigned for connection to CNodes

Comma separated list of port numbers 1-32, enclosed in square brackets []

Add the notation '/2' after each port number to indicate that each 100Gb switch port is split for connection to two 50 Gig CNode ports.

cnode ports: [7/2,8/2,9/2,10/2,11/2,12/2]

 

dnode_ports

Ports assigned for connection to DNodes

Comma separated list of port numbers 1-32, enclosed in square brackets []

dnode_ports: [1,2,3,4,5,6]

Host Configuration Parameters

Section in file

Parameter

Description and usage notes

Include for

Default

Format, Possible Values

Example

host_conf

 

Configures hosts

All installations

 

 

host_conf:
 

mode

Indicates number of NICs per CNode and transport protocol.

All installations

None

  • eth. One NIC per CNode to be deployed in Ethernet mode

  • ib. One NIC per CNode to be deployed in Infiniband mode

  • mixed. If more than one NIC per CNode

mode: mixed

 

mgmt

Specifies if the management network should be isolated from the data network or not.

All installations

None

  • outband. Management is isolated from management network

  • inband. Management is on data network.

mgmt: outband

CNode

 

Configures CNodes

All installations

 

 

CNode:
 

ext_interface

Specifies the management interface on every CNode.

Out of band management

bond0 (as needed for inband management)

String

ext_interface: eno1

 

technician_interface

Specifies the technician interface on every CNode

All installations

None

String

technician_interface: eno2

 

port_mode

Specifies the interface modes of the physical interfaces on the CNode NICs.

Installations can have one or two NIC(s) per CNode. Two NICs can be separately dedicated to the VAST internal or client data network.

All Installations

Factory default

  • eth. Sets both the internal and external interfaces to Ethernet mode.

  • ib. Sets both the internal and the external interfaces to Infiniband mode. Use this for IB only installations.

  • Comma separated list of type=mode pairs in order of PCI enumeration where type is the traffic type (internal or external) and mode is the interface mode (eth or ib). For example:

    • external=ib,internal=eth.

    • internal=eth,external=ib

port_mode:internal=eth,external=ib

 

eth_mtu

MTU of ethernet ports, in bytes.

If the MTU on the client data network is greater than 9000 bytes, set eth-mtu to the same as on the client data network.

If the MTU on the client data network is less than 9000, set eth-mtu to 9216.

All Installations with ETH mode ports

9216

Number of bytes

eth_mtu: 9000

 

internal_interfaces

Two interfaces over which to configure the "VAST internal" network.

For installations that have one NIC installed per CNode, this is automatically inferred and can be excluded or left blank.

Installations with 2 NICs per CNode

None

Two interface names separated by a comma.

internal_interfaces: ens785f0,ens785f1

 

external_interfaces

Two interfaces over which to configure the client data network.

These are automatically inferred with the correct interface names when there is only one NIC per CNode.

Installations with two NICs per CNode.

None

Two interface names separated by a comma.

external_interfaces: ib0,ib1

 

internal_virtual_interfaces

VAST Clusterconfigures a bond interface over two logical interfaces. It is used for internal management traffic between the VMS and the CNodes and DNodes.

This parameter specifies the two virtual interfaces over which the bond interface is configured. Each virtual interface should be bound to a different physical interface to enable fail over.

For installations with one NIC per CNode (two physical interfaces), you can leave this blank and the correct interface names will be inferred automatically.

Installations with two NICs per CNode.

None

Two interface names separated by a comma.

internal_virtual_interfaces: ens785f2,ens785f3

 

ib_mode

The mode of any IB networks.

IB and mixed ETH+IB installations

connected

Possible values:

  • datagram

  • connected

ib_mode: datagram

 

ib_mtu

The MTU of the IB network.

IB and mixed ETH+IB installations

None

Number of bytes

ib_mtu: 2044

 

template

The template for the subnets. In an IB configuration, there could be an IP address collision within our subnets (172.16.1.x, 172.16.2.x, 172.16.3.x). In such a case, enter a different template.

Can be relevant If client data network is Infiniband

172.16.{network}.{node}

IPv4 address format with {network} and {node} wildcards for last two bytes respectively.

template: 173.16.{network}.{node}

 

data_vlan

For Ethernet configurations VAST internal data is isolated from the client data network using this VLAN. In case of a conflicting use of the default VLAN (69), enter a different VLAN that is already not used on the client network.

Ethernet only

69

integer

data_vlan: 60

DNode

 

Configures DNodes

All installations

 

 

Dnode:
 

ext_interface

Specifies the management interface on every DNode.

Out of band management

bond0 (as needed for inband management)

String

ext_interface: ens6

 

technician_interface

Specifies the technician interface on every DNode.

All installations

None

String

technician_interface: ens4

 

internal_interfaces

Two interfaces over which to configure the "VAST internal" network.

This is automatically inferred and can be excluded or left blank.

 

 

 

 

 

internal_virtual_interfaces

VAST Clusterconfigures a bond interface over two logical interfaces. It is used for internal management traffic between the VMS and the CNodes and DNodes.

This parameter specifies the two virtual interfaces over which the bond interface is configured. Each virtual interface should be bound to a different physical interface to enable fail over.

You can leave this blank and the correct interface names will be inferred automatically.

 

 

 

 

 

port_mode

Specifies interface mode.

All installations

Factory default

  • eth. Sets the interfaces to Ethernet mode.

  • ib. Sets the interfaces to Infiniband mode.

port_mode: eth

 

eth_mtu

MTU of ethernet ports

Ethernet and Ethernet + IB

9216

Number of bytes

eth_mtu: 9000

 

ib_mode

The mode of the IB network.

IB only

connected

Possible values:

  • datagram

  • connected

ib_mode: datagram

 

ib_mtu

The MTU of the IB network.

IB Only

None

Number of bytes

ib_mtu: 2044

 

template

The IP template for the subnets. Should match the CNode template.

Can be relevant If client data network is Infiniband

172.16.{network}.{node}

IPv4 address format with {network} and {node} wildcards for last two bytes respectively.

template: 173.16.{network}.{node}

 

data_vlan

For Ethernet configurations we isolate VAST internal data from the client data network using the data vlan. In case of a conflicting use of the default VLAN (69), enter a different VLAN that is already not used on the client network. This must match the data-vlan for the CNodes.

Ethernet installations with VLAN conflict

69

VLAN number

data-vlan: 60
 
 

Related articles

Comments

0 comments

Article is closed for comments.