# Broadcom NetXtreme-E Linux Installer

## QUICKSTART: Single Host Installation with Default Configuration

The install.sh script in this directory can be used to install drivers, libraries, and firmware, as well as configure IP address and RoCE congestion control on a single host.

To install all software components, set up RoCE congestion control with default settings, and configure an IP address, issue the following command:

    bash install.sh -i <interface name or PCIe address> -a <ip address> -n <netmask>

**Note:** that for RoCE to function properly in the case of any congestion (many-to-one flows, link speed changes, etc) your network switch must be configured to match the host congestion control settings. Example settings are output at the end of the installation process.

## Introduction

This tool automates installation of the Broadcom NetXtreme Ethernet and RoCE drivers, as well as manages installing firmware for Broadcom NetXtreme network controllers. 

The installer has two modes:

 - Single Host: The install.sh script can be run on a single system to configure that host only
 - Cluster Mode: A list of servers can be specified for installation from a remote host

The following changes will be applied to your systems:

 - Install bnxt_en Ethernet driver
 - Install bnxt_re RoCE driver
 - Install libbnxt_re RDMA verbs library
 - Update device firmware
 - Configure IP address if requested
 - Configure RoCE congestion control if RoCE is installed

Driver modules will be installed in this priority:

  - RedHat/CentOS: DKMS, Source RPM, Binary RPM, Source
  - SuSE: Signed RPM if secureboot, DKMS, Source RPM, Binary RPM, Source
  - Ubuntu/Debian: DKMS, Source

## Requirements

### Supported operating systems

See the Supported Operating Systems for Ethernet NIC Controllers page for the full list of supported operating systems and platforms:

https://techdocs.broadcom.com/us/en/storage-and-ethernet-connectivity/ethernet-nic-controllers/bcm957xxx/1-0/software-installation/supported-operating-systems.html

### Supported Devices

The following Broadcom NetXtreme-E devices are fully supported by this tool:

 - Whitney+: [BCM9574xx](https://www.broadcom.com/products/ethernet-connectivity/network-adapters/25gb-nic-ocp)
 - Thor: [BCM9575xx](https://www.broadcom.com/products/ethernet-connectivity/network-adapters/100gb-nic-ocp)

All speeds, and cable/transceiver types supported by the above devices are supported by this tool.

### Software Requirements

The installer will attempt to automatically install all required packages using the native package management system. This requires that subscription-based systems have package repositories enabled:

 - RedHat: BaseOS
 - SuSE: PackageHub

If previous versions of Broadcom drivers are installed, potential conflicts between versions can be avoided by running the included uninstall.sh script, or removing all currently installed packages named bnxt* and libbnxt_re.

### SuSE

For unregistered SuSE systems:

 - Set enabled=1 in all files in /etc/zypp/repos.d/
 - Install ansible with: pip3 install ansible

## Secure Boot

Signed drivers compatible with secure boot are supported on SuSE installations. In order to successfully use signed drivers, follow the steps on this page to import the Partner Linux Driver Program key:

https://drivers.suse.com/doc/Usage/Secure_Boot_Certificate.html

When switching a system from secure boot to non-secure boot, or visa-versa, re-run the installer with the -f argument, to force reinstallation of the correct drivers, or uninstall then re-install.

## Single Host Installation

Before starting installation, please enable your distribution's package repositories. The installer will make use of these to install prerequisite packages. Without them, the installation may fail, or may need to fall back to installation from source, rather than source or DKMS packages.

The install.sh script is the best tool for installing all software on a single host. The simplest valid command is:

    bash install.sh -i <interface name or PCIe address>

This will install the software found distributed with the installer to the specified network device, upgrade the device firmware, log the installation process details to install.log, and on successful completion, output the configured values and example network switch configuration.

The default installation values are:

 - Software packages: bnxt_en / bnxt_re drivers, libbnxt_re library, bnxtqos and bnxtnvm utilities
 - Firmware for the selected device
 - RoCE enabled, with PFC and ECN
 - RoCE traffic tagged with DSCP value 26, on priority 3
 - RoCE CNP packets tagged with DSCP value 48, on priority 7

See `install -h` output for all options. The most common are:

 - `-a <address> -n <netmask>`: Specify an IP address and netmask for the device being configured
 - `-v`: Verbose mode. Output installation log while running
 - `-m <MTU>`: Specify the device MTU. Default is 1500
 - `-2`: Install Ethernet driver only. Do not install/configure RoCE
 - `-q <VID>`: Use Ethernet VLAN tag VID rather than DSCP
 - `-f`: Force installation, even if this causes a downgrade
 - `-s <source>`: Specify the software source. Either URL or directory name may be used. Default is `../../../`as the installer is assumed to be packaged with the source files

To configure additional interfaces, the install.sh script can be re-run with the additional interface name specified with `-i`.

## Cluster Installation

Installation and configuration of multiple servers can be accomplished with the cluster installation mode of this installer. This is accomplished by setting up a management host with [Ansible](https://www.ansible.com/), and creating an inventory file containing the details of the target hosts, and configuration.

### Management Host Setup

Most often, the automation scripts, based on Ansible (www.ansible.org) are run from a management host, rather than one of the servers to be managed, due to the fact that connectivity may be interrupted.

#### Management Host Requirements

 - Python 3: Some host package systems (RHEL 8+) no longer support Python2. Use python3 for best compatibility.
 - pyghmi Python module, if IPMI power control is to be used

#### Ansible Installation

The recommended way to install Ansible is via the system package manager:

Yum systems (CentOS):

    yum install ansible

Apt systems (Debian, Ubuntu):

    apt install ansible

RedHat Enterprise Linux:

    subscription-manager repos --enable ansible-2.9-for-rhel-8-x86_64-rpms
    dnf install ansible


It is also possible to install Ansible Python "pip" package manager. This can easily be done within a new Python virtual environment with commands like the following:

    virtualenv --system-site-packages ~/ansible
      source ~/ansible/bin/activate
      python3 -m pip install ansible


#### pyghmi Installation

If "boot_type=cold" will be used to power cycle systems after installing new firmware, the python pyghmi module must be installed.

Python virtualenv:

    pip install pyghmi

  Apt systems: apt install python3-pyghmi
  
  Yum systems: There is no OS-provided package.  Use a package from the RHEL OSP  (OpenStack Platform) or, for CentOS, the RDO repositories (example below):

    yum install http://mirror.centos.org/centos/7/cloud/x86_64/openstack-stein/python2-cryptography-2.5-1.el7.x86_64.rpm http://mirror.centos.org/centos/7/cloud/x86_64/openstack-stein/python2-pyghmi-1.2.16-1.el7.noarch.rpm

### Target Host Setup

Ansible uses ssh to configure the target hosts, so password-less ssh must be configured on the target hosts. The simplest process for this is:

On management host, as the user to be used to run the installer:

    eval `ssh-agent`
    ssh-keygen
    ssh-add

For each target host:

    ssh-copy-id root@<target host>

It is required to install python on the target hosts. It is also recommended to update the operating system packages on each host. For example:

 - For CentOS/RHEL: `yum update -y && yum install -y python`
 - For Debian/Ubuntu: `apt upgrade -y && apt install -y python`

### Inventory File Configuration

The hosts to be managed must be added to the `inventory` file in the ansible directory.

All hosts to be configured must be placed in the [nxe_servers] section. Examples are provided in the `inventory` file distributed with the installer.

Each host line has the following required fields:

 - Hostname: The name of the host. It is not required that this name resolves.
 - `ansible_host`: The IP of the host. May be omitted if the hostname resolves.
 - `priv_iface`: The name of the NetXtreme interface (netdev name or PCI address as in 0000:01:00.0)

Optional fields:

 - `priv_ip`: IP address to assign the interface from priv_iface
 - `priv_netmask`: IP netmask to assign the interface from priv_iface
 - `boot_type <warm|cold|ifupdown>`: After firmware upgrade, do warm reboot, IPMI power cycle, or ifup/ifdown (default)
 - `ipmi_ip`: If boot_type=cold, IP address to pass ipmitool
 - `ipmi_user`: If boot_type=cold, username to pass ipmitool
 - `ipmi_pass`: If boot_type=cold, password to pass ipmitool

Global variables are defined in the `[all:vars]` section of the inventory file:

 - `solution_dir`: The temporary directory to create on each target host to store source code, firmware images, etc.
 - `source_dir`: The location of the source archive. Can be either URL or directory name.  The source can be extracted onto the management host, or onto each target host.
 - `ethernet_mtu`: The Ethernet MTU. Only set when IP address is also specified for each host.
 - `force_install`: Force installation of software and firmware, even if a downgrade.


RoCE Congestion control parameters should be specified.  The default configuration specifies the preferred configuration: ECN-based congestion control, with DSCP for priority coordination with switch infrastructure with RoCE traffic on DSCP 26.  Alternatively, PFC, or PFC+ECN may be used.

RoCE without any form of congestion control is unsupported, and will result in very poor performance.

### Running ansible

The recommended way to run these scripts is to use the provided script, run.sh.  This script will set up the entire solution:

    cd ansible
    bash run.sh

By default, the Ansible scripts will exit if the scripts fail on any node. If you wish to change this then edit ansible/site.yml and set any_errors_fatal to false.

## Network Switch Configuration

In addition to basic link-level network configuration (speed, auto/static, FEC), RoCE requires congestion control to be configured for anything beyond basic connectivity tests. For this reason the installer will enable one of three congestion control modes on the host(s):

 - ECN: Explicit Congestion Control Notification (IETF RFC 3168) is used to mark packets being forwarded through Ethernet switch queues with a configured level of congestion. The receiving host will then send Congestion Notification Packets (CNP) to the sender, which will prompt a reduction in transmission rate.
 - PFC: Priority Flow Control (IEEE 802.1p) allows Ethernet switches to notify senders of congestion, requesting a pause in transmission.
 - ECN+PFC: ECN+PFC allows the fine-grained control of per-flow speeds of ECN alone, plus the quick response to bursts of PFC.

The default congestion control mode is ECN. This can be overriden with the install.sh -m command-line option, or the roce_cc_mode variable in the Ansible inventory file.

All three congestion control modes require Ethernet switch configuration. The example below can be used to aid in configuration of IOS-type switches. Please note that the DSCP values may need to be changed if a different value is
specified to the installer. Also the ECN marking thresholds are appropriate for 100G and 200G. If using other speeds, see the RoCE Users Guide for correct values. The install.sh script, if used, will output example switch configuration with correct priority values specified.

### ECN

    qos map dscp 26 to traffic-class 3
      qos map dscp 48 to traffic-class 7
      !
      interface Ethernet1/1
        qos trust dscp
        !
        tx-queue 3
          random-detect ecn minimum-threshold 500 kbytes maximum-threshold 1500 kbytes max-mark-probability 20

### PFC

      qos map dscp 26 to traffic-class 3
      !
      interface Ethernet1/1
        qos trust dscp
        priority-flow-control mode on
        priority-flow-control priority 3 no-drop

### ECN+PFC

    qos map dscp 26 to traffic-class 3
      qos map dscp 48 to traffic-class 7
      !
      interface Ethernet1/1
        qos trust dscp
        priority-flow-control mode on
        priority-flow-control priority 3 no-drop
        !
        tx-queue 3
          random-detect ecn minimum-threshold 500 kbytes maximum-threshold 1500 kbytes max-mark-probability 20

## GPUDirect

GPUDirect support may be enabled by setting the gpudirect=true in the inventory file, or using the -g argument to install.sh. This will force source compilation of the drivers, and install the ib_peer_mem kernel module along with bnxt_en and
bnxt_re.

## Un-Installation

The uninstall.sh script can be used to remove all installed packages. This script is run locally on any host desired.

## Frequently Asked Questions

### Installation fails with: ERROR: Resolve RDMA device name

This error usually indicates that the RNIC firmware is not configured to support RoCE. To enable RDMA in the firmware, execute the following commands from the `Linux_Installer` directory:

    chmod +x ../../bnxtnvm/Linux/Release/bnxtnvm
    ../../bnxtnvm/Linux/Release/bnxtnvm -dev=<IP Interface name> setoption=support_rdma:0#1 # Enable RDMA on first RNIC interface
    ../../bnxtnvm/Linux/Release/bnxtnvm -dev=<IP Interface name> setoption=support_rdma:1#1 # Enable RDMA on the second RNIC interface
    reboot

### Installation fails with Ansible errors

Some Linux distributions package old or incomplete Ansible packages. A work-around is to uninstall the distribution Ansible package, and install Ansible directly from its Python repository:

   which yum && yum remove ansible
   which apt && apt remove ansible
   which zypper && zypper remove ansible
   pip3 install ansible
