2. Preparing the Test Environment

Before you test the hardware, you must perform some initial setup steps. These steps are preparing ancillary hardware, configuring the SUT for testing, and configuring the test network.

2.1. Ensuring Equipment is Ready

The requirements for running the tests for a server are minimal. Ensure that you have:

  • Writable USB sticks with enough free space (> 256 MB). Each stick must contain a single partition with a writable FAT filesystem on it. Note that a USB stick with multiple partitions may cause problems, so if necessary you should repartition your device to have a single partition. Modern computers typically provide both USB 2 and USB 3 ports, but we require testing of only the fastest USB device – normally the USB 3 port. If you need to test more than one computer then bring enough USB sticks to test all the systems.

  • Writable SD cards configured with the same rules as the USB sticks. These SD cards are needed only on those (rare) servers that have external SD card slots.

  • A computer to function as an iperf3 Target on the test LAN. This computer must have network ports that equal or exceed the speed of the fastest network port on your SUTs.

  • A computer to function as a MAAS server on the test LAN. This server will provision the SUT. (Note that a MAAS server for certification testing should ideally be configured to automatically install the Server Test Suite on the SUT, which will not be the case for a “generic” MAAS server.)

Frequently, the MAAS server and iperf3 Target are one and the same computer; however, sometimes it is desirable to use two computers for these two purposes. The upcoming section, Preparing the Network Test Environment, covers these servers in more detail.

2.2. Configuring the SUT for Testing

The following should be considered the minimum requirements for setting up the SUT and test environment:

  • Minimum loadout

    • Minimum of 4 GiB RAM

    • 1 internal storage device (HDD, SSD, or NVMe); 2 with minimal RAID

    • 1 CPU of a supported type

  • Recommended (preferred) loadout

    • Maximum supported number of internal storage devices, especially if you can configure multiple RAID levels (e.g. 2 for RAID 0, 3 for RAID 5, and 6 for RAID 50)

    • The largest disk capacity available from the OEM – ideally, over 2 TiB on a single disk or RAID array

    • Maximum amount of supported RAM

    • Maximum number of supported CPUs

    • If the SUT has multiple disk controllers (such as a motherboard-based disk controller and a separate RAID controller), we strongly recommend that disk devices be connected to all controllers during testing.

  • All hardware, including CPUs, must be production level. Development level hardware is not eligible for certification.

    • This can be excepted on a case by case basis for pre-release systems where the Partner and Canonical have arranged certification to be published coinciding with a SUT’s release announcement.

  • If possible, as many processors as the SUT will support should be installed.

    • Note that systems that ship with processors from different families (e.g., Coffee Lake vs. Cascade Lake) will require extra testing.

    • CPU speed bumps and die shrinks do not require extra testing.

  • The SUT should not contain any extraneous PCI devices that are not part of the certification.

    • This includes things like network, SAN and iSCSI cards.

    • Hardware RAID cards are allowed if they are used to provide RAID services to the SUT’s onboard storage.

  • Virtualization (VMX/SVM) should be enabled in the BIOS/UEFI, when supported by the CPU’s architecture.

  • The SUT should be running release or GA level (not development level) firmware. Development level firmware is generally not eligible for certification.

    • The one exception to this rule is that you may use unsigned GA equivalent firmware if such allows you to flash the system up and down rev as needed. However, the test results must show the version to be equal to the publicly available version.

  • BIOS/UEFI should be configured using factory default settings, with the following exceptions:

    • Many systems ship with IPMI-over-LAN disabled by default. This will need to be enabled to ensure MAAS can remotely manage the SUT.

    • If the hardware virtualization options in the BIOS/UEFI are not enabled, enable them, save the settings and allow the SUT to reboot.

    • The SUT must be configured so that at least ONE network device is set to PXE-boot by default.

    • If the SUT’s firmware supports PXE-booting in UEFI mode, it must be configured to boot in UEFI mode, rather than in BIOS/CSM/legacy mode.

    • All non-network devices have been removed from the Boot Options so that Network Booting (PXE) is the only option. The Ubuntu installer will add an Ubuntu menu option and that is OK, but testing expects the Network Boot device is the first item in the boot menu.

      • On systems where it is impossible to remove non-network boot options, please ensure that the Network Booting option is the first (default) boot option so that the machine ALWAYS boots from network first.

  • The SUT Firmware should have verifiable identifiers in DMI Types 1, 2 and/or 3 that match the information entered in the hardware entry on C3.

    • For ODMs this means Make and Model data must be accurate in DMI Types 1 and 2.

    • For OEMs where Make and Model may not be determined yet, some other identifier must exist in DMI Types 1, 2 and/or 3 that matches data provided in the Hardware Entry on C3

  • Storage should be properly configured.

    • Disks must be configured for “flat” storage – that is, filesystems in plain partitions, rather than using LVM or bcache configurations. Similarly, software RAID must not be used.

  • The SUT’s BMC, if present, may be configured via DHCP or with a static IP address. If the BMC uses IPMI, MAAS will set up its own BMC user account (maas) when enlisting the SUT.

  • A monitor and keyboard for the SUT are helpful because they will enable you to monitor its activities. If necessary, however, certification can be done without these items.

2.3. Preparing the Network Test Environment

Particularly if you’re testing in a location where you’ve never before tested, or if you’re testing a SUT with unique network hardware, you may need to prepare the network environment. In particular, you should pay attention to the following:

  • In addition to the SUT, the network must contain at least one other machine, which will run MAAS and an iperf3 server; however, you may want to separate these two functions.

    • Certification Environment Setup Guide describes how to configure a MAAS server. The MAAS server computer itself does not need to be very powerful by modern standards. MAAS version 3.1 or later is required for certification work.

    • When testing multiple SUTs simultaneously, you will need multiple iperf3 Targets, one for each SUT. If your iperf3 Target has a sufficiently fast NIC or multiple NICs, you can assign the computer multiple IP addresses and treat each one as a distinct Target. This topic is covered in more detail in Appendix B: Network Testing Options. Alternatively, you can run network tests against a single iperf3 Target sequentially; however, this approach complicates submission of results. Note that poor network infrastructure may make multiple simultaneous iperf3 runs unreliable.

  • Ideally, the network should have few or no other computers; extraneous network traffic can negatively impact the network tests.

  • The network should have no DHCP or TFTP servers, other than the one provided by MAAS.

  • Network cabling, switches, and the iperf3 server should be capable of at least the SUT’s best speed. For instance, if the SUT has 100 Gbps Ethernet, the other network components should be capable of 100 Gbps or faster speeds. If the local network used for testing is less capable than the best network interfaces on the SUT, the network test won’t run, and those interfaces must be tested later on a more-capable network. If the test environment uses separate networks with different speeds, with the SUT cabled to multiple networks via different ports, you can specify multiple iperf3 servers, as described later.

  • If desired, the MAAS server may be run inside a virtual machine; however, it is advisable to run the iperf3 server on “real” hardware so as to minimize the risk of network tests failing because of virtualization issues.

  • Every network port must be cabled to the LAN and properly configured with either DHCP or static addressing. If a SUT has 4 NIC ports, then all 4 must be connected to the LAN.