Ubuntu Server Hardware Certification Test Case Guide

Introduction

This guide describes the jobs performed in Ubuntu server certification testing. A job can be anything executed by Checkbox, typically these are either information gathering jobs or test cases.

Test cases follow the format Category/TestName such as ethernet/detect. Some job names may simply be a TestName without the category designator. These are typically jobs that gather and/or attach hardware info for the submission file.

Jobs are grouped into three categories:

mandatory_include

Items that Checkbox will run every time. These jobs can not be skipped.

include

Items that Checkbox may run. These jobs can be skipped depending on the presence or absence of certain components, or software.

bootstrap_include

Items that Checkbox will run before the final test list is created. These jobs accomplish tasks such as gathering initial system information that is used to determine which test cases are applicable to the SUT.

This guide is based on the server-full-20.04.pxu list used for full Server Certification. Other lists in the canonical-certification-server UI are either subsets of this list, or not applicable to 20.04 certification, such as the 18.04 lists.

Tests

bootstrap include:

device

No description available

fwts

No description available

include:

benchmarks/disk/hdparm-cache-read_{name}

This test runs hdparm timing of cache reads as a benchmark for {name}

benchmarks/disk/hdparm-read_{name}

This test runs hdparm timing of device reads as a benchmark for {name}

cpu/clocktest

Runs a test for clock jitter on SMP machines.

cpu/frequency_governors

This test checks that CPU frequency governors are obeyed when set.

cpu/maxfreq_test

Use the Firmware Test Suite (fwts cpufreq) to ensure that the CPU can run at its maximum frequency.

cpu/maxfreq_test-log-attach

Attaches the log generated by cpu/maxfreq_test to the results submission.

cpu/topology

Parses information about CPU topology provided by proc and sysfs and checks that they are consistent.

disk/detect

Uses lsblk to gather information about each disk detected on the system under test.

disk/disk_cpu_load_{name}

Check to ensure CPU load of {product_slug} is not too high

disk/disk_stress_ng_{name}

Disk stress_ng test for {product_slug}

disk/read_performance_{name}

Verify that disk storage performs at or above baseline performance

disk/smart_{name}

This tests the SMART capabilities for {product_slug} (Note that this test may not work against hardware RAID)

disk/stats_{name}

This test checks disk stats, generates some activity and rechecks stats to verify they’ve changed. It also verifies that disks appear in the various files they’re supposed to.

info/hdparm_{name}.txt

Attaches the bootchart png file for bootchart runs

info/kvm_output

Attaches the debug log from the virtualization/kvm_check_vm test to the results submission.

mediacard/sd-preinserted-server

This is a fully automated version of mediacard/sd-automated and assumes that the system under test has a memory card device plugged in prior to checkbox execution.

memory/info

This test checks the amount of memory which is reporting in meminfo against the size of the memory modules detected by DMI.

memory/memory_stress_ng

Test to perform some basic stress and exercise of system memory via the stress_ng tool. This test also includes an over-commit function to force swapping to disk, thus SUTs should have suitably large swap files for the amount of RAM they have installed.

miscellanea/klog

Run Firmware Test Suite (fwts) klog tests.

miscellanea/klog_results.log

Attaches the FWTS klog results log to the submission

miscellanea/olog

Run Firmware Test Suite (fwts) olog tests (ppc64el only).

miscellanea/olog_results.log

Attaches the FWTS olog results log to the submission

miscellanea/oops

Run Firmware Test Suite (fwts) oops tests.

miscellanea/oops_results.log

Attaches the FWTS oops results log to the submission

networking/predictable_names

Verify that all network interfaces have predictable names.

optical/detect

Detects optical drives (CD/DVD) attached to the system.

optical/read-automated_{name}

This is an automated version of optical/read. It assumes you have already inserted a data CD into your optical drive prior to running Checkbox.

power-management/rtc

Verify that the Real-time clock (RTC) device functions properly, if present.

stress/cpu_stress_ng_test

Impose a high system load using the ‘stress_ng’ tool to exercise the CPU for several hours. The test is considered passed if the system does not freeze and if the stress_ng tool does not report errors.

usb/detect

Detects and shows USB devices attached to this system.

usb/storage-preinserted

Tests USB 2.0 or 1.1 ports on a system by doing write/read/compare tests on randomly created data. It requires that a USB stick is plugged into an available USB port before running the certification suite.

usb3/storage-preinserted

Tests USB 3.0 ports on a system by doing write/read/compare tests on randomly created data. It requires that a USB stick is plugged into an available USB port before running the certification suite. Additionally, it will only work with USB sticks and ports rated for USB 3.0 speeds or faster.

virtualization/kvm_check_vm

Verifies that a KVM guest can be created and booted using an Ubuntu Server cloud image.

virtualization/verify_lxd

Verifies that an LXD container can be created and launched