Ubuntu Server Hardware Certification Test Case Guide¶

Introduction¶

This guide describes the jobs performed in Ubuntu server certification testing. A job can be anything executed by Checkbox, typically these are either information gathering jobs or test cases.

Test cases follow the format Category/TestName such as ethernet/detect. Some job names may simply be a TestName without the category designator. These are typically jobs that gather and/or attach hardware info for the submission file.

Jobs are grouped into three categories:

mandatory_include: Items that Checkbox will run every time. These jobs can not be skipped.
include: Items that Checkbox may run. These jobs can be skipped depending on the presence or absence of certain components, or software.
bootstrap_include: Items that Checkbox will run before the final test list is created. These jobs accomplish tasks such as gathering initial system information that is used to determine which test cases are applicable to the SUT.

This guide is based on the server-full-20.04.pxu list used for full Server Certification. Other lists in the canonical-certification-server UI are either subsets of this list, or not applicable to 20.04 certification, such as the 18.04 lists.

Tests¶

bootstrap include:¶

device: No description available
fwts: No description available

include:¶

benchmarks/disk/hdparm-cache-read_{name}: This test runs hdparm timing of cache reads as a benchmark for {name}
benchmarks/disk/hdparm-read_{name}: This test runs hdparm timing of device reads as a benchmark for {name}
cpu/clocktest: Runs a test for clock jitter on SMP machines.
cpu/frequency_governors: This test checks that CPU frequency governors are obeyed when set.
cpu/maxfreq_test: Use the Firmware Test Suite (fwts cpufreq) to ensure that the CPU can run at its maximum frequency.
cpu/maxfreq_test-log-attach: Attaches the log generated by cpu/maxfreq_test to the results submission.
cpu/topology: Parses information about CPU topology provided by proc and sysfs and checks that they are consistent.
disk/detect: Uses lsblk to gather information about each disk detected on the system under test.
disk/disk_cpu_load_{name}: Check to ensure CPU load of {product_slug} is not too high
disk/disk_stress_ng_{name}: Disk stress_ng test for {product_slug}
disk/read_performance_{name}: Verify that disk storage performs at or above baseline performance
disk/smart_{name}: This tests the SMART capabilities for {product_slug} (Note that this test may not work against hardware RAID)
disk/stats_{name}: This test checks disk stats, generates some activity and rechecks stats to verify they’ve changed. It also verifies that disks appear in the various files they’re supposed to.
info/hdparm_{name}.txt: Attaches the bootchart png file for bootchart runs
info/kvm_output: Attaches the debug log from the virtualization/kvm_check_vm test to the results submission.
mediacard/sd-preinserted-server: This is a fully automated version of mediacard/sd-automated and assumes that the system under test has a memory card device plugged in prior to checkbox execution.
memory/info: This test checks the amount of memory which is reporting in meminfo against the size of the memory modules detected by DMI.
memory/memory_stress_ng: Test to perform some basic stress and exercise of system memory via the stress_ng tool. This test also includes an over-commit function to force swapping to disk, thus SUTs should have suitably large swap files for the amount of RAM they have installed.
miscellanea/klog: Run Firmware Test Suite (fwts) klog tests.
miscellanea/klog_results.log: Attaches the FWTS klog results log to the submission
miscellanea/olog: Run Firmware Test Suite (fwts) olog tests (ppc64el only).
miscellanea/olog_results.log: Attaches the FWTS olog results log to the submission
miscellanea/oops: Run Firmware Test Suite (fwts) oops tests.
miscellanea/oops_results.log: Attaches the FWTS oops results log to the submission
networking/predictable_names: Verify that all network interfaces have predictable names.
optical/detect: Detects optical drives (CD/DVD) attached to the system.
optical/read-automated_{name}: This is an automated version of optical/read. It assumes you have already inserted a data CD into your optical drive prior to running Checkbox.
power-management/rtc: Verify that the Real-time clock (RTC) device functions properly, if present.
stress/cpu_stress_ng_test: Impose a high system load using the ‘stress_ng’ tool to exercise the CPU for several hours. The test is considered passed if the system does not freeze and if the stress_ng tool does not report errors.
usb/detect: Detects and shows USB devices attached to this system.
usb/storage-preinserted: Tests USB 2.0 or 1.1 ports on a system by doing write/read/compare tests on randomly created data. It requires that a USB stick is plugged into an available USB port before running the certification suite.
usb3/storage-preinserted: Tests USB 3.0 ports on a system by doing write/read/compare tests on randomly created data. It requires that a USB stick is plugged into an available USB port before running the certification suite. Additionally, it will only work with USB sticks and ports rated for USB 3.0 speeds or faster.
virtualization/kvm_check_vm: Verifies that a KVM guest can be created and booted using an Ubuntu Server cloud image.
virtualization/verify_lxd: Verifies that an LXD container can be created and launched