Stable Release Update Certification Coverage for Ubuntu Core

Introduction

This document lists the coverage of regression tests performed for validating a Stable Release Update (SRU) on Ubuntu Core Certified systems. The test plan is not recommended for, nor will it be accepted for self-testing purposes.

The certification tests listed in this document are performed for a reference image. Custom images and test plans are created for individual projects under the Ubuntu Internet of Things Certified Hardware programme.

For each test job, one of the following certification statuses is specified:

Blocking

Features that are required for certification. If any of the blocking tests fails, the certification will fail.

Non-blocking

Features that are tested but not mandatory for certification. Failure in non-blocking tests will not prevent certification. However, a note will be added to the certificate to inform potential customers or users.

Note

Only categories of hardware are tested and not specific types of hardware. For example, tests are run to verify USB controllers work, but the type of peripheral(s) used during those tests are not specified.

Coverage is flexible based on customer requirements (for example, if a device’s use cases don’t require LEDs, then LEDs can be non-blocking)

Full test descriptions can be found in Canonical certification site for partners:

https://certification.canonical.com

sru-ubuntucore

Note

The certification tests presented in this document are validated by Checkbox version 4.2.0.dev76.

Non-blocking

Audio tests

The following test units are covered in this category:

audio/alsa-loopback-automated

Captured sound matches played one (automated)

Category ID:

audio
Status:

Non-blocking
Purpose:

Check if sound that is ‘hearable’ by capture device. This test is intended mostly for devices without internal speakers/microphones, but with a line-in and a line-out ports to which a loopback cable can be connected.
After-suspend:

True
Environment variable:

ALSADEVICE, ALSA_CONFIG_PATH, LD_LIBRARY_PATH
User:

root
Plugin:

shell

Bluetooth tests

The following test units are covered in this category:

bluetooth/detect-output

Store Bluetooth device information for reports.

Category ID:

bluetooth
Status:

Non-blocking
Purpose:

Automated test to store Bluetooth device information in the Checkbox report
After-suspend:

True
Plugin:

shell
bluetooth4/beacon_eddystone_url_interface

Test system can get beacon EddyStone URL advertisements on the {interface} adapter

Unit type:

template
Category ID:

bluetooth
Status:

Non-blocking
Description:

<missing description>
After-suspend:

True
From template:

bluetooth4/beacon_eddystone_url_interface
Template resource:

device
Template filter:

device.category == ‘BLUETOOTH’

CPU tests

The following test units are covered in this category:

cpu/cpuinfo_before_suspend

Verify that all the CPUs are online before suspending

Category ID:

cpu
Status:

Non-blocking
Purpose:

<missing purpose>
Description:

<missing description>
Plugin:

shell
cpu/offlining_test

Test offlining of each CPU core

Category ID:

cpu
Status:

Non-blocking
Purpose:

Attempts to offline each core in a multicore system.
After-suspend:

True
User:

root
Plugin:

shell
cpu/scaling_test

Test the CPU scaling capabilities

Category ID:

cpu
Status:

Non-blocking
Purpose:

<missing purpose>
Description:

Use Firmware Test Suite (fwts cpufreq) to test the scaling capabilities of the CPU.
Environment variable:

LD_LIBRARY_PATH, PLAINBOX_SESSION_SHARE, SNAP
User:

root
Plugin:

shell
cpu/scaling_test-log-attach

Attach CPU scaling capabilities log

Category ID:

cpu
Status:

Non-blocking
Purpose:

Attaches the log generated by cpu/scaling_test to the results submission.
Plugin:

attachment
cpu/topology

Check CPU topology for accuracy between proc and sysfs

Category ID:

cpu
Status:

Non-blocking
Purpose:

<missing purpose>
Description:

Parses information about CPU topology provided by proc and sysfs and checks that they are consistent.
After-suspend:

True
Plugin:

shell

Disk tests

The following test units are covered in this category:

disk/read_performance_name

Disk performance test for {product_slug}

Unit type:

template
Category ID:

disk
Status:

Non-blocking
After-suspend:

True
Environment variable:

DISK_MDADM_READ_PERF, DISK_NVME_READ_PERF, DISK_READ_PERF
From template:

disk/read_performance_name
Template resource:

device
Template filter:

device.category == ‘DISK’

Ethernet Device tests

The following test units are covered in this category:

ethernet/detect

Detect if at least one ethernet device is detected

Category ID:

ethernet
Status:

Non-blocking
Purpose:

Test to detect and return information about available network controllers on the system under test.
After-suspend:

True
Plugin:

shell
ethernet/info_automated

Gather info on the current state of network devices

Category ID:

ethernet
Status:

Non-blocking
Purpose:

Gathers some info on the current state of your network devices. If no devices are found, the test will exit with an error.
Plugin:

shell
ethernet/ping-with-any-cable-interface

Can ping the gateway with any cable Ethernet interface

Category ID:

ethernet
Status:

Non-blocking
Purpose:

Check any of the cable Ethernet ports available on the system can ping its default gateway.
After-suspend:

True
User:

root
Plugin:

shell

Informational tests

The following test units are covered in this category:

acpi_sleep_attachment

Attach the contents of /proc/acpi/sleep for further analysis.

Category ID:

info
Status:

Non-blocking
Purpose:

Attaches the contents of /proc/acpi/sleep if it exists.
Plugin:

attachment
codecs_attachment

Attach a report of installed codecs for Intel HDA.

Category ID:

info
Status:

Non-blocking
Purpose:

Attaches a report of installed codecs for Intel HDA
Plugin:

attachment
cpuinfo_attachment

Attach a copy of /proc/cpuinfo

Category ID:

info
Status:

Non-blocking
Purpose:

Attaches a report of CPU information
Plugin:

attachment
dmesg_attachment

Attach a copy of dmesg or the current dmesg buffer to the test results.

Category ID:

info
Status:

Non-blocking
Purpose:

Attaches a copy of /var/log/dmesg or the current dmesg buffer to the test results
User:

root
Plugin:

attachment
dmidecode_attachment

Attach output of dmidecode

Category ID:

info
Status:

Non-blocking
Purpose:

<missing purpose>
Description:

Attaches dmidecode output
User:

root
Plugin:

attachment
efi_attachment

Attaches firmware version info

Category ID:

info
Status:

Non-blocking
Purpose:

Attaches the firmware version
User:

root
Plugin:

attachment
lsmod_attachment

Attach a list of currently running kernel modules

Category ID:

info
Status:

Non-blocking
Purpose:

Attaches a list of the currently running kernel modules.
Plugin:

attachment
lspci_attachment

Attach a list of PCI devices

Category ID:

info
Status:

Non-blocking
Purpose:

Attaches very verbose lspci output.
Plugin:

attachment
meminfo_attachment

Attach copy of /proc/meminfo

Category ID:

info
Status:

Non-blocking
Purpose:

Attaches info on system memory as seen in /proc/meminfo.
Plugin:

attachment
modprobe_attachment

Attach the contents of /etc/modprobe.*

Category ID:

info
Status:

Non-blocking
Purpose:

<missing purpose>
Description:

Attaches the contents of the various modprobe conf files.
User:

root
Plugin:

attachment
modules_attachment

Attach the contents of /etc/modules

Category ID:

info
Status:

Non-blocking
Purpose:

<missing purpose>
Description:

Attaches the contents of the /etc/modules file.
Plugin:

attachment
net_if_management_attachment

Collect logging from the net_if_management job

Category ID:

info
Status:

Non-blocking
Purpose:

<missing purpose>
Description:

Allows logging of debug info and errors by the associated resource job
Plugin:

attachment
sysctl_attachment

Attach sysctl configuration files.

Category ID:

info
Status:

Non-blocking
Purpose:

<missing purpose>
Description:

Attaches the contents of various sysctl config files.
User:

root
Plugin:

attachment

Media Card tests

The following test units are covered in this category:

mediacard/sd-preinserted

Automated test of SD Card reading & writing (udisks2)

Category ID:

mediacard
Status:

Non-blocking
Purpose:

This is a fully automated version of mediacard/sd-automated and assumes that the system under test has a memory card device plugged in prior to checkbox execution. It is intended for SRU automated testing.
User:

root
Plugin:

shell

Memory tests

The following test units are covered in this category:

memory/info

Check the amount of memory reported by meminfo against DMI

Category ID:

memory
Status:

Non-blocking
Purpose:

This test checks the amount of memory which is reported in meminfo against the size of the memory modules detected by DMI.
User:

root
Plugin:

shell
memory/meminfo_before_suspend

Store memory info before suspending

Category ID:

memory
Status:

Non-blocking
Purpose:

<missing purpose>
Description:

<missing description>
Plugin:

shell

Miscellaneous tests

The following test units are covered in this category:

miscellanea/submission-resources

Check that data for a complete result are present

Category ID:

miscellanea
Status:

Non-blocking
Purpose:

<missing purpose>
Description:

A meta-job that verifies the data necessary for a complete result submission are present. Failure indicates that the results are incomplete and may be rejected.
Plugin:

shell

Non-device specific networking tests

The following test units are covered in this category:

networking/http

Ensure downloading files through HTTP works correctly.

Category ID:

networking
Status:

Non-blocking
Purpose:

Automated test case to make sure that it’s possible to download files through HTTP
Environment variable:

TRANSFER_SERVER
Plugin:

shell

Power Management tests

The following test units are covered in this category:

power-management/fwts_wakealarm

Executes ACPI Wakealarm test to validate functionality.

Category ID:

power-management
Status:

Non-blocking
Purpose:

Test ACPI Wakealarm (fwts wakealarm)
Environment variable:

PLAINBOX_SESSION_SHARE
User:

root
Plugin:

shell
power-management/fwts_wakealarm-log-attach

Attach and display fwts wakealarm test log.

Category ID:

power-management
Status:

Non-blocking
Purpose:

Attach log from fwts wakealarm test
Plugin:

attachment
power-management/rtc

Test that RTC functions properly (if present)

Category ID:

power-management
Status:

Non-blocking
Purpose:

Verify that the Real-time clock (RTC) device functions properly, if present.
After-suspend:

True
Environment variable:

RTC_DEVICE_FILE
User:

root
Plugin:

shell
power-management/tickless_idle_kernel

Verify if Tickless Idle (CONFIG_NO_HZ) is enabled in the kernel configuration.

Unit type:

template
Category ID:

power-management
Status:

Non-blocking
From template:

power-management/tickless_idle_kernel
Template resource:

model_assertion

Suspend tests

The following test units are covered in this category:

suspend/suspend-single-log-attach

Attaches the log from the single suspend/resume test

Category ID:

suspend
Status:

Non-blocking
Purpose:

<missing purpose>
Description:

<missing description>
Plugin:

attachment
suspend/suspend-single-log-check

Automated check of the suspend log for errors reported by fwts

Category ID:

suspend
Status:

Non-blocking
Purpose:

<missing purpose>
Description:

<missing description>
Plugin:

shell
suspend/suspend_advanced_auto

Automated test of suspend function

Category ID:

suspend
Status:

Non-blocking
Purpose:

This is the automated version of suspend/suspend_advanced.
Environment variable:

PLAINBOX_SESSION_SHARE, RTC_DEVICE_FILE
User:

root
Plugin:

shell

USB tests

The following test units are covered in this category:

usb/detect

Display USB devices attached to SUT

Category ID:

usb
Status:

Non-blocking
Purpose:

Detects and shows USB devices attached to this system.
After-suspend:

True
Plugin:

shell
usb/storage-preinserted

Test USB 2.0 or 1.1 ports

Category ID:

usb
Status:

Non-blocking
Purpose:

<missing purpose>
Description:

Tests USB 2.0 or 1.1 ports on a system by doing write/read/compare tests on randomly created data. It requires that a USB stick is plugged into an available USB port before running the certification suite.
After-suspend:

True
User:

root
Plugin:

shell

Wireless networking tests

The following test units are covered in this category:

wireless/wireless_connection_open_ac_nm_interface

Connect to unencrypted 802.11ac Wi-Fi network on {{ interface }}

Unit type:

template
Category ID:

wireless
Status:

Non-blocking
After-suspend:

True
From template:

wireless/wireless_connection_open_ac_nm_interface
Template resource:

device
Template filter:

device.category == ‘WIRELESS’ and device.interface != ‘UNKNOWN’
wireless/wireless_connection_open_bg_nm_interface

Connect to an unencrypted 802.11b/g Wi-Fi network on {{ interface }}

Unit type:

template
Category ID:

wireless
Status:

Non-blocking
After-suspend:

True
From template:

wireless/wireless_connection_open_bg_nm_interface
Template resource:

device
Template filter:

device.category == ‘WIRELESS’ and device.interface != ‘UNKNOWN’
wireless/wireless_connection_open_n_nm_interface

Connect to an unencrypted 802.11n Wi-Fi network on {{ interface }}

Unit type:

template
Category ID:

wireless
Status:

Non-blocking
After-suspend:

True
From template:

wireless/wireless_connection_open_n_nm_interface
Template resource:

device
Template filter:

device.category == ‘WIRELESS’ and device.interface != ‘UNKNOWN’
wireless/wireless_connection_wpa_ac_nm_interface

Connect to WPA-encrypted 802.11ac Wi-Fi network on {{ interface }}

Unit type:

template
Category ID:

wireless
Status:

Non-blocking
After-suspend:

True
From template:

wireless/wireless_connection_wpa_ac_nm_interface
Template resource:

device
Template filter:

device.category == ‘WIRELESS’ and device.interface != ‘UNKNOWN’
wireless/wireless_connection_wpa_bg_nm_interface

Connect to WPA-encrypted 802.11b/g Wi-Fi network on {{ interface }}

Unit type:

template
Category ID:

wireless
Status:

Non-blocking
After-suspend:

True
From template:

wireless/wireless_connection_wpa_bg_nm_interface
Template resource:

device
Template filter:

device.category == ‘WIRELESS’ and device.interface != ‘UNKNOWN’
wireless/wireless_connection_wpa_n_nm_interface

Connect to a WPA-encrypted 802.11n Wi-Fi network on {{ interface }}

Unit type:

template
Category ID:

wireless
Status:

Non-blocking
After-suspend:

True
From template:

wireless/wireless_connection_wpa_n_nm_interface
Template resource:

device
Template filter:

device.category == ‘WIRELESS’ and device.interface != ‘UNKNOWN’
wireless/wireless_scanning_interface

Test system can discover Wi-Fi networks on {{ interface }}

Unit type:

template
Category ID:

wireless
Status:

Non-blocking
After-suspend:

True
From template:

wireless/wireless_scanning_interface
Template resource:

device
Template filter:

device.category == ‘WIRELESS’ and device.interface != ‘UNKNOWN’

Appendix A. FWTS tests

As part of the certification process, we run a series of firmware tests that are part of the Canonical Firmware Test Suite. In general, any HIGH or CRITICAL error found in the fwts log can cause potential errors in the system and should be looked at by OEMs/ODMs.

Category

Test Item

Description

Information

acpidump

Check ACPI table acpidump.

Information

version

Gather kernel system information.

ACPI

acpitables

ACPI table settings confidence checks.

ACPI

apicinstance

Check for single instance of APIC/MADT table.

ACPI

hpet_check

High Precision Event Timer configuration test.

ACPI

mcfg

MCFG PCI Express* memory mapped config space.

ACPI

method

ACPI DSDT Method Semantic Tests.

CPU

mpcheck

Check Multi Processor tables.

CPU

msr

CPU MSR consistency check.

CPU

mtrr

MTRR validation.

System

apicedge

APIC Edge/Level Check.

System

klog

Scan kernel log for errors and warnings.