In order to ensure the best user experience possible when running Ubuntu on
Partner Servers, Canonical requires that hardware partners sufficiently test
the list of Vendor Approved Options that are applicable to a given Server Model.
As we understand this can be a very long list of devices, your Partner Engineer
will work with you to determine a minimal amount of testing necessary to
maximise coverage of options across all server models.
This means that not every single device will need to be tested, and none will
need to be tested more than once or twice. For this, we require a spec sheet
that includes a listing of any Vendor Approved Options for each Server Model
to be certified so that we can help determine the matrix of tests needed.
Hardware Partners can send Spec Sheets directly to your Partner Engineer, or can
provide a URL to online copies of Spec Sheets.
As this testing is required to help guarantee the quality of Ubuntu on your
hardware, and to ensure that end users have the best experience possible when
deploying your hardware into an Ubuntu environment in their datacenters,
failure to perform sufficient testing of options may result delayed or denied
certification requests.
The Server should be configured so that the only item in the boot order is a
single network device configured for Network Booting. If it is impossible to
remove HDD/SDD/NVMe devices from the boot menu, the Network Boot device should
always be in the first position to ensure the machine ALWAYS boots from
Network.
MAAS will instruct the server, during the network boot, to boot from onboard
storage if necessary.
Any CPU from a given CPU Family (e.g. Cascade Lake, Ice Lake, Rome, Milan) will
count for the entire CPU family. You do not need to re-test for each CPU unless
the CPU is from a different family. It is recommended that you do test different
models of CPU across a given server line to help broaden the scope of testing.
Any DIMM size may be tested, though we strongly encourage you to use the
largest DIMM size available.
Jumps in DIMM size do not require additional testing.
Systems that support Intel’s Optane DataCenter Persistent Memory Modules must
include those DCPMM devices. For convenience, DPCMMs can be configured in mixed
mode where supported in a mix of at least 30% RAM and 70% Storage. (That
percentage will vary based on the amount of DCPMM storage installed).
Where Mixed Mode is not supported, then testers will need to test the DCPMM
devices in both Memory Mode and App Direct (Storage) Mode, which requires
reconfiguring the DCPMM devices, then recommissioning in MAAS and re-running
the appropriate test command (one of test-memory or test-storage).
When configuring the DCPMMs for storage a percentage should be dedicated to all
block store modes: fsdax, sector, and raw. As with any other storage device,
each DCPMM storage volume should be properly partitioned, formatted, and
mounted prior to testing.
ALL RAID controllers must be tested. Exceptions can be made to this for very
minor variances in RAID Controller Models.
An example of this exception would be two SAS RAID Controllers where they are
essentially identical except for the addition or deletion of a Cache Battery.
Your Partner Engineer or the Certification Team will work with you to determine
if a RAID Controller is exempt from testing. This will be handled as we build
the test matrix for all Vendor Approved Options that apply to each Server
Model.
HBAs must be tested. We will work with you to determine which particular ones
from the list of Vendor Approved Options will need to be tested and which can
be risk assumed.
CNAs and NICs must be tested, CNAs should be tested in Network mode. Canonical
reserves the right to require further CNA testing in other modes as necessary
(for instance a CNA may need to be tested as an iSCSI initiator as well as a
Network card).
For Network Devices, all ports must be configured and tested on appropriate
network segments (e.g. a 100Gb port must be connected to a network segment that
is at least 100Gb and the iperf target must also be at least 100Gb).
GPGPU testing is supported for NVIDIA and AMD GPGPUs. This is a separate test.
When used on NVIDIA GPGPUs, the test requires the installation of drivers, a
system reboot, and running a separate test suite. All GPGPU models should be
tested, at this time there is no allowance for “representative” samples on GPGPU
devices. Please refer to Appendix G - Setting Up and Testing a GPGPU of the
Self-Testing Guide for more information.
Any device not explicitly outlined above must be tested. If you have any
questions about what should or should not be tested, please contact your
Partner Engineer.
Application of Test Results for Vendor Approved Options¶
Once a Vendor Approved Option has been tested in ANY Model of a Partner’s
Server Line, that Vendor Approved Option will be considered certified for the
full Server Line.
Thus, if a Server Line has 10 Models, and the Vendor sells 1 (one) model of
100Gb Network Controller, once that controller has been validated in Model 1,
it will also be automatically considered certified for Models 2 - 10.
This does not apply to Blade systems. A PCIe card tested in a rack or tower
server chassis does not remove the need to test a Mezzanine card with the same
chipset in a Blade or Compute Sled style system.
Once a Model is certified it will be publicly listed on the Ubuntu Certification
Website described later in this document. In addition to the Model, all Vendor
Approved Options that apply to the Certified Model will be listed alongside
that Model. Each Vendor Approved Option will show it’s status (e.g. Certified,
Untested, Unsupported, etc) and at least the Ubuntu Version or Kernel Version
that the Option was tested against.
This will give customers a more complete view of what Models and Options are
supported by Ubuntu Server.
This will also make creating BOMs for projects easier as there will no longer
be any question if the components in a given BOM have been certified.
Inadequate Test Coverage of Vendor Approved Options¶
We do not expect the Partner to test all Vendor Approved Options, which is why
your Partner Engineer will work with you to create a test matrix and highlight
the minimum devices to be tested to provide the maximum coverage of all Options
for the Partner Server Model.
Likewise, testing of Vendor Approved Options can be spread out over time and
across Certification tests for several servers. Thus if you have a list of 20
Vendor Approved Options and 10 Server Models, you could spread that test effort
out so that each Server Model only features 2 Options.
However, as alluded to above, if adequate coverage is not seen within a
reasonable amount of time, we reserve the right to delay or reject certificates,
and possibly revoke existing certifications as a last resort.
Certifications are available for the two most recent LTS versions of Ubuntu
Server. At this time, this includes 22.04 LTS and 24.04 LTS.
Certification is never granted for Interim releases of Ubuntu Server (non-LTS
versions such as 22.10, 23.04, or 23.10).
Certification Testing should be performed using the LTS release and GA kernel
initially (e.g. 24.04 LTS and the ga-24.04 kernel option in MAAS).
When hardware cannot pass certification because of hardware support issues with
the GA kernel, testers may use the most recent HWE kernel option (e.g. 24.04
LTS and the hwe-24.04 kernel option in MAAS) to perform testing.
Certification is valid from the certified kernel onward including all
subsequent kernel updates and HWE kernel releases for that LTS.
In other words, if a system is certified using 22.04 LTS and the 5.15 GA kernel,
that system is certified for the 5.15 (22.04 GA, 22.04.1 GA), 5.19 (22.04.2 HWE),
6.2 (22.04.3 HWE), 6.5 (22.04.4 HWE) and eventually 6.8 (22.04.4 HWE) kernels that
comprise the 22.04 LTS and LTS Point Release family.
If a system is certified using 22.04 LTS and the HWE kernel, then the certification
is valid from that HWE kernel version onward. Thus if the system was certified
using 22.04.3 LTS and the 6.2 HWE kernel, the system is considered certified for
the 22.04.3 LTS HWE Kernel version 6.2, the 22.04.4 LTS HWE Kernel version
6.5, and the 22.04.4 LTS HWE Kernel version 6.8.
Any exceptions to this policy will be decided on a case by case basis before
the certification can be issued.
Installation should be performed using the Ubuntu images and kernels provided
by the default MAAS image stream hosted at https://maas.io
Deployed OSs for Certification should not be updated with current package
versions unless explicitly instructed to by the Server Certification Team.
Testing should always start with the GA version of Ubuntu unless the HWE kernel
is necessary to provide support for newer hardware. Doing certification
primarily on the GA release will ensure the longest support lifetime for
customers, while the HWE release will ensure customers are able to use the
newest hardware.
Certification should always be performed using the most recent version of the
Certification Suite. This is installed separately after the OS Version has been
installed on the SUT. Installation is usually performed automatically as part
of the deployment process.
Figure 1: The typical Ubuntu Server Certification LTS Support Schedule. This
is also available for download and closer examination from the Certification
Portal: https://certification.canonical.com/server/lifecycle/
Certifications are valid from the point release they are issued against until
the end of the lifetime for that LTS. For example, a system certified on
Ubuntu 22.04.2 LTS will be considered certified for 22.04.2, 22.04.3 and 22.04.4
but will not be considered certified for 24.04 LTS or subsequent LTS
releases.
Those subsequent LTS releases will constitute a new LTS release family and will
require separate certification of each Model and Vendor Approved Option.
For the purpose of Ubuntu Server Certification, we encourage the testing and
certification of a Partner’s entire server line, including user selectable
Vendor Approved Options.
For each Server Model, we will list all Vendor Approved Components that were
tested. We will also list the supported/certified status of each of those
components and that information will be plainly visible on the public
certification website for each model the Vendor Approved Components are
applicable. In order to ensure coverage of as many component options as
possible, the Certification team will work with the Partner to decide on a test
matrix to cover a model and its full breadth of component options.
The Certification Suite is constantly evolving as new testing methods are
employed, as technology changes, and as bugs are discovered and resolved within
the Suite.
Changes to existing test cases in a given LTS will not change the test
requirements, and will likely only change the method used to test.
Newly introduced tests are considered a Suite change and will not block current
LTS certifications.
For example, the KVM test has moved from using qemu directly to using LXD to
manage the virtual machine. As this only changes how the test is conducted, the
VM test is still considered a blocking test Conversely, the addition of a
TPM 2.0 test would constitute a suite change and would not gate current
LTS certifications, but MAY become a blocker for future LTS releases.
Likewise, tests specifically for new technologies will not be blockers on the
current LTS, but could become blockers on the next LTS.
The requirements for Certification are considered fluid up to the day the LTS
is released. At that point, certification requirements are locked in and will
not change for the life of that LTS. Any new test cases will be introduced as
Non-Blocking items and will not gate certifications.
Note that this only applies to additions to the requirements. Requirements can
be eased (tests removed) at any time and will be applicable to all
certifications going forward. An example of this would be the removal of the
requirement for floppy disk testing as such devices are not in use any longer.
As the Suite is constantly evolving, there is a natural progression for tests
that is applied throughout the development cycle.
Any new test is introduced as a Non-Blocking item. This implies that the test must
be run, but will not gate the certification effort for the current LTS. As we
approach the next LTS, Non-Blocking tests are re-evaluated for promotion to
Blocking (Required) tests and likewise, Blocking tests are evaluated for
demotion to Non-Blocking or removal altogether.
As a more concrete example, let’s suppose a new Storage I/O stress test is
introduced after 24.04 LTS is released but before 26.04 LTS. That new test
would be introduced as a Non-Blocking test, thus any failures would not gate
24.04 certifications. This “Break-In” period is a chance to review and improve
the test as well as gather data from various testing scenarios to determine its
viability later on.
As we approach 26.04, we would re-evaluate this new Storage Stress test. If it
is seen as important and reliable enough, then it will be promoted to Blocking
for 26.04 and be required to pass for all 26.04 certifications.
Also keep in mind that even though this test would now be required for 26.04,
it will still remain a Non-Blocking item for 24.04 certifications.
The policies for Certification are subject to change at any time for any
reason. That said, we make every effort to minimize policy changes and make
modifications only where necessary for changing business needs.
For all Servers that support virtualization, the KVM and LXD tests must be
run with Ubuntu as the host OS. This will launch an Ubuntu guest and validate
that virtualization functions. Certification does not apply to using any other
operating systems as a virtualization guest OS.
In special situations, we will provide Certification of Ubuntu as a guest OS on
a different host OS. These certifications are provided on a case-by-case basis
and must be agreed upon by both Canonical and the Partner. Please discuss this
with your Partner Engineer if you need to certify Ubuntu as a guest on your
hypervisor.
Guests or VMs created for the purpose of certifying Ubuntu as Guest on a
non-Ubuntu host OS should have a minimum of 4 GiB RAM and at least 100 GiB of
disk space to ensure the tests run successfully.
Guests should also have at least one virtual NIC that can successfully ping the
MAAS server and iperf target.
Certifications of this type will use the “virtual-machine-full” test plan, which
is a subset of the full server suite defined by the “server-full” test plan.
KVM testing is generally not required for certification of Ubuntu as Guest
scenarios as nested virtualization (e.g. running KVM inside a VM is considered
an advanced/non-standard configuration.) This exception may not apply to
certain special situations that are business goal dependent. That
determination will be made by the Certification Team.
The Server Hardware Certification Team also provides System on Chip
Certification Services for companies that produce SoCs meant to be used in
server systems built by OEM/ODMs down the road.
SoC Certification applies only to Systems on Chip and reference boards that
showcase those SoCs. It does not apply to production server systems based on
SoCs.
Additionally there is no inheritance upstream. So though an SoC may be
certified by an SoC vendor like Ampere or HiSilicon, OEM/ODMs who build
servers based on that SoC cannot also claim certification for their server.
ARM64 SoC based servers can ONLY be certified if the SoC in use is also SoC
certified. This is due to the need for enablement and ongoing maintenance of
code specific to numerous SoCs.
Also, SoC certification does not imply Server certification and Server
certification does not imply SoC certification.
SoC certification is best thought of as a subset of Server Certification. The
tools and test cases are the same, but SoCs have less stringent requirements
for certification. For example, SoCs can have non-functional blocks at the
time of Certification.
The implication is that while an SoC may have non-functional blocks (such as
USB3), those blocks will and should be enabled by the time an OEM/ODM is
creating a server based on that SoC. Note that once a server product based on
a certified SoC is presented for Server Certification, it must adhere to the
more stringent Server Certification rules. In other words, once the SoC
becomes a server, all hardware must work, with the only exception being
non-accessible/non-included blocks. A compute card with no externally
accessible USB ports, for example, will not need to pass the USB tests.
Additionally, SoC certification does not imply any level of support beyond
basic functionality where Server Certification does imply a level of support
including Ubuntu Advantage and other avenues.
As noted above, SoC Certification is a subset of Server Certification with less
stringent rules. Thus exceptions can be made for items that are non-functional
at the time of SoC Certification. When these are encountered, the
certification will have a note attached indicating what items are not considered
certified and are non-functional or untested.
Canonical does not perform performance or benchmark testing as part of
certification. Any benchmark or stress tools utilized are used strictly with
the goal of introducing significant load to the system. It is the
responsibility of the Partner to properly benchmark their own hardware with
Ubuntu installed.
Third Party Testing means testing hardware on behalf of another company. This
happens when an OEM produces a system that is sold to a reseller who re-brands
the hardware and sells at retail under the reseller’s name and marketing model.
Third Party Testing for Certification is ONLY allowed on a case-by-case basis
and must be agreed upon by Canonical, the OEM who will be doing the testing and
the Reseller (who may also be an OEM) who will be selling the hardware at
retail.
Any system tested in this manner MUST be readily identifiable as being the
Reseller’s system. See System Identification for more information
Public certificates will include Make/Model, release and pertinent hardware
information including certification/supported status of any Vendor Approved
Options applicable to the certified Make/Model.
Public certificates will not include any Pass/Fail test information, private
system data or other details that are not meant to be publicly accessible.
Access to C3 is available only to Canonical employees and designated employees
of partners participating in the Programme.
The private site will provide the Partner with a history of all certified and
registered models and a history of all submitted test results and all
certificates.
People with access must have an account on Launchpad (https://launchpad.net)
and their account must be added to the appropriate access group by the
Partner’s PE or a member of the Certification team.
