Certification Process¶
Most of the Certification process is defined in the Self-Test Guide. This document is meant to provide additional information and guidance on the Certification process.
Timeframe¶
Depending on the activity, the following should apply as far as time estimates:
Self-Testing reviews should be completed within 2 business weeks from initial submission to completion or publishing.
Onsite testing should be completed within 2 business weeks from initial testing to completion or publishing.
Remote testing should be completed within 2 business weeks from initial testing to completion or publishing.
Publishing of certificates is instantaneous as soon as the certificate is marked as passing.
Replies to inquiries should happen within 3 business days (this only applies to replies, it does not imply that a resolution to any inquiry will occur within that time).
Hardware enablement or bug fixing has no set timeframe due to the nature of those issues. Bugs will be resolved as quickly as we can; however, due to the variations in severity, complexity, and impact on other releases and systems, the actual time to fix and SRU a bug fix can vary from a few days to several weeks. Additionally, hardware enablement may require hardware to be present in our lab and may take several weeks to develop and then push into the kernel, MAAS, or wherever is appropriate.
Test Lab¶
The test lab should be as clean as possible and should have as simple a
network as possible. Network segments need to match the fastest supported
speed of any NIC on the SUT. (e.g. a 100 Gb NIC must be connected to a 100 Gb
LAN and the iperf target must also have a 100 Gb connection).
The lab will work best when there is unfettered Internet access for downloading test tools and dependency packages as well as MAAS images, cloud images, and so forth.
If Internet access is spotty or not permitted, local repository mirrors can be employed, but those require additional setup and maintenance.
If the Certification Team requires access, access should be provided via VPN or some similar means of ingress.
SUT BMCs should be connected and configured as described in the Self-Testing Guide. (See the Documentation section above)
Certification requires MAAS to be used to deploy all test systems.
Hardware¶
Hardware to be Certified should be GA level hardware, not development level hardware, SDV, BBVT, FVT or any other non-ready-for-production level. The hardware should be the same hardware that customers are able to purchase.
Firmware¶
Firmware should be GA or similar level. In all cases, firmware should be GA level, with the only exception being the need to use unsigned versions in order to maintain the ability to flash revisions up or down as needed.
Firmware should be available somewhere online and not a secret build that is only available internally to the Partner or Canonical. The only exception here is for initial release firmware that comes on a newly released system.
You do not need to recertify on updated versions of firmware. We will list the firmware in use at the time of certification and will support users from that firmware level onwards.
That is not to say we won’t recommend a firmware update where appropriate, but rather sets a baseline firmware level with the expectation that the Partner will continue to ensure that updates to firmware do not cause issues with running Ubuntu on already certified hardware.
System Identification¶
Data in firmware must contain valid and correct identifiers for the make/model being tested. Typically this information is contained in DMI Types 1, 2 and 3.
If the system is sold by a Partner ODM, then the DMI data must include the correct Make and Model for the SUT.
If the system is sold by a Partner OEM and intended for resale by a different brand or under a different mark, then DMI must include SOME sort of verifiable identifier that shows the SUT is, in fact, the model being tested. This distinction is allowed as in many cases, OEM systems may not have the Make/Model fields filled out.
In cases where one OEM/ODM is performing testing on behalf of another OEM/ODM who resells hardware from the primary OEM/ODM, that hardware MUST be identifiable in firmware as belonging to the reseller OEM/ODM.
Example: If Vendor A is testing hardware on behalf of Vendor B, the firmware must clearly show that the hardware is a model produced by Vendor B. Typically, this requires that DMI Type 1 (System) shows Vendor B as the Manufacturer and uses the Vendor B Marketing Model for the Product field. In these cases, DMI Types 2 and 3 can indicate Vendor A as the manufacturer.
Installation¶
Installation must be performed by Canonical’s MAAS (Metal-As-A-Service). MAAS must use the default Ubuntu images and kernels provided via https://maas.io for all Certification deployments.
If a System cannot be deployed via MAAS and it is determined that this is a lack of support or bug in MAAS, then we will work with the partner and the MAAS development team and Server Hardware Enablement to resolve issues that prevent successful deployments of the SUT.
Custom Kernels and Drivers¶
Custom kernels are not allowed for Certification. Certified hardware must work with the standard Ubuntu kernel for the SUT’s architecture. No unaccepted kernel patches will be allowed.
The standard Ubuntu Kernel includes the GA kernel or any released HWE kernel.
The exception to this involves kernel modules as outlined below.
Modules injected via DKMS generally are not allowed for certification.
Third Party / Proprietary Drivers¶
Hardware should be tested and certified using in-band drivers provided by the Ubuntu kernel. In cases where hardware is not supported by the current Ubuntu kernel, testing should then focus on the current HWE kernel.
Partners should work with their IHV upstreams to ensure necessary driver support is present in the Linux Kernel and will thus land naturally in the Ubuntu kernel.
Out-of-Band (Proprietary) drivers may be considered for use in Certification but that must be approved by the Canonical Hardware Certification Team.
For more information, please contact your Partner Engineer.
Storage Options¶
Certification testing should be performed for each storage mode supported. Thus if a system supports JBOD and onboard RAID plus an optional PCIe add-in RAID card (that controls onboard disks), the storage tests should be run against all three configurations.
The Server Test Suite provides a storage-only test plan for this purpose.
Storage Management Software¶
Any storage management software meant to manage, modify, monitor storage devices including internal and/or external arrays, JBODs, or other storage subsystems should be provided to Ubuntu users in a manner equal to the way it is presented to users of other operating systems. The preferred method is via a Debian package provided via download from the Partner’s website, or as a Snap package available via the Snap Store. The package should be presented equally with the same software packaged for other Operating Systems.
USB Testing¶
USB Testing requires at least one USB stick matching the fastest type of port (USB2, USB3). Thus if a SUT has both USB2 and USB3 ports, you will need at least one USB3 thumb drive plugged into the appropriate port prior to testing.
Network Testing¶
Network testing requires a second system to serve as a network target running
iperf3 in Server modes.
Network devices must be connected to clean networks of at least the maximum
supported speed for the device. Thus, a 100 Gb NIC must be connected to a
100 Gb LAN and the iperf3 target must also have a 100 Gb NIC connected to
the same LAN. A 1 Gb NIC may be connected to either any LAN segment that is 1Gb
or faster.
All onboard network devices and ports MUST be tested.
As described in the Self-Testing Guide a separate server is NOT required as the
MAAS server can also function as the iperf3 target provided the MAAS server
has appropriate network ports to match the speed of the fastest NIC to be
tested.
Bugs¶
It is not normal to encounter significant bugs during certification, or at least it should not be expected; however, bugs are found from time to time and must be addressed. In general, your Partner Engineer will work with your teams to shepherd bugs through our bug process with some level of priority given to bugs that block certification.
Bugs from Tier 1 Partners¶
Bugs from vendors that recognize Ubuntu as a Tier 1 OS will receive priority over other bugs. This does not imply any specific SLA or time frame, but we will work with engineering teams within Canonical to escalate bugs from Tier 1 vendors with the understanding that those engineering teams will have the right to decline or delay fixing a bug depending on business needs at the time.
Test Suite Bugs¶
Bugs found in the Suite will be treated with priority over feature additions or other work. We will work with the partner to resolve any bugs in the Suite in a timely manner, and will provide modified versions of the various files affected if necessary to speed a certification along. It is very important for the Partner and Tester to work directly with the Certification Team to resolve any bugs found in the Suite, including being available to re-run tests, commands, participate in debugging, replacing or patching the Suite, etc.
When such bugs are discovered, please reach out to your Partner Engineer for guidance.
Hardware Bugs¶
Bugs found in hardware or firmware are solely the responsibility of the partner to fix. The timeframe for doing so is entirely at the Partner’s discretion, and thus could cause a certification to be significantly delayed.
OS Bugs¶
Certification blocking bugs found in the OS may be filed by either the Partner or the PE or Certification team.
It is up to the PE and Partner to work together to get any bugs filed against the OS resolved in a timely manner. Note that OS bugs could result in certification being delayed until the OS SRU process is completed and any fix has been introduced to the OS via the Updates repository.
Additionally, membership in the Canonical OEM Partner Programme does not imply any sort of special handling of OS bugs. We will make a best effort to escalate bugs from our Partners, but Canonical’s engineering teams are under no obligation to accept that escalation outside of a special engineering engagement.
Regression Bugs¶
Bugs found in the OS during re-certification, or during regression testing, will be handled with higher priority than normal bugs. A bug found in a later package version will not jeopardize an existing certification. In other words, if package X is version 1.01 in 20.04 when a SUT is certified and package X is version 1.10 in 22.04 and causes a failure during regression testing, your original 20.04 certification will not be affected, and the regression introduced in version between version 1.01 and 1.10 of package X will be treated with higher priority as a regression in the OS.
Enablement Bugs¶
Bugs determined to require hardware enablement for a Model to pass Certification will need to be examined and discussed with the Partner Engineer and Canonical to determine the best course of action. This may require a paid Non-Recurring Engineering engagement with Canonical to perform the enablement work necessary.
Examples of Enablement NRE would include, but are not limited to, a new server management engine that requires use of a special API for hardware management or the inclusion of a new driver that is not supported by the current HWE kernel.
We will, at our discretion, attempt to do cherry picks of enablement patches from the Mainline kernel into the LTS GA kernel but do not guarantee that service. It is performed as a courtesy and is entirely dependent on the Partner Engineer’s workload and acceptance by the relevant engineering team within Canonical.
Submission of Results¶
Results should be submitted using the process outlined in the Self-Testing Guide.
Requesting Certificates¶
Certificates should only need to be requested one time per System per Release.
If re-tests are needed to satisfy testing requirements, do not create separate certificates.
Certificates are not necessary for subsequent point releases. If a system is certified already on 20.04.2, you do not need a new certificate for 20.04.3 and 20.04.4.
Certificates are necessary for each LTS family. If a system is certified for 20.04.3, it does need a new certificate for 22.04 LTS.
Verifying Results¶
The Model certified should be maintained in the Partner’s Lab for a period of up to 30 days (unless discussed with your Partner Engineer) and said lab should be made accessible via VPN or similar access to the Hardware Certification team, including root/admin level access to the MAAS and iperf servers and BMCs so that the Hardware Certification engineers can periodically validate the test results with spot checks.
This policy does not imply that validation will happen with every submission, but upon request. If the hardware cannot be held for a reasonable period of time, the Partner should make arrangements to re-acquire the hardware for a brief period as soon as possible, OR communicate the need for urgency to the Hardware Certification Team or Partner Engineer for advice.
Private Certificates¶
Private certifications are available but are only allowed on a case-by-case basis. Private certifications are generally used for pending tenders that the Partner wishes to participate in, or for certification on a pre-GA system with the expectation that such certification will be made public after the system GAs. If you believe you need a private certification, please contact your Partner Engineer for more information.
Zero-Day Certification¶
We encourage “Zero-Day Certification” for a new LTS release. This allows our Partners to advertise certified status on the latest LTS release of Ubuntu Server on the day it is released. In order to participate in Zero-Day Certification, the following applies:
SUTs must be tested within the testing window prior to LTS release, usually a 2- to 3-week period before release. Testing is conducted on the RC or last Beta of the LTS release.
SUTs must subsequently be re-tested for official certification using the GA/Release version of the new LTS within 60 days of Release. Thus, if Server A is certified Zero-Day, it must also be re-tested for official certification within 60 days following the LTS Release Day (GA +60).
SUTs that are not re-tested within the Cert Window will lose their certified status until such time as they are tested on the GA version of the LTS in question.
All other requirements must be met for Zero-Day Certification (e.g. MAAS for deployments, GA firmware, etc).
Re-Testing¶
Occasionally, re-testing is necessary to satisfy testing requirements, resolve bugs or other needs. The Certification Team will assist with guidance on what to re-run and how/when to do so.
Results from re-runs should be submitted to the same hardware entry as the original certification results. A private “Note” should be added to any existing certificate request that provides a link to the new retest results.
Do not request further certificates each time retest results are submitted to C3.
When performing retests, you MUST run the full requested test plan. We provide
targeted test plans and launchers to assist with this. For example, if you are
asked to re-run a storage test, you should run the test-storage command
which runs a small storage only subset of tests.
You should never modify a test plan. Modifying a test plan will result in the
submission being rejected by the certification team. The only exception to
this rule is for test launchers such as test-network which do allow you to
deselect undesired network devices to save time on the retest.
Regression Testing¶
The Certification Team performs regression testing on a pool of certified hardware on a regular cadence to ensure and improve the quality of Ubuntu SRU kernels and point releases. Partners should likewise include a regression testing component in their own testing programs.
The Certification Team runs regression testing on hardware contained in the Certification Lab.
Any regressions discovered do not affect existing certifications.
Re-Certification¶
Re-Certification is necessary in certain circumstances. Primarily, when a new LTS is released, certification from the previous LTS does not carry forward, thus any currently certified system that should be certified for the new LTS will need to be re-certified on the new LTS version.
Additionally, there are occasional changes that mandate re-certification. Anything that fundamentally alters a SUT’s electronic profile requires re-certification. This includes, but is not limited to:
CPU Family updates (e.g. system refreshes that change from Cascade Lake to Ice Lake to Sapphire Rapids)
Memory technology updates (e.g DDR4 to DDR5)
Changing an on-board device, on-board meaning soldered to the main or daughter board. Changing out a PCIe or Mezzanine device may require testing of the new device, but will not trigger a full recertification.
The following are examples of things that do not require re-certification (again, this list is not limited to the following):
CPU Speed bumps or core count increases (e.g. AMD Milan -> AMD Milan X)
Memory amount changes (e.g. 4 GiB to 16 GiB) unless that includes an increase in the number of memory slots physically on the board.
Whenever a question of re-certification comes up, the Certification Team will investigate the situation and make a decision on a case-by-case basis.
Physical Certificates¶
Typically, the entry on the Ubuntu Certification Website (https://ubuntu.com/certified) is considered the “Certificate”; however, on occasion where a PDF certificate is needed, such as for a tender or business case, we will create and provide an official PDF Certificate for your system.
To request a physical certificate, please contact your Partner Engineer.
Hardware for Regression Testing and Other Needs¶
For members of the full SOA programme, Canonical reserves the right to purchase at a negotiated discount, up to two (2) of each certified model of server, and a selection of Vendor Approved Options for purposes including, and not limited to, regression testing, bug investigation, test development and other needs.
If your company is a member of the Small IHV program and purchasing Certifications ad hoc, you are required to provide Loaner servers to our lab.
Please refer to your Certification Supplement and contact your Partner Engineer for any questions about hardware purchases or loaners.