The Hardware Certification team routinely performs certification testing of Kernel SRUs for recent releases. The objective is to run a small test suite on every system that has been certified with stock Ubuntu and ensure no regressions have occurred. This compliments the testing done by QA through the sheer range of hardware components and drivers exercised in the testing. The goal is to protect the Ubuntu user with a certified system from suddenly finding that something important doesn't work. Currently the test suite consists of just over 20 tests as defined at https://wiki.ubuntu.com/Kernel/kernel-sru-workflow/CertificationTestSuite. These tests are unlikely to fail unless there has been a major regression and due to the automated nature of the testing (which is essential due to the volume of systems which need to be covered in a short period of time) there are no manual tests. So far this has meant that even though there have been regressions encountered in SRU updates, the certification testing process has not picked them up. Some have been bugs that could only be discovered through manual testing and some have been fairly obscure corner cases that the test suite simply didn't address. The goal of this blueprint is to significantly increase the coverage of the SRU test suite within a few limits, the first being that all tests must be automated and the second being that there needs to be a sensible limit on the execution time of the test suite (to be discussed, but a guideline would be 30 minutes to 1 hour) so that SRUs can still be released in a timely manner. Each test should have as high a value as possible, so must focus on the most important hardware functionality first. Wireless testing is non-existent and testing of graphics and audio is not as good as it should be for such important pieces of functionality, so we intend to increase coverage in these areas at least. When considering new tests to add we will be looking to keep the SRU test suite synchronized with the test suite used to certify systems. This means that in the first instance new SRU tests should be sourced from the certification test suite and if a suitable test is not found then any new test created should also be included in the certification test suite. Initial thoughts on what needs to be tested are here: https://docs.google.com/document/d/1fHfOnnnVCXsSayz3XlaL9hgrl-uTBXHx7Ijbkc6aKTo/edit?hl=en_US == Agenda == * Introduction and overview * Wireless testing * Video testing * Audio testing * Q&A/Comments

Every cloud has an Aubergine lining... Cloud computing is becoming increasingly pervasive and important in the world today. Cloud computing is used in datacenters all over for resource utilization, hosting, storage, providing media content, on-demand computing power and many other applications. Competition is growing at a rapid pace in this new infrastructure race with public cloud space available from Google, Amazon, Canonical, Apple and others. For this and other reasons, we need to be sure that Ubuntu functions well in the cloud as both a host and guest OS. From a hardware certification point of view, we need to ensure that Ubuntu can function properly as a Virtual Machine on popular virtualization products: VMWare, XenServer and KVM. This means that we need to come up with a plan for extending infrastructure, tools and processes to allow for dedicated host machines for each of the respective virtualization products. However, we also need to keep this framed within the boundaries of Hardware Certification testing, not Software QA testing which is currently handled very well by the Ubuntu Server team and the Platform QA Team. To get this done, several things need to occur. First, we need to beef up the server testing, this means adding new tests to the server test suite if necessary. Infrastructure needs to be improved to provide the means to do multi-network device testing. This will most likely mean some significant expense in purchasing hardware and will also rely on assistance from IS in getting that infrastructure in place where applicable. Additionally, possible test development time and debugging time. We need dedicated hardware to serve as XenServer hosts (VMWare already exists, but we may wish to update to a newer version). We need automated means for doing SRU testing on VMs and the means to certify Ubuntu as a Virtual Machine on both VMWare (mostly existing already) and XenServer. Finally, we should start also testing KVM. This could mean a third dedicated server to run the current LTS and host KVM vms for cert purposes.

Enable jack detection features in the "Sound Settings" page of GNOME Volume Control, and redesign the UI to be more user friendly (present "Port"/"Connector" as primary object, merge hardware tab into Input/Output tabs).

Review of the current Ubuntu Kernel delta. This session will look at the current delta comprised of both patches to the core and the ubuntu specific drivers. The aim is to record what we are carrying, review the reasons for that component(s) to be carried, and recommend replacements, updates, cleanups, upstreaming etc of those components.

Review of the kernel configuration for P. This will concentrate on confirming the policy for various option types, as well as new options. For major new options, discussion and confirmation of the selection of each.

SRU testing is about validating that package updates don't introduce problems on stable releases. The testing includes regression testing performed by the QA team, security testing by the Security team, certification testing by the Certification team and verification by the Kernel team. There is also some testing performed by the Ubuntu community coordinated by the QA team, but this is mostly achieved on a best effort basis. The purpose of this blueprint is to increase community participation and include their successful testing rather than just bug reports. The current process for getting community participation is by commenting on the bugs fixed by the SRU when the package is building for the proposed archive. The comment describes how to enable this archive where the package will be ready in a few hours and then asks the subscribers to please test with feedback. Another process specific for kernel packages is for development releases to be announced on voices.canonical.com and proposed kernels may also be announced eventually. As can be seen from the pending SRU reports, this has not resulted in much participation and the responsibility then falls upon the QA team. The first objective is to increase community participation for proposed kernels. The most significant value is to increase the probability that verification testing is performed by someone in the community within a reasonable delay. As a side effect, another value is to relieve the QA team from some testing which can otherwise result in additional delays. The SRU release team would benefit from having more people testing with minimal delays to push a new kernel from proposed to updates. The second objective is to include successful testing from the community rather than just bug reports. This will enable the SRU release team to also consider the number of people having tested the proposed kernels in addition to their existing reports. This should result in greater confidence before pushing a new kernel from proposed to updates. This is the story to achieve both these objectives: 1. A kernel is added to the proposed archive, so the user is notified that a new kernel is available; 2. The user opens Update Manager where she is informed that the kernel can be tested after rebooting; 3. After booting for the first time with a proposed kernel, Checkbox prompts the user to run the SRU suite; 4. If a test fails, Apport is invoked and tags the bug with "regression-proposed" for the current reports; 5. After testing, all test results are submitted to Launchpad where additional reports can be generated.

As usual the Canonical hardware certification team will be hosting a session at UDS to plan and gather feedback about the general run of the certification programme for Ubuntu. For 12.04 LTS these are the goals of the improvements of the hardware certification team: == Blocked systems follow-up == When a system fails certification for a particular system the hardware certification team is doing a good job opening bugs, helping developers to debug the error and testing candidate fixes and packages. This is a great start, but for 12.04 LTS we need to improve the way we follow up those blocked system. How do we know when a system has been unblocked? When do we recertify? We need a proper policy for this, plus the needed integration between the certification site and Launchpad to be able to do this in an automated fashion. Also, we need to have good reports to help us following up for bugs waiting verification on -proposed. == Improve the efficiency of regular testing == Right now, the hardware certification team performs two types of regular testing: weekly testing on the development release and SRU testing on the kernels in -proposed. The weekly testing is currently run against all the systems scheduled to be certified in the next Ubuntu release and the SRU against all systems certified for the Ubuntu release that the SRU is being released for. We want to improve the efficiency of this type of regular testing by reducing the number of systems tested. The idea is to maintain a list of the minimum set of systems required for each of the testing. == Functional and stress testing coverage == Every 6 months the hardware certification team revisits the list of components that are covered in the certification programme and the list of tests that are part of the certification site. We will revisit the list and come up with a list of things that we will need to include (or remove) to the list for 12.04 LTS certification. Some ideas:  * USB 3.0  * 30 cycles suspend and hibernate  * Multitouch  * Battery  * Wifi slider (FCC requirement?) * BIOS modes: uEFI vs Compatibility vs Legacy

The Kernel SRU cadence continues to march along. It has been tweaked a bit now and then as needed. It is time once again to see if it needs further refinement or is it the thing of beauty we believe it to be.

P-cycle discussion of the master git tree maintenance

Ubuntu certification is entirely performed by Canonical engineers, who receive systems to be certified and test them on a Canonical facility. It would be desirable to allow partners to perform a pre-test themselves on-site, so they can have a clear idea of whether a system is likely to be certified, and take any action they deem necessary. This eases the logistics and increases the amount of systems that can be tested by allowing partners to anticipate work that needs to be done. However, since a certification program implies a strong validation component, the process needs to be improved and formalized to better respond to the special requirements of this new arrangement. A third-party testing environment introduces the following challenges (note possible solutions in parentheses): - non-Canonical engineers won't be as familiar with the Ubuntu certification suite (make the tests less prone to failure and clearer in their instructions, have clear, unambiguous documentation on how to run tests and what to do when they fail, improve the tests and redesign the testing tool's user interface to make the process friendlier and more "foolproof"). - A certain motivation for "cheating" appears (automate as much as possible, so as to reduce reliance on a human's judgement at test runtime. do not show the results during the testing process. Instead, results are sent to Canonical for assessment.). - Partner testing labs may not have internet connectivity (ensure that the tools can reliably collect all needed data offline, and submit on a connected system). Particular suggestions include: - A complete redesign of the Checkbox UI to address usability concerns that have been raised both in the past and as part of the large amount of feedback gathered from Ubuntu Friendly. This is particularly important as the tool gets wider exposure and is compared to other, more polished applications present in Ubuntu. - Splitting the tests into categories:   1 - Tests that can be executed and verified automatically (no human interaction required - testing CPU frequency governors)   2 - Tests where the system can perform the required action and only confirmation is required from a human. (Changing display resolution)   3 - Tests where a human has to perform an action and then the system can test and decide whether the test passes (insert USB stick and automatically test read/write behavior, wiggling the mouse and checking that things work as expected) - For category 2, collecting the information for later review, possibly by Canonical engineers, so the partners will mainly collect information and won't necessarily receive feedback as to which tests passed and which failed. - An "academic exam" approach, where, as mentioned, the testing tools just collect raw data and send to Canonical for analysis. To avoid all possible forms of cheating, some form of video surveillance has been suggested. - Signing and/or encryption of test data, to identify and/or avoid tampering. This needs more thought as, being free software, the source code for the tools is widely available. - Looking at the way Checkbox is used currently and adding functionality that's needed (keeping old submissions, something that at the moment is done ad hoc, but whose frequent need indicates it should be doable through the UI). - Leveraging information gathered from the Ubuntu Friendly program. Since this will have widespread use, a lot can be learned about which tests are important to people, how people are running them, how to make them easier or automate them, and how to redesign the user interface for the testing tools to make the process as streamlined as possible. Ubuntu Friendly constitutes an external user running the test suite without prior knowledge and no help, so the usability and automation improvements that will derive from that program will prove valuable to enhance the partner self-performed pretesting experience. Partners will be benefited by faster certification times, since they can get faster local feedback on how their systems perform prior to sending them in for actual certification. Canonical hardware certification engineers will profit from a more-automated set of tests and tools, enabling them to certify more systems. End-users will benefit from a wider selection of Ubuntu-certified systems, with the confidence that results are trustworthy due to the verification processes in place to prevent faulty systems from gaining certified status. All three will also benefit from the widespread on-the-field experience, knowledge, user interface redesign and test improvements stemming from the Ubuntu Friendly program and results.

Currently all certification testing of kernel SRUs is performed in the Certification labs on systems connected to the labs 'satellite' servers. This allows systems to be configured to netboot and then be setup to perform SRU testing via preseed commands. This is a very efficient and flexible configuration for the Hardware Certification team to perform its duties and allows well over a hundred systems to be tested in the space of a few days. The major drawback with this method is that because the infrastructure is so complicated (not just software but servers and network equipment are required) it is next to impossible for just anyone to perform SRU testing in the same way that the Hardware Certification team does. There is also the sometimes encountered problem of systems that do not like to PxE boot (an essential part of the current process) thus stopping them from being SRU tested, even by the Hardware Certification team. The solution to these problems and the goal of this blueprint, is to have built automatically a CD image which contains all of the elements necessary to test each SRU for each release. This will obviously include the -proposed packages, but also the tools and settings normally used by the Hardware Certification team to do the testing. Creation of this image will have several benefits, the most obvious ones being that other parties can perform the testing as and when they wish and that systems which cannot PxE boot can be tested instead using the custom image. The other benefit is that the update phase of testing doesn't need to be done for each machine on demand as it is at the moment, but can be done just once when the SRU becomes available. The Offspring (https://launchpad.net/offspring) build system will be investigated for use as the tool to create these images and control access to them.

Discussions on the likely mainline kernel version and appropriate kernel flavors for P.

Discussion on Live CD/USB integration with Secure Boot. Will we need to create signed images? How will we maintain user-modifiability? Can we avoiding signed images?

Current status of UEFI support in Ubuntu. Discussion of pre-UDS testing, discussion of new areas we'd like to support, and bugs that need fixing.