Overview of Ubuntu Automated QA and high level plans going forward.

This meeting is to brainstorm on the current release process used Oneiric, what worked and what didn't, and ways they can be improved for Precise.

Session for discussing progress on QA CoP topics and interests

To more efficiently summarize and get an overview of the release, its important for tools to know which team to associate a specific package with for reporting, triage, etc. Ad hoc spreadsheets are being used at the moment, but it would be better if we had an open and flexible way to associate this property with the project in launchpad.

Today, testing a package may be slow because of the waiting period that compensate the lack of rigorous test suites to regression-test changes, lack of hardware for QA’ing, we are mainly relying on community for testing, the amount of patches that get rolled can be important,... All of this can lead to a bug with a known fix going unaddressed in Ubuntu for several months. We need to shorten the delay of validation for SRUs and being able to provide adequate resolution on an ETA for a fix being included in Ubuntu. The goal of this session is to discuss and decide how to improve Standard SRU testing (everything but kernel), make it more efficient, increase the coverage of the QRT and build a test harness to run daily.

Builds: start tracking the quality of the builds, define what a broken build is and track how long it takes to engineering teams to fix the problems they introduce that impact testing. By doing this, the aim is to raise awareness of impact of untested submissions and to be able to determine how many times we are held by bad submissions. Add a dashboard to have a consolidated view of unit testing status. Collect initiatives from different engineering teams and either add it to Jenkins or consolidate it to make them easy to read and triage in case of failures.

The Ubuntu Engineering team in Canonical is committing a rotation of developers to work solely on development release maintenance for a month or two at a time, with the goal of keeping the development release buildable, installable, and upgradeable at all times in order to allow other developers to work with fewer interruptions. What should our priorities be? How should we organise ourselves? Given the alignment between this and e.g. the historical activities of MOTU, how can we build community participation or link into existing activities?

Discuss improvement for users experience, what can be done (default applications, patches, ideas ...) Initial draft of work items is available on https://wiki.ubuntu.com/Lubuntu/Developers/P-Plan#Spec_User_experience

We have many reports that help us keep track of automatically-detectable problems in the development release (FTBFS, NBS, component-mismatches, the conflict checker, the transition tracker, etc.). These are all well and good, but they are rather disconnected from each other and in many cases do not provide very good facilities for distributing work among developers. If we want to drive these reports consistently to zero, some time spent on infrastructure would be worthwhile. What can we do to improve matters?

Find an appropriate test case management tool and install it, find a way to integrate it with http://testcases.qa.ubuntu.com/. 1 cycle should be enough to install the tool and get the existing test cases that is worth keeping in there, we can also pilot the usage of it within QA before we move to the community using it.

This session is to gather feedback from stakeholders and community members regarding the custom bug listing feature.

This spec is to come up with ideas and methods for running automation testing on bare metal hardware (i.e. no VM). While it is geared mainly for armel platforms, there are enough x86 systems on the market that could also benefit from this. Testing should focus on server first, as it should be the easiest to implement across the widest software base.

Optimizations that could be done during Precise cycle Initial draft of work items is here : https://wiki.ubuntu.com/Lubuntu/Developers/P-Plan#Spec_Clean_house

Some features were rolled out during Oneiric to make the weekly release meetings more efficient for tracking features and bugs of interest to the release team, but more improvement is needed.     Features via Topics in: http://status.ubuntu.com/ubuntu-oneiric/     Bugs of interest to release team in: http://reports.qa.ubuntu.com/reports/kernel-bugs/reports/rls-mgr-o-tracking-bugs.html Would like to discuss elements of what is useful for each team to share, and how the round table can be made more effective for the participants.

Discuss and decide on what should be tested on kernel SRUs, considering what is already available and what is missing.

Defect Analysts have hundreds, if not thousands, of bug reports at which they could be looking. Their dashboard will be a collection of the highest priority bugs to review on a daily basis with links to reports of bugs matching specific criteria.

Adding the -lowlatency kernel into the universe repository will allow Ubuntu Studio users to have a tuned kernel for audio work installed by default. This kernel will be built and maintained by the Ubuntu Studio team, therefore it presumably would be in the universe repository. Responsibilities also include rebasing the kernel for security updates and working with -security to push updates.

Come and join us for a brainstorming workshop on the profile page in launchpad.

Review LTS release schedule, coordinate across teams, plan delivery across milestones.

The Papercuts project failed to reach its target of fixing 100 papercut bugs in both the Natty and the Oneiric cycles. The initiative needs a new life.

Today you can buy a computer preloaded with Ubuntu from many different vendors. The OS image used at the factory is often modified to provide better hardware support. It may also contain customizations such as additional bookmarks or applications. As a result of these customizations, the upgrade path to the next version of Ubuntu is not as clean as it could be.

Followup on https://blueprints.launchpad.net/launchpad/+spec/other-o-bug-lifecycle, reviewing results of developer survey, and discussing proposed changes to bug states and workflows.

QA Requirements for submitting automated tests into the QA lab - Test outputs - Resources - Jenkins

There is a good codebase of test cases already automated or suitable for automation in the QA Regression Testing project. With additional specifications created and implemented, the tests could be a valuable automated regression testing tool.

We need to identify areas where the Ubuntu documentation is lacking, improve our own documentation of how to get involved, and advertise to recruit contributors to help fill in the missing documentation.

Meeting of release team to do some brainstorming on things to be improved in next cycle.

This session is a demo and requirements review of the new tool Test Manager

Discuss ways to sync ebook user data (bookmarks, notes, reading position) between ebook readers using Ubuntu One.

Discuss what works and what doesn't in our processes for new developers

There has been some very interesting ideas on improving the release process brought up by Scott James Remnant in his blog post, http://netsplit.com/2011/09/08/new-ubuntu-release-process/ Some of these we are looking at implementing for Precise (in other blueprints), other points need discussions as future options. This session is targetted to brainstorming what would be the impact of some of the more radical changes we're not going to fit into precise, since it is an LTS. Also what planning/blueprinting work should be done to assess if the process changes will improve the developer and user experience before the next cycle.

This session is to clarify what is a test case and what everyone means by saying test case. We will also be discussing what constitutes a good test case.

The process of mentoring new Bug Squad members will be reviewed as will feedback from people being mentored. A redesign of the mentorship program will be discussed.

Launchpad can currently target bugs either at the distro (or sourcepackage) without any series associated, or associate it to a particular release. This model has several drawbacks as it causes confusion (bug 314432) as well as performance problems. The Launchpad team is investigating the possibility of associating all bugs with a series. This session is to discuss the user impact of such a change, and make sure it's compatible with the distribution workflows.

To more efficiently summarize and get an overview of the release, its important for tools to know which team to associate a specific package with for reporting, triage, etc. Ad hoc spreadsheets are being used at the moment, but it would be better if we had an open and flexible way to associate this property with the project in launchpad.

Following on discussions the past few months on ubuntu-devel, review the current workflow for accepting new packages into Ubuntu, with an eye to refinements or improvements we could make this cycle. Includes submission to Debian, Backports, REVU (and potential for switching to DebExpo), and ARB.

Open forum and discussion on how words and language are used in Launchpad, in the form of titles, labels, status and guides.

What can we do to drive MOTU this cycle? What sprints should we aim for?

A first set of metrics are to be put in place during Precise. This session is to discuss about code coverage and test escape analysis.

Followup session for the ARB, to plan work items. See the etherpad page for the previous session: http://summit.ubuntu.com/uds-p/meeting/19549/community-p-app-review-board/

Come and talk to us about any usability issues you have, ideas for changes and find out how the Launchpad team looks at usability and how we make changes.

What metrics should the defect analysts be gathering for their engineering teams? What visualizations of this data would be useful and what tool(s) should be used to create the visualizations?