Ubuntu logo

Developer Summit

Cloud Image Testing and Automation

2012-10-31 09:00..09:55 in B3-M8

Currently the Cloud Images are built and released based on upstream tests. The idea is that if the upstream tests work, that the images will just work. However, due to bugs hit this past cycle, and with the proliferation of new clouds, the need to validate that the cloud images work where they should and characterisiticly are what we say they are, is becoming apparent.

Also, with the proliferation of new cloud vendors, there is a need to automate the release process. Some in the community have asked for a more frequent release cadance for the cloud images.

As a result, for the R-cycle, we will implement testing and automation that will be aimed at ensuring enhanced quality of the images.

1) Implement deeper testing for the Ubuntu Cloud Images   2) Improve the quality of the Ubuntu Cloud Images by pro-actively looking for regressions and stress testing.   3) Implement Automatic releases to increase the release cadance for the Cloud Images

The following Tests will be implemented: * Image Characteristic Tests:   - Define attributes of a Cloud Image, i.e. prescense of certain files, file formats, and package sets. For example:         - precences of /etc/cloud and files under it         - check boot loader configruation for grub and grub2         - deeper cloud-init testing         - check SSH configuration

  • Bug regression tests: look over the Cloud Image bugs related to the bulding of the images. For example:     - check that /var/log/{btmp,wtmp,lastlog} exist     - In order for a cloud image bug to be "fixed released" a test should be written to prevent future regressions.

  • Image boot tests    - Test that QCow images boot on KVM and OpenStack    - Test that outputed files are valid    - Validate boot loader and console configurations

  • Image proactive bug searching in Cloud instances    - Fully excersize disk, I/O and network looking for potential bugs    - Increase Amazon instance sizes tested    - Excercise -proposed updates to check for issues which may cause problems    - Use of Juju for testing Juju/Cloud Integration    - Mock workload tests

  • Image performance testing    - Measure I/O performance to identify problems with kernels or userland configurations caused by updates or build process

  • Explore implementation of UTAH testing framework

  • Integration and publication of results to Jenkins Instance    - Add the daily images to iso.qa.ubuntu.com    - Automatically update test results to iso.qa.ubuntu.com

For Automation: * Integrate testing into image build process.    - Daily images for the development release must pass characteristic and regression tests.    - Daily images for the stable releases must pass all tests to be published    - Milestones for the development image are subjected to all tests suites    - Introduce automatic promotion of QA'd daily images when the daily has a new kernel or boot-criticial (i.e. reboot required to use) package.        - Automate email announcement generation - Implement Twitter Announcements - RSS Feeds

Feedback Requests (via the two major Ubuntu Cloud Lists) https://lists.ubuntu.com/archives/ubuntu-cloud/2012-October/000839.html https://groups.google.com/forum/?fromgroups=#!topic/ec2ubuntu/HLjoxOgOJ10