Afternoon engineering and hacking session for Validation and Infrastructure Working Groups

Afternoon engineering and hacking session for Validation and Infrastructure Working Groups

Afternoon engineering and hacking session for Validation and Infrastructure Working Groups

Supporting large scale testing of kernel trees and defconfig in the LAVA lab is a high priority engineering effort across all Linaro engineering and Landing Teams for Q4.10. This organization wide effort imposes a growing list of requirements that LAVA team has started to work on during Q3 and will focus on driving forward during the Q4.2011. This effort involves improvements to the lava dashboard to ensure that the build and test results data can be efficiently consumed by kernel engineers improvements to the lava dashboard to make it easy to follow every step of a submitted job from the scheduler to the job submissions step. increase of capacity for all LEB board types deployed in the lava-lab to a level suitable to continuously track 20+ kernel branches with multiple defconfigs each. improvements to lava job submissions tools that make it easy for services like the kernel CI service to hook link from their build page to the scheduler job and the results. The goal of this session is to discuss user stories around LAVA components for driving continuous integration testing and results tracking through LAVA. Session Notes: Currently the kernel is cross built in ec2 by jenkins -- build results are submitted to the dashboard and if successful a test is run in LAVA. http://validation.linaro.org/lava-server/kernel-ci-views/index attempts to convey a summary of the results. Current process of getting your own tree is manual - talk to Deepti to get your tree included. Wishlist: Want to able to use the set up topic trees for temporary testing. Maybe need to make it easier for KWG devs to set up trees for themselves We might want to use per-user configs so that users can select the trees/configs they care about Per-user reports and a web form so they can submit git tree, branch and config for a one-time or regular testing of their tree per-tree specification of tests to run Determine which tests to run with the new kernel: ACTION: mwhudson to submit mp to change this ltp hw enablement tests PM functional tests Store the kernel version (uname) so that we can cross-reference this Build results notification (by email, RSS feed) Can just do this in Jenkins for now Want to do this in LAVA RSS ok for now Weekly build-status digest email for the community? Deepak to do this manually for now John's builds don't appear on the tree view Waterfall view perhaps not appropriate for one-time builds Should use tags for grouping results (now that tags exists) deepti to change jenkins jobs to use tags Add tagging support to dispatcher so the results get tagged similarly View for test/build combination Easy way to see last successful build for a test/build Waterfall-ish view of test suite vs test/build history Also improve bundle view Anti-wishlist: Builds of out of tree configs Basically, developer should commit the Kconfig changes to the tree ACTION: start tagging build and test results (with LAVA tags) Figure out how our read-only tags look like (since tags cannot be changed afterwards). Mouse-over git-describe: show the shortlog

Tests of client/server and distributed systems via LAVA requires the ability to launch and coordinate actions on multiple target machines. We should discuss the ways people would like to use this, how to specify something like this sensibly in the test jobs, and the interaction with other LAVA components such as the dispatcher. Session Notes: use cases * client/server functionality * performance - nuttcp over network topologies * stress - use client systems to generate load on a server - requires feedback from both sides Testing systems that have shared resources like a san Changes proposed for lava * Target specification (for more than one target) * result reporting separated by target * inter-client concurrency, synchronization, client actions, data sharing and config defines client groups with a name for the group, and specific targets specified * what if want a number of devices of a certain type rather than individual clients? * scheduler would need to somehow make sure *all* systems in a client group for a job are available before running ^^^ Or, alternatively, provide a way to request and script jobs for classes/types of machines rather than specific machines * make actions something that can be installed from out-of-tree source * extend results to allow hw/sw context for additional machines? * Which existing actions, such as deploy, would need to be modified to work on groups of machines? Steps to implementing this: Defining multiple targets * specified or by type * scheduler allocation Dispatcher handling of multi client/context * does this work with new client/connection split in trunk * synchronization action Test actions * pluggable actions * defining generic ones? * API for defining test results? Results aggregation * would see all the results pre-submit, and be able to modify or add to them

Right now, we're just using a single server, and deploying packages that line up with our monthly releases. As LAVA is quickly growing and becoming more and more important in Linaro, we should explore how to make it more scalable and how we can support a more agile cycle of development, testing, and deployment of lava components. Session Notes: Current performance is "decent" * LMC uses flock to serialize runs - ACTION: investigate doing this on a ramdisk - Offloading this to another machine that is not handling other dispatcher activities would be better - offload jenkins too - main host should just be for interactive things Celery needed to help us spawn workers - maybe can Using the cloud - some worker tasks with low transfer requirements could be used right away if we have anything like that - eventually, deploying web server nodes, database, etc to cloud would be possible, and keeping dispatcher local - use juju for easy deployment Measurements - collectd is running, but not terribly useful - database transactions/min would be useful - google analytics type hit rate counter - ACTION: investigate graphite and statsd - sentry monitoring for django apps - should we run in canonical datacenter for things other than the dispatcher? - deployment might be an issue there - submitting RTs for changes - could we experiment with running some secondary staging server there ACTION: Launchpad has a script that measures how long transactions run for. This should help us avoid making db transactions that take too long Postgres schema migrations are disruptive, distributed postgres might have issues with this many processes have to talk to the database (scheduler, dashboard,..) could they talk through the queue web server performance (responsiveness) database performance System load (with a notification) - investigate using nagios or new relic for this Memory/swap usage uwsgi has a feature to let you know and/or take action if a request takes longer than a given threshold ACTION: Make sure we're making use of this when we do the new deployment ACTION: Does postgres have a slow queries log? Caching * global enablement is not going to work * enable globally for anon users and dropping timeout to a few minutes would be better - measurements first * cache reports at api level * tests are cache aware because they will see the stale data - need to figure out how to turn off by testing * wall clock time is ok to check if this is improving * is there a way to measure cache hit/miss rate? - memcached could probably help measure this Sentry could be used to track lots of different kinds of errors, including deploy failures ACTION: Investigate sentry Zyga: look at getting caching enabled, celery, sentry Michael: statsd/graphite Dave/Paul: other monitoring things

Afternoon engineering and hacking session for Validation and Infrastructure Working Groups

Afternoon engineering and hacking session for Validation and Infrastructure Working Groups

Afternoon engineering and hacking session for Validation and Infrastructure Working Groups

Afternoon engineering and hacking session for Validation and Infrastructure Working Groups

Afternoon engineering and hacking session for Validation and Infrastructure Working Groups

Afternoon engineering and hacking session for Validation and Infrastructure Working Groups

What kinds of things could we add for enabling more tests and better testing? Some ideas are things like: * Jtag * wifi APs * audio connections * USB devices (would something like a usb stick in every board be useful? enough? too much? not enough?) * hdmi capture Session notes: Andy Green said that he doesn't want to commit regressions to public git did casual testing adhoc testing on his own, has tried to formalize when these things worked, which branch was tested, making an html table of these testable things as a text file sounds like a use for lava-qatracker as these are mostly manual tests Are there improvements we could make that would make it possible to automate these things? Tests we would like to be doing: audio capture can do this on the same board even hdmi capture, even video there is a card (black magic design) for roughly US$175 do these capture EDID? can you fake EDID? bootloader testing power management testing Anticipate to have first samples from Dave Anders in December for these devices a crappy ammeter will still be useful for catching regressions varous things involving JTAGs usb gadgets simulate attaching and detaching keyboards and mice? want to test both modes of OTG port on panda, ethernet is a host usb asset, and this is true of many other boards as well, so it's more important to test the musb WIFI AP connect/disconnect/traffic zyga concerned that it might be flakey Andy said that in his experience it's not so bad with wifi/bluetooth Bluetooth? Have something it can pair with and do something bluetooth access point and do obex push may be limited to 7-21 devices or so per AP device that converts serial to keyboard input ~$50 Add back scheduler ability to support tags on devices so that we can describe the capabilities needed Can we provide a framework for allowing others to do some of the work? Can we coordinate with Canonical's HW testing teams? Andy's opinion of priorities here: hdmi - this seems to break frequently audio wifi to a much lesser degree, because his patches don't risk breaking this so much ACTION: Talk to ricardo, deepak about the testsuites they are working on and their sense of priorities also 199$ HDMI + analog + audio capture card with linux support: http://www.blackmagic-design.com/products/intensity/ bootloader testing - implement soft solution now, evaluate HW solutions to see if we can do this universally

The Toolchain WG currently runs an extensive test suite hosted in Michael Hope's personal laboratory. The LAVA team will work with the TCWG team to ensure this setup is migrated, as-is, into the LAVA lab. We will also work to ensure that all the resulting toolchain build and test data gets properly submitted to LAVA, the dashboard providing the facilities needed to allow the Toolchain WG to analyze this data. It is expected that the reporting facilities in the LAVA dashboard will be driven and partly implemented by the Toolchain WG itself using the available extension mechanisms. == Validation lab next steps == We now have the toolchain build system running on boards hosted by validation. What do we do next, such as:  * Start capturing test results using LAVA  * Start regular benchmarks  * Capture benchmarks results using LAVA  * Shift components to Jenkins or LAVA  * Shift the x86 builds first? Every time we have a merge request, we want to test and make sure there are no regressions - They are doing this today on seabright Many tests will fail, they are interested in the delta, not the total failures This runs a test of every commit to the branch 1 test on v7 1 test on v5 (but they don't have v5 hardware so they test on v7 x86, x64 1. create scripts for the test suites that run right now 2. create results parser that will let us get those into lava dashboard - or at least make sure the existing one works 3. poll launchpad, find the merge request we need to pull, create tarball, create job in lava with that tarball location as a parameter (template for this testing that gets filled in) (This could be part of the extension listed below) - For the merge request, we have multiple templates that kick off tests on the different hardware types needed (v7, v5, x86, x86_64) 4. create lava extension for visualizing results and comparing tests to the baseline (merge request should be able to figure out what was HEAD, and we can compare against that) ACTION: mockup needed for visualization ACTION: investigate using the ssh client to run this test in a chroot, they do the 32bit testing on a 64bit machine in a 32bit personality chroot ACTION: zygmunt to see what we can reuse from tarmac - email notification of jobs startig could be done with tarmac also - there should be 2 comments, when starting and when finishing this part and the job is about to be created - also send link to the scheduler job so they can cancel if they see something wrong Hardware considerations: USB Sticks and scheduler support for tagging those machines Do we need swap enabled on those machines? Benchmarking =========== Needs a customized rootfs that is consistant and stripped down - make sure there are no extraneous services running project for building, keeping attachements of the binaries that we built, etc They can submit a branch for pulling a branch they have, pulling it, and building it The resulting artifact can be used in the benchmarking run IMPORTANT: for private tests, we also need to have private scheduler jobs so that the raw output is not public also B we benchmark on a8, a9, and a15 eventually They can build it and point us at the toolchain, or we can build it as part of the run - if we bould it, it should be built and feed into this Next we want to run the benchmark with build parameters some of their benchmarks need to be private because of licensing - and results need to be kept private

Afternoon engineering and hacking session for Validation and Infrastructure Working Groups

Afternoon engineering and hacking session for Validation and Infrastructure Working Groups

Afternoon engineering and hacking session for Validation and Infrastructure Working Groups

In order to bring the Validation Dashboard user experience one step forward the LAVA team will focus on a nice, comprehensive interface for the main use cases. Those were identified to be: * Platform Image Testing (example questions provided above.) * Kernel Continuous Integration ("Has Linus' kernel regressed on any of our boards over the last day?") * An individual jobs' click-flow: making it easy to follow a job from entering the scheduler to the actual result submission. * The validation.linaro.org frontpage, which should give an interesting overview of the most important tests being run. During this session, we would like to review the existing pages in LAVA to discuss whether they are useful, or should be modified or eliminated, and look for information that is missing or difficult to get at, and discuss how to improve that. Session notes: A few issues in the current proposal with width, layout, etc should need just a few fixes before we can roll it out Merge as-is, there is an issue with one of the reports we have on the front page being too wide, but that's really a bug in the report. Do we need the date? the date takes up a lot of space on there ACTION: make sure we have a bug on that report - it needs to be fixed regardless of whether we choose to have it on the front page Side bar - do we even need it? Some pages don't have enough information to warrant having a side bar - this can be dropped for those pages current side bar used has some visual issues, but in general people agreed that having a collapsable side bar would be useful. Screen width on some small screens is really bad, so we can't assume we have a lot of horizontal space to work with ACTION: Consider other sidebar widgets, ensure that other parts of the UI don't get hidden by the side bar Flow of submitting a job to seeing results through the lifecycle * having a "me" page - Should this be a global me page, or per app? - list of scheduler jobs that this user has scheduled - notification options - notify me by email or RSS feed for all my jobs? - ability to subscribe to subscribe to other events "actions on objects" would be subscribable from the context of the thing you want to be subscribed to - email notification could be jammed, could cause hangs if it doesn't timeout. We should put them in a message queue for the user, and those get processed by a job that runs frequently. We can mitigate with the RSS feed notification. -- I'm not sure this is totally needed, I think this could be set up so it doesn't have to block on the email. The mailer daemon should process the queue and we shouldn't have to care about it Front page: should load quickly - actual content is not easy to pick something that matters to everyone, so we could put just about anything here This is also a marketing page for lava, for linaro ACTION: Involve marketing in what should go here - get them to review what we do and make sure they agree theme/branding For lava as a project, we might want a generic lava logo rather than linaro branded (low prio) Menus: We need a top level menu that stats constant, apps can introduce 2nd level menus if needed example: dashboard currently changes the menu result view (/bundles/sha1) On the view that shows a list of test runs in a bundle it currently shows: test run - this links to the results and is fine test name - this links to some kind of confusing list of historic runs uploaded on - ok, some sense of time is good analyzed on - this is confusing, some people have said that they think this means that someone triaged this test result and filed bugs on it, etc. that's not what it means at all What we need here: some kind of pass/fail/totals - we should be able to see this at a glance, then drill down for details ***If we can determine context, then could we show the current results across the top, and below show historical results in comparison as a sort of waterfall view Could we build the context based on testing efforts? ACTION: Kiko is working on a set of mockups in basalmics to recommend to us - waterfall views - time based reports Should we have bundle streams? Should we name them something else? As a matter of fact, we explain what's a bundle stream at each sessions. Kernel waterfall is too busy, too many bricks Kiko has an idea to arrange this by trees - instead of having checkboxes for trees, have links for each kernel tree For boards where we can't run the test, we should have a red unsupported or broken block for the lava-test seciton

Afternoon engineering and hacking session for Validation and Infrastructure Working Groups

Afternoon engineering and hacking session for Validation and Infrastructure Working Groups