Since long time ago I was inspired of the features of LAVA (Linaro Automated Validation). Lava was developed by Linaro to do automatic test on real hardware. It's written in python and based on a lot small daemons and one django application. It's scheduling submitted tests on hardware depending on the rights and availability. Setting up an own instance isn't so hard, there is an video howto. But Lava is changing it's basic device model to pipeline devices to make it more flexible because the old device model was quite limited. Our instance is available under https://lava.coreboot.org. Atm. there is only one device (x60) and we're looking for help to add more devices.
coreboot is under heavy development around 200 commits a month. Sometime breaks, most time because somebody refactored code and made it simpler. There are many devices supported by coreboot, but commits aren't tested on every hardware. Which means it broke on some hardware. And here the bisect loop begins.
Lava is the perfect place to do bisecting. You can submit a Testjob via commandline, track Job and wait until it's done. Lava itself takes cares that a job doesn't take to long. To break down the task into smaller peaces:
- checkout a revision
- compile coreboot
- copy artefact somewhere where Lava can access it (http-server)
- submit a lava testjob
- lava deploys your image and do some tests
git-bisect does the binary search for the broken revision, checks out the next commit which needs to be tested. But somebody have to tell git-bisect if this is a good or bad revision. Or you use git bisect run. git bisect run a small script and uses the return code to know if this is a good or bad revision. There is also a third command skip, to skip the revision if the compilation fails. git-bisect would do the full bisect job, but to use lava, it needs a Lava Test Job. Under https://github.com/lynxis/coreboot-lava-bisect is my x60 bisect script together with a Lava Test Job for the x60. It only checks if coreboot is booting. But you might want to test something else. Is the cdrom is showing up? Is the wifi card properly detected? Checkout the lava documentation for more information about how to write a Lava Testjob or a Lava Test.
To communicate with Lava on the shell you need to have lava-tool running on your workstation. See https://validation.linaro.org/static/docs/overview.html
With lava-tool submit-job $URL job.yml you can submit a job and get the JobId. And check the status of your job with lava-tool job-status $URL $JOBID. Depending on the job-status the script must set the exit code. My bisect script for coreboot is https://github.com/lynxis/coreboot-lava-bisect
cd coreboot # CPU make -j$CPU export CPU=4 # your login user name for the lava.coreboot.org # you can also use LAVAURL="https://$LAVAUSER@lava.coreboot.fe80.eu/RPC2" export LAVAUSER=lynxis # used by lava to download the coreboot.rom export COREBOOTURL=https://fe80.eu/bisect/coreboot.rom # used as a target by *scp* export COREBOOT_SCP_URL=lynxis@fe80.eu:/var/www/bisect/coreboot.rom git bisect start git bisect bad <REV> git bisect good <REV> git bisect run /path/to/this/dir/bisect.sh