Binary Regression Testing

The purpose of the Binary Regression Test (BRT) project is to be able to execute regression test suites focused on specific proprietary binaries. Successes and failures will be mapped to the specific versions of software packages tested. A subset of these packages will be monitored and the test will be executed whenever they have changes.

When failures do occur, information to be able to regenerate the system and regarding the cause of the failure will be captured. The source of the failure will need to be determined. This could be a problem in the application being tested, the test being run, or any of the underlying packages.

The major goal is to shorten the passage of time in the development cycle between when a code change happens and when it is noticed to have an adverse effect on the execution of the binary. Specifically, this means we will design a system that should make it easy to run regression tests on a pre-defined linux system which will be upgraded in the time frame of the open source projects and compile the results of those tests. The system information and any desired result information will also be saved, as defined by the test users. We want to make this information readily accessible for testers and others interested in the results.

This web site is very new and will be actively being updated with information as the project develops. Join the binary_SIG mailing list to get notification on project and web site updates. The project source will be temporarily in a CVS repository developer.osdl.org. Details to be posted as soon as it is put with the production repositories.

NEWS

Test::Sut in Sourceforge Subversion

March 31, 2006The Test::Sut repository, which merges STP and BRT client-side code, has been moved to subversion. Here is the checkout command: 'svn co https://svn.sourceforge.net/svnroot/stp stp'. The format of the XML input files has changed, but the functionality is the same.

Client Updates

February 24, 2006 We are beta-testing the client to run on 'whatever is installed' on the SUT and capture all pertinent distribution/system information. This is in response to several projects which wish to run tests that way. System upgrades on a per distribution basis may or may not follow, depending on interest. We add distribution support through the modules Linux::Distribution and Linux::Distribution::Packages.

Sys-SyslogMessages-0.02

February 22, 2006 Sys-SyslogMessages, version 0.02 is now available on CPAN. The new feature is that it allows the user to specify a time interval to tail.

New Module Sys-SyslogMessages-0.01

February 18, 2006 Sys-SyslogMessages, version 0.01 is now available on CPAN. This module helps to find the syslog on any distribution using syslogd or syslog-ng and capture it to a file. This is another tool which should help us to run tests on many distributions and still get all the information we want to capture.

Linux::Distribution::Packages: version 0.03

February 7, 2006 Linux::Distribution::Packages, version 0.03 is now available on CPAN. Slackware added thanks to Alberto Re.

Linux::Distribution::Packages: version 0.02

February 3, 2006 Linux::Distribution::Packages, version 0.02 is now available on CPAN. The new feature added allows the user to specify an output file name as an option to the distribution_write command.

Linux::Distribution::Packages: System Information Gathering Tool

January 20, 2006 - Building on module Linux::Distribution, I have released Linux::Distribution::Packages, version 0.01 on CPAN. This tool will gather the package information for various distributions and write it out in native, xml or csv. This is one more step to getting generic tools to gather OS information for running tests flexibly on remote systems.

System Information Gathering Tool Collaboration

January 17, 2006 - With an eye to making the test client run on any distibution and toward more modularization, I have done some work on an existing CPAN module from Alberto Re called Linux::Distribution. It was designed to guess what distro is running on your (test) system. I added Ubuntu and added the capability to get the version as well. I intend to build on this to collect more system information such as the installed packages. We can add any distros we want to test on and hopefully the cpan location will allow others to use it more easily.

Testing, RSS feeds, IRC-chat oh my!

December 13, 2005 - It's been a while since the last update. We've been working hard with the Beijing Software Testing and QA Center and they are on the verge of sending in their very first test results. We've learned a tremendous about in terms of usability of the framework from them and it is very much appreciated. We've also hooked up our site to an rss feed (a big Thank You to Bryce Harrington for the script). We've also started an irc chat room #open_testing so that we can discuss any testing issues. It is extremely useful when trying to connect with our Chinese collaborators.

Beijing Software Testing and QA Collaboration

November 3, 2005 - We are currently in the process of beginning to work with the Beijing Software Testing & QA Center. We're having some preliminary discussions to determine how they can utilize the BRT framework in their existing test environment. Our goal is to have them up and running tests and submitting results as quickly as possible.

Running Tests and Improving Framework

October 27, 2005 - We currently have a series of Xorg tests running and producing results. We're also continuing our LTP test runs. Our focus now is shifting towards delivering on a set of binary applications that can be tested in the BRT framework. There is still ongoing improvements being made to the framework and results page. Simplification and ease of use has been improved on the user side allowing test runs to be generated quicker and less painfully. However, there is still room for more improvements which are being tackled. Users will also notice results formatting and display undergoing some renovation in hopes to produce a more asthetically pleasing imformative display of results.

Testing Collaboration with SpikeSource

August 29, 2005 - We are trying to start a testing collaboration effort with SpikeSource. Right now we are trying to install and configure the SpikeSource Core Stack on our base installation. We hope this will result in the generation of a build set in which SpikeSource may run some of their tests on and report results.

Working on Development Environment and X

August 5, 2005 - We are setting up a development environment internally for STP, PLM and BRT. This is taking much of our attention. However, we are also getting a build set ready for x.org testing. They have a great test suite and we believe this will facilitate use of Enterprise Linux on the desk-top.

BRT in Production

July 15, 2005 - The Servers are up. We are running tests and the BRT Search Page is available. There is a new link on the sidebar to indicate what tests we have currently available. You may also run your own. Anyone can register, run a test and post results. Read the User Guide for info. The source is under Downloads-version 0.1.5-r2.