Drupal Quality Assurance - Planet Drupal

Relaunch of the testbot on qa.drupal.org

admin — Fri, 13 Nov 2009 00:28:41 +0000

The testbot is now running pifr 2.x! The system has many new features and has been completely redesigned with the future of Drupal quality assurance in mind. More information on the new features and the roadmap for qa.drupal.org to come.

Please bear with us as we get the new system functioning on drupal.org and qa.drupal.org.

Testing server setup package

Sat, 28 Feb 2009 20:45:57 +0000

This package provides a command line tool to assist in the setup of a server for the automated testing framework located at testing.drupal.org. The package does all of the following:

Checks all server software dependencies.
Installs the Drupal codebase and supporting testing modules.
Configures PHP and an Apache virtual host for the website
Installs all necessary databases and configures access permissions for Drupal.
Installs a pre-built version of Drupal.
Creates user accounts for all testing administrators with SSH pubkey access.
Configures administrator permissions on the server (access to all Drupal-related files, ability to restart Apache/MySQL, etc).

The script checks for any updates to the server install package each time it's run, and checks if it's previously performed any work, only performing those actions which were not previously complete. This allows the script to be run over and over again for things like managing the list of current testing administrators.

Written by Chad Phillips

Test result quality

admin — Tue, 24 Feb 2009 04:04:24 +0000

The testing bot has been active for almost four months and during that time I have received a number of concerns from people about the quality of the results. Some of the concerns were valid and discovered issues, others were simply oversights or misunderstandings. I would like to take the time to describe the areas that cause invalid results and what to do if you feel the results on your test are invalid.

I would like to make it clear that the automated testing framework itself has had only one bug that has effected the results. The bug has been fixed since December and the testing framework has seen no other bug reports that effect the results. That is not to say that there have not been bugs in the framework, but that the results have not been incorrect due to the framework.

So if the framework is not the cause of invalid test results what is? Well, there is no simple answer to that. The following is a list of the areas that have been discovered to be sources of problems.

Insufficient memory due to increasing requirements of the tests.
Virtual machine resource problems.
Virtual machine network issues.
Drupal core tests being broken, failed test results show up on all tests.
Various server configuration issues.
Version specific PHP bugs.
Version specific cURL bugs.
Low-level OS library bugs.
Poorly written tests that use random values that collide on occasion.
Tests that reference settings or files that may not always be in the same location.

The server configuration issues have been the largest cause of problems. Items ranging from PHP short tags being enabled to some distributions having a separate PHP configuration for command line scripts that was overlooked during setup. Some testing servers are toggled on and off due to weird VM problems that seem to come and go. Others seems to remain consistent throughout.

Thankfully Chad "hunmonk" Philips has create a script that simplifies the process of setting up a test server and ensuring that a proper testing environment is setup. In addition I have been developing the next generation of the automated testing framework that will automatically remove servers that do not pass automated inspections.

In the meantime, if you run into results that you do not believe are valid please do not jump to conclusions.

Make sure that you are running the latest Drupal 7 core code and tests.
Some test have been know to fail randomly due to the way in which they were written. Please take a look at the tests that are failing.
Get someone else to confirm that the tests pass for them. Sometimes code is left in tests that does not work from machine to machine.

If you confirm that the result is still not making sense please report it to the Project Issue File Review queue.

Overall, I trust the results that the testing bot provides due to the stability of the automated testing framework and I hope you will do the same.

Bot re-enabled after drupal.org upgrade to D6

admin — Fri, 20 Feb 2009 01:43:39 +0000

The bot is now active again. There were a few issues with the newly upgraded drupal.org, but Chad "hunmonk" Phillips and myself "boombatower" where able to get them resolved.

Thanks to everyone involved in the D6 upgrade.

Bot caught up and system performing well

admin — Thu, 22 Jan 2009 07:28:25 +0000

It has been a while since the last update so I figured it is time for one. I lost my login credentials for testing.drupal.org during my graphics card failure and thus was not able to make an updates until a few days ago.

After a lengthy catch up period the bot is now caught up with the issue queue and has some impressive statistics to show for it.

The event log contains over 40,000 logged events.
The test suite has been run over 11,800 times.
Over 3,500 patches has been processed by the system at least once (most is 26).
The Drupal core SimpleTest suite of tests now contains over 9,000 assertions. (not bot related)

Work has begun on the next generation of the testing framework that will be easier to manage, self correcting, and offer more features. Along with my blog posts regarding the development I have created more specific issues for PIFR 2.0 and PIFT 2.0.

Development is currently postponed until Project has been ported to 6.x and drupal.org port to 6.x is secured. Once completed work will focus on PIFT 6.x upgrade and then 2.x development.

Any donations to the development of the automated testing system are much appreciated and let us know that you value the increased quality assurance that the framework brings to the Drupal community.

Theme needed

admin — Sat, 13 Dec 2008 08:18:46 +0000

If anyone is interested in developing, or knows of an existing one, a "bot like" theme for testing.drupal.org please make a comment.

The bot needs a bit of pizazz. (Drupal 6.x)

Two years 3 months 17 days in the making

Sun, 26 Oct 2008 01:20:43 +0000

We know you are excited, but let us get this working before you ask for any more features.

http://groups.drupal.org/node/976 - Sat, 2006-07-08 05:41