Drupal Quality Assurance

There was a conflict between two committed patches on Friday, June 5:

Typo in node_access_rebuild() -- db_insert('node_accesss')->db_insert('node_access')
Remove the 'post settings' admin screen and relocate contents

One patch added a test, the other changed a button name. Hilarity ensued.

The conflict was been cleaned up within an hour and testing bot turned back on.

The drupal.org client key changed and needed updating on testing.drupal.org. This caused all communication between the two servers to stop and thus created a backlog. The key has been updated and things should be back to normal.

A patch at Installer: Configure site text cleanup was committed which broke HEAD for a few hours tonight. The fix has been committed, and testing bot turned back on. If your patch got a "Failed: Failed to install HEAD." status assigned to it on Sunday evening, try marking the issue back to "needs review" so testing bot can have another crack at it.

UPDATE: a patch was committed to Drupal HEAD which resolved this issue. Testing has been enabled again.

The testing bot is currently disabled due to apparent false negatives in some path module tests.

Test slave #16 was recently pulled off the master server list, but it was done so in a way that caused data corruption. I attempted to correct the problem, but I made a mistake and reset 2,132 tests (trigger happy). Please be aware of this if you see oddities in the activity log or test results.

UPDATE: I was able to restore the results through a few interesting mysql queries. :)

Until http://drupal.org/node/445950 is resolved.

edit: now fixed (partial commit)

This package provides a command line tool to assist in the setup of a server for the automated testing framework located at testing.drupal.org. The package does all of the following:

• Checks all server software dependencies.
• Installs the Drupal codebase and supporting testing modules.
• Configures PHP and an Apache virtual host for the website
• Installs all necessary databases and configures access permissions for Drupal.
• Installs a pre-built version of Drupal.
• Creates user accounts for all testing administrators with SSH pubkey access.
• Configures administrator permissions on the server (access to all Drupal-related files, ability to restart Apache/MySQL, etc).

The script checks for any updates to the server install package each time it's run, and checks if it's previously performed any work, only performing those actions which were not previously complete. This allows the script to be run over and over again for things like managing the list of current testing administrators.

Written by Chad Phillips

The bot is currently disabled due to locale test failures. Form more information please see the following issues.

• locale test is failing
• Review locale module's simple tests

Edit by webchick 2009-Feb-24 @ 12:00pm EST: Locale tests fixed, testing bot back on!

The testing bot has been active for almost four months and during that time I have received a number of concerns from people about the quality of the results. Some of the concerns were valid and discovered issues, others were simply oversights or misunderstandings. I would like to take the time to describe the areas that cause invalid results and what to do if you feel the results on your test are invalid.

I would like to make it clear that the automated testing framework itself has had only one bug that has effected the results. The bug has been fixed since December and the testing framework has seen no other bug reports that effect the results. That is not to say that there have not been bugs in the framework, but that the results have not been incorrect due to the framework.

So if the framework is not the cause of invalid test results what is? Well, there is no simple answer to that. The following is a list of the areas that have been discovered to be sources of problems.

• Insufficient memory due to increasing requirements of the tests.
• Virtual machine resource problems.
• Virtual machine network issues.
• Drupal core tests being broken, failed test results show up on all tests.
• Various server configuration issues.
• Version specific PHP bugs.
• Version specific cURL bugs.
• Low-level OS library bugs.
• Poorly written tests that use random values that collide on occasion.
• Tests that reference settings or files that may not always be in the same location.

The server configuration issues have been the largest cause of problems. Items ranging from PHP short tags being enabled to some distributions having a separate PHP configuration for command line scripts that was overlooked during setup. Some testing servers are toggled on and off due to weird VM problems that seem to come and go. Others seems to remain consistent throughout.

Thankfully Chad "hunmonk" Philips has create a script that simplifies the process of setting up a test server and ensuring that a proper testing environment is setup. In addition I have been developing the next generation of the automated testing framework that will automatically remove servers that do not pass automated inspections.

In the meantime, if you run into results that you do not believe are valid please do not jump to conclusions.

• Make sure that you are running the latest Drupal 7 core code and tests.
• Some test have been know to fail randomly due to the way in which they were written. Please take a look at the tests that are failing.
• Get someone else to confirm that the tests pass for them. Sometimes code is left in tests that does not work from machine to machine.

If you confirm that the result is still not making sense please report it to the Project Issue File Review queue.

Overall, I trust the results that the testing bot provides due to the stability of the automated testing framework and I hope you will do the same.

The bot is now active again. There were a few issues with the newly upgraded drupal.org, but Chad "hunmonk" Phillips and myself "boombatower" where able to get them resolved.

Thanks to everyone involved in the D6 upgrade.