Drupal Quality Assurance

The testing bot has been active for almost four months and during that time I have received a number of concerns from people about the quality of the results. Some of the concerns were valid and discovered issues, others were simply oversights or misunderstandings. I would like to take the time to describe the areas that cause invalid results and what to do if you feel the results on your test are invalid.

I would like to make it clear that the automated testing framework itself has had only one bug that has effected the results. The bug has been fixed since December and the testing framework has seen no other bug reports that effect the results. That is not to say that there have not been bugs in the framework, but that the results have not been incorrect due to the framework.

So if the framework is not the cause of invalid test results what is? Well, there is no simple answer to that. The following is a list of the areas that have been discovered to be sources of problems.

• Insufficient memory due to increasing requirements of the tests.
• Virtual machine resource problems.
• Virtual machine network issues.
• Drupal core tests being broken, failed test results show up on all tests.
• Various server configuration issues.
• Version specific PHP bugs.
• Version specific cURL bugs.
• Low-level OS library bugs.
• Poorly written tests that use random values that collide on occasion.
• Tests that reference settings or files that may not always be in the same location.

The server configuration issues have been the largest cause of problems. Items ranging from PHP short tags being enabled to some distributions having a separate PHP configuration for command line scripts that was overlooked during setup. Some testing servers are toggled on and off due to weird VM problems that seem to come and go. Others seems to remain consistent throughout.

Thankfully Chad "hunmonk" Philips has create a script that simplifies the process of setting up a test server and ensuring that a proper testing environment is setup. In addition I have been developing the next generation of the automated testing framework that will automatically remove servers that do not pass automated inspections.

In the meantime, if you run into results that you do not believe are valid please do not jump to conclusions.

• Make sure that you are running the latest Drupal 7 core code and tests.
• Some test have been know to fail randomly due to the way in which they were written. Please take a look at the tests that are failing.
• Get someone else to confirm that the tests pass for them. Sometimes code is left in tests that does not work from machine to machine.

If you confirm that the result is still not making sense please report it to the Project Issue File Review queue.

Overall, I trust the results that the testing bot provides due to the stability of the automated testing framework and I hope you will do the same.