The testbot is now running pifr 2.x! The system has many new features and has been completely redesigned with the future of Drupal quality assurance in mind. More information on the new features and the roadmap for qa.drupal.org to come.
The script checks for any updates to the server install package each time it's run, and checks if it's previously performed any work, only performing those actions which were not previously complete. This allows the script to be run over and over again for things like managing the list of current testing administrators.
Written by Chad Phillips
The testing bot has been active for almost four months and during that time I have received a number of concerns from people about the quality of the results. Some of the concerns were valid and discovered issues, others were simply oversights or misunderstandings. I would like to take the time to describe the areas that cause invalid results and what to do if you feel the results on your test are invalid.
I would like to make it clear that the automated testing framework itself has had only one bug that has effected the results. The bug has been fixed since December and the testing framework has seen no other bug reports that effect the results. That is not to say that there have not been bugs in the framework, but that the results have not been incorrect due to the framework.
So if the framework is not the cause of invalid test results what is? Well, there is no simple answer to that. The following is a list of the areas that have been discovered to be sources of problems.
The server configuration issues have been the largest cause of problems. Items ranging from PHP short tags being enabled to some distributions having a separate PHP configuration for command line scripts that was overlooked during setup. Some testing servers are toggled on and off due to weird VM problems that seem to come and go. Others seems to remain consistent throughout.
Thankfully Chad "hunmonk" Philips has create a script that simplifies the process of setting up a test server and ensuring that a proper testing environment is setup. In addition I have been developing the next generation of the automated testing framework that will automatically remove servers that do not pass automated inspections.
In the meantime, if you run into results that you do not believe are valid please do not jump to conclusions.
If you confirm that the result is still not making sense please report it to the Project Issue File Review queue.
Overall, I trust the results that the testing bot provides due to the stability of the automated testing framework and I hope you will do the same.
Thanks to everyone involved in the D6 upgrade.
It has been a while since the last update so I figured it is time for one. I lost my login credentials for testing.drupal.org during my graphics card failure and thus was not able to make an updates until a few days ago.
After a lengthy catch up period the bot is now caught up with the issue queue and has some impressive statistics to show for it.
Work has begun on the next generation of the testing framework that will be easier to manage, self correcting, and offer more features. Along with my blog posts regarding the development I have created more specific issues for PIFR 2.0 and PIFT 2.0.
Any donations to the development of the automated testing system are much appreciated and let us know that you value the increased quality assurance that the framework brings to the Drupal community.
If anyone is interested in developing, or knows of an existing one, a "bot like" theme for testing.drupal.org please make a comment.
The bot needs a bit of pizazz. (Drupal 6.x)