Automated testing for contrib modules [#362931]

There is already some bot, who testing the patches:
http://testing.drupal.org/

Why not create bot, who testing the contributed modules?
One mistake of developer and hundreds of developers and thousands of thousands visitors are the most injured.

There are some very serious leaks in Drupal, which cause each time the famous white screens.
Developers will not learn each time what there are mistakes that they shouldn't do.

Example: #352344: Fatal error: Unsupported operand types in common.inc on line 1369
Many people voted for the patch for Drupal core, answer: Drupal core doesn't care of broken module which brokes the whole application, it's always a developer fault.
I think the fault is on both sides.

For me it's like M$ goals, system in the clean environment is always passing the tests. Virus installed on system automatically after 15min of connected to the internet? Of course, it's a user fault!

So, there should be some global solve for that, otherwise until the end of time and bad conversion from 4.x to 5.x, 5.x to 6.x, 6.x to 7.x, 7.x to ... Drupalers will have the nightmare.
And the most topics in FAQ will be how to check which module failed and blame the bad developer, because Drupal is the perfect (no patched needed).
Example FAQ: http://drupal.org/node/362799

Comments

Comment #1

Amazon commented 23 January 2009 at 01:48

In the long term, we would like to test the current releases modules. But we need to get the next generation of the patch testing done for managing servers first. If you want to help with the back port of the testing framework to Drupal 6 that would be a good initial step.

Comment #2

kenorb commented 23 January 2009 at 01:59

Some interesting detail:
Please type in Google (and see some Cache pages):
+"/includes/bootstrap.inc on line" -drupal
+"/includes/common.inc on line" -drupal
There are (xxx)k live websites with broken core.

Comment #3

dave reid

he/him

English

Nebraska USA

commented 23 January 2009 at 02:00

Of course in order for this to work, module maintainers actually have to write tests for their modules. You'll usually find the modules with tests are the ones that don't have silly errors that frustrate users. So I don't really see this helping much. What I think will help is project metrics on the project page (showing how many issues are still open, how often the developer is active, etc).

Comment #4

kenorb commented 23 January 2009 at 02:03

Yes, I would like to. My initial step was to apply for CVS account, which was refused;)
Any other initial steps?;)

Comment #5

kenorb commented 23 January 2009 at 02:16

Most of the module maintainers don't care about making the tests, they don't have time to do that, there is no any restriction rules to do that and each developer will say that for their code there is no needed any tests, because it's working and it's simple.
It's not the activity problem, people are people, quality of the proper code doesn't depends of developer activity.

Some people have good will, have time, but their are not good example of experienced programmers.

My idea is to create a quite big Drupal installation with all the modules installed which will be validated by some script (e.g. Simpletest or JMeter script) and report some all broken modules and notify the maintainers of their broken code.

Comment #6

dave reid

he/him

English

Nebraska USA

commented 23 January 2009 at 02:44

Title:

create some testing bot for validate the contributed code

» Automated testing for contrib modules

Module maintainers that care about their modules are writing tests. Here's a list of the current contrib modules that have .test files. Many of them are the high-profile modules like cck, coder, votingapi, date, webform, google_analytics, rules, location, image, etc. I don't disagree that having automated testing on contrib would be useful, but it will be up to the module maintainers (again) to make sure they're doing the right thing.

Comment #7

catch

he/him

English

commented 23 January 2009 at 09:54

Project:	Drupal.org infrastructure	» Drupal.org site moderators
Component:	CVS	» Redesign

We've discussed a 'click through' test for core - which would query the router table then visit as many paths as possible - mainly to pick up notices, and the test bot already picks up syntax errors in patches.

If we had php -l, a click through test, and then at some point integrate a coder review into the test framework - which could report back obvious things like the l() errors even if we didn't want full reports on code style and the rest - then potentially we'd have somewhat useful data for every contrib module, regardless of whether it comes with tests or not. Since this comes under metrics, moving it to the redesign component.