I'm not sure what we did, but so far, D7 is a disaster.

At least, for Mollom, I considered that plenty of stuff would get easier. Negative.

Required fix for this patch to produce any remotely useful test results at all: #297860-36: Regression: Convert session.inc to the new database API (and _sess_write should use a merge)

I'm not trying to leverage any Form API improvements (or any other improvements) in this patch yet, so this is a straight port.

However, for any reason, the "Form administration" tests are failing, and I have no idea why. I've changed that test to use the fake testing server... but. When looking at the actual verbose debug output, then the CAPTCHA image is served from xmlrpc.mollom.com and not from the fake server... (?)

In addition to that, various assertions that test for a certain message are reported as fails, but when looking at the actual verbose debug output, the expected message is properly displayed... (?)

Something goes horribly wrong there.

Lastly, Mollom is a very nice example for why the new categorization below "Configuration and modules" is a monster #fail and will lead to opposite of the desired effect, i.e. a much more worse UX. I seriously bet you won't find Mollom's admin pages without looking at the patch.

Support from Acquia helps fund testing for Drupal Acquia logo

Comments

Dries’s picture

Thanks for starting to work on an upgrade! :)

- I haven't looked at the patch closely enough yet, but given the many form API hacks that we had to put in the Drupal 6 version of the module, maybe they are no longer supposed to work in a straight port ...

- We need to fix #297860-36: Regression: Convert session.inc to the new database API (and _sess_write should use a merge) because those changes were required to make the XML-RPC error handling work; see #578470: XML-RPC error handling sometimes fails silently. It looks like that patch was accidentally reverted. Because it is accidentally reverted, XML-RPC errors are never reset, which makes it pretty unreasonable to do more than one XML-RPC request per HTTP request. If the first XML-RPC call causes an error, all subsequent calls will incorrectly have the same error set. Clearly, we need to fix that in core. It was pretty broken in Drupal 6 too, so it is not really a regression -- Drupal 6 was in an equally bad state ... ;-)

- As for the IA in Drupal 7 -- I guess I'd expect a link called 'Mollom spam protection' under 'Content authoring' or something. Not sure if that is what you did, but that is where I'd look initially.

- The other thing we'll need to "work around" in Drupal 7 is the new session handling. To enable more aggressive caching, Drupal 7 no longer keep a session if there is nothing to save in that session. As a result, session IDs are not maintained across page requests. Mollom uses the session ID to identify record-replay attacks across page requests. This also affects Pressflow in Drupal 6 because they backported the session handling: see #562374: Mollom incompatible with Pressflow's (and Drupal 7's) session handling -- causes CAPTCHAs to fail -- ideally we'd be able to backport that fix to the Drupal 6 module.

sun’s picture

FileSize
61.53 KB

Incorporated the patch from #562374: Mollom incompatible with Pressflow's (and Drupal 7's) session handling -- causes CAPTCHAs to fail, which fixes a couple of tests.

Also fixed a couple of more issues. Getting closer. ;)

re 1) Nope, my revamp actually removed all Form API hacks ;)

re 2) Thanks for committing that fix.

re 3) Good guess, but I seriously think that this category is not really appropriate...

sun’s picture

Alright. I was finally able to understand why the tests are failing... #663992: Verbose debug output is cached by the browser OMG. :P

Dave Reid’s picture

@sun: Hah. That is a DrupalWTF moment. :)

sun’s picture

FileSize
70.55 KB

I fear that this will end in yet another >100 KB patch...

sun’s picture

FileSize
70.81 KB

oh my. It definitely helps to clear the entire browser cache. Spent another couple of hours looking at totally weird but entirely stale debug pages. :-/

sun’s picture

FileSize
72.73 KB

1 failing test remains.

Dries’s picture

Thanks for the clean-up, sun. Great job. It's actually great to see such a patch, if only to evaluate how hard it will be for module maintainers to upgrade their modules. I only skimmed the code for now, but here is what I got.

+++ mollom.admin.inc	19 Dec 2009 18:08:30 -0000
@@ -370,13 +376,16 @@ function mollom_comment_admin_overview_s
-            db_query("UPDATE {comments} SET status = %d WHERE cid = %d", COMMENT_NOT_PUBLISHED, $cid);
+            db_update('comment')
+              ->fields(array('status' => COMMENT_NOT_PUBLISHED, 'cid'))
+              ->condition('cid', $cid)
+              ->execute();

Redundant 'cid' in the fields() statement?

+++ mollom.module	19 Dec 2009 21:09:20 -0000
@@ -498,9 +500,13 @@ function mollom_get_form_info($form_id =
-  $mollom_form = db_fetch_array(db_query_range("SELECT * FROM {mollom_form} WHERE form_id = '%s'", $form_id, 0, 1));
+  $mollom_form = db_query_range('SELECT * FROM {mollom_form} WHERE form_id = :form_id', 0, 1, array(':form_id' => $form_id))->fetchAssoc();
   if ($mollom_form) {
     $mollom_form['fields'] = unserialize($mollom_form['fields']);
+    // @todo Is this a bug in D7?
+    if (!$mollom_form['fields']) {
+      $mollom_form['fields'] = array();
+    }

I saw you did the same earlier on in the code -- if I'm not mistaken. It didn't had a TODO.

+++ mollom.module	19 Dec 2009 21:09:20 -0000
@@ -951,6 +949,11 @@ function mollom_process_mollom_session_i
 function mollom_validate_analysis($form, &$form_state) {
+  // Do not invoke validation if a form error occurred elsewhere.
+//  if (form_get_errors()) {
+//    return;
+//  }
+

Not sure what this is.

+++ mollom.module	19 Dec 2009 21:09:20 -0000
@@ -1070,26 +1090,36 @@ function mollom_pre_render_mollom($eleme
+  $timestamp = time();

We should probably use REQUEST_TIME for these.

sun’s picture

FileSize
73.56 KB

The remaining failing test/assertion cannot be resolved within Mollom module - see the @todo I added.

Actually, there are two larger @todos added in this patch. One is bound to Mollom.module's architecture, the other one is about D7's render system, and the latter is the one I mean here.

Note that this patch is still rolled against DRUPAL-6--1; I believe that it should apply cleanly against HEAD though.

Lastly, this patch is still a straight port of the current code, not taking into account any new options in D7. The latter should always be a separate step in porting modules. (You know what happened to CCK in D6, eh? ;)

sun’s picture

FileSize
75.22 KB

oh, I *love* your reviews! :)

Fixed those.

Not sure about the mollom_form_load() comment though... I guess you mean similar code in mollom.test -- but that code doesn't unserialize the configured fields, just checks whether a $form_id has been stored.

sun’s picture

Status: Needs review » Reviewed & tested by the community
FileSize
76.42 KB

Fixed some last bits. I think this is ready to fly. We cannot resolve the remaining fail without entirely rewriting the implementation (which we should still try, but I'm not even sure whether it's possible at all).

sun’s picture

Oh. My. God.

I quickly patched batch.inc + form.inc in D6 to get the total elapsed time for running the tests to compare them with D7....................

D6 == 1 min 37 sec
D7 == 4 min 5 sec

errr, and we want to release? No way.

Dries’s picture

While Drupal 7 is probably slower than Drupal 6, it obviously shouldn't be that much slower. The performance degradation is probably either the direct or indirect result of the testing framework. I believe installation of Drupal became significantly slower due to (i) switching to InnoDB, (ii) using batch API to install Drupal -- I believe we install one module per batch, and that we do expensive cache clearing in between each batch step, (iii) we do more extensive clearing between tests, (iv) etc. That said, I agree that running the tests is rather slow.

I'll do a more in-depth review tomorrow! :)

sun’s picture

uh oh - something went wrong during syncing DRUPAL-6--1 with HEAD... All synced files went into the tests/ subdirectory ;)

EDIT: See http://drupal.org/cvs?commit=304076 for what I mean.

Dries’s picture

Dang. I think I fixed it the broken CVS HEAD but I haven't really tested it yet.

Dries’s picture

Status: Reviewed & tested by the community » Needs review

Reviewing this patch, it is a rather straightforward port of perfectly solid code. It is hard not to like it, so I have committed to CVS HEAD. I made the following notes when reviewing the patch.

"Drupal 7"-ifications for a follow-up patch:

  1. Add the settings link to the info file so the module administration page points to the settings page.
  2. Get rid of functions like mollom_node_insert and leverage the fact that the node ID is available in the form submit handlers.
  3. Double check that all update functions in the install file have Doxygen comments.
  4. The "administer nodes" permission was split into "administer nodes", "access content overview", and "bypass node access" so we need to review Mollom's use of the "administer nodes" permission.
  5. #538164: Comment body as field might still go in and might affect the port.

Other things I noticed:

  1. I ran the tests from the command line using $ /Applications/acquia-drupal/php/bin/php scripts/run-tests.sh --url http://cvs.localhost:8082/ --php /Applications/acquia-drupal/php/bin/php Mollom, using the Acquia Drupal stack installer. I couldn't run them from within the administration UI -- see below. I get more than one error though:
    Mollom access checking 114 passes, 0 fails, and 0 exceptions
    Comment submission protection 157 passes, 0 fails, and 0 exceptions
    Contact form protection 132 passes, 0 fails, and 0 exceptions
    Data processing 122 passes, 3 fails, and 0 exceptions
    Fallback behavior 77 passes, 1 fail, and 0 exceptions
    Form administration 137 passes, 4 fails, and 0 exceptions
    Reporting functionality 42 passes, 0 fails, and 0 exceptions
    Key provisioning for Mollom reseller 4 passes, 0 fails, and 0 exceptions
    Server list recovery 16 passes, 0 fails, and 0 exceptions
    User registration and password protection 85 passes, 0 fails, and 0 exceptions
    
  2. When configuring the article and comment forms, it is slightly confusing that the author field and the homepage URL field are not mentioned as they are often a target for spammers; especially the homepage URL field in comments.

Drupal 7 core bugs that I ran into:

  1. The permission link on the module administration page is broken -- looks like an overlay bug in core. Unrelated to the Mollom module.
  2. When running the tests from within the administration pages, I got an error message: Notice: Undefined variable: args in overlay_close_dialog() (line 525 of modules/overlay/overlay.module).
sun’s picture

+++ mollom.install	19 Dec 2009 21:28:23 -0000
@@ -260,16 +258,28 @@ function mollom_update_6106() {
+    ->fields(array('did' => 'SUBSTR(did, 6)')) // @todo Expressions in UPDATE?
...
+    ->fields(array('did' => 'SUBSTR(did, 9)')) // @todo Expressions in UPDATE?

Need to talk to Crell or grep throughout core to find out whether this is possible.

+++ mollom.module	19 Dec 2009 21:51:45 -0000
@@ -125,28 +125,22 @@ function mollom_help($path, $arg) {
+function mollom_init() {
+  // Expire all mollom session IDs as soon as possible.

Rename to mollom_exit() and eventually move below mollom_form_submit().

+++ mollom.module	19 Dec 2009 21:51:45 -0000
@@ -288,12 +283,19 @@ function mollom_report_access($entity, $
+      'title' => t('Administer Mollom-protected forms and Mollom settings'),
...
+      'title' => t('Post content without checking'),

The human-readable permission strings need work.

+++ mollom.module	19 Dec 2009 21:51:45 -0000
@@ -288,12 +283,19 @@ function mollom_report_access($entity, $
+    'access mollom reports' => array(
+      'title' => t('Access Mollom reports'),
+    ),

Actually, I implemented this for testing purposes only. Can probably be removed, sorry.

+++ mollom.module	19 Dec 2009 21:51:45 -0000
@@ -498,9 +500,13 @@ function mollom_get_form_info($form_id =
 function mollom_form_load($form_id) {
-  $mollom_form = db_fetch_array(db_query_range("SELECT * FROM {mollom_form} WHERE form_id = '%s'", $form_id, 0, 1));
+  $mollom_form = db_query_range('SELECT * FROM {mollom_form} WHERE form_id = :form_id', 0, 1, array(':form_id' => $form_id))->fetchAssoc();
   if ($mollom_form) {
     $mollom_form['fields'] = unserialize($mollom_form['fields']);
+    // @todo Is this a bug in D7?
+    if (!$mollom_form['fields']) {
+      $mollom_form['fields'] = array();
+    }

Investigate whether the old code was wrong or whether D7 has a bug here, and why D6 behaved differently.

+++ mollom.module	19 Dec 2009 21:51:45 -0000
@@ -994,11 +992,22 @@ function mollom_validate_analysis($form,
+function mollom_validate_captcha(&$form, &$form_state) {
...
+  // @todo Problem: This error is output also in case we have no valid keys or a
+  //   communication error, in this request or perhaps even in the last already.
+  //   Thus, in case the fallback method blocks all submissions, a user gets TWO
+  //   error messages, and one of them being totally nuts, because *there is no*
+  //   CAPTCHA in the form, because we didn't reach the servers in the first
+  //   place. However, we also do not really want to invoke lots of XML-RPC
+  //   requests for the case when we (could) *know* that we won't get a usable
+  //   $result, because the servers are empty or keys are invalid - unless the
+  //   site administrator revisits Mollom's settings page again and fixes the
+  //   keys/servers.
+
   // Bail out if no value was provided.
   if (empty($form_state['values']['mollom']['captcha'])) {
     form_set_error('mollom][captcha', t('The CAPTCHA field is required.'));

@@ -1012,6 +1021,12 @@ function mollom_validate_captcha($elemen
+  // Cancel validation if we have communication issues.
+  // @todo These should really be bitwise OR'ed error code constants...
+//  if ($result == NETWORK_ERROR || $result == MOLLOM_ERROR) {
+//    return;
+//  }

We probably want to add an internal system variable to track the current "healthiness" of the module's configuration - i.e. if there are no keys, or invalid keys, or whether we have no valid server list, then always invoke the fallback behavior. To be implemented as new (first) #process or #element_validate of the 'mollom' element.

+++ mollom.module	19 Dec 2009 21:51:45 -0000
@@ -1057,6 +1072,13 @@ function mollom_pre_render_mollom($eleme
+    // @todo D7: A horrible effect of the new rendering system: If we cannot
+    //   reach Mollom servers, then mollom() will invoke _mollom_fallback(),
+    //   which outputs a message. However, this is #pre_render. The main page
+    //   content is render()'ed *within* page.tpl.php. Which is also where
+    //   $messages are rendered. But $messages are rendered before $content!
+    //   I already briefly discussed this chicken-n-egg problem with moshe, but
+    //   as of now, we have no ideas. 19/12/2009 sun
     $result = mollom('mollom.getImageCaptcha', $data);

I already spent hours tinkering about this, but still have no idea how we could resolve that.

That's quite a critical regression, because it basically means that no renderable array in the page template can output a message. The only idea I had so far was to eventually inject all messages in an ESI-style approach, i.e. replacing a replacement token/macro in the fully rendered page.tpl.php HTML markup with the actual messages. Quite a hack. Perhaps we simply went too far with usage of render() for the main page content (if the $content variable in page.tpl.php was an already rendered HTML string, then this problem would not exist).

As a workaround for Mollom, it would only be resolvable, if we would revamp the entire implementation, and perform all form alterations and validations during element and form validation.

+++ mollom.module	19 Dec 2009 21:51:45 -0000
@@ -1070,26 +1092,36 @@ function mollom_pre_render_mollom($eleme
-    // Otherwise, hide the CAPTCHA if there was a communication error.
-    elseif ($result === NETWORK_ERROR) {
+    // Otherwise, hide the CAPTCHA.
+    else {

There are a couple of fixes like this that need to be back-ported. Created #665270: Backport fixes from port to D7 for that.

+++ mollom.module	19 Dec 2009 21:51:45 -0000
@@ -1362,15 +1396,17 @@ function node_mollom_form_info() {
+function mollom_node_insert($node) {
+  mollom_data_save('node', $node->nid);
+}
...
+function mollom_node_delete($node) {
+  mollom_data_delete('node', $node->nid);
 }

You are right that we can now try to leverage the 'post_id' mapping property that was intended for this purpose (to be processed in mollom_form_submit()).

In addition to that, I considered whether we can perhaps simplify the current 'report delete callback' that can be defined in hook_mollom_form_info() - and change it to the $form_id of the real/original delete confirmation form of the entity instead.
Hence, instead of duplicating functionality (we always delete), we would only integrate with the existing deletion functionality/workflow.

+++ mollom.module	19 Dec 2009 21:51:45 -0000
@@ -1389,6 +1425,26 @@ function mollom_form_node_admin_content_
+function mollom_node_view($node, $build_mode) {
+  // Only show the links if the module is configured.
...
+    $node->content['links']['mollom'] = array(
+      '#theme' => 'links',
+      '#links' => array(
+        'mollom_report' => array(

This is another bug in Drupal core -- Node module registers $node->content['links']['node'] and assigns theme_links() for the defined links in there. The current design of node/comment links seems to have intended to build separate lists of links for each module.

That, however, looks very wonky in the rendered output, and also doesn't make much sense from the generated HTML markup.

Not sure whether there is already an issue for that, but since the fix is likely to be an API change (just skipping the second array level?), we should fix it soon. Not sure whether I'll have time for that.

This review is powered by Dreditor.

sun’s picture

FileSize
4.04 KB

Attached patch does the trivial fixes.

Dries’s picture

Committed the bug fixes in #18. Thanks.

Dries’s picture

With the latest bugfix patch applied/committed, I now get the following:

$ /Applications/acquia-drupal/php/bin/php scripts/run-tests.sh --url http://cvs.localhost:8082/ --php /Applications/acquia-drupal/php/bin/php Mollom

...

Mollom access checking 115 passes, 0 fails, and 12 exceptions
Comment submission protection 157 passes, 0 fails, and 12 exceptions
Contact form protection 127 passes, 0 fails, and 8 exceptions
Data processing 110 passes, 18 fails, and 9 exceptions
Fallback behavior 77 passes, 1 fail, and 12 exceptions
Form administration 140 passes, 0 fails, and 8 exceptions
Reporting functionality 44 passes, 0 fails, and 4 exceptions
Key provisioning for Mollom reseller 4 passes, 0 fails, and 4 exceptions
Server list recovery 16 passes, 0 fails, and 4 exceptions
User registration and password protection 84 passes, 0 fails, and 8 exceptions

Is that consistent with your results? You mentioned you only saw one fail so it sounds like it might be inconsistent, and that we have to do some more investigation to figure out why.

sun’s picture

I just ran the tests again (via GUI) and still get

894 passes, 1 fail, 0 exceptions, and 213 debug messages

whereas this 1 fail belongs to

Fallback behavior
77 passes, 1 fail, 0 exceptions, and 19 debug messages

and is the fail I documented + explained in-code as well as above.

Dries’s picture

Odd, I'm using the following PHP version on the command line:

deimos:drupal-cvs dries$ /Applications/acquia-drupal/php/bin/php --version
PHP 5.2.9 (cli) (built: Sep 18 2009 10:32:57) 
Copyright (c) 1997-2009 The PHP Group
Zend Engine v2.2.0, Copyright (c) 1998-2009 Zend Technologies

Anyway, I'll try switching to the UI instead.

Do you happen to run a patched core?

sun’s picture

FileSize
83.23 KB

Indeed, very odd.

But anyway, let's try whether we can revamp all of this for D7 now. :)

erm, also included -- can we remove that duplicate tests/tests/ directory? ;)

sun’s picture

FileSize
18.74 KB

Thanks for removing the test files!

Dries’s picture

FileSize
17.59 KB
+++ mollom.module	23 Dec 2009 22:21:42 -0000
@@ -991,9 +998,13 @@ function mollom_validate_analysis($form,
 
+  // Prevent the page cache from storing a form containing a CAPTCHA element.
+  $GLOBALS['conf']['cache'] = CACHE_DISABLED;

This shouldn't happen if there is a session.

Made some minor text changes. Update patch attached.

sun’s picture

FileSize
17.62 KB

uhm, wait - do we automatically disable page caching if there is a session for the current user in HEAD? If this is true, then we need to update Mollom's code in various locations. For now, I didn't merge that changed comment.

All tests pass for me.

What's left? - Entirely killing the #pre_render.

sun’s picture

+++ mollom.module	24 Dec 2009 14:21:01 -0000
@@ -991,9 +1000,13 @@ function mollom_validate_analysis($form,
 function mollom_validate_captcha(&$form, &$form_state) {
...
+  // Prevent the page cache from storing a form containing a CAPTCHA element.
+  $GLOBALS['conf']['cache'] = CACHE_DISABLED;

errr, this is the wrong location to prevent page caching ;)

I'm on crack. Are you, too?

sun’s picture

FileSize
17.38 KB

Fixed that.

Dave Reid’s picture

Issue tags: +D7CX
Dries’s picture

Great, I re-ran the tests with the latest patch and I got nothing but passes:

Mollom access checking 113 passes, 0 fails, and 0 exceptions
Blacklisting 21 passes, 0 fails, and 0 exceptions
Comment submission protection 161 passes, 0 fails, and 0 exceptions
Contact form protection 131 passes, 0 fails, and 0 exceptions
Data processing 125 passes, 0 fails, and 0 exceptions
Fallback behavior 80 passes, 0 fails, and 0 exceptions
Form administration 140 passes, 0 fails, and 0 exceptions
Language detection 26 passes, 0 fails, and 0 exceptions
Reporting functionality 43 passes, 0 fails, and 0 exceptions
Key provisioning for Mollom reseller 4 passes, 0 fails, and 0 exceptions
Server list recovery 16 passes, 0 fails, and 0 exceptions
User registration and password protection 82 passes, 0 fails, and 0 exceptions
Dries’s picture

Oh, I committed the patch in #28 to CVS HEAD.

sun’s picture

FileSize
7.58 KB

Resolved a couple of @todos:

- Removed #pre_render.

- Implemented hook_modules_uninstalled().

- Removed debugging code.

Dries’s picture

Status: Needs review » Fixed

Looks good to me, and all tests pass so committed to CVS HEAD.

TODO for Dries: I want to verify manually that the Mollom session IDs remain intact across different page views / form steps. That is critical for Mollom to work correctly.

Status: Fixed » Closed (fixed)
Issue tags: -D7CX

Automatically closed -- issue fixed for 2 weeks with no activity.

  • Commit 00073c4 on master, fai6, 8.x-2.x, fbajs, actions by Dries:
    - Patch #663246 by sun: initial port to Drupal 7.
    
    
  • Commit 0a4ae21 on master, fai6, 8.x-2.x, fbajs, actions by Dries:
    - Patch #663246 by sun: more Drupal 7 updates to remove the remaining @...
  • Commit 27f98fa on master, fai6, 8.x-2.x, fbajs, actions by Dries:
    - Patch #663246 by sun: removing Drupal 6 form API hacks now we fixed...
  • Commit edf6884 on master, fai6, 8.x-2.x, fbajs, actions by Dries:
    - Patch #663246 by sun: more bug fixes for the Drupal 7 port.
    
    

  • Commit 00073c4 on master, fai6, 8.x-2.x, fbajs, actions by Dries:
    - Patch #663246 by sun: initial port to Drupal 7.
    
    
  • Commit 0a4ae21 on master, fai6, 8.x-2.x, fbajs, actions by Dries:
    - Patch #663246 by sun: more Drupal 7 updates to remove the remaining @...
  • Commit 27f98fa on master, fai6, 8.x-2.x, fbajs, actions by Dries:
    - Patch #663246 by sun: removing Drupal 6 form API hacks now we fixed...
  • Commit edf6884 on master, fai6, 8.x-2.x, fbajs, actions by Dries:
    - Patch #663246 by sun: more bug fixes for the Drupal 7 port.