As discussed in IRC already, this patch adds configuration and a hook_cron implementation to flush messages that have been deleted by all users.

Features:
- On/Off Switch
- Configure how long the deleted messages should be kept, select which ranges from 0 - 100. 0 means on the next cron run. I don't think we need other units than days, as it isn't really important if the messages are cleared some hours sooner or later...
- HAVING-support for _privatemsg_assemble_query, so I re-introduced those changes from the patch in #351542: user interface
- Already deleted messages will be updated and considered to be deleted now as there is no way to guess the actual deletion time

Conflicts:
- same hook_update_N function as in #348907: Per thread/Multiple thread actions. This means that either this or the block_cache patch needs to be rerolled, but this is better than having a gap for -dev users imho.
- #288183: Provide api function for other modules to send messages.: There are probably conflicts and this functionality eliminates the need for the $delete_by_all switch in hook_privatemsg_message_delete but makes a hook_privatemsg_message_flush necessary
- #348907: Per thread/Multiple thread actions: privatemsg_thread_delete needs to use the timestamp instead of 1 too.

The commit order is however not important. if that one gets commited, the other patches need a re-roll and if one of those gets commited first, this one needs to be rerolled.

Comments

berdir’s picture

StatusFileSize
new6.99 KB

Removed the COUNT(), only used that to test it, it isn't needed...

naheemsays’s picture

Not tested, but I assume that

+    drupal_set_message($query);

Is just debug code?

berdir’s picture

Status: Needs review » Needs work

Oh, yes of course. Will update the patch..

berdir’s picture

Status: Needs work » Needs review
StatusFileSize
new6.72 KB

Removed the debug message... it was late...

berdir’s picture

StatusFileSize
new8.34 KB

I found out that it wasn't working on PostgreSQL yet. The issue is the count query in combination with GROUP BY and HAVING. I wrote a function which generates a special $str_select with only aggregate functions and subqueries. Those that are probably needed in HAVING. It's not possible to add all fields, because the group by is missing, so postgresql bails out.

It's not perfect...
- A query with HAVING without GROUP BY wouldn't work, but we don't have them yet..
- A HAVING statement which would use a non aggregate field would not work..

However, it really gets complicated, so I would suggest to only implement support for those features if we actually need to use them.

berdir’s picture

Status: Needs review » Needs work

TODO: Add hook_privatemsg_message_flush and remove $deleted_by_all

berdir’s picture

StatusFileSize
new8.3 KB

Updated:

- added new hook_privatemsg_message_flush
- removed $deleted_by all related code
- Added a configurable flush maximum per cron run, defaults to 200 (takes 1,3s on my pc)

berdir’s picture

Status: Needs work » Needs review

Forgot to update the status

berdir’s picture

StatusFileSize
new8.97 KB

And forgot to add the update hook, thanks nbz!

berdir’s picture

StatusFileSize
new9.34 KB

PostgreSQL fixes:

- use php's date() in the update hook, pgsql doesn't know unix_timestamp()
- use MAX/MIN in HAVING, pgsql can't handle the aliases

berdir’s picture

StatusFileSize
new9.37 KB

Next try, slight improvements of the wording.

naheemsays’s picture

Status: Needs review » Reviewed & tested by the community

Setting to rtbc - I tested on mysql, Berdir on pgsql and it all works as expected.

berdir’s picture

Status: Reviewed & tested by the community » Needs review
StatusFileSize
new12.12 KB

- To make sure this works correctly, I added some tests. The test creates 10 new messages, deletes a different sub-group of them for author and recipient, overwrites the deleted timestamp for some of the deleted messages and then checks if the messages are still there or if they have been flushed.

- While writing the patch, I noticed that _privatemsg_load does not check if the message exists at all. If it does not exist, a array with only the user key set is returned. That does not make sense, so I changed it to return FALSE if no message has been found. node_load for D6 and 7 does the same thing.

- Removed a debug statement in privatemsg_cron()

naheemsays’s picture

As mentioned in http://drupal.org/node/303087#comment-1365210, the current implementation of HAVING with sub queries gives a warning message, however that shortcoming does not seem to affect the usage in this patch as there is no subquery in the sql where HAVING is used.

berdir’s picture

StatusFileSize
new11.63 KB

Re-roll, changes:

- updated privatemsg_message_change_delete. $delete parameter should now be TRUE/FALSE and is passed down to the delete hook.
- re-included the count changes in nbz's inbox patch. I need to do more performance checks as I can't get MYSQL to stop sorting that *** subquery, but it works great.
- updated tests

naheemsays’s picture

Just an idea - if its only the HAVING that broke the old method for COUNT AND it turns out to be a performance hog , why not use the old method for all other queries that do not contain a HAVING?

Instead of removing the old method and adding

+    if (!empty($HAVING)) {
+      $str_having = '('. implode(') AND (', $HAVING) .')';
+      $query .= " HAVING {$str_having}";
+    }
+
+    // Also build a count query which can be passed to pager_query to get a "page count" as that does not play well with queries including "GROUP BY" or "HAVING".
+    // This should work with both mysql and pgsql as outlined in http://drupal.org/node/303087#comment-1370752 .
+    // The COUNT query is created before adding the ORDER BY as that is not needed and will just slow things down.
+    $count = 'SELECT COUNT(*) FROM ('. $query .') as count';
+

Do not remove the old count query builder and use:

+    if (!empty($HAVING)) {
+      $str_having = '('. implode(') AND (', $HAVING) .')';
+      $query .= " HAVING {$str_having}";
+      // queries containing a HAVING break the count query on pgsql.
+      // In this case, use the subquery method as outlined in http://drupal.org/node/303087#comment-1370752 .
+      $count = 'SELECT COUNT(*) FROM ('. $query .') as count';
+    }
+
naheemsays’s picture

Status: Needs review » Needs work

needs to be rerolled to remove the query builder changes as they have already gone in.

berdir’s picture

Status: Needs work » Needs review
StatusFileSize
new10.36 KB

Removed query builder stuff...

litwol’s picture

Status: Needs review » Needs work

localhost:privatemsg litwol$ patch -p0 < privatemsg.flush_messages11.patch
patching file privatemsg.module
Hunk #1 succeeded at 336 (offset 38 lines).
Hunk #2 succeeded at 395 (offset 38 lines).
Hunk #3 succeeded at 1175 (offset 4 lines).
Hunk #4 FAILED at 1415.
1 out of 4 hunks FAILED -- saving rejects to file privatemsg.module.rej
patching file privatemsg.test
patching file privatemsg.install

berdir’s picture

Status: Needs work » Needs review
StatusFileSize
new9.43 KB

Re-rolled, all tests pass again and improved some inline docs.

naheemsays’s picture

the update number needs to be bumped to 6003.

naheemsays’s picture

StatusFileSize
new9.02 KB

rerolling.

naheemsays’s picture

StatusFileSize
new9.29 KB

rerolled - I also got rid of hook_privatemsg_message_delete as it was now almost useless.

berdir’s picture

Status: Needs review » Needs work
+++ privatemsg.install	12 Sep 2009 05:39:28 -0000
@@ -417,4 +417,16 @@
+function privatemsg_update_6004() {

Once again, that needs to be updated, we are now at 6006 :)

+++ privatemsg.test	12 Sep 2009 05:39:28 -0000
@@ -110,6 +110,58 @@
+      privatemsg_new_thread(array($recipient), 'Message #'. $i, 'This is the body',array('author' => $author));

Really minor, there is a space missing before the second/last array.

Can you fix these two things and then we can finally commit and close this issue? :) It does have (imho quite well) tests and has been tested manually extensively on both mysql and pgsql.

This review is powered by Dreditor.

naheemsays’s picture

Status: Needs work » Needs review
StatusFileSize
new9.38 KB

rerolled.

berdir’s picture

Version: 6.x-1.x-dev » 7.x-1.x-dev
Status: Needs review » Fixed

Yeah!

Added to 6.x-1.x-dev and 7.x-1.x-dev!

Note that the D7 Port contains a few necessary changs to theme() calls since I only got fatal errors without these which made it quite hard to test...

Status: Fixed » Closed (fixed)

Automatically closed -- issue fixed for 2 weeks with no activity.