So, the Dojo lesson showed me the guts of the node access system. I am as guilty as the next developer for in not reviewing this before. node_access_rebuild iterates every single node in the system. Enabling, say, forum_access for drupal.org would very likely time out and leave a very broken system behind because of the incomplete node_access table update.

We have a solution at hand however for long running queries. The update system has a reload (with or without JS) facility. The query should only load, say, 200 nodes at a time and reuse this facility to reload itself. Also, there should be a warning in the docs of all modules that call this function that going into maintenance mode is recommended.

Comments

yched’s picture

yched CreditAttribution: yched commented

I have worked on some code to generalize update.php "progress-bar" operations.
It was originally to deal with mass deletions in CCK (http://drupal.org/node/97861 , http://drupal.org/node/123721) but the goal was to make a 'progressive ops' API available to other modules, and thus ideally have this integrated to core.

I did not really get to finish it, though - the thing works, but the shape of the API and the integration with drupal paths and form handling still needs some love.

I'll try to clean up what I have and upload it in my sandbox some time soon.

Dries’s picture

Dries CreditAttribution: Dries commented
Status: Active » Reviewed & tested by the community

I've committed this to CVS HEAD as a temporary fix. We can investigate better solutions in CVS HEAD.

Should go into DRUPAL-5 branch too.

Thanks Moshe.

yched’s picture

yched CreditAttribution: yched commented
Status: Reviewed & tested by the community » Active

Looks like Dries updated the wrong thread - it's http://drupal.org/node/108752 that got committed

yched’s picture

yched CreditAttribution: yched commented

Work in progress submitted at http://drupal.org/node/127539

yched’s picture

yched CreditAttribution: yched commented
Status: Active » Closed (duplicate)

I submitted a patch for 'batchifying' node_access_rebuild in http://drupal.org/node/144337

jlmeredith’s picture

jlmeredith
Location Denver, CO
CreditAttribution: jlmeredith commented

Is there currently a workaround for this issue?

yched’s picture

yched CreditAttribution: yched commented

you mean, for drupal 5 ? er, no, there is currently no workaround that I can think of.
The 'workaround' being discussed in http://drupal.org/node/144337 relies on the D6 batch API, which will not get backported in D5 core, and AFAICT, cannot live in a contrib module...

jlmeredith’s picture

jlmeredith
Location Denver, CO
CreditAttribution: jlmeredith commented

Just and FYI - Although not a work around, there is a solution for those trying to use node access based modules with 5.1. Use the current HEAD version of Drupal instead or patch based on this issue: http://drupal.org/node/123705. I have tested the patch from here with up to 20,000 nodes with no issues and four different node access modules.

dalin’s picture

dalin
he/him
Primary language English
Location 🇲🇽, 🇨🇦, 🌍
CreditAttribution: dalin commented

@jlmeredith You can hack a fix for Drupal 5. Simply add this line right after set_time_limit(240); in the node_access_rebuild() function:

ini_set('memory_limit', '256M');

Unless of course you're on a host that doesn't allow you to adjust the memory limit.