menu_router table truncated and site goes down

Explicit locking of tables was removed from core in D6. So if we're going to put it back, it may stop Drupal from working on some hosts. I don't have an alternative suggestion to fix this specific issue though so leaving at needs review.

Optalgin, usually bugs are fixed in the development branch then backported to D6 (also there are currently more people looking at issues for D7 core). There are exceptions, and if this really does require db_lock_table() it might be one of them, but it's worth trying to keep both branches consistent unless there's a particular reason not to.

Table locks are out in D6, so I think just shortening the window is a good start.

A discussion is going on in http://drupal.org/node/248739, which is a duplicate of that one (or the other way around).

A better patch (that further reduce the lock window), prepared for #248739.

Better title.

http://drupal.org/node/248739 was duplicate (and has discussion for potential fixes)

1. Add a menu_version table which has one auto increment field.
2. Add a version column to the menu_router table
3. Add the version to the router queries.
4. Add a cronjob that occassionally cleans up.

Yes. This sucks. However, it avoids a lock.

Can we get in a first patch to just move the DELETE query? This would alleviate (I think) a large fraction of the possible concerns.

I don't mind using a table lock for D6 if we did not officially remove LOCK TABLES from the requirements (and for sites where the lock fails they are not any worse off than now).

For D7, as above, let's get this as a transaction.

I suggested a patch implementing soft locking for that type of operation in #251792.

@Damien Tournoud - since D7 will almost certainly support transactions, I think the issue is really how do we get a stop-gap for D6.

@pwolanin: please read my discussion in the other issue: there is two types of concurrency issues, and transactions only fix one of them. Moreover, will transaction-safe database engines (ie InnoDB on MySQL) will be a strong requirement of Drupal 7? Let me doubt it.

@Damien Tournoud - it sounds somewhat like you want a better implementation of the sort of semaphore that cron uses to indicate that a concurrent execution is already in progress?

@pwolanin: exactly. Cron semaphore is broken because variable are asynchronous (they are cached). See issue #249185: Concurrency problem with variable caching leading to cache inconsistency for details.

I'm trying to solve the general problem once and for all. If the implemented solution gets positive review, the cron semaphore will also be replaced by the new locking framework.

Echoing pwolanin in #19... I've had to recover from this problem (an empty menu_router table) several times. In my experiences, the problem is wiping out the menu_router table prior to loading all the modules via module_implements to get the menu items.

This can happen while developing modules -- whenever we need to reset the menu router/cache (like via submitting /admin/build/modules or using 'reset cache' from devel.module), and it can also happen if you install a buggy, bum module with code problems. That was the case for me tonight... I wanted to test the module_builder for 6.x, so I attempted to install it and it crashed my site... after menu_router was wiped and obviously before it could be rebuilt (sure it was a -dev version and I was merely on my local test environment, but modules should fail to install more gracefully than knocking out the rest of the site, I think).

Simply moving that DELETE function /after/ all the 'module_implements' calls sounds like it would have resolved all the cases I've had to deal with -- by effectively making it an 'all-or-nothing' menu_router rebuild (if php crashes before DELETE, so be it, but at least I still have menu_router).

All that discussion re:concurrency sounds important too, but maybe this can be split up into separate issues?

ok, here's the trivial patch - let's at least get this into HEAD and 6.x and then think about additional fall-backs or locks for 6.x.

@pwolanin: please at least move the DELETE to _menu_router_build. This DELETE placed in a different function as the INSERT seems like a bug on its own anyway.

how about this.

@pwolanin: there is a typo in the watchdog message, but otherwise it looks good.

or this - even a little simpler.

Still the same typo in the watchdog message.

oops - we must have cross-posted. I didn't see the first message about the typo.

Perhaps this is also a place we should actually use a table lock? Apparently we can still use them in D6, even though an effort has been made to reduce them.

with typo fix

and the same with a table lock. Note, it seems that there are no locks at all in core in D6 at the moment...

There was a big effort to remove them, however this doesn't seem to have been reflected in CHANGELOG.txt or requirements - so if we needed to add one back in would probably be OK (if a bit of a shame, but so's people's sites getting hosed).

@pwolanin: so you are down to my initial proposal... putting the LOCK back in.

But... the requirement for the LOCK permission was removed [1], even if it was not reflected in the documentation.

Because we don't check this requirement anymore, the db_lock() operation will throw errors when the database user does not have the necessary permission. This is not acceptable (especially for the D6 part of this bug).

[1] http://cvs.drupal.org/viewvc.py/drupal/drupal/includes/install.mysql.inc...

The patch in #33 is a stop-gap fix for the issue, that is ready to go in Drupal 6 (I verified that it still applies).

For a longer-term fix, we will need both support for transactions (to prevent race conditions in DELETE-then-INSERT operations) and support for some kind of locking (to prevent several operations from running in parallel). I made a proposal for the latter in #251792.

Damien: thanks, I'd forgotten we'd taken it out of system requirements, my mistake - everyone please ignore #35.

@catch - obviously a number of us were confused about locks in D6...

I updated the requirements page to better reflect the current situation.

The patch is still applied to drupal 6.x , but for me with the last router patch, the number of queries generated by the menu API is always frightening (see devel log in attached file).

Executed 1524 queries in 6768.27 milliseconds. Page execution time was 7308.87 ms.

My Drupal/Server config :

Drupal	6.2
Access to update.php	Protected
Configuration file	Protected
Database updates	Up to date
File system	Writable (public download method)
MySQL database	5.0.51a
PHP	5.2.5
PHP memory limit	128M
PHP register globals	Disabled
Unicode library	PHP Mbstring Extension
Update notifications	Enabled
Web server	Apache/2.2.8 (Win32)

@lilou: well, I guess this is a different issue, but I concur: this is really frightening.

@damien : you're right, it's seem to be a flexifilter issue. Sorry.

requirements: http://drupal.org/requirements

Committed #33 to Drupal 6. RTBC for Drupal 7, and then feel free to come up with something else from there in Drupal 7.

bump - can we get this basic path in HEAD, while we're waiting for more substantial changes?

re-roll #33 for HEAD to remove offset.

Is this patch going to be put into the next version of drupal 6? I have 6.2 and I keep getting errors about duplicate entries in the menu_router table. I had a similar memory overload crash once and I think something got screwed up in the database.

Yes- this will be in the Drupal 6.3 release - it was already committed to that branch.

patch still applies with minor offset.

With patch all test pass - except known failures:

Site-wide contact form: 120 passes, 4 fails, 0 exceptions
Core filters: 48 passes, 12 fails, 0 exceptions
Poll create: 27 passes, 7 fails, 0 exceptions
XML-RPC validator 0 passes, 0 fails, 3 exceptions

+  if (!$menu) {
+    // We must have a serious error - there is no data to save.
+    watchdog('php', 'Menu router rebuild failed - some paths may not work correctly.', array(), WATCHDOG_ERROR);
+    return array();
+  }

How do we explain the fact that we can get a serious error?

I think this branch of the code sending the message is unlikely to be executed. Perhaps if there is a faulty hook_menu_alter implementation we could get and empty $menu. If the rebuild dies because (for example) a newly enabled module has a parse error, or PHP runs out of memory, we'll never send that message because the code will die before that point. The main (and important) change is that we don't delete the existing router until we have the new one in memory.

So can we remove the chunk from #50 then? If so, please re-roll and I'll commit it.

@Dries -having some version of that check is useful. How about this simplified code?

I committed this to CVS HEAD but I believe this will need to be backported to Drupal 6.

@Dries - the change or the whole patch? The version with the watchdog warning is already in 6.x

I installed the 6.x-dev version and confirmed that the changes in #33 are in the source but this has still not fixed the problem for me. I'm still getting lots of these errors on page loads. (not every load but it seems to happened randomly.)

@designerbrent - please clarify. Your comment doesn't really make sense. You are seeing watchdog errors? PHP errors? The code touched by this patch does not execute on most pages - one page that does execute it is the admin/build/modules page

@pwolanin - sorry. I didn't mean to be so vague.

Every few times I load my site, I get close to 301 error messages that state the following:

# user warning: Duplicate entry 'admin/content/node-type/featured-photo/fields/field_flickrid/rem' for key 1 query: INSERT INTO menu_router (path, load_functions, to_arg_functions, access_callback, access_arguments, page_callback, page_arguments, fit, number_parts, tab_parent, tab_root, title, title_callback, title_arguments, type, block_callback, description, position, weight, file) VALUES ('admin/content/node-type/featured-photo/fields/field_flickrid/remove', '', '', 'user_access', 'a:1:{i:0;s:24:\"administer content types\";}', 'drupal_get_form', 'a:3:{i:0;s:27:\"_content_admin_field_remove\";i:1;s:14:\"featured_photo\";i:2;s:14:\"field_flickrid\";}', 127, 7, '', 'admin/content/node-type/featured-photo/fields/field_flickrid/remove', 'Remove field', 't', '', 4, '', '', '', 0, 'sites/all/modules/cck/includes/content.admin.inc') in /path/to/drupal/includes/menu.inc on line 2366.

Now that I'm reading back through the full issue here, I'm thinking that it is not the same thing. I came to this issue from #238760: menu_router table truncated and site goes down which seemed close to the problem.

The only way I've found to fix the problem is to rebuild the menu's via the devel menu.

The menu's load every time, I just end up with these huge error messages.

I'm running Drupal 6.x-dev, but the same thing happened with Drupal 6.2. Apache 2.0.61, PHP Version 5.2.6.

If this is totally off base, please let me know and I'll be happy to move the issue elsewhere.

Thanks!

Well, if the same thing happened with 6.2, it cannot be due to this patch, which was committed after 6.2. So, perhaps you should open a separate issue, or look for another open one that's more precisely about the problem you're seeing.

@pwolanin - Ok.. sorry to clutter things up. I obviously misread this. I found one that matches closer #246653: Duplicate entry in menu_router: Different symptoms, same issue

Hmm, actually I think the patch that went in wasn't ideal - the if() where I put it misses the 2nd line that will kill the menu system, which is where we set the menu masks. Something like the D6 patch might be better - we can certainly remove the watchdog call. Or, we could just not check $menu, since be moving the delete query to after all the hook_menu invocations we've eliminated pretty much all the problems that were commonly occurring.

Hi,
I encountered this issue (or what seems very similar) today for the first time on D6.3 install.
My errors (a lot of them) are here in case there's anything potentially revealing about them: http://drupalbin.com/2637 (also attaching a .txt file, whichever's easier).

The errors showed up on /admin/content/types after I added a radio text widget to a new CCK content-type.

I'd like to follow this thread in case there's a patch backported to D6.3 -- or if you think it's not the same issue, I can move my issue elsewhere if this isn't the place (please let me know if you think that's the case).

All the best!
:) Scott

@scottrigby - this appears to be the opposite problem. The router is not cleared during the rebuild. There needs to be a separate issue for this, since I think it's a different bug you're seeing and in #58.

Hi,

So if I understand the status of this issue : A patch has been commited but it do not completely resolve the problem, isn't it ?

I have 6.3 version of drupal and I still have this problem of duplicate entry in menu.inc on line 2371.

@zmove - there is minor cleanup around the patch committed, which is why this issue is open. There seems to be a different bug, so someone should open a new issue and link to it

Leaving this issue to D7, but treating it as D6 in a separately issue #289618: menu_router table truncated and site goes down: Transaction or Lock required for D6 referencing this one.

#289618: menu_router table truncated and site goes down: Transaction or Lock required for D6 was a duplicate, in which arhak believe that this was not solved for D6.

Sorry Damien, but I'm stating that the issue is present on 6.3 on the third sentence of the issue #289618: menu_router table truncated and site goes down: Transaction or Lock required for D6
Please read carefully the second and third sentence below

I'm posting this here because the original issue #238760: menu_router table truncated and site goes down was moved to D7 branch (then back to D6, then to D7 and over and over again...).
I think that is incorrect because the D6 issue was fixed and it doesn't appear as such in any place.
Although, I don't think the issue was fixed for Drupal 6.3 at all.

May I reopen that issue for 6.3?
(I won't if you agree it's just a duplicate or whatever else, but I won't agree)

in reply to http://drupal.org/node/289618#comment-950417

With that patch, duplicate entry errors on menu_router can *only* happen when two requests try to concurrently rebuild the menu table and fall inside the (shorter) window of vulnerability.

a shorter window of vulnerability is still a window and thus a vulnerability
How many administrators are allowed by Drupal?
If the answer is "only 1" then we can rely in a shortened window and a workaround documentation for exceptional cases.
But two or more administrators might be doing different stuff and thus causing a race condition.

The process here, for Drupal core, is to fix issues affecting several branches in the *same* issue ticket.

That goes fine for the discussion about the issue as I stated on that issue of mine, but I also think that another opened ticked should exist to avoid forgetting for instance to backport it or whatever would be the final answer to the other branch.
That's why I don't agree with

Having several separate tickets for the same issue is the best way to forget to fix an issue in one of the branches.

but if it's that the way you go...

Also, our process involves fixing the issue in the development branch (today Drupal-7), before backporting in order to always have an up-to-date development branch.

How do you keep track of what needs to be backported or fixed in older versions if all the issues are marked as duplicates in that branch.
OTOH, think I'm a newbie to Drupal (which is not far from the truth, think I'm another newbie) and I find such problem with Drupal, I go to the issue tracker and search the branch 6.x to see if some body saw that too.
What do I find then?
Only duplicates.
Why?
Because patches, fixes, and everything else is in D7 branch. Which I won't look for because I'm not a HEAD developer.

Two things:
- I strongly believe that being the same issue should be discussed one place, but having to possible different final solutions must be different issues. Once the fix is done for 7.x there will be a D6 issue pending for backporting or whatever it's own final solution would be (even a "don't fix")
- The issue wasn't solved with neither of those patches. I've being working with Drupal 6.3

Do you work with two browser. I guess you do, so do I. Now you know how I run into this race condition from time to time. Also, when working on any system that uses database one test realized before starting working is support for suddenly database server shutdown. Drupal 6.3 doesn't pass this test because the lack of transaction and locking.

The process is not something you can or should discuss, at least not in the middle of an issue ticket.

I'm really sorry. I wasn't aware who had the truth. I just said what I thought was common sense. Look above this thread and tell me if this tracking system tells you for sure whether this is an issue present in 7.x, 6.x or both.
I always saw the issue tracking system as a tool which has that brief view on top letting me know quick and without reading the whole discussion the status of the branches. Allowing me also to query the health of a particular branch. If I find a cleared branch there is nothing more to talk about it: it's bug free! tested and approved by the community. Is that right?

I'm marking this one as a duplicate. Please don't reopen. If your problem is the same as #238760, it should be discussed there, regardless of the current version of that ticket.

My issue starts stating that is the same issue for a different branch, I also explain how D6 has patches improving it but not solving it. I also state that the discussion should be followed here. I made every issue/forum I found with the same or similar issue be aware of this discussion.
But I tell you again there is an issue in D6 which doesn't appear opened (for 6.3 I'm talking)
How ever searches the tracking system will find duplicated and the "original" ones are for D7. Thus, the issue will be opened over and over for D6, and other people will redirect them here and explain them why it's targeting D7.
Ok, this issue might be. But is a wrong policy for treating an issue queue. Same issues will keep showing up until people can search and found it reported, in the correct branch and with it's correct status, nobody will be hurt if the issues are different for live cycle and all of them point where the discussion must be.

I have no idea which issue to post this on, but I have suddenly developed this issue. I have been on 6.3 for over a week (new site), using the site frequently. No errors. Today, I upgraded to the most recent beta4 of BOTH Date and Calendar modules. And now, getting LOTS of this error. Not sure if it is related.... Also commenting on the other thread....
Maria

subscribing..

The D7 patch is wrong (sorry) it means we may leave the masks empty which will kill a site just as effectively.

Here's a D7 patch to bring it in line with the more correct logic of the D6 patch, but leaving out the watchdog call.

Committed #72 to HEAD, so at least 6.x and 7.x are on equal footing. I diffed the two functions and the only other changes are all related to file and file path being removed.

I've no idea what status to set this issue now. Active? Sounds like people are still having issues.

indeed

I'm still seeing this in D 6.4 quite a bit.

@webchick :

Yes, still have the issue and desesperately wait for a 6.5 version that correct it to put the website in production.

@zmove, designerbrent - this should not happen on 6.4 unless there is some additional error. Can you please check your logs for MySQL errors or anything else of note? Are you sure you are running a clean 6.4 core?

Yes, I'm sure to have a clean 6.4 (if clean mean that I didn't modify core file).

However, I installed several modules, and I get this error on all 6.4 websites I run. But there are some common modules to all these install, but these modules are not unknown (CCK, views, pathauto, admin menu...). Maybe this is related to my webserver config, I have a lot of performance issues with WAMP and Drupal 6, maybe this is the problem. But all my 3 6.4 website have these error (and it's not the same drupal root...

Sorry.. I got to following the wrong issue. The one that is really effecting me is #246653: Duplicate entry in menu_router: Different symptoms, same issue, not this one.

#78, I'm facing the same problems, the only errors I get are all for duplicate entry .... but I never got the error on a live site (lucky me :p), so I guess it's a race condition. I'm using 2-3 browser each with several tabs open while building a site, so there's a lot going on at the same time.

Running fresh d6.6 with views, og, fckeditor, imce, and a few others.

Same issue as #246653: Duplicate entry in menu_router: Different symptoms , not sure which issue to follow for D6 or if any fixes exist.

>Duplicate entry.........key 1 query: INSERT INTO menu_router ....../includes/menu.inc on line 2385.

I get a whole page of these errors all over the place. could anyone tell me if there is an issue for D6.6 that I should be following?

thanks,
Jen

Just another question, until this is resolved. Is this error a critical one? How does it effect things?Can I still run a live site and just turn off reporting errors to page?

it isn't a problem if you turn off error reporting, there's no problem for regular visitors

@g76 don't worry, there won't be data lost and everything is recoverable if the site crashes:
most of the time nothing happens more than lots of duplicated entries,
some times the menu router table might get truncated and the navigation of the site gets broken, at least until D6.4, but I think there were some patches applied which makes more difficult to let this happen, anyway, if it does there are several ways to get the site back an running quickly:
1- having a special page at hand which reconstruct the menu router table
2- manually changing a variable in DB stating menu should be rebuilt
3- having a query at hand which restores the menu entry for admin/settings/performance and then manually demand rebuild menu or cache clearance

The fastest might be having a php code in a separate page which demand menu rebuild, but that's only needed if site goes down, which seems not to be your case until now

Search the issue queues and you will find several maintainers screaming for sites broken (don't know if still happens for D6.6) and their respectively solutions

this one is done.

hate to re-open.. but is this fixed?? i have a Dr6.10 install and my client just reported site whitescreened when trying to submit a node

looking at logs i see 100's of errors (created on submit) similar to:

Duplicate entry 'admin/content/node-type/bookmark/groups/%/remove' for key 1 query: INSERT INTO menu_router (path, load_functions, to_arg_functions, access_callback, access_arguments, page_callback, page_arguments, fit, number_parts, tab_parent, tab_root, title, title_callback, title_arguments, type, block_callback, description, position, weight, file) VALUES ('admin/content/node-type/bookmark/groups/%/remove', 'a:1:{i:5;N;}', '', 'user_access', 'a:1:{i:0;s:24:\"administer content types\";}', 'drupal_get_form', 'a:3:

looks remarkably like this issue - i.e. menu_router handling doesn't appear to work

i realize quite a few posts about this (most seem to be marked as duplicates of this one), can someone point me to the more generic one that says there is still issues with menu_router when submitting content (which, unless this is caused by some poorly coded module???, would seem to imply Dr6 is unusable??).

I'm a newbie (been building drupal for a year). I'm getting these errors and it doesn't matter what module is installed. here is the gist:
user warning: Duplicate entry '_' for key 1 query: INSERT INTO menu_router VALUES in /menu.inc on line 2385.

I'm not sure how to use a patch, etc. but willing to try anything if someone can give me a breakdown as to exactly what to do.
Running Drupal 6.10 on Apache on Unix, MySQL 5.0.67, PHP 5.2.8.

That's not the same issue - it's caused by attempting to insert duplicate entries into your menu_router table, not an empty menu_router table. Please open a new issue for the error.

Marked #333428: Duplicate entries in menu_router table (6.x branch) as duplicate.

Related issue:
http://drupal.org/node/264545#comment-863438

In my opinion problem of truncated menu_router is not caused because of db transaction and locking the table doesn't solve that problem completely.
Here is a case when module_list() returns list of modules, which are loaded in "bootstrap mode".
http://drupal.org/node/496198#comment-1721218
In case when you have module which call module_list(TRUE) it will truncate your theme hooks, menu_router and cause very weird behaviour even with WSODs.
See: #496198: module_list() called with wrong arguments causing WSOD and breaking theme_registry
And this: #496170: module_implements() cache can be polluted by module_invoke_all() being called (in)directly prior to full bootstrap completion
(menu_router_build calling module_implements)

Subscribe

We had a similar issue occur today. The site did not go down, but we were unable to access a number of links. Fortunately, I had a fairly recent backup and was able to move over menu_router. In comparing the two tables, I noticed that every path with more than two "directories" (i.e. admin/build/block) was deleted. The only rows remaining contained either a direct path (admin) or path and one "directory" (admin/build). It is not clear how this problem even occurred.

I've added in dtools function: wsod_rebuild_menu_router_table() (dev version).
It will rebuild the whole menu_router table in case when it's broken (node item is missing).
You can use this code to rebuild that table or download dtools module and run it in emergency mode.

subscribe, massive php errors,

Duplicate entry 'admin/build/views/%/preview/%' for key 1 query: INSERT INTO menu_router (path, load_functions, to_arg_functions,

#97:
#333428: Duplicate entries in menu_router table (6.x branch)

#333428: Duplicate entries in menu_router table (6.x branch)

Wow. I was having this issue. For some reason, it was zeroed out (although the host has been having issues settling down from an upgrade). I just threw the above into a temporary file, uploaded it, ran it. Fixed the whole thing.

Thank you, thank you, thank you.

And yes, I need to upgrade. After a previous admin changed the sites administration password (on the host!) and didn't tell anybody... Grrrrr. It doesn't help that the person who has the email account gets scared when he sees a password reset come his way, even when he knows my email is just sitting above it...