Duplicate entry in menu_router: Different symptoms, same issue

Hey, don't try to pass this one off on me. The error being shown isn't even remotely related to Views. And what's a .odt file anyway? Doesn't look like anything I can read.

I have the same errors (one by path entry !) :

Duplicate entry 'admin/settings/logging/dblog' for key 1 query: INSERT INTO whv3_menu_router (path, load_functions, to_arg_functions, access_callback, access_arguments, page_callback, page_arguments, fit, number_parts, tab_parent, tab_root, title, title_callback, title_arguments, type, block_callback, description, position, weight, file) VALUES ('admin/settings/logging/dblog', '', '', 'user_access', 'a:1:{i:0;s:29:\"administer site configuration\";}', 'drupal_get_form', 'a:1:{i:0;s:20:\"dblog_admin_settings\";}', 15, 4, '', 'admin/settings/logging/dblog', 'Database logging', 't', '', 6, '', 'Settings for logging to the Drupal database logs. This is the most common method for small to medium sites on shared hosting. The logs are viewable from the admin pages.', '', 0, 'modules/dblog/dblog.admin.inc') in C:\www\public\site1\includes\menu.inc on line 2353.

Title changed ...

Duplicate : #248739: Duplicate entry errors - lots of them -> #238760: menu_router table truncated and site goes down

I am having this same issue with the site I am working on. Views is not being used on my site yet so it has nothing to do with it. The error occurs when I submit changes in content types, and sometimes when visiting random administration pages. This is the error below:

# user warning: Duplicate entry 'admin/reports/search' for key 1 query: INSERT INTO menu_router (path, load_functions, to_arg_functions, access_callback, access_arguments, page_callback, page_arguments, fit, number_parts, tab_parent, tab_root, title, title_callback, title_arguments, type, block_callback, description, position, weight, file) VALUES ('admin/reports/search', '', '', 'user_access', 'a:1:{i:0;s:19:\"access site reports\";}', 'dblog_top', 'a:1:{i:0;s:6:\"search\";}', 7, 3, '', 'admin/reports/search', 'Top search phrases', 't', '', 6, '', 'View most popular search phrases.', '', 0, 'modules/dblog/dblog.admin.inc') in C:\Documents and Settings\Aaron Hawkins\My Documents\websites\WaitingForTheStorm\includes\menu.inc on line 2353.
# user warning: Duplicate entry 'admin/settings/admin' for key 1 query: INSERT INTO menu_router (path, load_functions, to_arg_functions, access_callback, access_arguments, page_callback, page_arguments, fit, number_parts, tab_parent, tab_root, title, title_callback, title_arguments, type, block_callback, description, position, weight, file) VALUES ('admin/settings/admin', '', '', 'user_access', 'a:1:{i:0;s:29:\"administer site configuration\";}', 'drupal_get_form', 'a:1:{i:0;s:27:\"system_admin_theme_settings\";}', 7, 3, '', 'admin/settings/admin', 'Administration theme', 't', '', 6, 'system_admin_theme_settings', 'Settings for how your administrative pages should look.', 'left', 0, 'modules/system/system.admin.inc') in C:\Documents and Settings\Aaron Hawkins\My Documents\websites\WaitingForTheStorm\includes\menu.inc on line 2353.
# user warning: Duplicate entry 'search/node/%' for key 1 query: INSERT INTO menu_router (path, load_functions, to_arg_functions, access_callback, access_arguments, page_callback, page_arguments, fit, number_parts, tab_parent, tab_root, title, title_callback, title_arguments, type, block_callback, description, position, weight, file) VALUES ('search/node/%', 'a:1:{i:2;N;}', 'a:1:{i:2;s:16:\"menu_tail_to_arg\";}', '_search_menu', 'a:1:{i:0;s:4:\"node\";}', 'search_view', 'a:1:{i:0;s:4:\"node\";}', 6, 3, 'search', 'search', '', 'module_invoke', 'a:4:{i:0;s:4:\"node\";i:1;s:6:\"search\";i:2;s:4:\"name\";i:3;b:1;}', 128, '', '', '', 0, 'modules/search/search.pages.inc') in C:\Documents and Settings\Aaron Hawkins\My Documents\websites\WaitingForTheStorm\includes\menu.inc on line 2353.

Every few times I load my site, I get close to 301 error messages that state the following:

# user warning: Duplicate entry 'admin/content/node-type/featured-photo/fields/field_flickrid/rem' for key 1 query: INSERT INTO menu_router (path, load_functions, to_arg_functions, access_callback, access_arguments, page_callback, page_arguments, fit, number_parts, tab_parent, tab_root, title, title_callback, title_arguments, type, block_callback, description, position, weight, file) VALUES ('admin/content/node-type/featured-photo/fields/field_flickrid/remove', '', '', 'user_access', 'a:1:{i:0;s:24:\"administer content types\";}', 'drupal_get_form', 'a:3:{i:0;s:27:\"_content_admin_field_remove\";i:1;s:14:\"featured_photo\";i:2;s:14:\"field_flickrid\";}', 127, 7, '', 'admin/content/node-type/featured-photo/fields/field_flickrid/remove', 'Remove field', 't', '', 4, '', '', '', 0, 'sites/all/modules/cck/includes/content.admin.inc') in /path/to/drupal/includes/menu.inc on line 2366.

The only way I've found to fix the problem is to rebuild the menu's via the devel menu, but that is only temporary since that just clears the menu_router table.

The menu's load every time, I just end up with these huge error messages.

I'm running Drupal 6.x-dev, but the same thing happened with Drupal 6.2. Apache 2.0.61, PHP Version 5.2.6.

Attached is the complete error log the shows each time.

I don't think this is a duplicate as it seems to be different from #238760: menu_router table truncated and site goes down

I am a complete newbie to Drupal and I have experienced this exact problem. Upon loading the admin page (after what seems like several minutes) I get over 200 errors all having to do with user warning: duplicate entries -- every single one related to inserting values into the menu_router table.

After reading this thread, I went ahead and disabled and removed the Views module (I had installed Views 6.2.beta), truncated the menu_router table and then rebuilt it using a snippet of code I found, the problem went away.

Besides hoping to understand the error, my question is, why would loading the admin dashboard trigger all of these insertions in the menu_router table in the first place?? What is the trigger for that?

It does not seem to be locking the table when doing this. It seems to be a race condition. I did a little test inside of the menu.inc file

Line Number:2274

When the above code ran via me opening a whole bunch of tabs of the site, clicking various links the results were strange...

before delete 89 after delete 2
before delete 13 after delete 2

So to me it looks like while one load is creating the menu another load is tearing it down and rebuilding it.

This is just my observation, I could be wrong.

Until this bug is fixed you can use this small hack, that help me out. But you must use MySQL since other dbms doesn´t support "REPLACE INTO".

Drupal 6.3
Replace line 2356 of includes/menu.inc in the function _menu_router_build() {
db_query("INSERT INTO {menu_router}
with
db_query("REPLACE INTO {menu_router}

Dirty, but it works!

#15 works great! dirty indeed but it works...

so what's causing it?

Is it a race condition? I get it more when I insert menu_rebuild() into a module that I'm developing, to make updating the menu quicker.

Perhaps menu_rebuild() is getting called at roughly the same time for two different users, so two processes are trying to insert the new menu data into the table at about the same time, leading to duplicates?

I'm not using views either.

This is a duplicate of #238760: menu_router table truncated and site goes down. A fix was already released in 6.3 that should mitigate the issue.

This still can happen in 6.3 too, if menu_rebuild() is called by two site users at the same time.

I suspect it happens when two different users cause the menu_router table to be re-built at the same time, and it's a race condition caused by the lack of table locking. It appeared on my site much more often when I was calling menu_rebuild() on every page load (for lazy development reasons!).

I think what is happening is:

User 1: DELETE FROM {menu_router}
User 1: Do some INSERT INTO {menu_router} items (1,2,3)
User 2: DELETE FROM {menu_router}
User 1: Do some more INSERT INTO {menu_router} items (4,5,6)
User 2: Do some INSERT INTO {menu_router} items (1,2,3)
User 2: Do some more INSERT INTO {menu_router} items (4,5,6) => ERROR duplicate rows!!!

Replacing "INSERT" with "REPLACE" thus works around the problem, but if it is a race condition the menu routing might just end up slightly wrong depending on the order of calls to the query on line 2356.

To make this more likely, add menu_rebuild() into a module's .module file, so it gets called a lot. Then get several users to access the site at the same time.

Not sure about the duplicate link, as I _think_ these are different problems, albeit with the same section of code:

#238760: bug caused by code exiting while database state is invalid (between deleting old menu data and inserting new), so site's menu disappears.
#246653: bug caused by normal site use, when two processes try to rebuild the menu, site menu remains, but users see many error messages.

I think the fix for #238760 requires transactions, or equivalent, so menu table is always useable.
I think the fix for #246653 requires table locking, or equivalent, so only one process can update it at a time.

So not quite the same problem?

These are perfect duplicates, please comment on the other issue rather than on this one.

During the discussion in #238760 two things were committed: first, a stop-loss solution in case a module performed badly and the rebuild operation failed (hence the title), second a mitigation for the race condition you are describing.

The fix that was release in 6.3 is only a mitigation. The real issue cannot go away until Drupal has a locking framework, ie. a way to prevent two batch operations (like menu_rebuild) to happen at the same time. Anyway, in 6.3 you should only see that in extreme conditions (such as a broken module that calls menu_rebuild at every page load...).

Leaving this issue to D7 at #238760: menu_router table truncated and site goes down
but treating it as D6 in a separately issue #289618: menu_router table truncated and site goes down: Transaction or Lock required for D6
For the whole discussion, please refer to #238760: menu_router table truncated and site goes down
To know why this issue is in fact a duplicate of #238760 refer to #289618: menu_router table truncated and site goes down: Transaction or Lock required for D6

I don't think this is duplicate - this is a totally different (in fact opposite) problem from #238760

Yes it is the same issue.
Lets see
if you rebuild the menu with devel module and the problem goes off (if you ever get with this issue it will be appearing over and over almost for sure) then we are talking about the same issue.
Different symptoms, same issue
Once the menu fail to be rebuilt different symptoms might come up.
1- site goes down / administrative task are unavailable / some URLs are unavailable
2- site keeps complaining about duplicated keys in menu_router table

The cause is a timeout or whatever might interrupt a non-Transactional manipulation of the data (when I run manual test looking for robustness I suddenly shutdown the MySQL server), this is (IMO) a critical flaw of Drupal 6 (don't know what about 5 but I think it has some sort of table locking)
1- If the menu doesn't get rebuilt then the menu_router table might be almost empty causing the unavailability of the site
2- If the menu get partial rebuilt but isn't flagged as totally rebuilt the site might be usable and in further request it will attempt to rebuild it, but as it is partially built then it will complain about duplicated keys.

subscribing

I don't think this is duplicate - this is a totally different (in fact opposite) problem from #238760

Please, this issue was originally posted for 6.x
if this isn't a duplicated, then find, mark it as such, but don't change it to next generation of Drupal, the issue is present in 6.x and as such must remain until marked as fixed, won't fix, or whatever. Once the issue is marked maybe as "won't fix" then incoming issues might be marked targeting 7.x, but for now, this issue is present in 6.x, it's a major issue and as such deserves to be answered (in the worse case as "won't fix")
I won't mark it as duplicated again because you don't agree, but I'm returning it to the 6.x where it belongs to.

See what happened on #238760: menu_router table truncated and site goes down jumping to 6.x and to 7.x over and over, it's not traceable.
Please read #289618: menu_router table truncated and site goes down: Transaction or Lock required for D6 again and try to agree with me why this issue must be kept on both 6.x and 7.x but as different issues (for tracking purpose) and maybe both referencing a unique discussion until they split up (sooner or later one of them will be marked as fixed or won't fix while the other one will not)

@arhak - the process is that bugs are fixed in the newest version before backporting. The menu code in 7.x is very much the same as 6.x, so whatever bug is present should be addresed in 7.x first.

What I'm having a difficult time understanding is that there is a query to delete all entries from {menu_router} before any inserts are attempted. So, is that deletion query failing for some reason?

the process is that bugs are fixed in the newest version before backporting. The menu code in 7.x is very much the same as 6.x, so whatever bug is present should be addresed in 7.x first.

Well, I disagree with pointing every issue to D7 because there are proposed and tested patches for D6 (follow the references in my previous comment) and then the issue is changed back and forward every time someone think has the solution for D6 and later someone change it back to D7 because the patch is rather a improvement but not a solution.

What I'm having a difficult time understanding is that there is a query to delete all entries from {menu_router} before any inserts are attempted. So, is that deletion query failing for some reason?

The deletion always succeed. The insertion is what fails because of timeout. Now, if the menu is half built and some change (module enabling/disabling, accessing modules pages I think causes it also, and other situations) triggers the rebuild menu task, then there will be duplicate entries, of course. Now I don't remember when I looked into the code, but for some reason, rebuilding the menu with devel most of the time succeeds, deleting the whole menu_router table and recreating it again, but in other situations a rebuild is attempted without deleting the table first.
There are proposed patches that improve this awful situation in several ways without providing a definitive solution:
- the first one reorders the code so the computation is done first and the delete is near the insertion (originally the deletion is first, then computation and later an insertion loop, the time window for failing such task is huge)
- the second one remove the deletion attempt and uses ALTER instead of insert (valid only when menu is augmented, wrong otherwise)

please, follow the pointed issues on my previous comment

I have no idea which issue to post this on, but I have suddenly developed this issue. I have been on 6.3 for over a week (new site), using the site frequently. No errors. Today, I upgraded to the most recent beta4 of BOTH Date and Calendar modules. And now, getting LOTS of this error. Not sure if it is related.... Also commenting on the other thread....
Maria

@ahak - the already comitted patches (6.x and 7.x) move the deletion code to close to the insertion code. Your analysis does not make sense - the code should always delete all entries before attempting any insert. If only some subset was inserted, I'd expect you to 404 on the subset of pages corresponding to the missing items.

Hi guys, i had this same problems, i fixed commenting in my module the code>

/*
function artigo_init() {
// menu_rebuild();
}*/

then it work fine!

Doing a menu rebuild in a hook_init() function would (at the least) make the site very slow - why was that code there?

Let's close this one for good. This is a duplicate of #238760: menu_router table truncated and site goes down.

We *know* that a race condition can still happen. This will stay this way. menu_rebuild() is a costly non-transactional function should be called as few as possible, and clearly not on every page load.

If you use 6.6 the hack mentioned in #15 must change line 2370 in includes/menu.inc.

Number 15 also fixes it on D.10.

Sorry, I'm not a Core expert, but....will there be some change in 6.11 to account for this, or should I be managing my DBs to lock tables in some way, or...how do folks recommend avoiding this? It's a pretty ugly error, and lots of folks are experiencing it:

http://drupal.org/node/352978
http://drupal.org/node/333428
http://drupal.org/node/248739
http://drupal.org/node/375327 [ which indicates that just refreshing the page fixes the problem - not hacking core ]

subscribing - want to know when this is fixed (backported to D6)

I fail to see how this can be closed. #238760: menu_router table truncated and site goes down is marked as fixed and there are quite a few comments saying it is a different issue. The issue discussed in this thread is not fixed. I upgraded a production site to Drupal 6.10 yesterday and am seeing this in certain situations and it is ugly.

When I save a contact record in CiviCRM 2.2.0 I see this without fail. I think this is because the record can contain user information that might have changed, so the menus might need rebuilding to reflect any updates. There's a call to menu_rebuild() in some groups code that looks to depend on a user account changing but I can't read the code well enough to know for sure...

Saying that menu_rebuild() should not be called so often doesn't actually fix the problem! ;-)

So, can someone who understands what's happening please comment on the following?

• #15, above. Is it a safe workaround? If it is safe then why isn't it good enough to commit for Drupal 6.11 instead of leaving a known race condition in the code? ;-)
• #238760: menu_router table truncated and site goes down #83 seems to suggest that turning off error reporting to the screen is a good workaround. So, I just go to admin/settings/error-reporting and turn off errors to the screen? It's that easy? And recommended for a production site? I've read the comment on the error-reporting page but I'm wondering what people do on real systems... :-)

I just want to make an informed decision.

Thanks...

peace & happiness,
martin

* #15, above. Is it a safe workaround? If it is safe then why isn't it good enough to commit for Drupal 6.11 instead of leaving a known race condition in the code? ;-)

REPLACE is MySQL-specific. Plus, this basically only hide the error, it doesn't change the fact that two menu_rebuild() are running at the same time. The true solution is to implement a soft-locking framework, and that's being worked out in #251792: Implement a locking framework for long operations.

* #238760: menu_router table truncated and site goes down #83 seems to suggest that turning off error reporting to the screen is a good workaround. So, I just go to admin/settings/error-reporting and turn off errors to the screen? It's that easy? And recommended for a production site? I've read the comment on the error-reporting page but I'm wondering what people do on real systems... :-)

On real systems, like here on drupal.org, we don't have any problems. The main issue is badly written modules calling menu_rebuild() when they should not ;)

Thanks Damien. I've disabled errors to screen and I'm not seeing any problems.

subscribing...

subscribing

i've got what I believe to be the same problem in my 6.10 production site now. its causing me a great deal of slowdown for admin tasks - so much so that I can rarely load my admin/build/modules page without the script timing out.

I tried the fix in #15 on my site but it didn't work for me.

HELP!

In my case, a similar massive error message appeared when upgrading the views module to views-6.x-2.4.
The fact is that I was editing page content in parallel, in another browser tab, while upgrading the module.

Here's my workaround. Hope it shows some hint:
1) I deactivated completely the views modules
2) The nasty error message disappeared and the admin site load time returned to usual values.
3) I reactivate the views module, and again I was working in parallel editing some content -> OOPS! THE ERROR AGAIN!!!
4) Return to point 1) and 2)
5) This time I reactivate the views module making sure that I was not running any parallel operation in my drupal site.
6) The error message disappeared completely, and the site and views module worked as should once again.

I tried this. It seemed to work for a while, but after I logged back out and back in, the errors reappeared. I'm running 6.10 and the latest modules.

#238760: menu_router table truncated and site goes down is closed but doesn't fix this. I know some people *knows* about this race condition but there is no intention to fix it.

We have hit this bug sometimes and people are seen this on our sites when trying to delete 2 nodes with menu entries at concurrent page request (ie: 2 firefox tabs).

This patch seems to solve it for me and maybe for some of you too. Is there any downside in fixing it this way?

Code cleanup.

I applied the patch in #47 to a site that is seeing this issue.

It has not been there for a long enough time to be sure that it is gone forever, but will report back when I have something one way or the other.

Drupal 6 doesn't require the LOCK permission in MySQL anymore. Plus we have a db_lock_table() function that need to be used for that.

The true solution for this issue is #251792: Implement a locking framework for long operations, as I already stated above. Marking as a duplicate of this one. You will help everyone by reviewing the patch there.

The locking framework is a long term solution for Drupal 7.x.

It is the proper solution.

For the meantime, sites on Drupal 6.x that use modules that aggressively rebuild the menus will be around for a while, so it is best to keep this open, since it offers sites that see this issue solved via the above patch.

@kbahey actually the locking framework is intended for drupal 6

I understand why my patch is not the best solution but it can be used as a temporary workaround for people seen this problem often, because the locking framework issue is a year old and seems this bug is more easy to hit than you should expect.

I have a similar bug when updating or submitting new content. I'm testing the fixes from simplest to most complex (simplest being deactivate/uninstall views). I have uninstalled views per the instructions in #44, and things seem to work so far. I will post if anything else arises.

EDIT: #44 does not work. Will try a few of the other fixes for more clarification.

Thanks Drupal community for your help on this matter!

#47 worked for me. Thanks!

The patch in #47 eventually worked for me. It seemed to take a few days for the errors to stop occurring. This after clearing the cache.

same here, will try the patch now

I'm with kbahey in #50. ie. it would be great to have a solution based on db_lock_table(), in the meantime, until the locking framework is finished for D7, and then backported to D6.

subscribe

Just my 2 cents:
I am all in favour of a temporary solution, to fix this issue until the locking framework is finished.

Is there a better temporary solution than the one in #47? I'm not sure from reading #251792: Implement a locking framework for long operations. Also, I posted a script in that issue to rebuild the {menu_router} table in case yours gets trashed, which happens sometimes on my D6 site (presumably b/c the locking isn't good enough).

subscribe

subscribe

I'm having this same problem. I'm fairly new to Drupal, so please forgive my ignorance, but is there a specific method I need to use to apply a patch? I'm using Drupal 6.12. Is the patch in #47 appropriate for me to use?

@renee & tessmonsta, is the patch in #47 still working for you?

The patch works for me. It removes the error message.

As I can see, it does not reduce the queries needed for menu rebuild, or the time needed for each of these queries. (which can be quite long, see http://drupal.org/node/302638)

So far, I did not notice any evil side effects.

@aktary:
I am sure there are better ways, but I usually try to find the place in the code, and do it manually. Then I indicate where I applied the patch, with something like:

// PATCH FROM http://drupal.org/node/246653#comment-1482178 >>>>
// REMOVED  .... (old code)
// ... (new code)
// <<<< PATCH END

This helps to find these places again when I do a software update.

Subscribe

Subscribe

Subscribe

I seem to have beem able to resolve this problem for my site using the patch (as it exists so far for D6) from #251792: Implement a locking framework for long operations.

Note that the patch implements locking for menu_rebuild(), but I had to add an additional lock specifically for menu_router_build() because both Panels and Admin Menu call that function directly and can otherwise trigger the "duplicate entry" collisions.

I tried loading the Modules admin page and some Panels admin pages repeatedly, and couldn't get any errors to happen, whereas before I'd get pages and pages of errors without much effort using any of those modules.

PeteS: which patch from #251792: Implement a locking framework for long operations are you using? Are you using #65?

There's actually one more posted after that, #71, which I believe is the one I used

subscribing.

Proper duplicate link:
#333428: Duplicate entries in menu_router table (6.x branch)

Read more:
http://drupal.org/node/238760#comment-950563
http://drupal.org/node/238760#comment-1379898

subscribe

Any backport for 6.x?

subscribe

subscribe

Subscribing

I'm on Postgres and seeing this issue a lot, especially when editing two views in two different Firefox tabs at a time. It inserts inline into the views ajax interface and makes it really hard to navigate away.

I get this about every 2 or 3 minutes when I'm actively developing the site in admin.

I've solved it myself by putting the menu rebuild into a transaction (using raw BEGIN/COMMIT statements in _menu_router_build()). This works fine for me.

Like others, I see this as a major enough bug to need a fairly rapid fix in 6.x.

I turned off errors to screen just to get this out of the way, but I'm pretty sure this is a widespread and common issue in recent versions of 6.x.

hmm. This completely screws up mysql replication

ps. this can be worked around by adding slave-skip-errors=1062 to your mysl config file on the slave

Sorry for spamming everything..

here's a rewrite proposal for menu_router_build,
Rewrite the function menu_rebuild
(scroll to the end to find the latest patches)

The patch does not yet incorporate the locking framework, but it should be easy to add that later on. Plus, the number and chance for errors caused by concurrent requests will be much reduced even without the locking framework.

I haven't checked the code but looks to me like XML Sitemap Menu is rebuilding the menu every time cron runs. Could this be a problem too?

hotblack's solution works for me!

(http://drupal.org/node/246653#comment-922486)

Subscribing. Been having this problem since setting up multi-site. Slow speeds on both sites, and modules page displays many of these errors quite often

Shouldnt we have a solution if we replace the query of "insert into" by "insert ignore into"? I have not tried that yet but just putting forward my idea... the problem with me is that at time i see the error and at times not.

INSERT IGNORE
or INSERT IF NOT EXISTS.
or INSERT ... ON DUPLICATE KEY UPDATE
or REPLACE INTO
etc

Which of those work in all DB engines?
If we don't find a cross-db solution, maybe it should be abstracted away into drupal?

I used REPLACE INTO and I have not seen the issue arise since. I have been working with three new modules too and have been reloading the modules page a lot because of it. In the past I definitely would have seen it by now.

Nice catch.. Thanks.. I'll look into this further..

subscribing...

a nice drupal quirx, if it gets worse I'll try #15

Cheers!

using Drupal 6.13 on mysql - same problem here - will try #15, although - don't hack core...

subscribing... guess I'll end up using #15 too, as the site goes online tomorrow

I just noticed this duplicate entry problem on one of my sites running D6.13. At the time I was upgrading a few modules to their latest versions and I whenever I upgraded a module (I upgrade modules one at a time) I would see hundreds of these duplicate entry errors in the log.

I noticed that I had devel 6.x-1.16 installed, but it was disabled. I uninstalled the module and then installed devel 6.x-1.18. And since then I have not seen any more duplicate entry errors in the log after that.

I will update if I see them again, but for now it is looking good.

subscribe

If you want to reproduce this error, I found a way to get it consistently. If you have Views installed, edit one of your views. Press the Save button twice. Probably you'll get the error if you press the Save button twice on most content forms.

It's obviously a race condition, so apart from the solutions posted above, there are a couple of others. One is to reject any form that's been submitted more than once. Another would be to implement a module that disables the Save/Submit button on all forms the first time it's pressed, so it can't be pressed again. Unfortunately, the second solution could cause other problems. For instance, when I save views, I sometimes get an AJAX javascript error dialog and have to press Save again.

There has been a decent mention here of table locking for slow queries. Could it be beneficial in swapping this table to innoDB, as this supports row-level locking?

Actually in this case table-level locking is what you want. It's deleting all the rows from the table then reinserting them all. The error occurs when two cache-clears occur at the same time, and so there's two parallel processes doing the delete and reinsert. Usually what this means is that the second one will attempt to insert a row that the first has already put back. A full table lock would means that although the second cache-clear would have to wait until the first (making it take quite a bit longer), they wouldn't be working on the same data at the same time.

A row-level lock doesn't help, because all the rows are being purged.

We have this problem even on Postgres, but the solution is to simply take out an exclusive write lock on the menu_router table. It slows things down when you're working with multiple cache-clears (eg. the views UI on multiple tabs), but it stops all the errors, and the possibility of table corruption.

The main issue here is that Drupal 6 dropped the requirement (I believe) for Drupal to have the LOCK TABLE permission, therefore core doesn't rely on db_lock_table() working. Patching core to lock this table would require drupal to have this permission. Personally I think it should *try* to lock the table, but if it fails, just keep going anyway.

It's deleting all the rows from the table then reinserting them all.

Which is not necessary at all.

Well actually it is necessary as part of the menu rebuilding process, but maybe rebuilding the menu is not necessary in a lot of the circumstances where it's done.

Nope, it's not. We could just as well scan the old table contents, see what has changed, and write the changes to the database, as proposed in #512962: Optimize menu_router_build() / _menu_router_save(). It would be even easier in D7 than in D6, as it already happens all in one function. Just I'm not gonna make it myself.

It would still be a good idea to have concurrency control, but it won't be that important as it is with the current bulldozer approach.

Ah yes, in an ideal world where Drupal went for efficiency, yeah that would be a nice option. It would require a fair bit of memory though, as it'd need to do an in-memory compare. I suppose it's not that big though.

I wish this could apply to the node_access rebuilding process too though, it's such a pain to change access control rules and have to rebuild the permissions and have most of your content offline for hours.

Ah yes, in an ideal world where Drupal went for efficiency, yeah that would be a nice option. It would require a fair bit of memory though, as it'd need to do an in-memory compare. I suppose it's not that big though.

That's what a lot of people said in the other issue. But it's not a valid argument: We already have all the information in memory!!
http://api.drupal.org/api/function/_menu_router_save/7
I mean, how do you want to refill a deleted table if you don't have the information in memory?

And, if you look at the other functions involved, you will see that more stuff of the same magnitude is in memory during that rebuild process. And all passed around (and thus copied) as a by-value array arguments.

A diff-based table contents replace is easily possible with no or little additional memory use (compared to the bulldozer way):
Do a SELECT loop on the menu (with one select query), and while doing that, remember what needs to be deleted, inserted, updated. Then execute.

To further reduce memory in special cases (if you are really serious about it):
- If the deletion queue grows too big, go back to bulldozer strategy.
- Delete immediately, don't queue deletions. I'm not sure if that would disrupt the SELECT loop, so maybe not.

If there's no guarantee of table locking, then it's Flag Time.

So, why not do this...?

function menu_router_build() {
  while (variable_get('menu_router_rebuild_began', 0) != 0) {
    usleep(250000); // wait for 1/4 second
  }
  variable_set('menu_router_rebuild_began', time());

  // rebuild the menu as usual

  variable_del('menu_router_rebuild_began');
}

Well, you may say, that variable might not ever get unset, and menu_router_build() might just keep spinning until the script times out.

To mitigate against this rare event, menu_cron() could look for the flag and if it was set before the previous call to menu_cron(), unset the variable and call menu_router_build() as a precaution.

Sounds weird, but I'm really looking for something reasonable here.

Explain to me why this might not be a good approach.

All of my modules are completely up-to-date and I continue to receive this error. Subscribing.

subscribe

I copied a site from one server to another. I made an exact copy and only changed settings.php.

On one server it is slow, showing errors and I can repeat the bug by changing a view, click save and while it is reloading browse around the site (since reloading takes over a minute). I use another browser to click around the site while the views page is reloading. This itself is not painstakingly slow, but it does generate the ' duplicate entry' errors.

On the other server I cannot repeat the bug. As a matter of fact, the page reloads so quickly I barely have time to browse around other pages.

Also the database seems to be slow 'randomly' while visiting the site it might be slow or even report not to be able to connect to the database server.

my solution seems simple enough: move to the another server. But I will never know why?

What is the difference between the servers?
Both are shared hosting. One is expensive and generates the bugs. One is dead cheap and works fine.
The expensive one where drupal shows this bugs is running FreeBSD 4.10, php 5.2.6, Apache/1.3.37, mysql 5.0.
The cheap one where drupal does not show this bugs is running a linux 2.6.9 probably fedora, php 5.2.3, apache 2.0.54 and mysql 5.0.

Could it be that the one database is just responding so much faster that there is no time for this problem to occur?

I found one thing that helped (after applying the patch) was to truncate the cache_menu and menu_router tables manually- it seems flush caches wasn't cleaning them properly.

#15 worked for me as well.

subscribing

Subscribing.

I have the same problem as PeteS in #65. I already have the very latest version of the D6 patch from #251792: Implement a locking framework for long operations (http://drupal.org/node/251792#comment-2125124), but I still get this problem in _menu_router_build().

@111: if you are using Admin Menu then changes to some of its internals have removed the call to menu_router_build() that is causing the problem (see #276751-42: Allow to alter/customize/add links in administration menu). However the problem won't I think be fixed in the 6.x-1.x branch and work on the new branches seems to have slowed somewhat - the latest 6.x-3.x release (alpha3) was way back last August. But maybe it's fairly stable..

I am using admin_module, but unfortunately the alpha 3 release didn't work for me - more trouble than it was worth. I'll try with the latest snapshot, or failing that I guess I'll try backporting the patch. Thanks gpk.

I am running 6.15 and seeing this issue quite a lot (2 or 3 times per day, but with hundreds of entries in the dblog
Sometimes it is from people having access to the admin menu, but also some from "Visitors".

I am running a site where the core and all of my modules are up-to-date as of 2010-01-29. Last night, I got 400+ errors where menu.inc was trying to update the menu_router table with existing paths. Please let me know if there are any logs or configuration files that I can provide that might help with figuring this out.

@115: Sounds like a duplicate of #251792: Implement a locking framework for long operations. You might want to check that the error messages reported there are the same as what you have seen. The fix is to apply the patch at #224 in that issue, or upgrade to 6.x-dev. Some of the comments around there give some guidance.

Does the 6.16 drupal update fix this issue? (it says it implements table locking for long operations).

The fix from #251792 (which I believe was committed to go into 6.16) did not fix this issue for me on Postgres. I still get the errors just as before.

Using real table locks (db_lock_table('menu_router')) instead of the new locking framework did solve the problem for me though.

So I guess the new locking framework is a bit of a dud...

@118: Strange, it seems to work for people on MySQL. I can't see anything in the patch at #251792-224: Implement a locking framework for long operations that wouldn't work on Postgres but maybe I'm missing something. Did you see the lock appearing in the {semaphore} table during menu rebuilds? Also I think there were 1 or 2 contrib modules that also caused this error even after applying that patch (Admin menu among them possibly), but the latest versions play nicely with the fix.

Right. After posting that comment I went to try and replicate the errors by clearing all cache in two tabs at the same time (using admin menu) but I couldn't fault it.

I have seen the menu router errors though, but only when saving two views at the same time, so I suspect it is a contrib issue. I'll have a probe later on to see if I can find the culprit code in views that might be bypassing the lock somehow. I've only been able to get the errors to come up by chance so far (typical of a race condition), but it is still many pages of duplicate key errors, which was the original problem.

It may require a patch to views to fix, but if it's the locking framework's fault, I'll see if I can patch it.

It's strange that using db_lock_table() works in menu.inc though, it suggests there's a corner case.

Are you using a table lock in the identical place to the new lock acquire/wait in http://api.drupal.org/api/function/menu_rebuild/6? There was discussion in the other issue about whether and additional lock was needed somewhere else, but IIRC the conclusion was that if contrib modules use the API correctly (i.e. only invoke the public functions) then all should be well.

There was also another edge case I thought I'd identified in the other issue (but have not had time to follow up) - the process that gets its menu rebuild blocked may not have its changes to the menu incorporated into the menu router table. It will get the FALSE back from menu_rebuild() so in theory a caller could try the menu_rebuild() again, but it doesn't look like the return value is actually examined anywhere. I don't think this would cause the duplicates problem though, just might mean that something is missing or wrong in the {menu_router}.

Suggest you refer back to #251792 for more info...

Just got this on a 6.16 site. 435 consecutive errors on one page load. Here's the first 5:

    * user warning: Duplicate entry 'admin/user/user/list' for key 1 query: INSERT INTO speed_menu_router (path, load_functions, to_arg_functions, access_callback, access_arguments, page_callback, page_arguments, fit, number_parts, tab_parent, tab_root, title, title_callback, title_arguments, type, block_callback, description, position, weight, file) VALUES ('admin/user/user/list', '', '', 'user_access', 'a:1:{i:0;s:16:\"administer users\";}', 'user_admin', 'a:1:{i:0;s:4:\"list\";}', 15, 4, 'admin/user/user', 'admin/user/user', 'List', 't', '', 136, '', '', '', -10, 'modules/user/user.admin.inc') in /home/arencambre/drupal6/includes/menu.inc on line 2457.
    * user warning: Duplicate entry 'admin/build/modules/list' for key 1 query: INSERT INTO speed_menu_router (path, load_functions, to_arg_functions, access_callback, access_arguments, page_callback, page_arguments, fit, number_parts, tab_parent, tab_root, title, title_callback, title_arguments, type, block_callback, description, position, weight, file) VALUES ('admin/build/modules/list', '', '', 'user_access', 'a:1:{i:0;s:29:\"administer site configuration\";}', 'drupal_get_form', 'a:1:{i:0;s:14:\"system_modules\";}', 15, 4, 'admin/build/modules', 'admin/build/modules', 'List', 't', '', 136, '', '', '', 0, 'modules/system/system.admin.inc') in /home/arencambre/drupal6/includes/menu.inc on line 2457.
    * user warning: Duplicate entry 'admin/settings/biblio/basic' for key 1 query: INSERT INTO speed_menu_router (path, load_functions, to_arg_functions, access_callback, access_arguments, page_callback, page_arguments, fit, number_parts, tab_parent, tab_root, title, title_callback, title_arguments, type, block_callback, description, position, weight, file) VALUES ('admin/settings/biblio/basic', '', '', 'user_access', 'a:1:{i:0;s:17:\"administer biblio\";}', 'drupal_get_form', 'a:1:{i:0;s:21:\"biblio_admin_settings\";}', 15, 4, 'admin/settings/biblio', 'admin/settings/biblio', 'Preferences', 't', '', 136, '', '', '', -10, 'sites/all/modules/biblio/biblio.admin.inc') in /home/arencambre/drupal6/includes/menu.inc on line 2457.
    * user warning: Duplicate entry 'admin/settings/wysiwyg/profile' for key 1 query: INSERT INTO speed_menu_router (path, load_functions, to_arg_functions, access_callback, access_arguments, page_callback, page_arguments, fit, number_parts, tab_parent, tab_root, title, title_callback, title_arguments, type, block_callback, description, position, weight, file) VALUES ('admin/settings/wysiwyg/profile', '', '', 'user_access', 'a:1:{i:0;s:18:\"administer filters\";}', 'drupal_get_form', 'a:1:{i:0;s:24:\"wysiwyg_profile_overview\";}', 15, 4, 'admin/settings/wysiwyg', 'admin/settings/wysiwyg', 'Profiles', 't', '', 136, '', '', '', 0, 'sites/all/modules/wysiwyg/wysiwyg.admin.inc') in /home/arencambre/drupal6/includes/menu.inc on line 2457.
    * user warning: Duplicate entry 'admin/content/comment/new' for key 1 query: INSERT INTO speed_menu_router (path, load_functions, to_arg_functions, access_callback, access_arguments, page_callback, page_arguments, fit, number_parts, tab_parent, tab_root, title, title_callback, title_arguments, type, block_callback, description, position, weight, file) VALUES ('admin/content/comment/new', '', '', 'user_access', 'a:1:{i:0;s:19:\"administer comments\";}', 'comment_admin', 'a:0:{}', 15, 4, 'admin/content/comment', 'admin/content/comment', 'Published comments', 't', '', 136, '', '', '', -10, 'modules/comment/comment.admin.inc') in /home/arencambre/drupal6/includes/menu.inc on line 2457.

@#122 that doesn't seem to belong here
it looks like a custom module issue (speed_menu_router or speed_menu) rather than a Drupal issue

EDIT: sorry, you have a table prefix "speed", right?
then it should be posted over here #251792: Implement a locking framework for long operations since they attempted to solve that issue, and it seems they didn't

Subscribing.

subscribing.

Yeah, we're still seeing it in 6.16 :\

@joshk: when do you see it in 6.16? I've never seen it in a long time...

I see it in Drupal 6.16 too :( Unfortunately, I couldn't tell you exactly what the circumstances were that led up to it, but I do know my logs are stuffed full with this error, still.

Unfortunately this has been difficult to reliably reproduce. I do seem to notice it appear after a cache clear (but not everytime).

I am working on the same site that joshk mentioned in #126. I should mention it is based off of pressflow. For completeness, I attached a list of enabled modules on our site.

@mrfelton: I may know why I don't see it, because I am using Pressflow Drupal 6.16 and have turned these tables to InnoDB. Since the Pressflow version of the menu_rebuild() code uses the application-level locking framework from #251792: Implement a locking framework for long operations, but also wraps the actual delete, etc. in a transaction, the problem shouldn't arise.

I think doing a LOCK TABLES/UNLOCK TABLES also works, since that's the solution I was using prior to 6.16. But apparently that is no longer a requirement for Drupal installation, so I guess that's out of the question.

Maybe we should look more at the patch from #512962: Optimize menu_router_build() / _menu_router_save(): if the menu_rebuild() database operation makes significantly less queries, than it is less likely to get into a race condition, at least I would hope. But, I've been wrong before (as the comments in the first part of that issue will evidence).

@jonskulski: Thanks for the list. It was very helpful. I think, but am not sure, of course, your problem may be admin_menu. I believe that at least some versions of that module call the private function _menu_rebuild(), instead of menu_rebuild(), due to limitations of the Drupal 6 menu system. Thus, the locking framework gets bypassed.

Could you perhaps try disabling admin_menu on a test version of your site, try stress-testing menu_rebuild() by saving some CCK nodes, modules page, views, etc., and see if you get the errors?

I think this would be helpful knowledge for the whole community.

@EvanDonovan - ahh. admin_menu. Completely forgot about #111 - #113! I see there is a new alpha4 version of admin_menu now. I'll give it a try (alpha3 didn't work for me)

Ah, I'm using Admin Menu, too.

EDIT: Oops, I am on Admin Menu 6.x-1.5 on the site that had the above error, so not sure if the Admin Menu 3.x issues matter for me.

Current status of Admin Menu looks like some of the issues in 3.0-alpha3 may be resolved. Some of them looked they were fixed in core: http://drupal.org/project/admin_menu. Perhaps sun would be able to give us a more accurate update on status?

Correction to #132 -same as poster in #133... I'm also using admin_menu 6.x-1.5 on the site that has this issue, so I don't think that issue is the problem here.

We migrated to Pressflow 6.16 and converted the database to InnoDB and this seems to have alleviated the problem.

@chrism2671: Yeah, that should work for most people. Of course, if there is a vanilla 6.16 issue, that still needs to be addressed (somehow).

It could be related to multiple requests during the same window of time during the rebuild. We seem to be getting this more rather than less with the update to 6.16, but we had more developers working on the same db installation. Without looking at the code, are the flags are set in the best atomic way possible? Relying on the $config variable to start will not fix the issue as the bootstrap process of two different requests will share the same $config flags, but surely this must have been considered.

Links for admin menu are common in the errors, but about 10% are related to other modules or core router items.

@131 et seq: there is no private function _menu_rebuild() - you are probably thinking of menu_router_build() which as described in #251792-196: Implement a locking framework for long operations is the culprit in admin menu, and is not protected by the lock (which is used in http://api.drupal.org/api/function/menu_rebuild/6). menu_router_build() is called in the 6.x-1.x branch of admin menu here: http://drupalcode.org/viewvc/drupal/contributions/modules/admin_menu/adm.... This is only fixed in the 6.x-3.x branch of admin menu, which is still awaiting some core bugfixes before a formal release http://drupal.org/project/admin_menu.

@138: "Links for admin menu are common in the errors, but about 10% are related to other modules or core router items"
If you are using the 6.x-1.x branch of admin menu then it is probably causing all those errors. If necessary you could probably work round the problem by adding your own lock_acquire() around the menu_router_build() call in admin_menu.inc.

@gpk: Thanks for the correction. Sorry, all, for the error. It's been a while since I looked at that code.

If we're going to resolve this issue, people are going to have to start attaching fairly complete logs of the database errors they are receiving, and what modules they have enabled. Otherwise, it's just guesswork.

+1. On and off for me.

I'm using drupal 6.16 and admin_menu 1.x branch and I get lots of these every once in a while(when two admins access the admin area):

Duplicate entry 'admin/content/node-type/photograph/fields/field_photograph_sourc' for key 1 query: INSERT INTO menu_router (path, load_functions, to_arg_functions, access_callback, access_arguments, page_callback, page_arguments, fit, number_parts, tab_parent, tab_root, title, title_callback, title_arguments, type, block_callback, description, position, weight, file) VALUES ('admin/content/node-type/photograph/fields/field_photograph_source/remove', '', '', 'user_access', 'a:1:{i:0;s:24:\"administer content types\";}', 'drupal_get_form', 'a:3:{i:0;s:25:\"content_field_remove_form\";i:1;s:10:\"photograph\";i:2;s:23:\"field_photograph_source\";}', 127, 7, '', 'admin/content/node-type/photograph/fields/field_photograph_source/remove', 'Remove field', 't', '', 4, '', '', '', 0, 'sites/all/modules/cck/includes/content.admin.inc')  in /var/www/html/content.ctc.com/includes/menu.inc on line 2457.

Hope that helps.

Backtrace:

user warning: Duplicate entry 'admin/content/nodewords/meta-tags/other/edit/%' for key 1 query: _menu_router_build /* admin : _menu_router_build */ INSERT INTO menu_router (path, load_functions, to_arg_functions, access_callback, access_arguments, page_callback, page_arguments, fit, number_parts, tab_parent, tab_root, title, title_callback, title_arguments, type, block_callback, description, position, weight, file) VALUES ('admin/content/nodewords/meta-tags/other/edit/%', 'a:1:{i:6;s:19:\"nodewords_page_load\";}', '', 'user_access', 'a:1:{i:0;s:20:\"administer meta tags\";}', 'drupal_get_form', 'a:2:{i:0;s:20:\"nodewords_pages_edit\";i:1;i:6;}', 126, 7, '', 'admin/content/nodewords/meta-tags/other/edit/%', 'Edit page meta tags', 't', '', 4, '', '', '', 0, 'sites/all/modules/contrib/nodewords/nodewords.admin.inc') in includes/menu.inc on line 2463.

trigger_error(string)[database.mysqli.inc:141];
._db_query(string)[database.mysql-common.inc:42];
..db_query(string)[menu.inc:2463];
..._menu_router_build(string)[menu.inc:1744];
....menu_router_build(a:0:{})[admin_menu.inc:9];
....._admin_menu_rebuild_links(a:0:{})[admin_menu.module:142];
......admin_menu_footer(a:1:{i:0;i:0;})[?:?];
.......call_user_func_array(a:2:{i:0;s:17:"admin_menu_footer";i:1;a:1:{i:1;i:0;}})[module.inc:483];
........module_invoke_all(a:2:{i:0;s:6:"footer";i:1;i:0;})[theme.inc:1611];
.........theme_closure(a:0:{})[?:?];
..........call_user_func_array(a:2:{i:0;s:13:"theme_closure";i:1;a:0:{}})[theme.inc:656];
...........theme(a:1:{i:0;s:7:"closure";})[theme.inc:1877];
............template_preprocess_page(string)[?:?];
.............call_user_func_array(string)[theme.inc:697];
..............theme(a:3:{i:0;s:4:"page";i:1;s:38:"The requested page could not be found.";i:2;b:0;})[common.inc:384];
...............drupal_not_found(a:0:{})[system.admin.inc:17];
................system_main_admin_page(a:2:{i:0;s:5:"build";i:1;s:7:"modules";})[?:?];
.................call_user_func_array(a:2:{i:0;s:22:"system_main_admin_page";i:1;a:2:{i:0;s:5:"build";i:1;s:7:"modules";}})[menu.inc:348];
..................menu_execute_active_handler(a:0:{})[index.php:18];
...................index.php

When installing new module.
Using admin_menu - 6.x-1.5

This confirms what has been said before: admin_menu.inc calls _menu_router_build() explicitly, and therefore it is the admin_menu module that is the culprit here.

@kbahey: So if this issue is confined to the admin_menu module, should we close it as fixed, since people will only encounter it if they have that module installed?

Does this still happen with admin_menu 3.x ?

With admin menu 1.5 i was getting these errors every 10-20 page loads. 3.0alpha4 seems to resolve it.

Tracking. Same errors, lots of errors, with vote_up_down/vud.theme.inc in includes/menu.inc line 2457.

Thanks

How is vote_up_down/vud.theme.inc implicated?

This is one of hundreds:

Duplicate entry 'vote/%/%/%/%/%/%' for key 1 query: INSERT INTO menu_router (path, load_functions, to_arg_functions, access_callback, access_arguments, page_callback, page_arguments, fit, number_parts, tab_parent, tab_root, title, title_callback, title_arguments, type, block_callback, description, position, weight, file) VALUES ('vote/%/%/%/%/%/%', 'a:6:{i:1;N;i:2;N;i:3;N;i:4;N;i:5;N;i:6;N;}', '', 'user_access', 'a:1:{i:0;s:16:\"use vote up/down\";}', 'vud_vote', 'a:6:{i:0;i:1;i:1;i:2;i:2;i:3;i:3;i:4;i:4;i:5;i:5;i:6;}', 64, 7, '', 'vote/%/%/%/%/%/%', 'Vote', 't', '', 4, '', '', '', 0, 'sites/all/modules/vote_up_down/vud.theme.inc')  in /home/.../includes/menu.inc on line 2385.

Thanks

That error has occurred inserting a menu item provided by vud.module into the {menu_router}. Likely as not, the error is caused elsewhere. In Drupal 6.16, using admin_menu 6.x-1.x is usually the culprit. Possibly another module though is calling menu_router_build() directly, which is what tends to cause the error.

For me updating admin_menu.module solved the problem.

+1 Same here

As per #144 shouldn't this be sitting then with admin menu if that is the culprit?

Coming from #251792: Implement a locking framework for long operations. Subscribing.

@154 looks like there may be other culprits also e.g. http://drupal.org/node/251792?page=1#comment-3006924.

Let's open a separate issue against admin_menu to deal with the known problem there and leave this one as a catch-all for any other related issues..

#808526: admin_menu.inc calls menu_router_build() explicitly, causing Duplicate entry error INSERT INTO {menu_router} (race condition)

Continuing from http://drupal.org/node/251792?page=1#comment-3006924...

@stripped your speech: suggest you try running a grep on your codebase to see where menu_router_build is being called from, assuming that's the problem.

I can't see anything wrong in admin.module, though I've only had a quick look and may have missed something.

Why not employ the use of database transactions so two operations can't occur at once or if there is any issue changes are not committed?

http://dev.mysql.com/doc/refman/5.0/en/commit.html

> Why not employ the use of database transactions
There is some discussion (from the core developers) of the rationale for the approach adopted in 6.16 and 7.x in the other issue I think, or possibly elsewhere. As far as 6.x goes, I don't think the Database API supports transactions.

The problem you are having is probably that a contrib module is not utilising the menu API in the right way so the locking mechanism is being circumvented.

@stripped your speech:

That was already discussed, and cannot be done, since the permission to use transactions is not a requirement for the installation of Drupal 6.x.

I believe this issue should be closed, since the problem has been resolved in Drupal via the API change in #251792: Implement a locking framework for long operations, and there should be documentation, if it does not already exist, added to alert contributed module developers that it is critical for them not to call _menu_router_build directly, as per #808526: admin_menu.inc calls menu_router_build() explicitly, causing Duplicate entry error INSERT INTO {menu_router} (race condition).

The locking framework accomplishes the purpose of transactions without using actual transactions.

Also, note that the Pressflow Drupal distribution uses transactions in conjunction with the locking framework, for those of you who are able to install it. (It is MySQL-specific.)

back with a vengeance. Apparently updating the admin menu was temp fix. worked great for months up till today. Uninstalled admin menu. site back to normal and no more duplicate entries .........

If modules out there invoke menu_router_build explicitly because this was just introduced as an API change in 6.16, I don't see how the issue is solved. Would it not make menu_router_build a private API call and not to be used by modules at all? Is there a way to adjust menu_router_build to not execute if the code calling it does not have permission? Then it could just throw a watchdog error so developer knows to use the locking framework.

True, while it could be considered 'solved' fundamentally, I'm sure there are lots of modules invoking menu_router_build, which causes the problem anyway.

@Kevin Quillen: I believe that actually making functions public or private is a PHP5 language capability, but Drupal 6 does not have PHP5 has an install requirement. (That was added for Drupal 7.) So the burden is on contributed module developers to respect the "suggestion" that _menu_router_build() not be called publically.

Not sure how many modules call _menu_router_build() directly - if people who encounter this error could run the following command: grep -ri "_menu_router_build" sites/all/modules from the DocumentRoot of their website, that would help track down the culprits.

Spent 4-5 hrs on this issue this morning.
After reading entire thread, disabling admin mod and others, tried to access database, 1-2 mins for page loads there as well. (Network Solutions) Called tech support, they fixed something on their end. site flies now.
re-enabled admin panel, views, the last 4 modules I installed prior to problem. No issues. Maybe it wasn't me. Always start with the simple stuff.
Al

I grepped, and this was all that came up:

-sh-3.2$ grep -ri "_menu_router_build" modules
modules/themekey/themekey_build.inc: * (based on _menu_router_build() in includes/menu.inc)

Is there another function to look for that when called, it itself calls menu_router_build? Otherwise, I am stumped.

@165, 163: that's the wrong command. The function name doesn't start with a _ (that may be a typo of mine that's unfortunately been replicated..!)

grep -ri "menu_router_build" sites should do the trick.

There is a setting in Boost:

Clear all cached pages in a menu on an insert/update/delete operation:

Disabled
Only Flush Menu Parents, Siblings & Children
Flushes Entire Menu Tree

We have Only Flush Menu Parents option selected. Related?

@167: your guess is as good as mine, since I don't use Boost... looks like you might have picked up something in themekey though in #165, but I can't find any reference to menu_router_build in themekey_build.inc http://drupalcode.org/viewvc/drupal/contributions/modules/themekey/theme... so maybe you have an older version??

It's a comment that references the function the code after it was based on.

What version of themekey?

Hmmm. I don't know the exact version (cant access the site outside of a certain IP) but it was whatever was newest in Dec/Jan.

Ah, there are rather a lot of versions from around then...

ThemeKey 6.x-1.1

Hmm that looks pretty harmless - it doesn't insert anything into the DB AFAICS..

Maybe do a grep of your codebase for menu_router

Should pick up functions containing that string and also and direct manipulation of the {menu_router} table..

Returns 4 results:

sites/all/modules/ubercart/uc_store/includes/summaries.inc
sites/all/modules/admin/admin.module
sites/all/modules/ctools/includes/menu.inc
sites/all/modules/ctools/page_manager/plugins/tasks/page.admin.inc

Admin module

/**
 * Menu access callback for admin landing pages.
 */
function admin_landing_page_access($path) {
  static $paths;
  if (!isset($paths)) {
    $cache = cache_get('admin_paths');
    $paths = ($cache && !empty($cache->data)) ? $cache->data : array();
  }
  if (!isset($paths[$path])) {
    $item = db_fetch_array(db_query("SELECT mlid, menu_name FROM {menu_links} ml WHERE ml.router_path = '%s' AND module = 'system'", $path));
    $result = db_query("
      SELECT m.load_functions, m.to_arg_functions, m.access_callback, m.access_arguments, m.page_callback, m.page_arguments, m.title, m.title_callback, m.title_arguments, m.type, m.description, ml.*
      FROM {menu_links} ml
      LEFT JOIN {menu_router} m ON ml.router_path = m.path
      WHERE ml.plid = %d AND ml.menu_name = '%s' AND hidden = 0", $item['mlid'], $item['menu_name']);
    $paths[$path] = FALSE;
    while ($item = db_fetch_array($result)) {
      $item['options'] = unserialize($item['options']);
      if ($item['external']) {
        if (!empty($item['options']['alter'])) {
          drupal_alter('translated_menu_link', $item, $map);
        }
        if ($item['access']) {
          $paths[$path] = TRUE;
          break;
        }
      }
      else {
        $map = explode('/', $item['link_path']);
        _menu_link_map_translate($map, $item['to_arg_functions']);
        $item['href'] = implode('/', $map);
        if (strpos($item['href'], '%') !== FALSE) {
          continue;
        }
        if (!isset($item['access'])) {
          if (!_menu_load_objects($item, $map)) {
            continue;
          }
          _menu_check_access($item, $map);
        }
        if (!$item['access']) {
          continue;
        }
        $paths[$path] = TRUE;
        break;
      }
    }
    cache_set('admin_paths', $paths);
  }
  return $paths[$path];
}

Ubercart summaries.inc

$accessor = "LIKE '%s/%%'";

  // If no_recur is TRUE, only look for the parent
  if ($only_parent) {
    $accessor = "= '%s'";
  }

  $result = db_query("SELECT path FROM {menu_router} WHERE path ". $accessor ." ORDER BY weight", $path);

  while ($row = db_fetch_array($result)) {
    $item = menu_get_item($row['path']);
    // Only allow items the user can access.
    if ($item['access'] === FALSE) {
      continue;
    }

    if ($item['page_callback'] == 'drupal_get_form') {
      if ($item['type'] == MENU_DEFAULT_LOCAL_TASK) {
        $parent = menu_get_item($item['tab_parent']);
        $href = $parent['href'];
      }
      else {
        $href = $item['href'];
      }

      $form_id = $item['page_arguments'][0];

      if (!function_exists($form_id)) {
        require_once($item['file']);
      }

      $form_state = array('storage' => NULL, 'submitted' => FALSE);
      $form = drupal_retrieve_form($form_id, $form_state);
      drupal_prepare_form($form_id, $form, $form_state);

      $summary_items = summarize_form($form);

      if (!$trim || $trim && count($summary_items) > 0) {
        $summaries[] = array(
          'path' => url($item['path']),
          'href' => $href,
          'title' => $item['title'],
          'items' => $summary_items,
        );
      }
    }
  }

  return $summaries;
}

CTools menu.inc

 $parents[] = '0';

        // Use array_values() so that the indices are numeric for array_merge().
        $args = $parents = array_unique(array_values($parents));
        $placeholders = implode(', ', array_fill(0, count($args), '%d'));
        $expanded = variable_get('menu_expanded', array());
        // Check whether the current menu has any links set to be expanded.
        if (in_array($menu_name, $expanded)) {
          // Collect all the links set to be expanded, and then add all of
          // their children to the list as well.
          do {
            $result = db_query("SELECT mlid FROM {menu_links} WHERE menu_name = '%s' AND expanded = 1 AND has_children = 1 AND plid IN (". $placeholders .') AND mlid NOT IN ('. $placeholders .')', array_merge(array($menu_name), $args, $args));
            $num_rows = FALSE;
            while ($item = db_fetch_array($result)) {
              $args[] = $item['mlid'];
              $num_rows = TRUE;
            }
            $placeholders = implode(', ', array_fill(0, count($args), '%d'));
          } while ($num_rows);
        }
        array_unshift($args, $menu_name);
      }
      else {
        // Show only the top-level menu items when access is denied.
        $args = array($menu_name, '0');
        $placeholders = '%d';
        $parents = array();
      }
      // Select the links from the table, and recursively build the tree. We
      // LEFT JOIN since there is no match in {menu_router} for an external
      // link.
      $data['tree'] = menu_tree_data(db_query("
        SELECT m.load_functions, m.to_arg_functions, m.access_callback, m.access_arguments, m.page_callback, m.page_arguments, m.title, m.title_callback, m.title_arguments, m.type, m.description, ml.*
        FROM {menu_links} ml LEFT JOIN {menu_router} m ON m.path = ml.router_path
        WHERE ml.menu_name = '%s' AND ml.plid IN (". $placeholders .")
        ORDER BY p1 ASC, p2 ASC, p3 ASC, p4 ASC, p5 ASC, p6 ASC, p7 ASC, p8 ASC, p9 ASC", $args), $parents);
      $data['node_links'] = array();
      menu_tree_collect_node_links($data['tree'], $data['node_links']);
      // Cache the data, if it is not already in the cache.
      $tree_cid = _menu_tree_cid($menu_name, $data);
      if (!cache_get($tree_cid, 'cache_menu')) {
        cache_set($tree_cid, $data, 'cache_menu');
      }
      // Cache the cid of the (shared) data using the page-specific cid.
      cache_set($cid, $tree_cid, 'cache_menu');
    }
    // Check access for the current user to each item in the tree.
    menu_tree_check_access($data['tree'], $data['node_links']);
    $tree[$cid] = $data['tree'];
  }
  return $tree[$cid];
}

CTools page.admin.inc

 // Check to see if something that isn't a page manager page is using the path.
  $path = implode('/', $path);
  $result = db_query("SELECT * FROM {menu_router} WHERE path = '%s'", $path);
  while ($router = db_fetch_object($result)) {
    if ($router->page_callback != 'page_manager_page_execute') {
      form_error($form['path'], t('That path is already in use. This system cannot override existing paths.'));
    }
  }

That all looks pretty benign. I take it you are using 6.16+ and still having the problem?

Also @162 "lots of modules invoking menu_router_build"
I'm not sure that's the case. It's a very expensive call (can take a few seconds to run). I think someone may have grepped the contrib repository to check.

Yep, 6.16.

I'm running 6.15 with the same error being thrown. Of the 4 modules listed in #175, the only one I have in common is the Admin module. I'm also running boost, but it's currently disabled.

@178: you can expect to see this error in 6.15 since it was only fixed in core in 6.16. From a quick look I don't see any problems likely to be caused by the 4 modules mentioned in #175.

@177: getting to the bottom of it probably requires running a backtrace when the error occurs so that the originating bit of code that causes it can be found.

I made a mistake, I am on 6.16, but I have yet to upgrade to 6.17

Subscribing, I'm running into this same issue on 6.17. I get an entire list of menu_router errors like the authors on random page saves.

Error on fresh install (very fresh, after I enter site info and click save). I noticed that Garland theme was not enabled but default was selected. Not sure if this is related?

attaching error log

Duplicate entry 'themes/bluemarine/bluemarine.info' for key 1 query: INSERT INTO system (name, owner, info, type, filename, status, throttle, bootstrap) VALUES ('bluemarine', 'themes/engines/phptemplate/phptemplate.engine', 'a:13:{s:4:\"name\";s:10:\"Bluemarine\";s:11:\"description\";s:66:\"Table-based multi-column theme with a marine and ash color scheme.\";s:7:\"version\";s:4:\"6.17\";s:4:\"core\";s:3:\"6.x\";s:6:\"engine\";s:11:\"phptemplate\";s:7:\"project\";s:6:\"drupal\";s:9:\"datestamp\";s:10:\"1275505216\";s:7:\"regions\";a:5:{s:4:\"left\";s:12:\"Left sidebar\";s:5:\"right\";s:13:\"Right sidebar\";s:7:\"content\";s:7:\"Content\";s:6:\"header\";s:6:\"Header\";s:6:\"footer\";s:6:\"Footer\";}s:8:\"features\";a:10:{i:0;s:20:\"comment_user_picture\";i:1;s:7:\"favicon\";i:2;s:7:\"mission\";i:3;s:4:\"logo\";i:4;s:4:\"name\";i:5;s:17:\"node_user_picture\";i:6;s:6:\"search\";i:7;s:6:\"slogan\";i:8;s:13:\"primary_links\";i:9;s:15:\"secondary_links\";}s:11:\"stylesheets\";a:1:{s:3:\"all\";a:1:{s:9:\"style.css\";s:27:\"themes/bluemarine/style.css\";}}s:7:\"scripts\";a:1:{s:9:\"script.js\";s:27:\"themes/bluemarine/script.js\";}s:10:\"screenshot\";s:32:\"themes/bluemarine/screenshot.png\";s:3:\"php\";s:5:\"4.3.5\";}', 'theme', 'themes/bluemarine/bluemarine.info', 0, 0, 0) in /hsphere/local/home/friedelectronics.com/modules/system/system.module on line 822.

Duplicate entry 'themes/chameleon/marvin/marvin.info' for key 1 query: INSERT INTO system (name, owner, info, type, filename, status, throttle, bootstrap) VALUES ('marvin', '', 'a:13:{s:4:\"name\";s:6:\"Marvin\";s:11:\"description\";s:31:\"Boxy tabled theme in all grays.\";s:7:\"regions\";a:2:{s:4:\"left\";s:12:\"Left sidebar\";s:5:\"right\";s:13:\"Right sidebar\";}s:7:\"version\";s:4:\"6.17\";s:4:\"core\";s:3:\"6.x\";s:10:\"base theme\";s:9:\"chameleon\";s:7:\"project\";s:6:\"drupal\";s:9:\"datestamp\";s:10:\"1275505216\";s:8:\"features\";a:10:{i:0;s:20:\"comment_user_picture\";i:1;s:7:\"favicon\";i:2;s:7:\"mission\";i:3;s:4:\"logo\";i:4;s:4:\"name\";i:5;s:17:\"node_user_picture\";i:6;s:6:\"search\";i:7;s:6:\"slogan\";i:8;s:13:\"primary_links\";i:9;s:15:\"secondary_links\";}s:11:\"stylesheets\";a:1:{s:3:\"all\";a:1:{s:9:\"style.css\";s:33:\"themes/chameleon/marvin/style.css\";}}s:7:\"scripts\";a:1:{s:9:\"script.js\";s:33:\"themes/chameleon/marvin/script.js\";}s:10:\"screenshot\";s:38:\"themes/chameleon/marvin/screenshot.png\";s:3:\"php\";s:5:\"4.3.5\";}', 'theme', 'themes/chameleon/marvin/marvin.info', 0, 0, 0) in /hsphere/local/home/friedelectronics.com/modules/system/system.module on line 822.

Duplicate entry 'themes/bluemarine/bluemarine.info' for key 1 query: INSERT INTO system (name, owner, info, type, filename, status, throttle, bootstrap) VALUES ('bluemarine', 'themes/engines/phptemplate/phptemplate.engine', 'a:13:{s:4:\"name\";s:10:\"Bluemarine\";s:11:\"description\";s:66:\"Table-based multi-column theme with a marine and ash color scheme.\";s:7:\"version\";s:4:\"6.17\";s:4:\"core\";s:3:\"6.x\";s:6:\"engine\";s:11:\"phptemplate\";s:7:\"project\";s:6:\"drupal\";s:9:\"datestamp\";s:10:\"1275505216\";s:7:\"regions\";a:5:{s:4:\"left\";s:12:\"Left sidebar\";s:5:\"right\";s:13:\"Right sidebar\";s:7:\"content\";s:7:\"Content\";s:6:\"header\";s:6:\"Header\";s:6:\"footer\";s:6:\"Footer\";}s:8:\"features\";a:10:{i:0;s:20:\"comment_user_picture\";i:1;s:7:\"favicon\";i:2;s:7:\"mission\";i:3;s:4:\"logo\";i:4;s:4:\"name\";i:5;s:17:\"node_user_picture\";i:6;s:6:\"search\";i:7;s:6:\"slogan\";i:8;s:13:\"primary_links\";i:9;s:15:\"secondary_links\";}s:11:\"stylesheets\";a:1:{s:3:\"all\";a:1:{s:9:\"style.css\";s:27:\"themes/bluemarine/style.css\";}}s:7:\"scripts\";a:1:{s:9:\"script.js\";s:27:\"themes/bluemarine/script.js\";}s:10:\"screenshot\";s:32:\"themes/bluemarine/screenshot.png\";s:3:\"php\";s:5:\"4.3.5\";}', 'theme', 'themes/bluemarine/bluemarine.info', 0, 0, 0) in /hsphere/local/home/friedelectronics.com/modules/system/system.module on line 822.

Duplicate entry 'themes/pushbutton/pushbutton.info' for key 1 query: INSERT INTO system (name, owner, info, type, filename, status, throttle, bootstrap) VALUES ('pushbutton', 'themes/engines/phptemplate/phptemplate.engine', 'a:13:{s:4:\"name\";s:10:\"Pushbutton\";s:11:\"description\";s:52:\"Tabled, multi-column theme in blue and orange tones.\";s:7:\"version\";s:4:\"6.17\";s:4:\"core\";s:3:\"6.x\";s:6:\"engine\";s:11:\"phptemplate\";s:7:\"project\";s:6:\"drupal\";s:9:\"datestamp\";s:10:\"1275505216\";s:7:\"regions\";a:5:{s:4:\"left\";s:12:\"Left sidebar\";s:5:\"right\";s:13:\"Right sidebar\";s:7:\"content\";s:7:\"Content\";s:6:\"header\";s:6:\"Header\";s:6:\"footer\";s:6:\"Footer\";}s:8:\"features\";a:10:{i:0;s:20:\"comment_user_picture\";i:1;s:7:\"favicon\";i:2;s:7:\"mission\";i:3;s:4:\"logo\";i:4;s:4:\"name\";i:5;s:17:\"node_user_picture\";i:6;s:6:\"search\";i:7;s:6:\"slogan\";i:8;s:13:\"primary_links\";i:9;s:15:\"secondary_links\";}s:11:\"stylesheets\";a:1:{s:3:\"all\";a:1:{s:9:\"style.css\";s:27:\"themes/pushbutton/style.css\";}}s:7:\"scripts\";a:1:{s:9:\"script.js\";s:27:\"themes/pushbutton/script.js\";}s:10:\"screenshot\";s:32:\"themes/pushbutton/screenshot.png\";s:3:\"php\";s:5:\"4.3.5\";}', 'theme', 'themes/pushbutton/pushbutton.info', 0, 0, 0) in /hsphere/local/home/friedelectronics.com/modules/system/system.module on line 822.

To track this down I suggest you do a debug_backtrace() in drupal_error_handler, or use http://drupal.org/project/drupal_tweaks.

same here

Tracking. I also have these errors (the menu_router flavor). They occur in D6.17 and I recall seeing them as far back as D6.9 at least.

I have the same problem in drupal 6.17
Could we implement the solution #47 (even though it is temporary)?
Is there any other solution to this immediate?

subscribing

subscribing

guess im not the only one!!

@bisuteria, et al. - see my comment #160 and comment #166 by gpk. This error is most likely caused by contributed modules that call menu_router_build() directly, and should be fixed in those modules.

I grepped my code base in #175 and found nothing calling menu_router_build() directly. It must be another reason?

@191: see #183.

I just ran grep -R menu_router_build in the sites/all directory (where I put all my modules) and didn't get a single hit.

There are other ways for a rogue module to cause this problem than by calling menu_router_build(). Hence #192.

I've enabled Drupal Tweaks and don't see anything immediately wrong.

How can I use this and/or debug_backtrace to determine when the menu breaks and pin down why?

Use option to show or log backtrace on Drupal errors.

If you have a "method" for getting the duplicate entry error to occur then go ahead and do it!! And look in the Drupal Tweaks output for clues as to what's causing it. The only snag with this approach is that since we are talking about a race condition the pageview/thread which is the root cause of the problem (i.e. where the suspected rogue module is doing its dastardly stuff) may not always be the one that gets reported by Drupal Tweaks ... however over time you should (I would hope) get a backtrace which leads back to the root cause.

The only other approaches I can think of are
- disable modules and see if it goes away
- grep codebase for other suspect code ... e.g. direct manipulation of {menu_router} database table
- maybe use the devel module's query log functionality to try to trap where {menu_router} is getting modified (other than via http://api.drupal.org/api/function/menu_rebuild/6)

My instinct tells me it happens during cron, on this particular site cron runs every 12 minutes because its integrated into an inventory system, powering the products in ubercart. It also has Boost and XML Sitemap, both heavy processors during cron. Other than that there doesn't seem to be a way to trigger it manually by a user.

I'll try adding devel and looking for menu_router changes.

#197: It's definitely not related to cron for me, because I have them on a development site where cron is disabled. Unfortunately I have not been able to find any steps to reproduce them; they just crop up sometimes. I'll see if I can get it to log backtraces.

#196 (gpk):

If you have a "method" for getting the duplicate entry error to occur then go ahead and do it!! And look in the Drupal Tweaks output for clues as to what's causing it. The only snag with this approach is that since we are talking about a race condition the pageview/thread which is the root cause of the problem (i.e. where the suspected rogue module is doing its dastardly stuff) may not always be the one that gets reported by Drupal Tweaks ... however over time you should (I would hope) get a backtrace which leads back to the root cause.

A more reliable way to get a backtrace for each menu_router_build could be to write a custom hook_menu or hook_menu_alter implementation, and then ddebug_backtrace (with devel) and some way to get this into a system message or into a log. Such a monitoring module could do a silent job for a while, until you (or the one who wants to try this) see the error occur again. This thing could use a static var or its own semaphore to detect race conflicts.

Shouldn't be duplicate of:
#512962: Optimize menu_router_build() / _menu_router_save()
#251792: Implement a locking framework for long operations
and
admin_menu #808526: admin_menu.inc calls menu_router_build() explicitly, causing Duplicate entry error INSERT INTO {menu_router} (race condition)
?

@kenorb:

#512962: Optimize menu_router_build() / _menu_router_save() is related, but separate, since it is a performance improvement, whereas this is a bug.

#808526: admin_menu.inc calls menu_router_build() explicitly, causing Duplicate entry error INSERT INTO {menu_router} (race condition) is a specific example of a contrib module causing this issue.

This issue is more of a catch-all; it should have been fixed by #251792: Implement a locking framework for long operations, but some people are still reporting it. I believe that it is being caused by contrib modules, which is why #196 & #199 discuss how to debug it.

Finally caught one of these in a backtrace. For me it is indeed admin menu calling menu_router_build(), in _admin_menu_rebuild_links() which apparently gets called in a theme hook (admin_menu_footer()).

If you are getting these error messages and you have the admin menu module enabled, it is probably the cause. See the issue below. A patch you can try is available there.

#808526: admin_menu.inc calls menu_router_build() explicitly, causing Duplicate entry error INSERT INTO {menu_router} (race condition)

Would changing the menu.inc (line 2457) from "INSERT INTO ..." to "INSERT REPLACE INTO ..." not solve this issue? Seems like that would avoid this coming up again in the future with some other module or modules.

@Zahor
That is DB specific syntax for MySQL. It would crash and burn on other platforms using Postgres.

If you have admin menu enabled, follow this issue #808526: admin_menu.inc calls menu_router_build() explicitly, causing Duplicate entry error INSERT INTO {menu_router} (race condition).

Otherwise, let the developers here know exactly what you set up is:

Drupal version
All contributed modules that are enable and their versions

Unlikely, but could help

PHP version
DB type and version

I agree with EvanDonovan that this is most likely a GIGO issue related to a contributed module, but the odd comment above suggest otherwise.

I.E. "fresh install", but no confirmation of NOT having any contributed modules enabled and no mention of the Drupal version.

Ah, yes, just reading about PosgreSQL not supporting it. Ok, fair enough.

Having same issue on 6.19, subscribing.

Here's to the hope that this bug (reported ~2.5 years ago) will get attention from somebody with the skills to resolve it...

@jbeall: The issue here is not lack of skill, it's lack of data. For most people, this bug should have been resolved by patches that have already gone in. See #201, 204 for more.

The above patch #47 or #46 not working for 6.19 version .

the solution with 6.19 is very easy.

I had the same problem , i did lot of things to make it correct . But when i closed the browser and reopened the error was gone.

Also try to open site in other browser if this problem occurs.

subscring

If the only reason why Drupal does not use this method is "other dbms doesn't support replace command" then why not using a pair of DELETE/INSERT commands instead of a REPLACE ?

@211, 203: there is discussion of using REPLACE way above (it would hide the problem but not get rid of the underlying bug). The bug in Drupal core has actually now been fixed by implementing a locking system, however not all contrib modules use the API correctly. If you are still seeing this error then you are probably using admin_menu 6.x-1.x. That module's maintainer is now recommending 6.x-3.x, see #808526: admin_menu.inc calls menu_router_build() explicitly, causing Duplicate entry error INSERT INTO {menu_router} (race condition).

Update Jan 2013: Best to use the recommended version of admin_menu from the project page. admin_menu 6.x-1.x was fixed in 1.7 to use the locking mechanism properly (see issue linked just above) and so this branch is now safe to use; at the time of writing this, 6.x-3.x still has outstanding issues caused by core bugs.

subscribe. over 1K log entries from this. Will try upgrading admin_menu.

I have the same issue as well...Lots of duplicate entry errrors. Subscribing.

I have the same issue as well...Lots of duplicate entry errors. Subscribing.

Ha ha @new_B you also have a duplicate entry error in your comment. ;)

@#212, upgrading to 6.x-3.x didn't work for me, made it even worse, see http://drupal.org/node/863092#comment-4300002

I'm having an issue that seems related. Whenever I enable a module, save a view, or or perform a number of other admin functions it takes a VERY long time for the process to complete. Many times over a minute.

I think this has to do with the menu's being rebuilt multiple times. I went as far as to disable and remove all contrib modules and all core non-required modules and although speeds increased it still took WAY too long to complete simple tasks.

Anyone else seeing this?

#218: Sounds familiar. Do you have a large site menu?

I have about 60 items in primary links. Besides that, just the standard navigation menu.

#218: Had same problem. For me, disabling the "database logging" module before running update.php fixed the problem.

In the absence of any fresh reports marking this issue as fixed.

Please open a new issue with full details of how to reproduce should any similar problems recur.

Also, per #217 and pace #212, the best version of admin_menu 6.x to use would be the one from the Recommended releases section of the project page. At the time of writing this is latest 6.x-1.x release rather than 6.x-3.x which does not yet have a full release.

Certainly nothing was done to fix this. If anything, it couldn't be reproduced without contrib.

Hi
it is not a good solution . it make website dramatically slow , Rebuild Every time !!

Just ran into this on drupal 6.26. No idea how to reproduce yet.

There were a bunch of database table locks (MyISAM) and with dblog it brought the site down.

Another problem the site is having is various pages serving.the wrong content due to I assume corruption in the menu_router table. This Is causing the site to serve unpublished nodes for other published content. Thus this could also be viewed as a security problem as well.

My site is using admin_menu 1.18. Does anyone know if this module is the cause? And if so what patches could be used to resolve the issue.

@225: I suggest you open a new issue to discuss your particular site problems since they may be unrelated to this. Hard to diagnose without more detailed info.

Yes admin_menu 6.x-1.8 is currently the one to use on D6 and it includes a fix to prevent these errors. See #808526: admin_menu.inc calls menu_router_build() explicitly, causing Duplicate entry error INSERT INTO {menu_router} (race condition) and #1189532: function _admin_menu_rebuild_links() calls menu_router_build() which may bypass the locking mechanism.

Sounds like your site needs at least to have all its caches cleared...

@gpk & update #212: My client just ran into this problem again yesterday. I can confirm I'm using admin_menu 1.8, which has the fix.

Does anyone know of any other modules which cause this problem? path_redirect? pathauto?

Something definitely borking my menu_router table.

Additionally on my live set up, we have two web servers and another dedicated database server. I believe the concurrency perhaps causes this to happen more often, but right now it's happening every couple days / weeks and no end in sight. If no one notices, the website will continue to provide bad content until the caches are cleared.

Something that could be remedied by Pressflow perhaps?

Ok, I looked into this a little more today. I can now (kind of) duplicate it. At least I know what's happening.

Some assumptions:

- In order for this bug to happen, clearing your cache needs to occasionally take more than the acquire_lock is held for. acquire_lock is currently held for 30 seconds.

- This will most likely only happen to you, when you've got a setup with a slow database. In my case, my "slow" database is a 16 core 32GB server over a 1000T connection, 1 hop to my dedicated web servers (2x). The slow part is the additional latency caused by TCP connections and networking...and well Drupal does way too many queries. While I'm able to kind of trigger this bug on my development server, I've not been able to get the "messed up content" part of the bug locally. You'll need a slower database to catch it.

What's happening in the code:

Let's assume you have two processes.

Process #1 goes and clears cache for what ever reason. Process #1 lock_acquires('menu_rebuild'). Let's assume this process will take 100 seconds.

Process #2 comes in at 35 seconds and clears the cache as well.

Process #2 will attempt to lock_acquire('menu_build').

Process #2 It will not find the lock in $_GLOBALS['lock'], so it will attempt to INSERT it INTO {semaphore}, which will fail.

Process #2 will then call lock_may_be_available('menu_rebuild'), which will find the lock from process #1 in SELECT {semaphore}.

Process #2 will then check and see if the lock from Process #1 has expired (it has), and it will acquire the lock itself.

Now Process #1 is 1/2 way through rebuilding the menu, and process #2 will DELETE what he's already done. Process #1 will continue to INSERT entries into {menu_router}, and then rebuild a 1/2 assed {menu_router}.

If at any point between Process #1 being done and Process #2 being done, a surfer hits your site....your caches in some strange state and some (not all) of your pages could be completely messed up.

How I duplicated this bug

This bug is currently happening on my production server every couple weeks. It completely messed up the site and until someone clears the cache manually (and the CDN cache) it continue to be messed up.

As for duplicating this bug manually here's the code I've added to lock.inc and menu.inc to help you debug the code and duplicate it yourself.

For me, I applied this patch, which displays via watchdog how much time is spent in between the menu_rebuild() locks. Locally this is roughly 2 seconds with out load, and can go up to 40 under load. Average locally was around 8 seconds under semi load. I set my lock_acquire() and lock_wait() timeouts to 4 seconds so I can catch the bug.

I opened 10 or so tabs logged in as admin with admin_menu. I clear menu_cache on each of them. I try to stagger it every 2 seconds so that they'll expire each others locks.

When in another "anonymous browser" as an anonymous user I open a bunch of menu items which get inserted first in the menu rebuild process. I refresh them, until I can find one that's messed up.

It's not easy to duplicate, but as I mentioned it seems to be happening on my live server often enough that I can't dismiss this.

---

You could run into my problem as mentioned previously if clearing your menu_router for what ever reason hovers around 30 seconds, and gets triggered twice around the same time.

----

I need help

I need help tracking down what in Drupal calls menu rebuilds and now I can minimize this issue. Another other suggestions (aside from speed up your database idiot, i know) would be appreciated.

The site has 26,000 nodes. I've got a sitemap 1.x on this site, which seems to rebuild menus and could be the cause. I've also got some scripts will pull data from a SAP server and synchronizes content via a bunch node_save()'s. Also messes around with some SEO stuff in nodewords tables.

For now I think I'm just going to bump up the lock timeouts when rebuilding the menu's. I also personally think that a multiple INSERT query needs to get built in _menu_router_build(), and after it's built, the DELETE is called, then immediately the multiple INSERT is called. It's between these two timings, that the problem exists I believe.

PS. This bug could probably be duplicated more easily by adding sleep()'s inside the _menu_router_build() for($menu as $path => $v) just after the INSERT on smaller / faster installs.

You'd still need to test on an pre-existing site with an appropriate amount of content and not just a base core install.

>help tracking down what in Drupal calls menu rebuilds
grep?

As far as core goes the list is here of course: http://api.drupal.org/api/drupal/includes!menu.inc/function/calls/menu_rebuild/6

I wonder if you could also be hitting a theoretical problem I wondered about a long while back. Suppose process 2 comes in before the 30 secs are up. Say at T+10 secs. Then it will get held in limbo for 30 secs by lock_wait() (i.e. finishing at T+40 seconds) "Wait for another request that is already doing this work." as the comment says - and menu_rebuild() will then return FALSE (not that I have ever seen any caller do anything with this return value, just as menu_rebuild() ignores the return value from lock_wait().

In terms of your situation, when the second process reaches the end of its limbo (T+40) it will carry on doing whatever it was doing and will (if it ignores the return value from menu_rebuild() ) assume that the menu_rebuild() is complete. Whereas there are still another 60 seconds to go (100 - 40). Recipe for disaster???

(The specific potential problem that I had thought about previously was if you enabled a module during a menu rebuild its menu items might not get added to the {menu_router} table, especially if the module name began with "a" or had a low weight or something! Not quite as catastrophic as what you are seeing anyway.)

Yes it might be worth disabling sitemap to see if that helps; 30 secs is already a long time for the whole site to hang during a rebuild although I can see why you might want to increase that.

[and if XML sitemap is triggering the problem it might be worth trying 6.x-2.x which has at least as many users as 1.x]

Let's patch against 6.x dev.