Problem/Motivation

Getting a bunch of WSOD in the last weekish on latest -dev.

PDOException: SQLSTATE[23000]: Integrity constraint violation: 1062 Duplicate entry 'javascript_parsed' for key 'PRIMARY': INSERT INTO {variable} (name, value) VALUES (:db_insert_placeholder_0, :db_insert_placeholder_1); Array ( [:db_insert_placeholder_0] => javascript_parsed [:db_insert_placeholder_1] => a:2:{s:13:"refresh:en-CA";s:7:"waiting";s:13:"refresh:en-US";s:7:"waiting";} ) in variable_set() (line 991 of includes/bootstrap.inc).

Proposed resolution

Remaining tasks

User interface changes

API changes

Support from Acquia helps fund testing for Drupal Acquia logo

Comments

mikeytown2’s picture

Title: PDOException: Integrity constraint violation: 1062 Duplicate entry 'javascript_parsed' » locale_js_alter() - PDOException: Integrity constraint violation: 1062 Duplicate entry 'javascript_parsed'

Most likely the result of locale_js_alter() as this the only place where that variable gets used. It gets reset inside of drupal_clear_js_cache(). AdvAgg used to cache calls to hook_js_alter() but that resulted in some headaches with things not working correctly with 3rd party code. I've looked into this more and the fix (Using a cache level of 5 to cache hook_js_alter) will be in this issue #2458871: Promote the cache level of 3 to normal; create new level for aggressive cache

Using the variable table the way core's css/js handling does usually results in bad things happening like you pointed out here. Couple of issues where this was brought up but never fixed:
#886488-39: [D7] Add stampede protection for css and js aggregation
#1048316-2: [meta] CSS and Javascript aggregation improvement
#1014086-34: Stampedes and cold cache performance issues with css/js aggregation

The fact that the locale module is causing issues is quite disturbing. As that means that even if I cache hook_js_alter() bad things can still happen given enough requests coming in. Long story short the way variable_set is used inside of locale_js_alter() is broken. This can be fixed without a core patch but it'll take some effort.

Some of the work that might be related to the eventual locale_js_alter fix would be located in here #2468603: Submit AdaptiveTheme page when responsive css changes I'll be working on a way to generalize the adding of subfiles to files as I'll have done it in 2 places, that issue and in https://www.drupal.org/project/css_emimage

As of right now setting the cache level to 5 manually in your settings.php will mitigate this issue.
$conf['advagg_cache_level'] = 5;

Currently using my local copy of https://www.drupal.org/sandbox/greggles/1481160 to check if javascript_parsed gets used elsewhere in 3rd party code.
grep -l -r -i "'javascript_parsed'" ./

joelpittet’s picture

Love that gotta get them all module:) I also want to refactor it finds fresh modules easier than the manual update he has now.

Thanks for the reading material, I'll set the caching level to 11 5 for now and see what happens.

mikeytown2’s picture

https://www.drupal.org/project/snapit seems to be only only module that uses the code outside of tests so it seems like using an alternate storage method is possible.

joelpittet’s picture

I get a WSOD with only a $conf['advagg_cache_level'] = 5; change on flush all caches. Same error log error as in the IS.

mikeytown2’s picture

Even with a cache level of 5 the locale_js_alter will sill need to be ran once before it is cached. You remember this issue right? #2042851: PDOException: SQLSTATE[23000]: Integrity constraint violation: 1062 Duplicate entry 'rules_event_whitelist' for key 'PRIMARY it was fixed with this #2190775: Misuse of variables table with rules_event_whitelist. I'll have to do the same with locale.

joelpittet’s picture

Oh I know those well:) I'll test out any patch you think will help!

mikeytown2’s picture

_locale_invalidate_js() inside of locale.inc makes not using a variable very hard to do. Going to wrap the call to locale_js_alter with locks to see if that fixes it for you.

mikeytown2’s picture

Status: Active » Needs review
FileSize
1.44 KB

You'll have to flush caches; this will replace the call to locale_js_alter with _advagg_locale_js_alter.

joelpittet’s picture

Giving it a try, thanks

joelpittet’s picture

Status: Needs review » Reviewed & tested by the community

Seems to do the trick, thanks @mikeytown2

mikeytown2’s picture

Status: Reviewed & tested by the community » Fixed

Thanks for testing!

  • mikeytown2 committed 8efeb4a on 7.x-2.x
    Issue #2469063 by mikeytown2: locale_js_alter() - PDOException:...
joelpittet’s picture

Status: Fixed » Needs work

I managed to see this again:(

Integrity constraint violation: 1062 Duplicate entry 'javascript_parsed' for key 'PRIMARY': INSERT INTO {variable} (name, value) VALUES (:db_insert_placeholder_0, :db_insert_placeholder_1); Array\n(\n    [:db_insert_placeholder_0] => javascript_parsed\n    [:db_insert_placeholder_1] => a:2:{s:13:"refresh:en-CA";s:7:"waiting";s:13:"refresh:en-US";s:7:"waiting";}\n)\n in variable_set() (line 991 of /html/includes/bootstrap.inc)., referer: https://domain.com/ca/admin/modules
mikeytown2’s picture

I'll wrap it in a try catch block, that should catch the PDO error. Will also make the lock 10 seconds instead of 5

  • mikeytown2 committed 74234b7 on 7.x-2.x
    Issue #2469063 by mikeytown2: Wrap the call to locale_js_alter() in a...
mikeytown2’s picture

Status: Needs work » Fixed
mikeytown2’s picture

joelpittet’s picture

Status: Fixed » Needs review
FileSize
397 bytes

Thanks again @mikeytown2, just ran into this error after applying the latest -dev:

PHP Parse error: syntax error, unexpected '{', expecting '(' in /advagg.inc on line 206, referer: /us/admin/config/development/performance/advagg/operations

Looks like it was in the patch... strange it didn't end up in -dev? Or maybe it's just d.o packaging delay?

mikeytown2’s picture

mikeytown2’s picture

mikeytown2’s picture

Status: Needs review » Fixed

I've committed #18 in the other issue.

joelpittet’s picture

Oh totally unrelated to this patch, just saw the error after updating and saw the try/catch was used...

Status: Fixed » Needs work

The last submitted patch, 18: locale_js_alter-2469063-18.patch, failed testing.

mikeytown2’s picture

Status: Needs work » Fixed
joelpittet’s picture

I'm sorry this one keeps coming back with a vengance:

Spotted this in my log:

[Tue Apr 28 02:08:51.886101 2015] [:error] [pid 25323] [client 208.118.141.179:51718] \xc3Title|https://domain.com
|severity=error
|type=php|ip=208.118.141.179
|uri=https://domain.com/us/admin/commerce/config/commerce-entitycache
|referer=https://domain.com/us/admin/commerce/config/commerce-entitycache|uid=249
|link=
|message=PDOException: SQLSTATE[23000]: Integrity constraint violation: 1062 Duplicate entry 'javascript_parsed' for key 'PRIMARY': INSERT INTO {variable} (name, value) VALUES (:db_insert_placeholder_0, :db_insert_placeholder_1); Array\n(\n    [:db_insert_placeholder_0] => javascript_parsed\n    [:db_insert_placeholder_1] => a:2:{s:13:"refresh:en-CA";s:7:"waiting";s:13:"refresh:en-US";s:7:"waiting";}\n)\n in variable_set() (line 991 of /var/www/html/includes/bootstrap.inc)., referer: https://domain.com/us/admin/commerce/config/commerce-entitycache
joelpittet’s picture

Status: Fixed » Needs work

Going through the code, it looks like _advagg_locale_js_alter() is a private function not a hook_js_alter() nor is it getting called anywhere... maybe this is part of why this is showing up still?

mikeytown2’s picture

If you go to "admin/config/development/performance/advagg/info" under Advagg CSS/JS Hooks Implemented by Modules at the bottom you should see _advagg_locale in there. I replace the locale hook call with _advagg_locale here: http://cgit.drupalcode.org/advagg/tree/advagg.module#n317

If this is not working flush the caches as hook_module_implements_alter is cached.

mikeytown2’s picture

Is _advagg_locale being called instead of locale?

joelpittet’s picture

re: #27 OH that is magic... scary magic.

I see it there

js_alter: 1
  _advagg_locale
mikeytown2’s picture

Status: Needs work » Fixed
FileSize
832 bytes

Committed this patch; reopen if it doesn't work

  • mikeytown2 committed 9102b59 on 7.x-2.x
    Issue #2469063 by mikeytown2: catch PDOException instead of Exception.
    
joelpittet’s picture

FileSize
79.78 KB

I've flushed the caches and crossing my fingers;)

I maybe should open up a new issue for this? Variable debug is reporting advagg_bundler_info and javascript_parsed as possible candidates to port to a custom table and I'm curious your thoughts on the matter since you seem to be all about performance.

Though it's semi related:

mikeytown2’s picture

Open up an issue for advagg_bundler_info; that looks like a mistake. javascript_parsed is core and I've done as much as I can to fix it; also at ~9kb it's not huge.

joelpittet’s picture

Thanks added here: #2479755: Remove 'advagg_bundler_info' variable and yes agree on the javascript_parsed one, just pointing as it may be related to this issue.

joelpittet’s picture

Just noticed this little gotcha:

Notice: Undefined variable: return in _advagg_locale_js_alter() (line 402 of advagg.module).

And looking at it I could be wrong but locale_js_alter() never returns a value, maybe the $return is not needed?

joelpittet’s picture

Status: Fixed » Needs review
FileSize
519 bytes
mikeytown2’s picture

Status: Needs review » Reviewed & tested by the community

If you got that notice & not the PDOException then it sounds like this is close to being fixed. Will commit your change tomorrow

The last submitted patch, 17: advagg-2469063-15-try-catch.patch, failed testing.

The last submitted patch, 30: advagg-2469063-30-catch-pdo.patch, failed testing.

joelpittet’s picture

I hope we got it *crosses fingers*

mikeytown2’s picture

Status: Reviewed & tested by the community » Fixed
FileSize
1023 bytes

Committed this. Will do a variable read from the db if javascript_parsed is empty; should help the number of writes to the variable table.

  • mikeytown2 committed 9e5221c on 7.x-2.x
    Issue #2469063 by mikeytown2: locale_js_alter() - PDOException:...
mikeytown2’s picture

$conf needs to be global

joelpittet’s picture

Was just about to mention that:P

joelpittet’s picture

Status: Fixed » Needs work

This feels like a white whale. Hmm got this after rebuilding the registry from an anonymous user error log:

PDOException: SQLSTATE[23000]: Integrity constraint violation: 1062 Duplicate
entry 'javascript_parsed' for key 'PRIMARY': INSERT INTO {variable} (name,
value) VALUES (:db_insert_placeholder_0, :db_insert_placeholder_1); Array
(
[:db_insert_placeholder_0] => javascript_parsed
[:db_insert_placeholder_1] =>
a:2:{s:13:"refresh:en-CA";s:7:"waiting";s:13:"refresh:en-US";s:7:"waiting";}
)
in variable_set() (line 991 of
/html/includes/bootstrap.inc).

mikeytown2’s picture

Thinking this is coming from somewhere else. Can you patch core with this #1158322-199: Add backtrace to all errors so we can see what the cause it.

joelpittet’s picture

Good suggestion, I'm a bit concerned about putting that in production... more nervous but I may give it a try, thanks for the suggestion

mikeytown2’s picture

I haven't worked on that issue because I've been using in prod without issue for some time; it's low on my priority list currently.

mikeytown2’s picture

Status: Needs work » Postponed (maintainer needs more info)

Moving the status as I need more info before I can act on this.

mikeytown2’s picture

Any news on what function call is causing the error?

joelpittet’s picture

Status: Postponed (maintainer needs more info) » Fixed

I've not seen this in a bit... I'll re-open if it, rears it's head again.

joelpittet’s picture

Status: Fixed » Postponed (maintainer needs more info)

It's back, usually happens when I rebuild registry, but my backtrace is currently quite useless...

PDOException: SQLSTATE[23000]: Integrity constraint violation: 1062 Duplicate
entry 'javascript_parsed' for key 'PRIMARY': INSERT INTO {variable} (name,
value) VALUES (:db_insert_placeholder_0, :db_insert_placeholder_1); Array
(
     [:db_insert_placeholder_0] => javascript_parsed
     [:db_insert_placeholder_1] =>
a:2:{s:13:"refresh:en-CA";s:7:"waiting";s:13:"refresh:en-US";s:7:"waiting";}
)
  in variable_set() (line 991 of
html/includes/bootstrap.inc).

It's starting at emaillog_watchdog which is who is providing the backtrace... so not helpful.

Steady’s picture

Removed

joelpittet’s picture

@Steady that is completely unrelated. You may want to post that in Drupal core's issue queue as a new issue for drupal 7.x

This issue is specifically about locale and advagg: javascript_parsed

Would you mind editing your comment about to remove the details so people don't flood this issue with related questions? You may want to try drush registry_rebuild to help out of that mess.

Steady’s picture

Thanks for your message.

I will edit my previous comment.

Unfortunately I have no idea what drush registry_rebuild means.

I am just a lowly drupal user on mini sites and I was devastated to get the message when I was a good boy upgrading from 7.36 to 7.37

I have no idea whether the upgrade is sound or not. Whether the database is corrupt or not. The content seems to be there on the site, but this drupal stuff is now getting very frustrating and I am loathe to go back to wordpress, but it is now getting tedious how difficult drupal is to get up and running without coming across something that seems to mess up or complicate things on a weekly basis.

Rant over.
Mr Fed up

joelpittet’s picture

Oh I forgot the link to that... meant to include it: https://www.drupal.org/project/registry_rebuild

It's got me out of jams before. But since you had errors that said "Server went away" it may be a problem with MySQL...

Thanks for removing your comment. The details you had there were great for a new issue:
https://www.drupal.org/node/add/project-issue/drupal

I just upgraded to 7.37 and I got a message about "Cache rebuild lock hit: data" but didn't have the error you are seeing, sorry to hear that! I bet the core will be able to help and feel free to cross link the new issue you create here.

The upgrade was fairly minor(I read all the changes before posting it)

Steady’s picture

Thanks for the data Joel.

mikeytown2’s picture

Looking into where the variable_set('javascript_parsed', $parsed) could be coming from outside of locale_js_alter() and _locale_invalidate_js() is the only other place.

It can be called from these functions:
locale_add_language() which on a normally running website can be called by these 2 submit functions: locale_languages_predefined_form_submit() & locale_translate_import_form_submit().
locale_translate_delete_form_submit().
locale_translate_edit_form_submit().
_locale_import_po which is called by locale_translate_import_form_submit().

This all leads me down to submit functions from a form or advagg's implementation of hook_module_implements_alter() not working all the time. It could also be a 3rd party module; I've only searched core but http://www.drupalcontrib.org/api/drupal/drupal!includes!locale.inc/funct... doesn't seem like the answer.

3rd party module code could be calling variable_set('javascript_parsed' direct; mind searching your code for that string?

mikeytown2’s picture

joelpittet
Any luck with the backtrace on this?

joelpittet’s picture

Status: Postponed (maintainer needs more info) » Fixed

Sorry nope and i've not seen it again in my travels... let's move this back to fixed.

I've got no other module that seems to touch 'javascript_parsed' in all ~500 modules I have installed except advagg.

mikeytown2’s picture

Thanks for the followup :)

I'm going to guess hook_module_implements_alter() not working all the time as the issue.

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.

interdruper’s picture

In my case, this exception arises without using the advagg module. The origin is inside _locale_invalidate_js() function, from core Locale module, so IMHO it is a core problem.

Installing advagg with this patch probably mitigates or even fixes the issue.