I've noticed sometimes when viewing the themes selection page that all the themes are disabled even the default theme when several themes were in fact suppose to be enabled. We've had a site that has had it's theme switched back to the default when it wasn't suppose to be and we don't know why but believe it has something to do with this. We do use Sections module to have some pages one theme and other pages another theme, but we don't believe the problem lies within the Sections module as the aforementioned issue with the disabled themes on the themes selection page occurs randomly on other Drupal 6 sites we have used.

CommentFileSizeAuthor
#195 305653-drush-updatedb-theme-duplicate-error-195.patch1.87 KBigor.ro
#140 system-rebuild-theme-data-305653-140.patch13.71 KBDavid_Rothstein
#136 system-rebuild-theme-data-305653-136-REMOVE-CACHE_1.patch13.51 KBaspilicious
#130 system-rebuild-theme-data-305653-128-REMOVE-CACHE.patch14.09 KBDavid_Rothstein
#130 system-rebuild-theme-data-305653-128-CLONE-CACHE.patch4.99 KBDavid_Rothstein
#128 system-rebuild-theme-data-305653-128-REMOVE-CACHE.patch14.09 KBDavid_Rothstein
#128 system-rebuild-theme-data-305653-128-CLONE-CACHE.patch4.99 KBDavid_Rothstein
#128 system-rebuild-theme-data-305653-128-TESTS-SHOULD-FAIL.patch5.32 KBDavid_Rothstein
#128 system-rebuild-theme-data-305653-128-REMOVE-CACHE-more-readable.txt6.17 KBDavid_Rothstein
#126 system_info_cache.patch3.23 KBdonquixote
#124 system_info_cache.simplify.patch6.22 KBdonquixote
#118 system_rebuild_theme_data.clone-multicache.patch3.55 KBdonquixote
#123 system_info_cache.patch3.56 KBdonquixote
#121 system_info_cache.patch3.37 KBdonquixote
#120 system_rebuild_theme_data.clone-multicache.patch3.47 KBdonquixote
#119 system_rebuild_theme_data.clone-multicache.patch3.47 KBdonquixote
#115 _system_theme_data.clone_.patch711 bytesdonquixote
#113 _system_theme_data.clone_.patch705 bytesdonquixote
#109 system-rebuild-theme-data-305653-109.patch12.75 KBDavid_Rothstein
#103 system-rebuild-theme-data-305653-103.patch12.11 KBDavid_Rothstein
#90 drupal-update-themes-305653-90-D6.patch1.37 KBJohnAlbin
#88 drupal-update-themes-305653-88-D6.patch841 bytesJohnAlbin
#81 drupal.update-themes_0.patch838 bytessun
#60 drupal.update-themes.patch838 bytessun
#46 drupal6.update-themes.patch958 bytessun
#45 drupal.update-themes.patch1.47 KBsun
#42 theme-disabled-305653-42-D6.patch1.43 KBcdale
#42 theme-disabled-305653-42-D7.patch1.92 KBcdale
#38 themes-disabled-1.patch1.02 KBDave Reid
#38 themes-disabled-2.patch1.26 KBDave Reid
#29 update-disable-themes-305653-29-D7.patch1.34 KBcdale
#29 update-disable-themes-305653-29-D6.patch809 bytescdale
#18 update-disable-themes-305653-18-D7.patch531 bytescdale
#17 update-disable-themes-305653-D6.patch568 bytesDave Reid
#17 update-disable-themes-305653-D7.patch946 bytesDave Reid
#13 7.x-theme-disable-update-305653-13.patch2.01 KBcdale
#13 6.x-theme-disable-update-305653-13.patch2.03 KBcdale
#11 system.module.diff1.11 KBsnowball43
#11 update.php_.diff301 bytessnowball43
#8 update.php_.diff301 bytessnowball43
#8 system.module.diff1.1 KBsnowball43
Support from Acquia helps fund testing for Drupal Acquia logo

Comments

designerbrent’s picture

subscribe

snowball43’s picture

Priority: Normal » Critical

Changing this to critical as it affects sites which allow selection of different themes for their users or sites which use one of the modules that allowing different themes to be used for various pages of the site.

Jacine’s picture

I'm experiencing this as well, but not only with themes - it's happening with modules also, so I'm pretty sure it's not being caused by the theme or theme settings. I've been trying to recreate the problem, but I can't tell when it's being disabled or why because it's very sporadic.

I've also noticed that when I make a change to post information visibility, by node, the changes are not reflected unless I go to the theme specific settings page, and resubmit the form, which is very odd. This may or may not be related, but I figured it was worth mentioning here.

I will keep trying to figure it out...

WindSlash’s picture

I am also experiencing this same problem with Drupal 6.4.

atiras’s picture

Same problem here -- it may be coincidence but it often seems to happen after updating a module.

modctek’s picture

I too am experiencing this particular problem. It's driving me crazy!

designerbrent’s picture

After doing some testing, it appears that the problem lies in the update.php file. If you run that, it will disable all your themes.

snowball43’s picture

FileSize
1.1 KB
301 bytes

I've narrowed the issue down to the function system_theme_data() being called in update.php. The $themes_info array inside the _system_theme_data() function is cached and by resetting the cache, update.php respects each themes current status.

I've made a couple of patches, I'm not sure if this is the best method since these functions do work as is outside of update.php.

NOTE: These patches were created against Drupal 6.4.

snowball43’s picture

Status: Active » Needs review

Forgot to change the status of the issue.

Status: Needs review » Needs work

Patch failed to apply. More information can be found at http://testing.drupal.org/node/14508. If you need help with creating patches please look at http://drupal.org/patch/create

snowball43’s picture

Status: Needs work » Needs review
FileSize
301 bytes
1.11 KB

Let's try again.

Babalu’s picture

thx

cdale’s picture

Version: 6.4 » 7.x-dev
Component: theme system » update system
FileSize
2.03 KB
2.01 KB

This issue also appears to be in 7.x.

I've attached an updated patch against the latest CVS for both 6 and 7, and changed the component to the update system as this seems to more accurately reflect the cause, as the function appears to work fine outside of update.php.

Dave Reid’s picture

See also #147000: Rewrite module_rebuild_cache() and system_theme_data(). Calling system_theme data will do a complete delete and insert for all themes in the system table, which creates a race condition and creates the same problem that has been described here.

cdale’s picture

Steps to reproduce for anyone trying to verify/test this:

1. Go to Site Building -> Themes
2. Make sure that one or more themes are enabled
3. visit update.php (You don't have to run any actions, just going to the page will trigger the bug)
4. Go back to Site Building -> Themes

Without the patch, enabled themes will no longer be enabled. With it, they should still be enabled.

Dave Reid’s picture

Confirmed by visiting update.php with and without site in maintenance mode.

Dave Reid’s picture

Title: Themes get disabled » All themes disabled in update.php
FileSize
946 bytes
568 bytes

Actually, this should be fixed just by changing the call from system_theme_data to _system_theme_data (which is supposed to be used since we don't want to hit the database) instead of adding an unneeded parameter to system_theme_data. I confirmed this patch fixes the problem on a patched HEAD by visiting update.php. My themes were no longer disabled and it did not change the update process.

cdale’s picture

Status: Needs review » Reviewed & tested by the community
FileSize
531 bytes

Nice. :) I didn't even try that, or think that it would work, but I suppose it makes sense. +1 from me.
(I had trouble applying the D7 patch, so I re-rolled it).

Dave Reid’s picture

Patch in #17 should apply cleanly. It passed testing.d.org: http://testing.drupal.org/node/15099

Dave Reid’s picture

Title: All themes disabled in update.php » Themes disabled during update

Clarifying title

cdale’s picture

@ Dave Reid in #19
The D6 patch applied cleanly for me, and the D7 one applied, but spat out a whole bunch of other stuff I haven't seen before, and then patch crashed. It was most likely something quirky on my end. The patch does look fine though. I can't explain it.

webchick’s picture

Version: 7.x-dev » 6.x-dev

To reproduce, enable a couple themes, go to update.php (don't need to do anything else), then revisit the themes page. Your themes are no longer enabled.

After patch, checkboxes stay checked.

Committed, thanks!

Moving back to 6.x.

Pasqualle’s picture

subscribe

najibx’s picture

no success for me.

Unfortunately, applying the patch, change to $themes = _system_theme_data(); in update.php does NOT do any different. The Enabled check box is not checked anymore, but Default yes. So the site is still running with the theme.

ealier without realizing this, whenever configuring blocks, I get Access Denied , because no theme is enabled. err ... iam lucky I realize the issues and found this thread.

Pasqualle’s picture

Status: Reviewed & tested by the community » Needs work

I can confirm, the D6 patch does not work.

Note: to reproduce the error, it is not enough to visit the update.php page, you need to click on the update button..

Dave Reid’s picture

I'll look into this shortly...

Dave Reid’s picture

Ok, wow. I hate these functions so much. Finally figured out what is going wrong. I'll try to explain as clear as I can.

  • On an update page, the first relevant call is list_themes()
  • list_themes() calls _system_theme_data() and then uses:
        $themes = _system_theme_data();
        foreach ($themes as $theme) {
          ...
          // Status is normally retrieved from the database. Add zero values when
          // read from the installation directory to prevent notices.
          if (!isset($theme->status)) {
            $theme->status = 0;
          }
          ..
        }
    
  • When list_themes() modifies that data, those modifications are also saved to the static data variable in _system_theme_data since it is automatically returned from _system_theme_data by reference.
  • Now when whenever we call system_theme_data (which calls _system_theme_data within it), all the $theme->status variables are set to 0.
  • system_theme_data calls system_get_files_database on the data from _system_theme_data.
  • Since the status key is set, the system_get_files_database function will not override the status key with the current status from the {system} table.
  • This data (including every theme's status->0) is now saved to the {system} table within system_theme_data, disabling all themes.

Where can we go from here to solve this?
1. We implement a $reset = TRUE parameter as was proposed in #13 for _system_theme_data so that system_theme_data will always get a fresh copy and not the saved static copy that could have been modified. Any calls to system_theme_data should result in the correct data.
2. Change system_get_files_database so that if a column in the {system} table is different from a key in the module/theme data, grab the value from the {system} table record. Any calls to system_theme_data should result in the correct data.
2. Make list_themes work a little smarter and not have to set status = 0 for all the themes.

I'll work on a patch...

I hate these system_theme_data functions. A lot.

detot’s picture

subscribe

cdale’s picture

Status: Needs work » Needs review
FileSize
809 bytes
1.34 KB

I must admit, I didn't test the latest patch all the way through an update, I did however test the one in #13 all the way through, so I know that that solution will work, however I do not really like it, as caching is a good thing. So that rules out option number 1.

Option 3 seems impossible to achieve. At least I can't find away.

So this leaves option 2. In this light, I've put together a nice simple patch that I've tested and seems to work well.

Let me know what people think.

cdale’s picture

Version: 6.x-dev » 7.x-dev

Moving back to 7.x

Dave Reid’s picture

Re #29, this is more what I was thinking:

     if (isset($files[$file->name]) && is_object($files[$file->name])) {
       $file->old_filename = $file->filename;
       foreach ($file as $key => $value) {
-        if (!isset($files[$file->name]) || !isset($files[$file->name]->$key)) {
+        if (!isset($files[$file->name]->$key) || $files[$file->name]->$key != $value) {
           $files[$file->name]->$key = $value;
         }
       }

The check for !isset($files[$file->name]) is removed since it will never hit because we check isset($files[$file->name]) three lines above it.

Dave Reid’s picture

Status: Needs review » Needs work

To test this on your own:

$changed_themes = _system_theme_data();
foreach ($changed_themes as $a_theme) {
  $a_theme->foobar = 'ferzle';
  $a_theme->status = 0;
}
$unchanged_themes = _system_theme_data();
var_export($unchanged_themes);

You will see that the $unchanged_themes variable has the changes made in the $changed_themes variable. A solution to fix this issue should not let that happen.

snowball43’s picture

#29 - cdale
Caching is a good thing though I do not think it is needed to the same extent as a regular Drupal page for update.php as update does not need to run all too often. The caching is what is creating the problem because somehow the $themes_info array is not getting fully set like it does on a normal Drupal page when the function _system_theme_data is called. If we can find the reason for the $themes_info array not getting fully loaded correctly, the issue will be solved.

cdale’s picture

I believe I've found the problem, although I'm not sure on the best way to fix it.

From a comment I found at php.net

On array copying a deep copy is done of elements except those elements which are references, in which case the reference is maintained. This is a very important thing to understand if you intend on mixing references and recursive arrays.

As each of the themes within the returned array is an object, a reference to the object is returned, causing any changes done, to actually affect the static $themes_info in _system_theme_data.

Does anyone have any suggestions on how we could cleanly make sure the returned objects in the array are copies? It would seem that looping might be the only way to achieve it.

bengtan’s picture

Hi all,

Here's a totally different approach to the problem.

Everyone's concentrating on the fact that status gets set to 0 for these themes.

Is anyone questioning why it gets written to the database?

It's a cached copy, isn't it? Why are we doing database inserts of (arguably incorrect) cached copies which we (arguably) may not have to?

update.php calls update_check_incompatibility() which calls system_theme_data() which unconditionally always writes the theme info structure to the database, including incorrect status values.

Can we add a param $update_db = FALSE to system_theme_data() so it doesn't write to the database when it is called by update.php?

[EDIT]

Hmmm... update.php also calls drupal_flush_all_caches() which calls system_theme_data(), so that's another write to the database which needs to be neutered.

Dave Reid’s picture

@35. The problem is still any call to either _system_theme_data or system_theme_data will return the invalid status=0. So if were trying to list themes, show current theme, do anything with themes, we're now dealing with invalid data. True, this should also avoid loading back to the database. I'm going to work on an end-all patch tonight for this issue.

bengtan’s picture

@36:

Ah okay, fair enough. I didn't think of it that way.

You (and others) have spent more time on the subtleties, so I will defer.

Thank you.

Dave Reid’s picture

Ok after a metric-crap-load of testing, I can really only see two approaches to fix this problem of the themes array being deep copied.

1. Perform a shallow copy on the themes array in _system_theme_data so that modifications to the themes array don't modify the static variable.
2. Implement a $reset parameter in _system_theme_data that will not use/trust the cached static variable in the cases that data will be written to the database.

Personally, I prefer approach #2, but let's get some more input.

cdale’s picture

Good work on sorting this all out Dave.

I'm impartial to which method is ultimately chosen, so long as it works.

Here's what I think the pro's and con's of each method are:

Method 1: Implicit Shallow Copy
Pros:
  1. No change to current interface
  2. users of system_theme_data do not need to consider if they need a 'clean' copy, and they don't need to worry about the case where another function/module has altered the data when perhaps it should not have.
Cons:
  1. Potential for higher memory usage, and longer execution times.
Method 2: Explicit reset
Pros:
  1. Caller is guaranteed to get a clean copy of the data when they request it.
  2. Caller can use the static value if they don't care about the integrity of the data.
Cons:
  1. To guarantee integrity, callers will have to use the reset.
  2. Forces the caller to use reset when they are saving information to the database.
  3. More potential for subtle bugs. (Easily fixed by adding the $reset = TRUE, but only if found.)

Given all that, I would say that I would lean more towards number 1. These are my opinions, and I'm happy to have them debated and proven wrong. :) Just looking for the best solution.

mlncn’s picture

Congratulations, Dave Reid.

Tested patch #2 from comment 38 and it applies fine to Drupal 6 with a fair amount of fuzz. It fixes the problem of no theme enabled for a non-default installation profile.

I would say this is ready to go in, but I know it has to be tested and applied against Drupal 7 first.

@cdale, I think calling reset when data is involved is best– this really applies to installation and updating, so it's Drupal core's problem, correct?

I fear this will go back and forth a few more times but, already, thanks very much for work, in particular by Dave.

benjamin, Agaric Design Collective

Pasqualle’s picture

Status: Needs work » Needs review

I think with #38 patch 2, you did not solved your test case in #32. The question: is it necessary to pass that test or that test is wrong? What is the expected functionality for _system_theme_data() function?

Also patch 2 is an API change. So to get accepted for D6, then it should be verified that it is the best solution.

I would like to see this bug fixed before the next release..

cdale’s picture

I've been using patch #1 and it works great. No issues what so ever.

Having looked at patch #2, if that is the method that people prefer, then I think some other changes might be required. For example, going though core, the majority of calls are to system_theme_data, not _system_theme_data, and the calls to _system_theme_data are only on maintenance pages, or pages that are not allowed to hit the db. So with patch #2, what is the purpose of leaving $themes static, as from what I can tell, $themes will be rebuilt every time they are requested under normal usage as $reset will always be true. Is this ideal?

I'm not meaning to sound overly critical, I think Dave Reid has done a great job, I'm just trying to find the best solution to a problem, and I personally feel that patch #2 is not it given the methods that system_theme_data is used throughout core.

Is there any particular reason people prefer patch #2 over patch #1?

Perhaps another approach could be to use the clone method, but only when the caller knows it will be modifying the data in a way that other calls would not desire.

I've attached a patch that I think demonstrates what I mean by that approach. The patch also reverts the change committed in #22.

Does this seem like a better idea to most?

snowball43’s picture

The solution in #17 works perfectly except for when the updates are actually made, that is when the batches get processed. If we can find what is most likely calling system_theme_data and change it to _system_theme_data or if another method is to blame, fix it there.

sun’s picture

Status: Needs review » Needs work

Testing this faster in current HEAD:

UPDATE system SET schema_version = '7010' WHERE name = 'system';

Afterwards, execute update.php, update, and see no theme enabled.

As for the patches, I tend to the $reset variant. Analyzing now.

Also, in D7 use "clone", in D6 we use drupal_clone() to clone data structures.

sun’s picture

Status: Needs work » Needs review
FileSize
1.47 KB

The solution is actually trivial:

In maintenance mode, do not update theme data in the database at all.

update.php's update_check_incompatibility() already cares for disabling modules and themes that are incompatible with the current version of core, so we can safely avoid this database update.

This patch also reverts the original patch the has been committed. So after applying this patch, just visiting update.php is sufficient to see that themes are no longer disabled.

sun’s picture

FileSize
958 bytes

Same patch for 6.x.

catch’s picture

Status: Needs review » Reviewed & tested by the community

Confirmed the bug and that the D7 patch fixes it. RTBC.

Pasqualle’s picture

Status: Reviewed & tested by the community » Needs review

I think in D6 the maintenance_mode is not set at update..

this code is used in D6 for the update script check:

strstr($_SERVER['SCRIPT_NAME'], 'update.php')
Pasqualle’s picture

Status: Needs review » Reviewed & tested by the community

cross post

sun’s picture

@Pasqualle: It is, see line 22 of update.php.

cdale’s picture

+1 from me on sun's patch. It's so simple, and works beautifully.

It still does not pass the use case in #32, but I feel that this may be outside the scope of this issue now, as this is only a problem when running in maintenance mode.

sun’s picture

I have no clue why the D7 patch was reported to fail. Detailed results page even mentions that it was not tested in testbed at all. Patch still applies cleanly.

Dries’s picture

Why do we call system_theme_data() to begin with?

sun’s picture

ok, after analyzing the whole process once more:

  • update.php invokes system_theme_data() to gather all themes and test them for Drupal core compatibility.
  • update.php also invokes _drupal_maintenance_theme(), which invokes list_themes() and _system_theme_data(). Note that those calls do not cause this bug, because _system_theme_data() does not alter the database. Only system_theme_data() alters the database.
  • However, #35: When update.php finishes, we call drupal_flush_all_caches(), which invokes system_theme_data().

This effectively means that we could probably place the same or similar code into drupal_flush_all_caches(), and also ensure that we ALWAYS invoke _system_theme_data() in update.php and theme.maintenance.inc... - actually no, since we still want to flush all theme data when drupal_flush_all_caches() is invoked in maintenance mode.

So the proper place to prevent a database update with malformed values is in system_theme_data(). Additionally, I think we should avoid invocations of special maintenance mode functions where possible and leave it to the subsystem to properly handle the maintenance mode case. That helps in demystifying our install/update/maintenance mode process.

Babalu’s picture

subscribing

bejam’s picture

Will the patches fix this problem on an existing DB i.e. on that is already getting the problem? Or do I need to do something manually to clean out dirty data?

bengtan’s picture

@46: Don't use this patch!!

Sorry, Sun, I'm not rubbishing your patch. I'm only shouting so people don't use it.

The patch in #46 fixes update.php, but unfortunately, it breaks a fresh install of drupal.

@53: I concur with this line of questioning.

bengtan’s picture

Going along the lines of reasoning of #35 and #53 ... why does drupal_flush_all_caches() want to call a function that writes to the database?

This just seems wrong (IMO). Flushing implies reading new data from the database, not writing data to the database (regardless of whether said data is correct or not).

BTW, the call to system_theme_data() in drupal_flush_all_caches() was added in includes/common.inc 1.756.2.27, between Drupal 6.2 and 6.3, for #239958: Clearing cache does not immediately reload theme's .info file

sun’s picture

Status: Reviewed & tested by the community » Needs work
sun’s picture

Status: Needs work » Needs review
FileSize
838 bytes

Reverting back to second option mentioned in my last summary. Works from my own testing.

To test this:

1) Go to admin/build/themes and ensure that at least one theme is enabled.
2) Execute the query in #44.
3) Visit update.php, but do not execute updates.
4) Go to admin/build/themes and confirm that the theme(s) is/are still enabled.
5) Visit update.php and execute updates.
6) Go to admin/build/themes and confirm that the theme(s) is/are still enabled.
7) Nuke your Drupal HEAD test site.
8) Install from scratch and confirm that Garland theme is enabled in admin/build/themes.

And, yes, this adds cruft to drupal_flush_all_caches(). However, I do not think that duplicating that function's contents into update.php would be a good idea.

bengtan’s picture

I got a bit confused about versions of core, so just clarifying ...

My comment #57 refers to Drupal 6.6. Sun's patch #60 refers to Drupal 7.x.

Some testing:

I tried the patches from #46 and #60 together against Drupal 6.6 (I backported #60 by hand myself, as it's not meant for 6.6).

Result: update.php no longer disables theme's, but when installing a fresh installation, the theme is not enabled. Breakage.

sun’s picture

Forget previous patches. Only apply #60. It's supposed to be applied on D7/HEAD, since D6 needs an additional tweak to get this working. However, D6 will be fixed after this patch hit D7.

hass’s picture

sub

starkos’s picture

subscribe

dbeall’s picture

subscribe

Frank Steiner’s picture

This bug has ugly side effects, at least in D6. E.g. that custom blocks created by users are not written to the blocks table in block_add_block_form_submit (as it checks for enabled themes) but only in _block_rehash. This causes the block title which the user entered in the form to be lost.
And it makes hook_block('save') in any custom module completely useless for user-created blocks because the block is not yet written.

Hoping for a soon fix in D6...

korvus’s picture

subscribe

xmacinfo’s picture

The side effects are for all functions that lists modules. For example themes are not longer tracked by update status. The System info module gives out an empty theme list, etc.

I believe that Drupal 7 will be fixed before D6, see comment #62.

Status: Needs review » Needs work

The last submitted patch failed testing.

Dave Reid’s picture

Status: Needs work » Needs review

Failed due to #74645: modify file_scan_directory to include a regex for the nomask.. Setting back to code needs review.

wu-wei’s picture

subscribe

encho’s picture

subscribing

catch’s picture

This is an easy patch to test, why not try it and report back? Might even get fixed that way...

Frank Steiner’s picture

I would like to do that but I don't have D7, only D6 and sun said that there was some additional patch needed which we get only after D7 has been fixed :-(

I can report that it works fine for my installed D6, knowing that it is supposed to break a fresh installation.

Dave Reid’s picture

webchick, I think this should be a 7-0-UNSTABLE-4 blocker to agree on a solution and get this fixed and then backported.

hass’s picture

This should also become a blocker for a D6.7 release.

Pasqualle’s picture

it does not blocked D6.6 so I don't see the point why it would block the next release..
It's just a critical D6 bug as any others

Frank Steiner’s picture

Hmm, most of the post in the list you linked to are not really critical bugs as they don't influence any other stuff. And many of the post end with "Oh, found the bug, was my fault" or sth.

This bug influences a lot of other modules, like stopping custom blocks and hook_block from working correctly (just to name the side effect that affects me), and this is bit more severe than most of the other "critical" bugs in your link...

toddchris’s picture

subscribe

spopovits’s picture

I have noticed that theme gets disable after cron runs too! Is this related to this issue or separate (this is for drupal 6 but thought it might give you more information)

sun’s picture

FileSize
838 bytes

Forcing re-test of last patch.

JohnAlbin’s picture

Status: Needs review » Reviewed & tested by the community

If your system has "no theme enabled", custom blocks (and Menu block module blocks) can't save their settings because those settings are saver per-enabled-theme. :-p Pretty critical. But debating whether this is a 6.7 blocker is pointless since a security bug will always force a new 6.x release regardless of any pending critical bugs.

Too much subscribing (we really need to fix the comment-to-get-email-updates "bug") and not enough testing!

Before I applied the patch, I created a bogus book_update_6001() to test. I can confirm that running update.php will nix any enabled themes. I can also confirm that running cron.php does not disable themes, spopovits. This is with all core modules enabled and no contrib modules.

After I applied the patch, I created a bogus book_update_6002() to test. And I can confirm the patch fixes the issue.

hass’s picture

John: You don't need to add any bogus update hooks... simply run update.php to clear caches and all themes are disabled.

nonsie’s picture

subscribe

ñull’s picture

subscribe

pierrelord’s picture

subscribe

webchick’s picture

Version: 7.x-dev » 6.x-dev
Status: Reviewed & tested by the community » Patch (to be ported)

Committed this to HEAD. Thanks!

Moving back to 6.x.

JohnAlbin’s picture

Status: Patch (to be ported) » Needs review
FileSize
841 bytes

Trivial re-roll of sun’s patch. Should be RTBC, but I’ll let someone else double check.

sun’s picture

Status: Needs review » Needs work

For D6, we also need at least the changes from the patch in #18.

JohnAlbin’s picture

Status: Needs work » Needs review
FileSize
1.37 KB

Done.

domesticat’s picture

The patch in #90 applied cleanly for me on two different sites running 6.6 and appears to fix the problem.

sun’s picture

Status: Needs review » Reviewed & tested by the community

I'd say RTBC then.

andypost’s picture

subscribe, patch applied for live site which often breaks

andypost’s picture

I've open #344625: Themes disabled on cron because this patch work only for update.php which is deleted on site but after 7 hours site "forget" about custom theme again

bengtan’s picture

+1 The patch in comment #90 works fine for me for both invoking update.php and a fresh install.

Gábor Hojtsy’s picture

Status: Reviewed & tested by the community » Fixed

Committed to 6.x, thanks!

Status: Fixed » Closed (fixed)

Automatically closed -- issue fixed for two weeks with no activity.

donquixote’s picture

I'm afraid you fixed only symptoms of the bug with _system_theme_data().
See #632080: _system_theme_data() should clone the returned objects. .

donquixote’s picture

This issue's reported problem might be fixed, but the underlying technical problem is still waiting for the next chance to make trouble. I mark this as "needs work", because I would like the attention of those people who worked on this issue.

----

A full explanation is in the issue linked to in #632080: _system_theme_data() should clone the returned objects., but here the short version:

The cache in _system_theme_data() is designed to remember the results from a system directory scan. This information does usually not change within one request, so it should not be a problem.

system_theme_data() brings in additional information from system_get_files_database(), which is not supposed to be cached. We expect fresh information if we call system_theme_data() a second time. Nothing should break!

And here's the problem: The information from _system_theme_data() contains objects, which are passed as pointers (by-reference, if you want). Now the information from system_get_files_database() is stored in these objects and thus cached in the static variable of _system_theme_data(). A second call to system_get_files_database() will not overwrite these values.

----

Please, guys, have a second look at this!
The other issue contains a patch.

----

EDIT:
Reading more of the old posts, I noticed that #27 (Dave Reid) has already seen this. And #39 and #42 already propose to clone stuff. So, I was too fast, again.

But:
For some reason the clone idea was rejected for the final patch. The issue focused on fixing update.php. But, #632080 shows there are other situations where the side effect is undesired. The one situation that I found might be a rare case (only applies with devel automatic redirection turned on), but I would not be surprised for other situations where system_theme_data() is called twice.

I think the clone idea should be reconsidered.
If this all worked in PHP4, which does automatically clone everything, I don't see why explicit clone would cause any problems. The safest would be to always clone, the only problem with this might be performance.

klonos’s picture

Status: Closed (fixed) » Needs work

subscribing...

Jerome F’s picture

Does this explain why i get this error message ?

warning: array_map() [function.array-map]: Argument #2 should be an array in /Applications/MAMP/htdocs/elephant_site_pilotage_dev_v_0_1/modules/system/system.module on line 1019.
warning: array_keys() [function.array-keys]: The first argument should be an array in /Applications/MAMP/htdocs/elephant_site_pilotage_dev_v_0_1/includes/theme.inc on line 1817.
warning: Invalid argument supplied for foreach() in /Applications/MAMP/htdocs/elephant_site_pilotage_dev_v_0_1/includes/theme.inc on line 1817.

I recently switched from one theme to another one, so this could come from elsewhere I just wonder.

butler360’s picture

Subscribing

David_Rothstein’s picture

Version: 6.x-dev » 7.x-dev
Status: Needs work » Needs review
Issue tags: +D7 upgrade path
FileSize
12.11 KB

@donquixote is correct - this bug is not fixed yet.

All it takes to trigger this again is one contrib module which happens to call system_rebuild_theme_data() somewhere in the update process. I experienced some pain with this last week, and I think the "culprit" wound up being in Drush, although it doesn't really matter since this is ultimately a core bug... In fact, this actually occurs in core alone right now, but only during the D6->D7 upgrade, since block_update_7004() forces a theme rebuild. Sites upgrading from D6 to D7 currently get all their themes disabled.

We need to make system_rebuild_theme_data() safe to call from anywhere. I think the simplest fix is just to remove the static cache from _system_rebuild_theme_data(). Drupal has a habit of statically caching everything just because it can, but this is a bad habit :) In particular, this function is designed to only get called very very rarely, in cases where we don't care much about optimizing performance. Furthermore, to actually take advantage of the static cache requires that it be called twice on the same page load. That may well be happening, but I would suggest that if system_rebuild_theme_data() is actually getting called twice on the same page load, that is something that should be fixed in the calling code.

Removing the static cache also has the following benefits:

  1. We get rid of some random drupal_static_reset() calls that are pretty ugly.
  2. We have two places in core that currently call _system_rebuild_theme_data() without ever using its return value, which can serve no purpose unless that somehow bizarrely primes the static cache for real use later on? The one in modules/system/system.updater.inc (see attached patch) seems like it's surely a bug that prevents that functionality from working correctly (I haven't tested, though).
  3. This static cache caused some bizarre problems in #651086: Cache clearing is an ineffective mess in module_enable() and system_modules_submit() as well, and I found at the time that removing it helped things a lot.
  4. Finally, I think the original fix committed above introduced a bug of its own? When running update.php, we do need to make sure that the rebuilt theme data gets inserted into the database as early as possible (because people very commonly go to update.php after changing the code in their filesystem, so the data in the {system} table is likely out of date and needs to be replaced). I have added a code comment to this effect in the attached patch - one could reasonably argue that update_check_incompatibility() is not the best function to do this in, but changing that would be outside the scope of this issue.

Note that the attached patch is much smaller than it looks, since most of it is just indentation changes after removing the static cache. My light testing suggests that this does not noticeably affect performance during installation or updates, but if it does, as mentioned above I think we need to look at why there is calling code that rebuilds the theme data multiple times in a row, and fix it there - probably in a separate issue.

andypost’s picture

I think we should find all cases when we really need to scan themes' files.

PS: http://api.drupal.org/api/function/_system_rebuild_theme_data/7 does not show all references to this function...

David_Rothstein’s picture

Yes, several places in core probably still scan the filesystem and do a full rebuild, even though calling http://api.drupal.org/api/function/system_get_info/7 or an equivalent custom database query would be sufficient. I think that is probably a separate issue.

For _system_rebuild_theme_data(), I think the only place this patch misses that we might want to change someday is list_themes(), which does this:

    if (!defined('MAINTENANCE_MODE') && db_is_active()) {
      foreach (system_list('theme') as $theme) {
        if (file_exists($theme->filename)) {
          $theme->info = unserialize($theme->info);
          $themes[] = $theme;
        }
      }
    }
    else {
      // Scan the installation when the database should not be read.
      $themes = _system_rebuild_theme_data();
    }

In theory we want to remove the MAINTENANCE_MODE check and leave only the db_is_active() one. Naively removing it leads to some subtle errors, though, so I did not include it in the above patch... I think it would be better as a potential followup issue.

catch’s picture

I'm not sure if that static is new or old, but system_rebuild_theme_data() in system_install() is likely one of the reasons for #686196: Meta issue: Install failures on various hosting environments. This whole area is a huge mess, but if we remove the static we also need to ensure it's not being called twice in the same request in core anywhere, otherwise our memory / timeout issues are only going to go downhill.

andypost’s picture

donquixote’s picture

I just want to point out that D7 and D6 look very different. system_rebuild_theme_data() does not even exist in D6. We will need different solutions for each.

Shouldn't we get #632080: _system_theme_data() should clone the returned objects. into D6 already? There will be no classical backport path, if D7 looks so different.

David_Rothstein’s picture

@donquixote, system_rebuild_theme_data() was previously called system_theme_data() in D6, but otherwise I don't think they're that different. However, in D6 we definitely want the least disruptive fix possible.

@catch, it looks to me like the static was added in Drupal 6. (Interestingly, it doesn't seem like there was ever an equivalent static for scanning modules, even though I assume that is more expensive since most sites have a lot more modules than themes lying around...)

I checked for duplication and it does appear that on every page request during install or update, the maintenance theme is needlessly calling _system_rebuild_theme_data() twice. The attached patch adds a fix for that by simply reusing list_themes(), which has its own static cache. This gets it down to a maximum of one call in most cases. However, we still get multiple calls when we are explicitly rebuilding the theme data (one call for the theme system, one call for the rebuild) - and system_install() is an example of that. However, I don't get the impression that system_rebuild_theme_data() is the bottleneck for the crazy timeout issues in #686196: Meta issue: Install failures on various hosting environments at all - do you? That function call was there in D6 as well, and doesn't seem to have changed much internally except for the stream wrapper stuff. Probably the solution to that issue is going to involve breaking system_install() up into multiple steps anyway.

Also, I would assume that getting rid of a needless static cache would (if anything) reduce memory consumption rather than add to it, right? Some quick testing with the command line installer suggests that this is the case, although the reduction is probably negligible, ~35 KB or so.

donquixote’s picture

Re: David Rothstein's patch (#103, #109)
Totally removing a static cache from system_theme_data() / system_rebuild_theme_data() will invite calling functions (core or contrib) to introduce their own static cache for the same data. These new static caches could be even more out of control than the existing one.

So, wouldn't it be better to think about what exactly we want to cache, and when it needs to be reset?

-----------

system_rebuild_theme_data() was previously called system_theme_data() in D6, but otherwise I don't think they're that different. However, in D6 we definitely want the least disruptive fix possible.

Interesting!
Some thoughts (a bit unstructured and brainstormy-like):

I see different cases:

  1. We scan the filesystem for theme definitions, and write that information to the database. This is what "rebuild" is about. Scanning the filesystem should not be done more than 1x per request - trying to catch up with possible file changes during one request would be insane. So, we can safely cache this information in a static var, BUT we should really only cache the stuff that is coming from the filesystem, not anything else!
  2. Usually a rebuild process takes more than just information from the filesystem. In our case, it depends on previous contents of the system table (theme enabled or disabled, etc). If we cache this info, we need to make sure it is reset when necessary.
  3. We load stuff from the database (system table), but we don't rebuild anything. We can cache this, but we need a reset option. The reset will only be called if some other function is writing to the database (switching a theme on or off, etc), or doing a full rebuild. It can be a lazy reset, and we don't need to re-scan the filesystem.

It would be nice if these differences were reflected by code / function names / documentation with more clarity.

The new function name suggests that calling the function will actually do a rebuild. Thus, with the new name it is only logical to remove the cache. Still, there could be some functions that don't need a rebuild, but just want the data from the database, or from a static cache. Should we provide a separate function for this use case? Or does it exist already?

----------

Some useful links:
http://api.drupal.org/api/function/system_theme_data/6
http://api.drupal.org/api/function/_system_theme_data/6
http://api.drupal.org/api/function/system_rebuild_theme_data/7
http://api.drupal.org/api/function/_system_rebuild_theme_data/7

Btw, I noticed that _system_rebuild_theme_data() still has the same problem of objects that are not cloned.

David_Rothstein’s picture

I forgot to mention above that I also tried poking around other pages in Drupal that would be expected to call system_rebuild_theme_data() and found that most only call it once per page request currently (visiting the modules page, submitting the modules page, hitting the "clear caches" button, etc). Visiting admin/appearance was an exception. There it is called twice, once by the menu callback function and once by update.module when adding its hook_help().

Other than that, it appears that outside of install.php and update.php, this static cache has zero benefit in Drupal core itself.

Still, there could be some functions that don't need a rebuild, but just want the data from the database, or from a static cache. Should we provide a separate function for this use case? Or does it exist already?

There is http://api.drupal.org/api/function/system_get_info/7 which is useful for some of these cases, but not all of them.

One thing that does occur to me, if we want to retain some kind of static cache - perhaps what we really want to retain is the fact that the filesystem was already scanned on this page request? But we don't necessarily need to cache all the data itself. It should be cheap to get that from the database again. That would not help install.php, but would be very clean for everywhere else. Something like:

function system_rebuild_theme_data() {
  $file_scan_performed = &drupal_static(__FUNCTION__, FALSE);
  if (!$file_scan_performed) {
    // ... Do exactly what the function does now. Then before returning, set this...
    $file_scan_performed = TRUE;
  }
  else {
    // Query the database and return everything in the same format that is returned above.
  }
}

This would have the advantage that multiple modules could declare "I really need to make sure I'm getting the most up-to-date data from the filesystem" without ever triggering multiple filesystem scans on the same page request (and without maintaining a giant buggy static cache that is rarely going to be used).

I think that any of these more complex solutions are good, but require more of a rewrite of these rebuild functions than we ideally want to do in this issue :) On the other hand, it's quite possible that some kind of major rewrite is going to be required to correctly solve #651086: Cache clearing is an ineffective mess in module_enable() and system_modules_submit() anyway.

donquixote’s picture

Did a bit of digging.
_system_rebuild_theme_data() gets everything from the filesystem, scanning .info files. The scan depends on the site name, but we can assume that this won't change during one request.

There is one exception: drupal_alter('system_info', ..). Duh! It can easily happen that this will yield different results if called a second time, if in the meantime we changed some configuration here and there.
http://google.com/codesearch?hl=de&lr=&q=function%5C+%5Cw%2B_system_info...

Aside of this one issue, it would be safe to cache the result of _system_rebuild_theme_data() in a static var, and this is exactly what happens in the current implementation. The idea is to cache what is coming straight from the filesystem.

But, there is another problem, more severe than the drupal_alter() thing: The result of the cache is returned as an object, (that means, by reference / as a pointer, or how you prefer to call it). This means that calling code can easily change the contents of the static cache, and it does! Which totally kills the idea of caching only what comes from the filesystem. Which is exactly what #632080: _system_theme_data() should clone the returned objects. attempts to fix. As I see now, #632080 can be perfectly adapted for D7.

With #632080 done, the remaining issue will be the drupal_alter() call.

Solution: Split the function in two parts.
The first does only scan the *.info files, and cache that information in a static var. The second will take this information, call drupal_alter(), and do some minor computations to determine the base theme etc (this happens after drupal_alter(), so we can't cache it). This second function can also have a cache, but if it does, we need to make sure that we don't forget to reset it when necessary.
Both parts of the function will clone the cached data, and never return the original, OR the data has to be wrapped into an object with read-only interface.

We could also implement this cache directly into file_scan_directory(), so it would also work for *.module or *.tpl.php scans.

I could prepare a patch for this solution, but I think we already win much with #632080 alone.

-----------

One thing that does occur to me, if we want to retain some kind of static cache - perhaps what we really want to retain is the fact that the filesystem was already scanned on this page request? But we don't necessarily need to cache all the data itself. It should be cheap to get that from the database again.

I could agree with this, but:

  • The information from file_scan_directory() is modified before it goes to the database, and the modifications can depend on a lot of dynamic things, unpredictable for us. The contents of the system table does not reliably reflect the contents of the info files.
  • _system_rebuild_theme_data() does not directly write to the system table. In fact, we have no guarantee that a function (indirectly) calling _system_rebuild_theme_data() will ever write anything to the database. This means, the second time that _system_rebuild_theme_data() is called, the database could still contain outdated content. We don't want to change this, because _system_rebuild_theme_data does not have all the information it would need, before it can write to the system table.

If you want to replace the static cache with a db solution, you need to introduce a place in the database to store the raw, unaltered *.info data. This could be a good idea anyway, to avoid too many directory scans! Still, I think a static cache wouldn't hurt that much:

maintaining a giant buggy static cache that is rarely going to be used

One side effect of the "clone the returned objects" is that the original cache will not consume as much memory as it would if other modules have the chance to stuff it with additional data. For a site with 20 themes, this is the data from 20 theme info files - usually not that much, compared to other data we keep in memory. And the few people that have 50 or more themes on one site can be expected to have bigger memory.

donquixote’s picture

Very simple D7 clone patch.

Status: Needs review » Needs work

The last submitted patch, _system_theme_data.clone_.patch, failed testing.

donquixote’s picture

late night brings stupid mistakes

donquixote’s picture

Status: Needs work » Needs review

..

donquixote’s picture

Duh!
drupal_system_listing() depends on the install profile:
global $profile
I hope this doesn't change too much.

donquixote’s picture

Status: Needs work » Needs review
FileSize
3.55 KB

A more complete solution that attempts to solve the drupal_alter problem as described in #112.
Probably buggy. I don't have a D7 testing infrastructure at the moment.

I think we should do #115 first, and then plan this as a second step.

Btw, _system_rebuild_module_data() has the same problem.
EDIT: No, it does not. module data is not cached.

donquixote’s picture

the usual problems: line endings etc.

donquixote’s picture

#118 is going to break thanks to a missing ")".

donquixote’s picture

FileSize
3.37 KB

Alternative solution, where the cache cloning goes into a reusable function.
Probably buggy.
I still recommend #115 for an immediate solution.

Status: Needs review » Needs work

The last submitted patch, system_info_cache.patch, failed testing.

donquixote’s picture

FileSize
3.56 KB

Finally have my testing environment :)

donquixote’s picture

Status: Needs work » Needs review
FileSize
6.22 KB

Same as #123, but replaced $themes[$key] with $theme in the foreach() loops.
Btw, I have the feeling that system_find_base_themes() could be a lot simpler.

EDIT: "Image field validation tests (ImageFieldValidateTestCase) [Image]"
How is this related??

Status: Needs review » Needs work

The last submitted patch, system_info_cache.simplify.patch, failed testing.

donquixote’s picture

FileSize
3.23 KB

#123 had a print_r().
And sorry for the patch spam. If you know a better way, let me know.

donquixote’s picture

I fail to understand why #124 fails, while #126 is fine.

Plan of action:
We can either go for #115, which doesn't need much change imo, and postpone #126.
#126 was written with patch readability in mind, it needs some cleanup before commit.
The changes in #124 are purely cosmetic.
I wait for feedback from others, before I continue.

@David Rothstein:
#109 might be an alternative, but ...
could you upload a version of #109 without the indentation changes? if (TRUE) can do wonders :)
Patch reviewing on Windows is a pain.
(or is this too much lazyness on my side?)

David_Rothstein’s picture

Let's go with one of the simpler solutions to fix this bug and postpone the rest until later. I think the other work you started might definitely come in handy over at #651086: Cache clearing is an ineffective mess in module_enable() and system_modules_submit() though...

Actually, whichever patch we go with here should contain most of the cleanups I had in #109, since those were generally in the codebase as a workaround for this bug and therefore aren't (or at least shouldn't be) necessary anymore. Removing them as part of this patch is therefore important and helps test that it works. I've also gone ahead and written an actual test as well :)

So, attached we have:

  • A patch which removes the static cache, based on #109, and removes all the workarounds.
  • A patch which clones the static cache, based on #115. It turns out we still can't get rid of the drupal_static_reset() calls in this case, because it leads to the infamous "toolbar in the overlay" bug from #651086: Cache clearing is an ineffective mess in module_enable() and system_modules_submit() - so with this solution, the static cache is still somewhat problematic and requires a workaround or two - at least until we deal with that other issue. (However, the other workarounds are removed in this patch.)
  • A patch which only removes the workarounds and adds the test. This is just to make sure the test fails when we want it to.
  • Another version of the first patch, but without all the indentation changes. This is just for reading and ease-of-reviewing pleasure :)

Let's let reviewers decide which to go with.

Either of the first two seem OK to me, but I personally prefer the first one a bit since I think it leads to a simpler, cleaner codebase. Plus, as mentioned, we already don't have a static cache on _system_rebuild_module_data(), even though that function is probably more expensive than this one (and gets triggered two or even three times per page request when visiting admin/modules) - and the world hasn't ended as a result of that :)

Status: Needs review » Needs work

The last submitted patch, system-rebuild-theme-data-305653-128-TESTS-SHOULD-FAIL.patch, failed testing.

David_Rothstein’s picture

Well, that proved my point - but now let's reupload the actual two patches so the testbot will leave things at "needs review" :)

catch’s picture

Status: Needs review » Needs work

While the clone cache patch looks OK, the remove cache patch has dodgy comments in it:

+  // Find themes
+  $themes = drupal_system_listing('/\.info$/', 'themes');
+  // Find theme engines
+  $engines = drupal_system_listing('/\.engine$/', 'themes/engines');
+

I think David's right that the themes directory is usually much smaller than the modules directory, so could probably live without the static cache otherwise.

David_Rothstein’s picture

Status: Needs work » Needs review

All I did in the middle part of that function was adjust the indentation due to the removal of the if() statement. I didn't try to improve any of the existing dodgy comments :)

donquixote’s picture

andypost pointed out in #632080-11: _system_theme_data() should clone the returned objects. that there are some D6 sites such as ThemeGarden, that use a lot of themes.

If we notice performance issues after removing the cache, I propose that we add a cache to drupal_system_listing(). This can be discussed in a separate issue.
With a clever implementation, this cache will be totally independent of global state, and will only have to be reset if someone actually does change in the filesystem.

Would we need to be afraid of memory issues? I think not. We will only cache a minimum of data for every file found in a filesystem scan, and we make sure that no other functions can spoil this cache with additional data. Other cached variables are bigger - think of theme registry.

catch’s picture

Status: Needs review » Reviewed & tested by the community

First patch from #130 looks RTBC. Just in case, it's this one: http://drupal.org/files/issues/system-rebuild-theme-data-305653-128-REMO...

Dries’s picture

Status: Reviewed & tested by the community » Needs work

This patch no longer applies. Can we get a quick reroll of the 'REMOVE-CACHE' patch?

aspilicious’s picture

Hopefully I got everything...

aspilicious’s picture

Status: Needs work » Needs review
aspilicious’s picture

#136 is plain wrong lots of errors

Status: Needs review » Needs work

The last submitted patch, system-rebuild-theme-data-305653-136-REMOVE-CACHE_1.patch, failed testing.

David_Rothstein’s picture

Status: Needs work » Needs review
FileSize
13.71 KB
catch’s picture

Status: Needs review » Reviewed & tested by the community

One day someone will add a period to that comment.

Dries’s picture

Status: Reviewed & tested by the community » Fixed

Committed to CVS HEAD. Thanks.

David_Rothstein’s picture

donquixote’s picture

Version: 7.x-dev » 6.x-dev
Status: Fixed » Patch (to be ported)

Nice, nice.
So what about D6 ? Remove cache, or rather clone?

andypost’s picture

I think D6 just needs clone, because removing could break some custom code.

mrtoner’s picture

Okay, I'm just talking out of my hiney here, but while David Rothstein says in #103 "All it takes to trigger this again is one contrib module which happens to call system_rebuild_theme_data() somewhere in the update process." -- well, it seems to me this problem transcends the update process (or cron, as documented separately).

I stumbled upon this issue when I began receiving these errors in D6.15:

* warning: array_map() [function.array-map]: Argument #2 should be an array in /path/to/modules/system/system.module on line 1015.
* warning: array_keys() [function.array-keys]: The first argument should be an array in /path/to/includes/theme.inc on line 1771.
* warning: Invalid argument supplied for foreach() in /path/to/includes/theme.inc on line 1771.

and experienced my default theme being disabled. It's still the default, but it's no longer enabled -- and the system table no longer has an entry for that -- and only that -- theme.

However...

This is happening neither during an update nor a cron run (at least, not when I run cron manually) -- it's happening at least daily. I am at my wits end with this problem. Am I wrong to think my problem is related to this issue?

(BTW, a Google search of this error brings up hundreds of sites that are experiencing this error.)

kmgflavin’s picture

same here. I re-enable the themes, do what i need to do, which is usually moving blocks around. I've installed Organic Groups, too. It seems like the themes will work, but I need to re-enable them any time I need to make adjustments that call on the themes, like blocks does. It's making me very nervous. I wish I knew more about how to help, testing or anything.

mrtoner’s picture

It seemed to me that the problem was occurring after a status update and -- sure enough -- a check of the core Update module shows that system_theme_data() is called there, too. My problem was specifically caused by me playing around with multiple site configurations, using the same database, and a cron job for two sites running at the same time. My fault, but it does point out that there's another point of failure for the system_theme_data() function.

andypost’s picture

boazr’s picture

sub

mr.andrey’s picture

subscribing...

manos_ws’s picture

Status: Patch (to be ported) » Needs review
andypost’s picture

Status: Needs review » Patch (to be ported)

@manos_ws dont try to re-test old patches. The question is asked in #144

medimacs’s picture

This is indeed frustrating many administrators of drupal sites. Will there be a solution in 6.17?

chweetraju’s picture

subscribe

bramface’s picture

Another cause of this error would be changing the permissions of sites/default/themes so that the default enabled theme were no longer readable.

-B

nilsja’s picture

subscribing

nilsja’s picture

does just using one theme work around the problem?

nilsja’s picture

perhaps this is related: http://drupal.org/node/794736 ?

Happens also with just one (fusion) theme activated. Does not only occur while updates, but apparently after cron. Also after applying the clone patch.

edit: same case as mrtoner describes in #146. but without multisite configurations...

does not occur with the default garland theme, but with fusion (core) and the skinr module.

boazr’s picture

Closed for me.
I guess this was caused by some other module, as it is magically fixed now - no patching to core (using vanilla 6.16). Now the problematic site can updatedb with no problems (and still retain its 10+ themes active).

nilsja’s picture

not for me.

the only way to work around this problem was selecting the garland theme. even if i had just the acquia marina or zen theme active it suddenly got deactivated. really ugly bug.

boazr’s picture

I'm just happy my iteration of this bug got magically fixed :)

andypost’s picture

There's a patch for D6 #632080-17: _system_theme_data() should clone the returned objects.
Suppose this is enough to fix

nilsja’s picture

@andypost, the patch did not fix it in my case.

mstef’s picture

subscribe

juan_g’s picture

After the D7 commit (comment #142), is this bug affecting just minor D6->D6 updates now, or still affecting major D6->D7 upgrades (like in #103)? Is the D7 upgrade path tag (blocking Drupal 7 beta) still valid here?

Anyway, during upgrades between major versions (D6->D7) it's recommended to switch to a core theme, such as Garland...

jmcerda’s picture

subscribing

klonos’s picture

Is this supposed to be fixed in D6.19? I am asking because I just got hit by the disabled themes bug while testing Open Atrium 1.0b8 (that is based in the D6.19 code).

I tried reading through the whole issue -especially after what's been done during 2010- and I am quite lost from successive switching back and forth between 6.x and 7.x and various commits. So, what do I need to test in order to help with this issue? Should I apply #17 from #632080: _system_theme_data() should clone the returned objects. and see?

ManyNancy’s picture

Component: update system » ajax system

Same question as above, what's going on here?

ManyNancy’s picture

Component: ajax system » upload.module

ehrm, it seems like the component 'update system' no longer exists, and that was why the component was changed.

David_Rothstein’s picture

Component: upload.module » database update system
mandreato’s picture

Same problem here on D6.20: after updating a contrib module, all the enabled themes were disabled.
I'm not sure on what to to now... maybe apply #632080: _system_theme_data() should clone the returned objects. ?

Agileware’s picture

Subscribing.

Agileware’s picture

From #144:

So what about D6? Remove cache, or rather clone?

boabjohn’s picture

Version: 6.x-dev » 7.4

Well, I'll join the queue here wondering what to do with a clean/simple D7.4 install that exhibits this problem. It's one of two sites managed through aegir...the other one (Bartik based) keeps its theme set while this one (using Marinelli) does not. Watchdog shows:

Warning: uasort() [function.uasort]: The argument should be an array in system_themes_page() (line 237 of /var/aegir/platforms/d7smallsite-222/modules/system/system.admin.inc).

I'm also taking the (perhaps ill-advised) step of setting this to 7.4 in hopes of at least having the proper issue thread shouted at me...

catch’s picture

Version: 7.4 » 6.x-dev

I think you have another issue there, I'd recommend opening a complete new issue with that error message, and ideally steps to reproduce from a clean install. Moving this back to 6.x.

boabjohn’s picture

Okay, thanks catch for the advice: I just did not want to fork the issue unnecessarily.

deepeddy’s picture

subscribing

Fogg’s picture

subscribe

xmacinfo’s picture

@Fogg and @deepeddy: Please stop suscribing like this. Instead press the Follow button above the Issue Summary.

Fogg’s picture

Is anything still happening in this issue? I am currently setting up a new drupal page and have this issue. Looks like the themes are deactivated over night - every night.

I am running a multisite installation and on the same installation at least 3 additional drupal pages are installed and working just fine. So I would guess the issue is with some module/theme. Any ideas where to look?

Thanks,
Fogg

donquixote’s picture

I have not seen this issue occur since the patch from
#632080: _system_theme_data() should clone the returned objects. is in core.

However, there could be other reasons for this still happening to other people.
You need to investigate yourself, using watchdog() and friends.

yvesdc’s picture

Issue tags: +theme disabled

Hi everyone. My themes are disabling constantly.

I am running drupal 6.22 and everything is up to date. I am using Themekey too.

This bug does not occur on the identical site on my local only on the live server!!!

Can the above patch be used on this update or is it already in core now and I need something else.
I have tried so many things with no luck.

Does anyone know of a workaround. Could the database tables be locked in some way so the themes cannot be disabled????

Any other clever fix suggestions?

Lars Toomre’s picture

This bug continues to haunt me at frequently but inconsistently. It has been going on for more than two years and I am getting tired of always going to admin/build/themes.

From what I can tell, my problem is not tied to either cron or update.php. Rather it occurs as a result of serving some unknown page that calls one of about 50 modules installed on the website. My suspicion is that its in fact one or more malformed contrib/custom modules that is causing the root problem. I cannot provide more information yet, but am starting to do further debugging because it simply is so frustrating to deal with.

Solving this will be sort of a Christmas present to myself.

donquixote’s picture

Maybe you could modify your _db_query() or db_query(), and catch any
"UPDATE {system}" or "DELETE FROM {system}".
And then a SELECT to check if any theme is missing or disabled, make a backtrace and log it (watchdog or something else).

Yep, there is already a mechanism for this kind of thing - just look at _db_query() in database.mysqli.inc.
But, a custom hack could be a lot cheaper on system resources.

Check how devel query log does it.. and then do it whatever way you want :)

donquixote’s picture

Here is some code that could help you with debugging.
No guarantees, just as a starting point.

It is a replacement of db_query() in database.mysql-common.inc, for D6.

function db_query($query) {
  static $_stack_count = 0;
  $babysit_themes = !$_stack_count && preg_match('/^UPDATE {system}/', $query);
  ++$_stack_count;
  $args = func_get_args();
  array_shift($args);
  $query = db_prefix_tables($query);
  if (isset($args[0]) and is_array($args[0])) { // 'All arguments in one array' syntax
    $args = $args[0];
  }
  _db_query_callback($args, TRUE);
  $query = preg_replace_callback(DB_QUERY_REGEXP, '_db_query_callback', $query);

  if ($babysit_themes) {
    $sql = "SELECT name, filename, status FROM {system} WHERE type = 'theme'";
    $rows = array();
    $q = db_query($sql);
    while ($row = db_fetch_object($q)) {
      $rows[$row->name]->before = $row->status;
    }
    $result = _db_query($query);
    $q = db_query($sql);
    while ($row = db_fetch_object($q)) {
      $rows[$row->name]->after = $row->status;
    }
    foreach ($rows as $name => $ba) {
      if ($ba->before && !$ba->after) {
        // A theme was disabled. This might be intended, or not.
        $msg = '';
        foreach (debug_backtrace() as $call) {
          $msg .= "\n<tr><td>$call[function]()</td><td>in line $call[line] of '$call[file]'</td></tr>";
        }
        $msg = '<table><tbody>'. $msg .'</tbody></table>';
        _babysit_theme($name, $msg);
      }
      elseif ($ba->after) {
        _babysit_theme($name, FALSE);
      }
    }
  }
  else {
    $result = _db_query($query);
  }

  --$_stack_count;
  return $result;
}

function _babysit_theme($name = NULL, $msg = NULL) {
  static $_memory;
  if (empty($name)) {
    // We are on shutdown.
    foreach ($_memory as $m_name => $m_msg) {
      watchdog('babysit theme', "<p>Theme '$m_name' was disabled.</p>$m_msg");
    }
  }
  else {
    // Called from db_query().
    if (!isset($_memory)) {
      // That's the first run.
      $_memory = array();
      register_shutdown_function(__FUNCTION__);
    }
    if ($msg === FALSE) {
      // False alarm, this theme is enabled again.
      unset($_memory[$name]);
    }
    else {
      $_memory[$name] = $msg;
    }
  }
}

What it does:
Whenever a theme is disabled, and is not re-enabled during the same request, then it will write the stack trace of db_query() to the watchdog.

Lars Toomre’s picture

Thank you for the suggested debugging function @donquixote. I am not at the right computer right now, but late this afternoon, I will incorporate your function into the code on the troubled site. Hopefully, this and some of my own debugging statements will help us pin down better where this elusive bug might be coming from. FYI - The troubled site is running Drupal 6.20.

Lars Toomre’s picture

@donquixote - Last night I stalled the code in #186 as you suggested. Checking today, I see that I have had several instances of the themes becoming unset. However, there is no output/watchdog data from the above code. I will need to debug this new code to make sure I completely understand what it is doing and where it might not be working as designed.

The log files do show that one occurrence must be somehow related to the hook_cron() code in the update module. The other random occurrences do no yet make any sense.

donquixote’s picture

Maybe the preg_match('/^UPDATE {system}/', $query) is too strict?
Maybe you also need to catch "DELETE FROM {system}".
And be more tolerant with whitespace and linebreaks.

Fogg’s picture

Hi all,

I am facing the same issue on of our my sites in a multisite environment. May be as additional input:

I cannot believe that a permission on the cronjob should be the issue. What I have found today is
a) baseurl was setup wrong (www2.domain.com instead www.domain.com)
b) cron.php was called multiple times from different crontabs (at the same schedule)

I changed this now, will update this issue within the next days if it works now or not.

Fogg

Fogg’s picture

my issue is solved. Most likely the wrong baseurl was the root cause. Hope this helps also others.

donquixote’s picture

I also had this happen to me again.
Looks like we need a way to catch what is going on.

budda’s picture

Suffered from this problem a few times in 2012 on Drupal 6 sites. Especially when updating to the latest Drupal 6 core.

igor.ro’s picture

Do not know if my problem have the same reason
but on run drush updatedb I got

Duplicate entry &#039;themes/garland/minnelli/minnelli.info&#039; for key &#039;PRIMARY&#039;                                                     [warning]
query: INSERT INTO system (name, owner, info, type, filename, status, throttle, bootstrap) VALUES (&#039;minnelli&#039;,
&#039;themes/engines/phptemplate/phptemplate.engine&#039;,
&#039;a:14:{s:4:\&quot;name\&quot;;s:8:\&quot;Minnelli\&quot;;s:11:\&quot;description\&quot;;s:56:\&quot;Tableless, recolorable, multi-column,
fixed width

the the same for all my themes.
Looks like the problem was in function system_theme_data() in this logic case

if (isset($theme->status) && !(defined('MAINTENANCE_MODE') && MAINTENANCE_MODE != 'install')) {
        db_query("UPDATE {system} SET owner = '%s', info = '%s', filename = '%s' WHERE name = '%s' AND type = '%s'", $theme->owner, serialize($theme->info), $theme->filename, $theme->name, 'theme');
      }
      else {
        $theme->status = ($theme->name == variable_get('theme_default', 'garland'));
        // This is a new theme.
        db_query("INSERT INTO {system} (name, owner, info, type, filename, status, throttle, bootstrap) VALUES ('%s', '%s', '%s', '%s', '%s', %d, %d, %d)", $theme->name, $theme->owner, serialize($theme->info), 'theme', $theme->filename, $theme->status, 0, 0);
      }

here is a patch for this

Gábor Hojtsy’s picture

websage_gr’s picture

Status: Needs work » Needs review
Issue tags: +D7 upgrade path, +theme disabled, +d6 upgrade path, +theme path, +drush update

warning: array_map() [function.array-map]: Argument #2 should be an array in /var/www/clientsdomain/httpdocs/modules/system/system.module on line 975.
warning: array_keys() [function.array-keys]: The first argument should be an array in/var/www/clientsdomain/httpdocs/includes/theme.inc on line 1728.
warning: Invalid argument supplied for foreach() in /var/www/clientsdomain/httpdocs/includes/theme.inc on line 1728.

I updated using drush and after that accessing the page in the browser it had thrown the above. I realized the trouble starts
if you use for example a framework for the theme. In my case that is blueprint. The same applies for fusion framework.
If your theme is in a path like /sites/all/themes/blueprint/mytheme or /sites/all/themes/fusion/mytheme then drush deletes the subdirectory of a theme.

So, if you use subthemes make sure you put the subtheme on the main themes/ path before you run update.
I read somewhere it is no longer important to have a subtheme in the subdirectory of the main theme/framework, in my case this solves the problem.

Version: 6.x-dev » 6.26
Component: database update system » system.module
Priority: Critical » Normal
Status: Patch (to be ported) » Needs work
Issue tags: -D7 upgrade path, -theme disabled

The last submitted patch, 305653-drush-updatedb-theme-duplicate-error-195.patch, failed testing.

Status: Needs review » Needs work

Status: Needs work » Closed (outdated)

Automatically closed because Drupal 6 is no longer supported. If the issue verifiably applies to later versions, please reopen with details and update the version.

palogan20’s picture

Issue summary: View changes

how to add new image to website. please update panelistas website

apaderno’s picture

Issue tags: -D7 upgrade path, -theme disabled, -d6 upgrade path, -theme path, -drush update

I am removing the tags used in just few issues. I apologize for bumping this issue.