Meta: update.module RAM consumption

Yeah, unfortunately, update_status just has a lot of data to process, especially if you have a ton of modules installed. I don't know what to say. Someone could potentially go through and try to optimize RAM consumption -- there are a few places we might be passing around more data than we need. That said, I'm not really interested in spending much time on this, since this sort of refactoring is very unlikely to get into the version of update.module in D6 core (unless we can show it's a real bug), and I don't want to put effort into the 5.x version that won't benefit the 6.x version.

I don't see myself upgrading several of my Drupal sites to 6 for many months until the needed modules are upgraded, and I absolutely loved the info provided by update_status. Sadly, I can't use it anymore, as it just won't load on some of my more complex sites. 1GB of PHP Memory?!? My hosting provider would hunt me down! I've been incrementally increasing it trying to find out what would be an acceptable amount, and so far even 64MB doesn't do it anymore. Not sure when I hit the tipping point, but by my count, we are using 60+ modules (though some of them are modules like CCK with multiple sub-modules) on our biggest site, and update_status just won't work anymore.

Anyone know of a way to track module updates without having to crawl each module page manually? I like to keep the modules updated to the latest versions if at all possible, and it's just become a nightmare without update_status!

First of all: #155450: backport separate {cache_update_status} table and private non-volatile cache API from D6 core ;)

Secondly, yes, it's a problem that update_status is doing everything all at once, however, precisely because it's in core, it's now very difficult to do anything about that. :( Unless I can convince Gabor that a serious reorganization of the code is necessary to get it to work on sites with more than ~30 modules, we're doomed in D6, since that's now in bug-fix-only mode.

So, sadly, the first place to do anything about this is in D7 core, which means no one will actually see it for ~1 year from now. I'm not going to restructure the D5 contrib module in such a way that isn't going to go into D6 core...

Side note: I bet one of the other problems are modules where the maintainer is irresponsible and generating tons of releases. People who think they should make a new official release after each CVS commit, etc. :( If you happen to be using one of these modules, feel free to submit a bug report to their issue queues, asking them to slow down with the official releases and not be so wasteful.

If this is bringing sites down, and there wouldn't be any breakage of external APIs, then I don't see why something couldn't get into D6 since it probably counts as a critical bug. Can we move this to core/D7 maybe?

*sigh* ... It's really a shame that more people didn't help test update_status and help move it into core when I was doing all that work (mostly on my own).

Anyway, I sent a note to the devel list to start a new thread about an old topic: http://lists.drupal.org/pipermail/development/2008-May/029947.html

In terms of changing this in D6 core, I guess we'll just have to try to see what Gabor/Dries think, but I'm not terribly optimistic they'll support a big refactoring in a stable version of core.

@dww - have not really looked at the internals yet, but is it possible for update.module (or update_status) to fetch the information for one module/package at a time? Also, since you are rarely using this data, perhaps it could be cached in smaller chunks - like per module?

1GB for update_status? I just can't believe that update_status data can be that much.. Can somebody make a proper measurement of memory usage?

@dww

Would it be possible for the update module to use a table and process things one module/version at a time? Whether it is retrieval from d.o, or checking that against the local?

By moving this from RAM to the database, it may help the situation?

- This isn't a minor support request, this is definitely a task that should be done, and perhaps considered critical.

- The most important question is can this be solved in D6 core or not?

- I don't believe 1 gig for update status alone. I think the problem is on sites where there are tons of modules already, bloating RAM usage on their own, and update_status just pushes it over the top. That said, I would like to see real numbers to know exactly what we're dealing with to decide if this is normal or critical.

- I'm not convinced that just caching per module will actually help total RAM consumption if that's all we change. I'm sure that "By moving this from RAM to the database, it may help the situation?" is fantasy. The data is only useful to update status once it's in RAM and being processed. The fact we store it in the DB is meaningless unless we load it into RAM to compute the status. And, all the data is already in the DB, since we catch everything we fetch, and everything we compute about the current status.

That said, if we split up the functions and data structures so that we do everything on a per-project granularity (not per module, btw), throw out data we don't need, and try to avoid processing multiple projects in the same page load, we might be able to greatly reduce the footprint. We'd have to:
A) Fetch and parse the XML release data only a project or two at a time, ideally via batch processing.
B) Try to throw out as much of the data as possible (e.g. all releases older than the currently installed release, fields included in the XML we don't actually care about for update status (if any), etc).
C) Compute the status for each project separately, also via batches.
D) Once we compute the status for a given project, cache just the status info in the DB, and only load that in update_requirements() and on the update status report.
E) Aggressively clear this cache of the current status for a project if any of the .info files change (pretty sure we already do this, but we should make sure it's still happening after whatever other changes we might make).

If we did all that, I think we'd be in much better shape to avoid running out of RAM on sites with tons of modules and/or projects that make way too many releases. The point is not that the data lives in the DB instead of RAM, it's that we ditch as much of the data as possible, and try to only process a subset of the data on any given page load.

I fear Gabor/Dries will refuse such a patch in stable core, unless there's clear evidence that this is a serious bug in the current design. But, it's going to require API changes (at least in the internal API of update.module itself), and those almost never change during a stable series unless required to plug a security hole. As an abuse of this exception, we could try to claim that sites disabling update.module due to the RAM requirements would be a security liability for Drupal, but I doubt that's going to fly. ;)

We need:

1) Documented, reliable RAM usage profiling of update(_status).module on various sites, showing the number of modules enabled, the RAM usage before update(_status)? and the RAM usage after enabling it.

2) Comment by Dries/Gabor about the feasibility of changing any of this in D6.

If other people can wrangle these, I'm willing to do the refactoring.

I would rather hack core than disable the update_status module.

subscribing so I can read this in more detail later.

Given that I have a not-really-interesting D6 site that takes up 40 MB on an alarming basis, it looks like D6 in general has some memory issues to work out. I am therefore going to bump this up to critical. I agree that we do need some better profiling to figure out exactly why D6 is so memory hungry, especially after we chopped out 25% of the memory usage via code refactoring already.

Hello,
Is there an easy ways to measure the resources consumed by various modules?

subscribing to help test patches

We maintain a few very large sites (definitely 30+ modules). But they're running fine with a PHP memory limit of 64MB with a few LAMP tweaks which any average sysadmin should be able to carry out did the trick for us. In fact, we've grouped over 10 of these sites onto one box and it's running fine after the initial problem was discovered.

I don't believe for a second that allocating 1GIG to PHP is a smart and common sense thing to do. Have you evaluated other alternatives like an op code cache, optimised apache and the database? Some OS (FreeBSD for example) use different memory management so try switching your setup.

While I am fully supportive of making core as resource friendly as possible out of the box, I agree with dww's concerns. This issue could be blown out of proportion by a few bad modules that have been installed. I guess we need some tests to identify where this excess load is coming from.

@KingMoore: I believe there is a project by a SoC student that returns this info.

Agreed with Crell in #17. If update_status is the culprit, it should be showed, then fixed in D7 (show benchmarks that it is fixed) and hopefully backported if possible with a reasonable fix. Looks like there would be enough interest, so it would not be dww only to take this on, right?

Unfortunately, some of us are running on shared environments where our options to change setups/tweak server settings are very limited. And I'd still very much like to have something I can use on the 5.x platform until I can reasonably migrate our site to 6.x

Not that there are any hard numbers to go on in #23, but that supports my theory that part of the problem is the huge number of releases. In addition to some shared per-project data, each release of a module adds ~600 bytes to the XML file. I have no idea how much RAM the XML parser chews up when parsing that additional 600 bytes, but lets say pessimistically between holding the data itself, overhead from the XML parser, and update_status internal data structures, we're talking 1KB per release. As a general rule, D6 modules don't have nearly as many official releases as their corresponding D5 versions do.

<speculation class="wild">So, 45 modules in D6 might be ~100 total releases, whereas 45 modules in D5 might be ~400-800 releases. So, that could be roughly on the order of 500KB additional RAM, just from the extra releases.</speculation> ;) 1/2 meg on its own certainly doesn't explain what people are talking about here. Again, I'm guessing that in the D5 case (without all the RAM reduction benefits of module splitting in D6), that the site is hovering close to the RAM limit as it is, and with D6, it's back under the limit.

However, all this is guess-work. What we most need is someone with a good RAM profiling setup to actually drill into this and come up with some hard data for various configurations.

Ok, well then perhaps we can optimize by just throwing away information about old releases?

I'd say a mixture of throwing out old release info we don't need and splitting up the cache to separate entries would go a long way toward reducing memory consumption. While that would increase the number of cache requests necessary, I don't think it's checked often enough or on enough pages that it would make a difference in normal operation.

subscribing

Why not let d.o provide this as a web service?

I have had a bit of a think about this over the week. Wouldn't it make a whole lot of sense if Drupal.org offered a web service where you could simply send d.o a module name and it would return to you the releases in XML format? Since Update Status is now in Core, the number of sites crawling d.o for releases is only going to go up. A dedicated service seems to be the way to go, and Update Status could then be patched to get it's data in this way, rather than the curent method. This should significantly lessen the load on both d.o and also the clients running the Update Status module.

@KingMoore: Er. That's exactly what we have now?

http://updates.drupal.org/release-history/drupal/6.x

@panatlantic: Is there any way you could put a temporary hack in Update Status module's code to log the size of the XML files / current ram usage in both 5.x and 6.x to compare?

lol I thought it was crawling the d.o pages. Maybe that was D5. Doh. Glad to see this was implemented before I thought of it :D

subscribing - I'm running a number of D5.7 sites and just fell over the edge with one of them, which means that I will need to disable update_module on that site. That site runs approx 35 modules.

I think the idea of having a parameter to allow checking of [all, 10,20, whatever] modules at a time is an excellent one.

Best regards
Pete

Forgot to mention that all my D5 sites are running on shared hosts, so, as with #22, I am unable to make any parameter changes. So for now, I have had to turn off update status on all my sites, because it is unusable.

I don't know much about D6 yet, but if the behavior is the same (and is enabled by default) then I would think that this has the potential to kill D6 on shared hosts...

subscribe

@dww - looking at the information sent by d.o - it's sending information about all releases - is that needed? On the d.o side can we only build information about the most recent official and -dev release for each branch?

@pwolanin: No, because we don't know what version they're running and what version(s) are security updates. Trivial case:

5.x-1.4 is latest
5.x-1.3 was a security update

if they're running 5.x-1.3, we need to tell them to upgrade to 5.x-1.4 because they're out of date.
if they're running 5.x-1.2, we need to tell them to upgrade to 5.x-1.4 because they're missing a security update.

it gets even more complicated with:

a) different branches/major versions
b) releases marked as unsupported
...

:(

@dww - hmm, so we need some of this history, but maybe only the most recent version plus the most recent security fix?

@pwolanin: And the info about *any* releases that are marked as unsupported. And, we have to include both the most recent (could be a beta/rc), and the most recent "real" release. And all this info for every available branch (most recent, most recent security release), etc. And, we'd have to double-check that existing update_status clients won't start freaking out with bogus results if it only sees this data. In fact, I'm pretty sure we're screwed there, since if it can't find the currently installed release in the info about what's available, it thinks that's really bad and generates some kind of warning. So, we'd have to either provide the slimmed data at a new location that newer versions of the clients know where to fetch it, or give up on this line of attack.

Plus, it's at least possible other people are using this data for other things, and they'll be very sorry if all of a sudden, most of the data just disappears. That'd be another reason to provide the full data in the current location and the slimmed data somewhere else.

Of course, we probably can/should change things in the 7.x version (using the PHP5 XML parser might make it easier to use a more fancy XML schema, we might want to include less/different info, etc, etc) -- it'd be (relatively) easy to have the history generating scripts include different info in a different format in the .../7.x/* files it generates, and have the update.module in 7.x core expect and handle different data.

But, since this issue is about 6.x (and 5.x contrib), I don't really think it's viable to change the format of the data at the server.

Sorry,
-Derek

@dww - if it's not viable, that's totally fine - I just want to consider the possibility. Though each module sends its version information, right? So a site with a 5.x-1.5 module doesn't need to know anything about versions before 1.5, right?

Yeah, I appreciate the attempt to explore other options, that's great...

Though each module sends its version information, right?

Not as of update_status 5.x-2.* and beyond. Each site only fetches data from a URL that includes the project name and the version of core. Sites never send specific info about what versions they're using. I could dredge up the issue numbers where all of this was originally discussed and decided if you want to read the history. ;)

Really? Looking at this code it seems like the version is included: http://api.drupal.org/api/function/_update_build_fetch_url/6

and if I hack that function to drupal_set_message() each url, I get values like:

• http://updates.drupal.org/release-history/drupal/6.x?site_key=1789&version=6.4-dev
• http://updates.drupal.org/release-history/node_clone/6.x?site_key=1789&version=HEAD
• http://updates.drupal.org/release-history/cvs_deploy/6.x?site_key=1789&version=6.x-1.0

Gah, I totally forgot we added that for tracking usage stats. ;) Tee hee.

@dww - ok! Does the 5.x version work the same way?

Above you also talked about changing the internal way that update module processes data to keep less in memory. Which do you think is the faster/simpler fix?

There were all kinds of server-side performance issues with putting too many brains in the server to send the client only the info it needed to know. That's what lead to the complete redesign for 5.x-2.* and the move to the XML files. I think it'd be better to just make the clients smarter about not saving/caching all the extra data, than reopening the can of worms about how updates.d.o serves up data in the first place.

@dww - well, we should certainly look into the latter, but can't we just pre-build an XML file for each module/core version - or smarter, perhaps, build it for the most recent 2 or 3 releases of each project, and have the PHP wrapper that servers the XML files just fall back to the big (current) file if you have a version it can't find?

FYI: he didn't post it here, but chx did some memory profiling last night. Even with a D6 site with modules with the most releases (I queried the DB to help him with a list of modules sorted by the most official releases -- he had something like 11 modules and 1 theme that between them had nearly 200 releases), he discovered that update.module was a drop in the bucket of RAM consumption relative to other parts of D6 such as the theme layer, the menu system, and views. See #259479-34: Views needs a lot of memory for his initial findings. Hopefully he'll post here with more of the update.module details he discovered (he was just pasting various results into IRC). So, I'm not quite as worried about this issue as I used to be. There's still room for improvement, and maybe even still in D6, but I'm not convinced this is a "the sky is falling" kind of problem anymore...

@dww - well, it seems like comments above suggest the module could not be used for D5 sites in some cases - since we are potentially looking at the number of releases for D6 modules increasing a lot over the coming moths, I'm still wondering if we need to tackle this.

Yeah, I left it "critical". I still think it's worth fixing. But, I don't believe it's quite as dire as I initially feared, that's all. I still believe the basic plan I outlined in #14 is what needs to happen. We can consider other ways to change the data format and optimize in D7, but for D6 (and perhaps D5), we need something pretty close to #14, and see how much that helps.

BTW, for testing, it'd be fairly easy to generate some bogus release history files with 100s of releases per project, and point a test site at those (since you can specify the location of your release history in the .info file for each project). So, that'd be another way to test in D6 without waiting for lots of real releases.

I did a number of investigations and the memory that update status consumes is insignificant compared to the memory mere code inclusion eats.

chx: did your investigations include a considerable number of contrib modules used on the site, and/or modules with lots of releases?

@chx - I'm a little surprised, considering that dww was saying there might be 400k sent per project (x 20 projects? on a big site)

Yes, there were no less than 178 releases. The total fetched XML is 105 767 bytes and parsing it takes 1 383 960 bytes. The site eats a whopping 25M on the empty homepage and 35M on admin/build/modules. That 1MB is nothing... I enabled all core modules and every module that let itself enable from the projects cck contemplate countdowntimer dhtml_menu everyblog flashvideo og scheduler views weather weblinks and added contrib theme fourseasons.

updatestatus.module memory consumption is a problematic part of Drupal especially its in the D6/D7 core

One relatively easier solution i can think of to reduce RAM resource consumption would be;

1. suggestion
To make an optional for updatestatus.module NOT to check against current installed modules (thats where/when the memory consumption occurs, right??)
BUT the module to list latest versions from d.o below the each module -NOT AGAINST- the installed ones- (sorry for the repeat it is for emphasis)

SO with this option of the updatestatus.module, determination of comparison will be by good old human eye -checking by eye against the installed and the latest (its only requires brain server overload instead of computer server overload :)
---------------------------------

2. suggestion
or... alternative solution could be
- introduce a simpler jq selector example of a check all/ check/off options rather than a full drop-down 'filtering' like here;
http://jqueryui.com/download (the red indication)

3. suggestion

filtering, checking - than running the module updates (attachment #1)

- filtering à la Drupal Project Module Issues right in this page! (attachment #2)

reducing the number of modules to update will reduce the memory consumption which is troublesome as many issues post on this topic and majority turned the disabled module as it makes managing Drupal more difficult rather than easy -contradictory to the purpose of the module itself.

i am novice drupal user using Drupal 6.12
and i am running on my localhost via xampp

since the number of modules are growing i had increased my memory limit;
ini_set('memory_limit', '40M');
i was able to recover from the error below;
Fatal error: Allowed memory size of 16777216 bytes exhausted (tried to allocate 41 bytes)

Anyway now i am facing a new error as below and attached;
Fatal error: Maximum execution time of 60 seconds exceeded in D:\xampplite\htdocs\drupal\drupal-6.12\includes\file.inc on line 891

please help me to overcome this error!

Thanks ;)

@aslamc: That error doesn't have anything to do with this task. In fact, it's not obvious it even has anything to do with this module. Please search the forums and issue queues for other instances of that error message and do a little research, first. If it turns out it's really related to update.module, then you should try upgrading to 6.x-dev (or 6.13 when that comes out) since we fixed something much more closely related to this via #243253: Update status module can cause white screen if the check for updates takes too long. Either way, this issue is about reducing the RAM footprint, not about timeouts, so let's leave it that way. Thanks.

p.s. Since this is going to require some pretty major API and code refactoring, I think it's safe to say this is only going to happen in D7.

Looking at http://updates.drupal.org/release-history/drupal/6.x, I see plenty of duplicated entries:

<term>
<name>Release type</name>
<value>Bug fixes</value>
</term>

The next best question is whether we really need to define

<version_major>6</version_major>

for every release? If Update module is requesting http://updates.drupal.org/release-history/drupal/6.x, all returned releases are for 6.x anyway, no?

@sun: #380814: Duplicate entries in some release history XML files

@dww - It would seem to me that we usually only need in the xml at most 3 entries for any branch: the most recent (if any) security release, the most recent official release, and the dev snapshot.

Maybe this code is just very conservative in that it assumes that version numbers and timestamps may be irrational and hence the actual history has to be examined?

"Finally, and most importantly, it keeps scanning the release history until it gets to the currently installed release, searching for anything marked as a security update."

http://api.drupal.org/api/function/update_calculate_project_data/6

Looking through the code, the main advantage of keeping all is only that we can mark a particualr release as revoked. But it seems that essentially we can still satisfy this by including revoked (unpublished) releases in the xml, but omitting ones that are published but not current?

I think for what I'm suggesting, we just need to change:

        // Stop searching once we hit the currently installed version.
        if ($projects[$project]['existing_version'] === $version) {
          break;
        }

To something that implements a version comparison - and stops if we are >= $version. Perhaps we could even use the built-in http://www.php.net/manual/en/function.version-compare.php ?

@pwolanin: We've been through exactly this proposal in this very issue. ;) Please re-read your question in #34 and my followup answers.

@dww - well, at least I'm consistent ;-)

So is this a "won't fix" or is the problem enough to try to make D7 changes?

I am glad that I found this thread after much searching. The error messages relating to db cache go away after disabling the update.module. One of the error messages I saw in syslog says "cron has been running for an hour and is most likely stuck."

Looking at comment # 60 about when a fix will be available. Then reading the rest, I wonder about what seems to be an agreement that this module needs attention (thread opened a year and a half ago). There is also agreement that this module is important and should be enabled. Other benefits for fixing it:

- less support for non-core module developers. Some threads for support (listing similar error message) border on enmity becuase the newb user can't give a complete account of the problem (is it php or mysql) and the developer can't debug an issue that they can't find in their code. Few answers recommend increasing memory.
- installing a 1 GB memory for a new implementation may not be feasible as the majority of sites start small and then grow as resources demand. This also affects adoption.

Solution:
I tend to agree (I am a systems and networks guy - not a programmer) with comments # 3 and 6: write to a table in db and parse one row (or a few) at a time while doing comparisons, results for modules that needs updating can be written to temp table. Comments # 56 and 58 can be used to reduce processing time. I think it's worth having to wait for processing than to disable the module all together.

@fk: I hear you, and I agree. I'd like to fix this. It's listed prominently on the update status community initiatives page. We've actually had a flurry of recent activity and progress on update module since I've carved some time out of my insane schedule for it, and I also managed to get another victim to agree to co-maintain with me (#588574: Make Dave Reid a co-maintainer for update status).

However, a prerequisite for fixing this at all (and it'll definitely need to be fixed in D7 first, then we can argue about the feasibility of back-porting to D6) is to get a lot more solid automated tests working, so that as we refactor the code, we know we're not breaking various edge cases. So, the next step here is to get #591632: Refactor tests to allow testing contrib committed. Then, we'll probably need another test-specific issue to flesh out a lot more cases than are currently covered (#253501: Tests for update.module was an extremely limited first attempt). Once the test coverage is more substantial, we can look more closely at fixing this and have a prayer that the core maintainers will actually commit it.

I really hate to belabour the point, but I think pwolanin is on the right track. Marking this as a D7 issue (#55) negates some of the points made in #37 against providing full vs. partial information. Here is another attempt a minimal, useful set (per #14, item B):

For each branch which contains a supported release:

1. The recommended release.
2. If different from #1, the latest release.
3. If different from #1 and #2, the latest security release (otherwise tag #1 or #2 appropriately).
4. If different from #1, #2, and #3, the oldest supported release.

This does not allow for update_status to consider:

• Updating from, e.g., -1.6 to security release -1.7 when security release -1.9 is also available.
• Updating to any release that is supported but not the latest, the latest security release, or the recommended release.
• Situations where, e.g., -1.4 and -1.6 are supported, but -1.5 is not.
• Probably other situations.

...but it means, at most, 4 releases per branch, and probably* 2 or 3 in most cases*.

* weasel words

The one solution could be introducing a hook like hook_menu, which check the new version of a module and notify. The hook should be implemented by all the modules and should check for update only when requested. i.e. by clicking on 'Check for update'.
Instead of rewriting core, this is the best and easy way. Of course, there can be a facility using which all the modules can check for update at once.

Regards.

.

subscribing

subscribing

There are a *lot* of different things we can do to help here. Instead of trying to handle all of them in a single issue with monster patches, I decided (and webchick agreed in IRC) that it'd be best to turn this into a meta issue, and just add links here to sub-issues for specific approaches to tackle certain problems in here. First up:

#669554: Reduce RAM bloat: only save attributes from .info we care about

Another spin-off: #669626: Reduce RAM bloat: split update status logic into a per-project helper and only save available release info we care about

Seems like all these made it in, demoting from critical. dww, please move back if it still is, or close if we're done.

Yeah, sorry, no. We got a lot further, but we still end up processing all releases for all projects in a single page load, so we've still got the same problems. It's a lot easier to fix now, but we're not there yet. I brain-dumped a whole plan to Dave Reid a few weeks ago and he said he was going to work on it, but that never happened. Alas.

If there was any chance this would still be committed to D7, I'd be happy to either write up the detailed plans and/or work on the patches myself, but webchick has been telling me that the chances of my Update manager patches still landing are getting vanishingly small at this point...

But, AFAIK, we're still at nearly the same RAM footprint in D7 as we've been since D5. The only reason it's not more visibly a problem right now is that very few D7 projects have enough releases (and D7 test sites don't use dozens of contribs yet) so it doesn't seem so bad...

Overall RAM consumption is still a critical bug in D7 (over 32mb for install, not great on other pages) - given that, without the fixes here the chances of hitting memory limits somewhere are going to be even higher than D6.

I don't really see why very minor API changes which only affect modules you maintain would be worse than leaving these critical bugs unfixed.

However, if that's really the case, this should be bumped to Drupal 8.

I don't see the problem in outlining what you have in mind here in this issue. We can always evaluate from there whether it's D7 or D8 material. But yes, certain API breaking patches, such as critical performance fixes might still be able to make it in, but they absolutely need to be finished up ASAP. All module porting is on hold until we call an API freeze and actually mean it, and we do not want a repeat of the Drupal 6 release cycle...

@webchick: I think catch's point is that "all module porting is on hold" is a bit misleading in cases like this. The only modules that might be effected by any of these potential changes are cvs_deploy and update_advanced, both of which I maintain, too. ;)

But okay, cool, I'll try to make some time tomorrow morning before I go offline for a few days to write up some proposals + plans here, hopefully with sufficient detail that anyone who was really motivated could work on a patch without me. Dave Reid should be able to help someone if needed, too...

Re: #76, #77: You people are talking about .install files and update.php. This issue is about the update (status|manager) module that tells you if your code is out of date and helps you upgrade it if so. hook_update_N() and hook_schema() have nothing to do with this bug/task. Please continue your discussion elsewhere. ;) Thanks.

edit: I just unpublished those comments, actually, so as not to confuse people who are genuinely interested in helping work on this issue.

It seems like even if the API change is small, we'd ideally want to get it in before beta or at least have it outlined here so a decision could be made on applicability to D7 or D8 prior to shipping a beta. So any update? This is one of only ~15 non-meta critical issues without a patch.

bump - still critical? Still someting we can/should do for Drupal 7?

Yes, IMHO, still critical, but I have basically no bandwidth to drive this forward. :/

Here is a short patch which excludes any modules in the $module/tests/ subdirectory from being processed by the update module. It cuts 3.25mb off of of the peak memory usage on a default d7 install.

Unfortunately I don't know enough about the update module to know if this will break anything.

That shouldn't break anything afaik. In fact if we don't mind breaking sites which put random contrib modules into /modules, we could also whitelist that directory to only check for system.module or simiiar.

I've tested this locally, and I've noticed very little speed difference between the patched version, and the unpatched version, however it does cut .5mb of required ram.

I'm confused why this patch is having any impact at all. From a few lines further down in _update_process_info_list() in update.compare.inc:

    // Skip if it's a hidden module or theme.
    if (!empty($file->info['hidden'])) {
      continue;
    }

I thought all test modules are supposed to be hidden. If there are modules that live in */tests/* that don't have hidden = TRUE in their .info files, that seems like a bug...

#82: 238950.patch queued for re-testing.

Agreed with dww. How did the tests fail? This patch shouldn't be changing anything if tests are properly hidden.

Never mind, I'm an idiot. The tests check for updates to modules that are in modules/update/tests/. If we block */tests/* the tests fail.

#82 is useless though. Any test modules (besides the one in update) should be hidden.

Still, this issue has a patch, so "needs work".

@yoroy: But that patch is not at all viable for any problem this issue is trying to address. ;) If that proposal was in its own issue, I'd have marked it "won't fix" or "by design" or something...

Furthermore, as per #68 and #70, this should be a meta issue that people can use to point interested parties to spin-off issues that are trying to tackle specific things. Not sure why catch un-meta'ed this in #73, but I'm putting it back.

Fair enough. I was just poking some active criticals to see if there'd be life in them :)

In preparation for the new "major" issue priority, I'm tagging this (not actually critical) issue as priority-major.
See: http://drupal.org/node/669048#comment-3019824 (#669048: We should have four priorities not three (introduce a new 'major' priority)) for more information about the coming major issue priority.

Meta tasks shouldn't be marked as critical, except if we have some actionable thing to do. Here, it's seems that the memory consumption is more by design :)

Changing to major as per tag.

Tag update

Repeating #94.