Problem/Motivation
drupal_static() should ideally be obsolete, but there are many procedural places in core that use them.
Other properly injected OO code needs to use drupal_static_reset() because of that, which leads to include-hell for tests.
Proposed resolution
Deprecate drupal_static() and drupal_static_reset(). Provide no replacement as using static variables is a bad design. Document that new code should not use drupal_static().
Remaining tasks
None.
User interface changes
None.
API changes
Both drupal_static() and drupal_static_reset() are now deprecated.
Comments
Comment #1
tim.plunkettComment #2
dawehnerSeriously .. there is really little value in doing this here, if you ask me, because most instances should be already converted and the other should be converted.
Comment #3
msonnabaum commented@dawehner what about FormBuilder? Is there a clear path to removing drupal_static_reset from it?
If not, I'd argue this is still worthwhile.
Comment #4
tim.plunkettUnless we make the parent issue a release blocker, I think this has merit.
Comment #5
dawehnerWhat about cats? Aren't they worthwhile to be added to core? Yes I think so. FormBuilder for example just uses a reset on drupal_html_id, which will be dropped anyway.
Comment #6
ParisLiakos commentedi agree with dawehner
Moving on #2121713: Move drupal_html_id() and drupal_html_class() to Drupal\Component\Utility is certainly more valuable
Comment #7
tim.plunkettThat is one out of 39 statics, just the example I picked:
Comment #9
ParisLiakos commentedWell, sooner all later most of them will go away. Opening issues to kill them one by one and assigning #1577902: [META] Remove all usages of drupal_static() & drupal_static_reset() as parent is probably a better course of action.
Dont get me wrong, i am just saying.. you can of course spend your time as you like :)
Comment #10
tim.plunkettSpoke with @alexpott in IRC, he said removing drupal_static did not qualify for a beta or release blocker, but agreed it should be a beta target.
Therefore, I think we should continue here to ensure we're not stuck with that for another cycle.
Comment #11
sunI don't really get the point of this issue... bootstrap.inc won't go away anytime soon, so it's not about auto-loading. The change makes
DrupalStaticitself injectable, but it's still using a static variable itself, so it's still a global state construct, and thus the dependency still exists as-is, it's just moved from A to B. I'm not able to see the improvement of doing that.Instead of duct-taping
drupal_static(), I'd rather refactor the actual dependency (e.g.drupal_html_id()) into a class/service. Even if that would end up to just be a nonsense/mini/stateful service, it would at least be a step in the right direction.But of course I agree with @ParisLiakos, this is just some sanity feedback; you can obviously use your time to work on whatever you want ;)
The reason this fails is that the NULL behavior for resetting all statics is not covered.
To my knowledge, the NULL case is actually the primary use-case in HEAD now. (although it's possible that I've been working on too much low-level stuff recently)
Comment #12
tim.plunkettOkay, after more though, ditching the over-architected Event-based approach.
All of these statics should ideally be replaced, but let's at least provide a way around this for now.
No interdiff since I kinda rewrote it.
Thanks for the pointer @sun, that was indeed what I missed.
Also I wrote a unit test, and then realized we had a web test, so I just removed that.
Comment #13
sunThis looks much better and simpler now, but I still don't think this is helpful, so I don't want to RTBC this change.
Comment #14
jibranI think we should profile this patch as well.
Comment #15
tim.plunkettIf you want to compare static variables and static properties, and profile the internals of PHP, feel free.
This issue is done, it just needs to be committed. Otherwise, the parent issue can be bumped to a release blocker, and this can be won't fixed.
Comment #16
jibranIt was just a suggestion but according to @tim.plunkett in IRC
so I withdraw my suggestion. :)
Comment #17
jhedstromThis needs a reroll. Not sure if it's too late for 8.0?
Comment #18
ajitsRerolling patch from #12. Had to do a little cleanup in
core/testsDrupal/Tests/Core/Form/FormTestBase.phpBasically I removed the following code:
Please let me know, if I did it right. The code did not make sense to me and looked like a test code. If this is wanted, I can add it back. Rest of the conflicts were pretty easy to fix.
Comment #20
cilefen commented@AjitS
test_form_id()is needed. The changes in FormTestBase from #12 are irrelevant now that there is an Html utility class. Some of the methods in TestFormBuilder should be using it. However, I think those methods are never called. They could possibly be removed in a separate issue.Comment #21
cilefen commented#2456983: Five methods in class TestFormBuilder are dead code. Remove them.
Comment #23
dawehnerLet's improve the patch a bit:
Comment #24
cilefen commentedReroll.
Comment #25
cilefen commentedI think we wanted the drupal_static() docblock mostly emptied. Could whoever rolls this next take care of that and let's do #23.
Comment #26
cilefen commentedThis is #25 and #23-1. We still need to:
Comment #27
cilefen commented"function" should be "class".
The examples are demonstrating the global functions instead of this class.
The StaticStorage docblock needs a good looking-over.
@dawehner After that is done, I think changing usages should be follow-ups. There are a lot.
Comment #28
dawehnerI think we should try to ensure that the fast pattern can actually used in an OO version
Comment #29
cilefen commented@dawehner Fair enough - could you please explain comment #23 in more detail?
Comment #30
cilefen commentedI fixed up some of the documentation before moving on.
Comment #31
dawehnerWho else thinks this is evil :)
Comment #32
fabianx commented#31: I think this looks pretty genius actually.
Discussed with msonnabaum and dawehner in IRC:
This allows two things:
a) Reset
b) Scoping
Both are important.
While it would be way better to not have any statics and have everything in classes, this is not gonna happen and there is also legacy code.
=> Lets get this in ...
Comment #33
pwolanin commentedDo we want to use it in SafeMarkup? it seems like the current class static is good enough?
Comment #34
dawehnerNo, you want to have a central place to reset those statics ... see the comment from fabianx
Comment #37
cilefen commentedComment #38
pwolanin commentedI don't see how this provides any scoping?
Also - do we know if there is a performance difference in calling the static method rather than a function?
Comment #39
tim.plunkett8.0.x, 8.0.0
obsolete
Comment #40
jeroentAddressed feedback in #39. Patch attached.
Comment #41
cilefen commentedA documentation specialist ought to look over the docblock.
Comment #44
anavarreComment #45
cilefen commentedA reroll.
Comment #46
cilefen commentedOops, ResettableStaticUnitTest crept in.
Comment #47
mikeryanWell, I'd like to see this, but it seems clear it isn't going to make 8.0.
Comment #48
dawehnerI think this certainly can still land in 8.0.x as it solves problems we have.
Comment #49
fabianx commentedI agree it could land in 8.0.x to make our tests easier, etc.
Comment #50
dawehnerAdapted the issue summary
Comment #51
joelpittetPossibly documentation duplication here: #422352: Improve code comments for static caching API
Comment #52
dawehnerThis is 8.1.x material now, this is for sure.
Comment #53
cilefen commentedTrying the reroll against 8.1.x.
Comment #54
mile23Currently 46 calls to
drupal_static()in core, according to api.drupal.org.I like
StaticStorageas an OOP replacement for our static pattern. Just one thing:If you pass in a generated name that doesn't exist, you end up losing the benefit of caching for everything. Could we add
resetAll()and be explicit? And then make the$nameargument non-optional. If it doesn't exist, do nothing.Comment #55
benjy commentedI wonder if a better name would be GlobalStorage or even GlobalStaticStorage? That way it's clearer that the storage is global to the entire request?
Comment #56
dawehnerGreat idea!
Having the static storage in there is IMHO an important bit. I'm happy with GlobalStatic, do you agree?
Comment #57
benjy commentedYes, I like GlobalStatic. +1 from me.
Comment #59
mile23Nice.
Not sure about the test failures... I guess everything uses
drupal_static(), eh?Couple things:
The only reason to have a default of NULL is so that you can call
reset()and have nothing happen. :-)Don't need the $name==null part in the documentation.
Comment #60
dawehnerGood points @mile23
Comment #61
mile23The patch in #60 addresses concerns in #59, and still applies.
The new static class fills a gap, though I hate statics. :-)
We have BC with the old global methods.
We have tests, and can migrate towards using this new class as needed.
I've added the 'Needs change record' tag, since it's a deprecation.
Updated the issue summary.
Generally RTBC-worthy except for some paperwork.
Comment #62
dawehnerThank you @mile23
Filed the change record
Comment #63
dawehnerComment #64
mile23Looks good. Let's go. :-)
Comment #65
catchThis was already discussed on #1577902: [META] Remove all usages of drupal_static() & drupal_static_reset() (edit: hmm partly on that issue, but I think more on another one), but the result of that discussion was just to gradually convert the procedural code using drupal_static() to services. Then any 'static caching' could be on protected properties of those classes then, if it's still needed.
I can't yet see the benefit of making drupal_static() itself 'nicer' - it was never supposed to be nice, it was a horrible hack to get around static caching issues in SimpleTest.
If there's a use case for it other than 'code that's not services yet', then it'd be good to see one. #33-35 mention SafeMarkup, but that's gone now.
Also is there any profiling done here? Without conversions that's hard to do, and I guess if there's a hit we could make the threshold for using the drupal_static_fast pattern lower, but also if it's sufficiently bad it starts to make the static caching in some cases pointless anyway.
Comment #66
dawehnerThroughout the cycle in 8.0.x we have actively added some static caches in places which didn't used to be some, like the filecache component, the extension discovery etc., so the base of the discussion changed over time.
Those bits currently have a normal static, so those caches cannot be cleared which already bites us in some places.
Comment #67
alexpottYep a static in a service is not really any better. Having the ability to clear static caches makes it easier to test stuff.
Comment #68
dawehnerOH yeah no quesiton. Its horrible that we have those kind of workarounds. Having though one central place for them, makes it really obvious how bad it actually is.
Comment #69
mile23So the problem is: Items use static caching which we can't control, therefore we're complicating the testing process because there's a bunch of static data keeping the code from using our mock. We're also eating up memory that won't be garbage-collected during the request.
If the policy is that we shouldn't migrate away from
drupal_static()(toStaticStorage), then we can still use this patch which has a BC layer leavingdrupal_static()in place. Code that usesdrupal_static()can have its cache cleared by a test usingStaticStorage::reset(), without having to jump through hoops to autoload an.incfile.The deprecation message could be to stop using statics rather than to start using
StaticStorage, which could then be documented as a backend fordrupal_static().Comment #70
mile23Hmm. I guess I should have marked this 'needs work.'
Updated the issue summary based on #65 onward.
If there's documentation on how to prove things with profiling, I'd love to see it. :-)
Comment #71
mile23Comment #72
mile23The patch in #60 needed a reroll, so I did that, and also updated the docblocks to reflect #69.
If this is acceptable we'll still need to work on the change record.
Also, @catch #65:
Services are really supposed to be stateless, but I think that boat has sailed.
Comment #73
dawehnerThese changes look great! Now its just about time to find someone to review it.
Comment #74
alexpottAs long as the test runner for things that use WebTestBase or BrowserTestBase are able to access the database and services of the site under test we have this problem. Saying that services shouldn't use this seems premature. The test runner makes far too many API calls at present in the name of speed but not sanity and proper test coverage.
Comment #75
mile23Updated the change record to reflect that we shouldn't use statics at all, and that this change is for testing legacy code. https://www.drupal.org/node/2661204
Updated this to be marked as deprecated in 8.1.x.
Also marked
StaticStorageas deprecated with the same message.Comment #76
dawehner+1 for telling people that statics aren't a good idea.
Comment #77
dawehnerThis difference would totally disappear when once we would have converted the entries over, but of course, for all other static caches, this would be worse again.
Comment #79
alexpottI'm not sure what is meant by
global static variables. A variable in the global PHP scope is not static ie something likeglobal $user... this seems an odd turn of phrase.Also isn't something like
ExtensionDiscovery::$filesshouldn't that be using StaticStorage.And adding something already deprecated seems premature. I don't think this is a promise we can keep.
Comment #80
dawehnerI agree, we should update the comment
Comment #81
mile23'Global static PHP variables' is a bit awkward but it's supposed to mean that you can use statics as long as they're not global-scope, like those wrapped by
drupal_static()or justglobal static $foo.I suppose telling people to use
StaticStorageis the best solution right now, since the problem we're trying to solve is being able to reset the static during tests without having to include an .inc file.So:
Un-deprecated StaticStorage.
Changed the deprecation messages to be 8.2.x.
Hopefully clarified the language in the deprecation messages.
Removed 'needs profiling' tag re: #77
Comment #82
dawehnerThank you @mile23!
Comment #83
catchI'm still missing the use-cases for this, as asked for in #65.
ExtensionDiscovery looks like it has a bug where it's not keying the $files static by hash of $this->searchDirectories and whether tests are included. Also ExtensionDiscovery is the opposite of 'properly injected OOP code', see ExtensionDiscovery::setProfileDirectoriesFromSettings()
I don't know of any other classes in core that are using static properties that are causing problems in tests, and none are listed here apart from ExtensionDiscovery and FileCache.
The only non-test classes that are calling drupal_static_reset() in core are in migrate, shortcut and taxonomy - none of these are good examples either. For example TermStorage::resetCache() calls drupal_static_reset('taxonomy_term_count_nodes') - that function doesn't even exist in 7.x let alone 8.x
BrowserTestBase has three calls for non-existent statics too.
Comment #84
dawehnerThese are examples of static properties ... which all could be problematic in tests or for example when someone would run a PHP daemon.
Comment #85
catchThanks for the list. Had a look through all of them.
See below.
Already has an issue to remove it, StaticStorage doesn't help with that.
#1305882: drupal_html_id() considered harmful; remove ajax_html_ids to use GET (not POST) AJAX requests
Has its own API to clear the static. Should not be globally reset since that would break timers.
Horrible, but also has a setter for testing. Should not be globally reset since it is set during bootstrap and not on demand, and there's no useful default.
Has a setter, should not be globally reset since the value is set in DrupalKernel::preHandle() based on a container parameter, meaning there's no useful default if it gets reset.
These are only statics because PHP < 5.6 doesn't allow array class constants. We don't ever want people to be able to change that list globally.
Should not be globally reset since it exists very specifically to ensure that it only gets run once per PHP process. Method could possibly be made re-entrant so it doesn't need the static, or possibly already is and the static is cruft.
'array constant' again.
This is a static so that the example always shows the exact same date for a request - i.e. so that two elements don't show 17:35:32 and 17:35:33
Could add a setter. Should probably however get request time from a request object and not be a static at all, or be replaced in JavaScript or something different entirely, or just remove it and accept the second rollover issue, since right now it'd break with render caching anyway.
StaticStorage doesn't help long running processes here, since in those cases you'd still want all date elements rendered from the master request to have a consistent date example, assuming we don't fix it properly above. Also no memory implications since it's a single object.
Discussed above and below.
Discussed below.
There are three calls to FormState::hasAnyErrors() in core.
#1863020: View's build fails when an unrelated form on the same page has validation errors and #1305882: drupal_html_id() considered harmful; remove ajax_html_ids to use GET (not POST) AJAX requests would remove two.
The third is in FormBuilder:
Looks dodgy in the same way that the call in ViewExecutable is - it's checking if there are any errors in any forms, not just this one.
So needs an issue for that last one, and should be @deprecated.
Makes my eyes hurt. But from a quick look could probably be replaced by passing in $urls as an array and adding to it by reference, since it is reset then checked within the same method and nowhere else.
Static was added in 2012 when the class went in, as far as I can see should be a constant like the other constants for that class.
Another 'array constant'.
Of those, several should definitely not under any circumstances be changed to either drupal_static() or StaticStorage. Three main categories for these:
1. protected static = array() because we don't require PHP 5.6, after which they could be constants. Just needs a @todo. In the case of Xss tag lists, converting those to StaticStorage would allow contrib/custom code to introduce security issues where they currently can't (if one altered the global list to save passing in a method param, much worse things have been done before).
2. timers, environment status, re-entrance checks, all things which can't be globally reset, and have setters for testing. Ugly, but StaticStorage doesn't help.
3. Bugs/travesties, for which StaticStorage will not make them less of a bug/travesty, some of those have had issues open to refactor for some time. Conversion won't help those issues move faster, might hold them up.
The borderline ones for me are ExtensionDiscovery, InfoParser and FileCache, which are all maintaining static caches - so actually storing things that they explicitly don't want to recalculate during a request.
FileCache already injects a cache backend. So I wondered why it doesn't inject a chained cache backend fronted by MemoryCache and drop the static property, or use MemoryCache directly and do it's own chaining internally. Then I read the issue and found this comment #2395143-86: YAML parsing is very slow, cache it with FileCache by a certain @dawehner. So not sure, but I think that could use its own issue to revisit if we need to. For example we could potentially replace that with a 'statically caching memory backend' that has lots of comments explaining why it does it. Cache backends have a deleteAll() method which would cover the long-running process/memory reclaim issue.
ExtensionDiscovery and InfoParser are both mostly called by https://api.drupal.org/api/drupal/core%21modules%21system%21system.modul...
That entire area is very messy, and if we continue refactoring it, could probably end up with class properties rather than statics, although we'd likely run into the same issue with container rebuilds then in the same way that FileCache does.
Those three, very specific and similar cases in more or less the same subsystems do not really support adding a generic API for static storage, especially not one that is described as a replacement for drupal_static().
Comment #86
mile23Given #185 should we rescope this issue to deprecate drupal_static() and pals, without adding StaticStorage? Then make follow-ups to do all the work suggested?
Comment #87
catchLinked to one wrong issue in #85, the Html::id() issue that's actually open is #1090592: [meta] Use HTML5 data-drupal-* attributes instead of #ID selectors in Drupal.settings, #85 links to the previous one.
@Mile23 there's a meta for drupal_static() at #1577902: [META] Remove all usages of drupal_static() & drupal_static_reset() , but as discussed in that issue, usually the drupal_static() removal is a by-product of a wider refactoring, so a single meta to track them is hard.
However in #83 I found at least two calls to drupal_static() that are dead code and could be removed, we could open a single issue to find and remove those calls since there may be others. That would then make it easier to see what's left.
Some of the cases in #85 could definitely have individual issues opened, like the array class constants and MailHandler etc.
I think it's worth an issue to talk about ExtensionDiscovery/FileCache/InfoParser more - potentially this issue could stay open for that?
Comment #88
berdir+1 to #86.
For example, we already have a static cache bin that's just a memory backend. We could use that for some cases. One of the benefits we have there is that we have cache tag based invalidation. So maybe we can also get rid of some manual cache invalidations through that.
Comment #89
mile23The use case for
StaticStorageis admittedly weak: We wanted a replacement fordrupal_static()that could be reset without loading .inc files or without booting a kernel in order to load an .inc file. This was for testing purposes.Generally, use of statics like this, especially hidden behind a non-autoloaded global function, is bad form, so the desire was to provide an alternative.
However, it's clear the
StaticStoragepart is a no-go, which is why I asked about rescoping for deprecation in #86.This issue would not be a meta for deprecation, but to apply the following patch which would mark
drupal_static()anddrupal_static_reset()as@deprecatedfor removal in Drupal 9.0.0.That way, we can signal to contrib that they should stop using it, and also have a clear reason for core maintainers to reject its use. This would also make all those other issues implied in #85 actionable.
Comment #90
donquixote commentedRelated issue I just opened, which is hopefully not too confusing:
#2729725: Reset API / ResettableInterface (brainstorming)
Comment #96
claudiu.cristeaThe question now is how to remove the usage from procedural code. If we cannot, we can't unpostpone this one.
Comment #97
claudiu.cristeaThis is the parent issue of #1577902: [META] Remove all usages of drupal_static() & drupal_static_reset() , not the other way around.
Comment #98
mile23Added #3037054: Deprecate drupal_static_reset() without @trigger_error()
Comment #105
claudiu.cristeaI think we still can deprecate the usage for contrib and custom code callers. Then we gradually deprecate core usages by progressing with #1577902: [META] Remove all usages of drupal_static() & drupal_static_reset() . Some may argue that the solution is ugly but I find it effective.
The MR is not a continuation of previous PRs, even we can already add the `@deprecation` tags to both functions.
Comment #107
daffie commentedShould we not remove all usage of the two functions before we deprecate them?
Comment #108
catchYes let's explicitly postpone this on #1577902: [META] Remove all usages of drupal_static() & drupal_static_reset() .
Comment #109
daffie commented