Closed (fixed)
Project:
Drupal core
Version:
8.0.x-dev
Component:
base system
Priority:
Critical
Category:
Task
Assigned:
Unassigned
Reporter:
Created:
3 Aug 2015 at 21:33 UTC
Updated:
7 Sep 2015 at 18:15 UTC
Jump to comment: Most recent, Most recent file
Comments
Comment #1
Anonymous (not verified) commentedyes please.
Comment #2
msonnabaum commentedComment #3
catchYes I'd assumed it was possible to pre-compile Twig templates in a build step (and/or potentially d.o packaging of themes and modules). But given the same filename would be used for different resulting templates, that's not possible at the moment. This came up on #2497281: Add an intermediate cache for Twig templates to avoid regenerating identical templates too.
Comment #4
fabianx commented#3: We could overwrite the used hash based on loaded extensions, which does not change per request.
Then the only thing is when code changes, but never on run-time.
And if someone has a crazy workflow with an extension changing on run-time then likely they already now need to overwrite the TwigExtension service, so that is pre-existing.
Comment #5
dawehnerSo currently the twig filename is determined by using PHPStorage which takes into account:
* The filename
* The secret, which is the hash salt by default.
* the directory mtime
Can't we expand the secret with something like
serialize($active_modules)(which is information available in the container), so this will automatically invalidate all usages of the templates?Comment #6
xjmComment #7
catchWe discussed this on the critical triage call with myself, Alex Pott, Alex Bronstein, xjm and webchick, and agreed with removing the requirement (I'm not sure if we 100% decided it was a release blocker, so not adding the triaged tag just yet).
In general I think the main change we should do in this issue is #5. Ensure that a changed module list results in different Twig template files on the filesystem (and also add the settings.php $deployment_identifier so a new filename can be forced if necessary). Then two things become possible:
1. Compiling templates in a build step - since you can have the old and new compiled templates next to each other, and the production site should switch from one to the other when the hash updates on production.
2. Putting the templates on a local filesystem, since they never get invalidated, just superceded by new file locations.
That combined with #2497243: Replace Symfony container with a Drupal one, stored in cache means that an individual Drupal install is not required to write PHP to a shared filesystem at all.
However, I don't think we can remove MTimeProtectedFileStorage - since you'll still need to write PHP for Twig on production if you're not pre-compiling - we don't want to make pre-compile of templates a requirement either (especially since that can't happen as part of d.o packaging). So we should also fix #2527478: Resolve infinite stampede in mtime protected PHP storage so that things work for sites that aren't relying on it. We can document strongly that it's there for Twig and discourage anyone else from using it though.
Comment #8
webchickJust for clarification:
Can't because it's dependent on site-specific things that Drupal.org can't know about? Or can't because no one has done the work?
Comment #9
catchThis.
The eventual compiled template depends on twig extensions, modules can define twig extensions.
Unless we remove that ability (i.e. support a fixed list of core extensions and nothing else), then compiled templates depend on the exact site configuration and state of the code base at the time.
That's also the reason why a shared fileystem is currently a requirement - if the compiled template from a specific twig tpl was always identical we'd not need to do extra work here with the filename hash.
Comment #10
dawehnerSo do we need more than this here?
Comment #11
gábor hojtsyMinor cleanups to make it easier to review.
Comment #12
fabianx commentedI don't think we even need the list of enabled modules.
The list of extensions that the Twig_Extension knows should be more than enough. Possibly even the hash of the concatenated spl_object_hash thereof as we have the extensions loaded already anyway.
And I think spl_object_hash is very fast as its just reading an internal value.
Also while we override this thing anyway, I would like to add the basename to the filename, so its easier to find templates, which has been bugging me forever - though that can also be a non-critical follow-up.
Third:
With phpstorage, we don't need the complicated subdir structure, as it is just replacing '/' with '#' anyway.
e.g.
would be my proposal.
Comment #13
znerol commentedWhat is the cost in terms of performance if the templates are not dumped at all by default? After so much work has been invested into making every bit of markup cacheable, compiling the templates to PHP files is maybe not strictly required anymore?
Comment #14
fabianx commentedAttached patch should do the trick and also has a nicer filename structure:
e.g.
As original author of the code I took the liberty to improve some missing docs.
Comment #15
fabianx commentedX-POST with #13:
Not 100% sure, gonna measure real quick.
Comment #16
fabianx commented#14 did not take $this->cache into account as I was trying something else.
Fixed here.
Comment #18
fabianx commented#13: Unfortunately that won't be possible:
Number of Function Calls 47,999 235,045 187,046 389.7%
Incl. Wall Time (microsec) 85,800 271,556 185,756 216.5%
So around 4x slower ...
--
We can write a test for this like in core/modules/system/src/Tests/Theme/TwigSettingsTest.php.
Comment #19
chx commentedI think this is a duplicate of #2301163: Create a phpstorage backend that works with a local disk but coordinates across multiple webheads but I also would like to make all the issues referred from #2547827: PhpStorage: past, present and future postponed and make the decisions there first.
Comment #20
dawehnerThis sounds like something you can easily calculate on container build time, can't you?
Comment #21
dawehnerLet's move it there.
Comment #23
dawehnerAlright, let's be nice today and use valid PHP.
Comment #25
wim leersAddressed everything below, plus some nitpicks.
s/$this/$container/ -> this will fix the test failures. :)
Comment inconsistent with other parameter-setting compiler passes (such as
ListCacheBinsPassandCacheContextsPass).Doesn't this mean that we're no longer writing these to.php.Confusingly, the implementation in the parent class sets a
.phpsuffix, but that doesn't appear to be important at all.Comment #26
dawehnerWell right, file extensions don't matter most of the time. I think we though still append .php, just so we have this tiny little bit of protecting to never show the file content to the user, in case someone manages to find the filename itself
Comment #27
wim leersYep,
*.phpis still appended by the caller of this method. I didn't make that clear, but at this point it's the parent class that is confusing me.But in any case, I can confirm from brief manual testing that this still works as expected.
Comment #28
dawehnerAlright working on some simple tests now.
Comment #29
dawehnerThere we go.
Comment #30
fabianx commentedLooks good to me :).
Comment #31
alexpottDon't we need to hash by the mtime of the files that contain the class? As a change to the class could result in changes to the templates right?
Why add \Drupal::version twice?
Comment #32
dawehnerWell, this is just the default value, but you could choose your own custom deployment identifier. Note:
\Drupal\Core\DrupalKernel::getClassNameuses the same pattern.Comment #33
dawehnerThere we go.
Comment #34
alexpottYou're going to hate me - but we need both the mtime and the classname. Sorry I should have been more explicit.
Comment #35
fabianx commentedYes, lets add the classname or the filename, then back to RTBC, while that is the pattern to test for freshness in theory we would need to check mtime of all NodeVisitors and Token Parsers registered in each extension, so this was only intended for when the list of extensions changes.
Any changes to the extensions itself would likely be a new deployment identifier, but both works equally well.
Comment #36
gábor hojtsyIs there an update related requirement here? We are changing the compiler passes, do we need to provide some update for this or is the container rebuilt on update?
Also any change notices that should be made? It seems like an internal change for what the file naming is, so it would be more "informational", so not sure...
Comment #37
catch#2507509: Service changes should not result in fatal errors between patch or minor releases was designed to fix this at least for patch-level core releases so it shouldn't need anything specific.
Comment #38
chx commentedComment #39
chx commentedComment #40
chx commentedComment #41
dawehnerThere we go.
Comment #42
catchClever. Had been concerned about this one and there's a fix for it right there in the patch.
\Drupal::VERSION and deployment identifier.
Settings argument could be left off since version is already in the hash?
Comment #43
dawehnerPuh, we had some space left on this row.
Nice!
Comment #44
catchOK so more of an actual review.
- So the filemtime is great for picking up changes to individual classes
- with it, I was wondering if we need the Drupal::VERSION and/or $deployment_identifier bits after all, since the exact classes and age results in a new hash anyway.
- then I thought they'd be useful anyway, since that would allow you to precompile templates to a specific hash location in advance of deployment (to allow read-only filestorage to be used, not just write-once).
- then I realised that depending on how code is deployed, the mtime might be different so you couldn't predict the hash.
So having the mtime in the hash makes 'write once to a local filesystem' much easier, but it also makes 'compile in a build step and deploy templates with code updates' potentially harder. I think it's fine to say that if you compile in a build step, you also need to make sure your deployment process preserves mtime on files. Just bringing it up as a trade-off.
And given all that, since the hash is going to depend on mtime, I'm really not sure we need the version and $deployment identifier in there after all.
Comment #46
dawehnerYou could argue that the deployment identifier is basically used to clear those stored PHP files, so you could assume that they are cleared if you change the identifier.
At least for now though, the deployment identifier just talks about the container, and not about the twig files.
Comment #47
dawehnerI agree, for now its fine to not take into account the deployment identifier. In contrast to the container we know what the template will vary by.
Comment #48
jibranDo we need a change record for new compiler pass? Other then that this is RTBC.
Comment #49
larowlanagree
Comment #50
alexpottThere's a bit of issue management we have to do before committing this.
Comment #51
alexpottThe patch looks great and the new thinking about Drupal version and the deployment identifier makes lots of sense. Nice work. Below are a few of nits...
reflection misspelt.
every time is two words.
Not used anymore.
Comment #52
dawehnerChanged the IS
Comment #53
gábor hojtsyComment not accurate anymore AFAIS(?)
Comment #54
dawehnerFixed those points.
Comment #55
dawehner.
Comment #56
jibranChanges look good. Let's create the change record and be done with it.
Comment #57
dawehner@jibran
There is one https://www.drupal.org/node/2551435
Comment #58
jibranOk thanks @dawehner.
Comment #59
alexpottLet's done the hashing once in the compiler pass. Was less hashing and also that makes the container parameter
twig_extension_hasha hash rather than data to hash and therefore the parameter name and value are a better match. The other thing to consider with crc32b is how worried about collisions are we?Comment #60
dawehnerReally strong point!
So a) yes, let's not hash on runtime. Once we have done that, we can actually simplify the code a bit.
Regarding cr32b vs. sha ... It doens't matter on runtime at all, so we can choose what we want. This part of the code is not about security, so I always kinda like to avoid using sha.
I like the additioinal semantic information you transport when you use sha vs. not.
Let's get also to a real point, these extensions wont' be changed often, especially given that hopefully not many modules provide twig extensions.
Comment #62
wim leersCRC32 has >4 billion possible values, so collisions are very unlikely.
Comment #63
dawehnerReupload the patch, this could be a random fail.
Comment #64
wim leersHrm, yes, the 5 failures in #60 look completely unrelated to the changes in this patch, so they're extremely likely to be random failures. Back to RTBC.
Comment #66
wim leersThis was another case of #2552687: Test failures in ConfigFormOverrideTest and ContainerRebuildWebTest on newly spun up testbot instances.
Comment #67
alexpottWith my comment about
I didn't mean that we should change the algo I meant we should consider the issue. With
crc32bthe directory names stay sane with sha256 they get insane. If we need sha256 we should consider the directory structure. Sorry.Comment #68
wim leersIt's not clear to me what you mean. Hopefully it's clear to @dawehner.
Comment #69
alexpottWhat i meam is:
SHA256 twig directory names
crc32b directory names
Comment #70
dawehnerAlex was faster ...
Expanded the docs a bit.
Comment #71
wim leersUgh, somehow I misread the #60 interdiff, I thought the patch already was using
crc32b! My bad.Comment #72
wim leersComment #73
alexpottCommitted b05a606 and pushed to 8.0.x. Thanks!
Comment #76
star-szrInline templates + Windows = #2558885: TwigEnvironment is unable to cache inline templates because it sends invalid filenames to MTimeProtectedFastFileStorage, need some test coverage and eyes there please :)
Comment #77
david_garcia commentedTemplate contents were leaking into filenames, so I guess this will also affect POSIX systems if the template contains characters that are invalid on that platform. So this is not just Windows specific, but more likely to happen there because a bigger range of characters are disallowed in path/file names.