Problem/Motivation
In #3082958: Add gitignore(s) to Composer-ready project templates, we want to add initial .gitignore files to our two Composer Project Templates to give users guidance on how to set up these files based on their preference to either commit the vendor directory or not.
The problem we face in that issue is that these two files are identical, save for slight variations in the path. It would be preferable if we could maintain but a single .gitignore, and place it in the right location with the scaffold plugin. In order to do that, we would need to interpolate replacement tokens inside the file in order to generate the correct relative path for ignored files inside the Drupal root.
Proposed resolution
Add an 'interpolate' flag to 'replace' operations:
"extra": {
"drupal-scaffold": {
"file-mapping": {
"[project-root]/.gitignore": {
"mode": "replace",
"path": "assets/example.gitignore",
"interpolate": true,
"overwrite": false
}
}
}
}
Once interpolation is enabled in a scaffold file, replacement tokens will be substituted in the file when it is copied into place. For example, given:
[web-root]/index.php
The result will be either ./index.php or web/index.php (or similar), depending on the value of [web-root] in the project's "locations" section.
Remaining tasks
None.
User interface changes
None.
API changes
None.
Data model changes
See example "replace" operation above.
Release notes snippet
It is now possible to use template substitutions in scaffolded files.
| Comment | File | Size | Author |
|---|---|---|---|
| #33 | 28-to-33-interdiff.txt | 5.67 KB | greg.1.anderson |
| #33 | 3086819-33.patch | 26.86 KB | greg.1.anderson |
| #28 | 3086819-28.patch | 32.53 KB | greg.1.anderson |
| #28 | 25-to-28-interdiff.txt | 4.35 KB | greg.1.anderson |
| #25 | 20-to-25-interdiff.txt | 1.84 KB | greg.1.anderson |
Comments
Comment #2
webchickOh, wow. This would be really great!
Comment #3
greg.1.anderson commentedHere's a start. This should pretty much do the trick, but will require some documentation updates, etc.
Comment #4
greg.1.anderson commentedRun the tests.
Comment #5
aaronmchaleNice! Also “27,404 pass”, hehe
Comment #6
greg.1.anderson commentedUpdated with docs.
Comment #7
greg.1.anderson commentedProvided a change record, so update issue summary.
Comment #9
MixologicTest fail appears to be a random fail - but there are 8 cs issues with the patch. I've requeued to verify the random fail.
Comment #10
greg.1.anderson commentedCode style fixes.
Comment #11
jibranI know we are not changing the project root file here but there are some lock file updates missing if you run
composer update drupal/core. Can we add those updates here as well?Comment #12
greg.1.anderson commentedRegarding #11, I think I have seen these updates already on one of the patches. Rather than risk a conflict, I'm going to leave them off here. We could make a separate issue if we thought it would be more help than harm vis-a-vis the 11 October feature freeze. My instinct at the moment is to not take up folks' time with this until after that date has passed.
Comment #13
mbayntonIt looks like the ensureLocations() method was removed and its array_map operation not moved elsewhere. Could you provide a little background on how that's safe / why it's no longer needed?
Comment #14
greg.1.anderson commentedensureLocationswas serving two purposes:1. It converted the paths from relative to full
2. It ensured that the parent directory of each location exists
The first was actually a side effect.
In order to perform substitutions on path locations, e.g. [web-root], it was necessary to convert the path variables from full paths to relative paths. That is, we want [web-root] to become "docroot", not the full path to where docroot happens to be located when the file is generated. Converting from full paths to relative paths turned out to be trivial to do, as paths relative to the cwd still worked.
It turns out that the ReplaceOp already ensures that the parent directory of the destination file exists, so the other half of the responsibilities of ensureLocations was therefore also unnecessary, and the entire method therefore disappeared.
Comment #16
mbayntonI've reviewed the code and extensively tested this. The same path determination and interpolation that's been modified here to meet the needs of template substitution was already being used to determine actual destination paths where scaffolded files should be layed down on the disk, and so my greatest concern was that we might cause regressions with that, but I have been unable to find any issues with the scaffolded files being placed in the wrong location. In particular,
composer --working-dirinvocations still place scaffold files correctly.However, I've tested using this with a modified `drupal/core` that includes interpolation instructions for a scaffolded
.gitignore, and discovered it does not quite meet our needs for #3082958: Add gitignore(s) to Composer-ready project templates. The problem is that we are defining the location of the web-root to be.for LegacyProject, which gets rendered out to the.gitignorefiles. Surprisingly, a.anywhere in a gitignore rule breaks it. E.g., git does not treat.in .gitignore paths like Linux/Windows does.There is no value we can provide for web-root that both causes the scaffold files to write out to the correct location AND generates a working .gitignore.
Comment #17
greg.1.anderson commentedTo address #16, I added an
interpolatePathmethod to the Interpolator class. This allows us to adopt the convention that paths are .gitignore-style paths (e.g.[web-root]/index.phpproduces/index.php), and then fix up that assumption if we are in a context where we know that the substitution is being interpreted as a path (i.e./index.phpis fixed up to./index.phpif it is a path).The interpolatePath method calls through to the usual interpolate method with $fix_paths = true so that we can in the future, if we desire, add a flag to the replace operation to allow replacements to be done path-style instead of .gitignore-style. I don't think we need this feature today, so I did not provide it, but with this implementation it will be easy to add in the future if needed.
Comment #18
mbayntonI've reviewed #17. It knowingly does some massaging of the template data that you wouldn't ordinarily expect a template substitution system to do in order to solve the specific problem raised in #16. In my opinion, this is okay, because, like all scaffolding operations, these new template substitutions are configurable. This will allow us to start with the current behavior and if needed add support for more generalized templating in future by requesting it through to-be-added configuration.
Also did an end-to-end test of #17 to verify it solves our "templatize .gitignore" use case.
Comment #19
greg.1.anderson commentedThis updated patch allows us to make substitutions of values such as
/[web-root]/index.phpin a scaffold file, and still avoid doubling up any directory separators (e.g. the result of replacing[web-root]with"./"will be/index.phprather than/.//index.phpor//index.php.Comment #20
jibranNew changes in #19 make sense so back to RTBC. Rebased the patch as well.
Comment #21
hussainwebI have been seeing the usage of constants such as `true` and `false` in lower case in past few patches. Have our coding standards changed? The page at https://www.drupal.org/docs/develop/standards/coding-standards still shows that they have to be upper case.
Comment #22
hussainwebIsn't it better to just
str_replace("//", "/", $path)? Is there a use-case where doing all this work is needed?Comment #23
greg.1.anderson commentedRegarding true vs. TRUE, it's true in PSR-2, so I have a bad habit of slipping the wrong one in. The last test has three coding standards violations, for example.
Doing a simple str_replace will not fix the double directory separators, because the double separators do not exist in $path. Here's an example of the scenario.
Scaffold file contents:
If
[web-root]isdocroot/, then the result of the substitution would be:We cannot simply replace // with / in the resulting template file, because there might be a // somewhere unrelated to any of the paths we substituted. We therefore add another substitution. In addition to
'[web-root]' => 'docroot/', we also add'[web-root]/' => 'docroot/'. In other words, we add a trailing slash on the end of the replacement key so that it will consume the extra slash in the scaffold file source. We put this pattern in first so that it will be found / replaced first, and any[web-root]in the file that is not followed by a/will still be replaced withdocroot/.We need to do this work because double slashes in .gitignore files break the ignore rule.
Comment #24
hussainwebAh, got it. We are not parsing the gitignore file, we are just replacing it in the whole file contents.
Comment #25
greg.1.anderson commentedCode style.
Comment #26
greg.1.anderson commentedI imagine we should commit this to 8.9.x-dev first, and then discuss whether it should be backported.
Comment #27
alexpottWhy is the default FALSE? Or to put this question a different way - why does interpolatePath call interpolate? I think we could copy the interpolate lines into interpolatePath and also put fixDoubleDirectorySeparators() inline and we'd have less API change and less API to support in the future - ie. other code that calls interpolate never has to consider whether it should set $fix_paths to TRUE.
I was wondering is this should be
isInterpolated(). Existing methods on the OperationData class are likeSo I guess this is okay but it's a bit confusing given it's not interpolating.
Comment #28
greg.1.anderson commentedHere's an update for #27
Comment #30
greg.1.anderson commentedRe-queued tests; they may pass now that the Message Plugin has been reverted.
Comment #31
greg.1.anderson commentedcore/tests/Drupal/BuildTests/Composer/Template/ComposerProjectTemplatesTest.phpdoes not belong in this patch. That is from another issue, included by mistake (durn patch workflow).I'll remove it when I'm not in the middle of traveling.
Comment #32
greg.1.anderson commented#31 should have set this back to 'needs work'
Comment #33
greg.1.anderson commentedHere's the reroll.
Comment #34
jibranBack to RBTC as #27 is addressed.
Comment #35
greg.1.anderson commentedPostponed on #3092563: Avoid overwriting .htaccess changes during scaffolding > security problem
Comment #36
greg.1.anderson commentedThis issue does not have to be postponed on all of #3092563: Avoid overwriting .htaccess changes during scaffolding > security problem; the blocking portion of that issue was split out into #3103090: Avoid re-scaffolding unchanged files (and printing scaffold file information over and over), which was committed to 8.9.x and 9.0.x. We could therefore re-roll and continue here.