Problem/Motivation

In #3082958: Add gitignore(s) to Composer-ready project templates, we want to add initial .gitignore files to our two Composer Project Templates to give users guidance on how to set up these files based on their preference to either commit the vendor directory or not.

The problem we face in that issue is that these two files are identical, save for slight variations in the path. It would be preferable if we could maintain but a single .gitignore, and place it in the right location with the scaffold plugin. In order to do that, we would need to interpolate replacement tokens inside the file in order to generate the correct relative path for ignored files inside the Drupal root.

Proposed resolution

Add an 'interpolate' flag to 'replace' operations:

  "extra": {
    "drupal-scaffold": {
      "file-mapping": {
        "[project-root]/.gitignore": {
          "mode": "replace",
          "path": "assets/example.gitignore",
          "interpolate": true,
          "overwrite": false
        }
      }
    }
  }

Once interpolation is enabled in a scaffold file, replacement tokens will be substituted in the file when it is copied into place. For example, given:

[web-root]/index.php

The result will be either ./index.php or web/index.php (or similar), depending on the value of [web-root] in the project's "locations" section.

Remaining tasks

None.

User interface changes

None.

API changes

None.

Data model changes

See example "replace" operation above.

Release notes snippet

It is now possible to use template substitutions in scaffolded files.

Comments

greg.1.anderson created an issue. See original summary.

webchick’s picture

Oh, wow. This would be really great!

greg.1.anderson’s picture

StatusFileSize
new26.42 KB

Here's a start. This should pretty much do the trick, but will require some documentation updates, etc.

greg.1.anderson’s picture

Status: Active » Needs review

Run the tests.

aaronmchale’s picture

Nice! Also “27,404 pass”, hehe

greg.1.anderson’s picture

Issue summary: View changes
StatusFileSize
new28.93 KB
new2.51 KB

Updated with docs.

greg.1.anderson’s picture

Issue summary: View changes

Provided a change record, so update issue summary.

Status: Needs review » Needs work

The last submitted patch, 6: 3086819-6.patch, failed testing. View results
- codesniffer_fixes.patch Interdiff of automated coding standards fixes only.

Mixologic’s picture

Test fail appears to be a random fail - but there are 8 cs issues with the patch. I've requeued to verify the random fail.

greg.1.anderson’s picture

Status: Needs work » Needs review
StatusFileSize
new29.32 KB
new3.25 KB

Code style fixes.

jibran’s picture

I know we are not changing the project root file here but there are some lock file updates missing if you run composer update drupal/core . Can we add those updates here as well?

greg.1.anderson’s picture

Regarding #11, I think I have seen these updates already on one of the patches. Rather than risk a conflict, I'm going to leave them off here. We could make a separate issue if we thought it would be more help than harm vis-a-vis the 11 October feature freeze. My instinct at the moment is to not take up folks' time with this until after that date has passed.

mbaynton’s picture

-  protected function ensureLocations() {
+  protected function getVendorRelativePath() {
+    $vendor_dir = $this->composer->getConfig()->get('vendor-dir');
     $fs = new Filesystem();
-    $locations = $this->getOptions()->locations() + ['web_root' => './'];
-    $locations = array_map(function ($location) use ($fs) {
-      $fs->ensureDirectoryExists($location);
-      $location = realpath($location);
-      return $location;
-    }, $locations);
-    return $locations;
+    return $fs->findShortestPath(getcwd(), $vendor_dir);
   }

It looks like the ensureLocations() method was removed and its array_map operation not moved elsewhere. Could you provide a little background on how that's safe / why it's no longer needed?

greg.1.anderson’s picture

ensureLocations was serving two purposes:

1. It converted the paths from relative to full
2. It ensured that the parent directory of each location exists

The first was actually a side effect.

In order to perform substitutions on path locations, e.g. [web-root], it was necessary to convert the path variables from full paths to relative paths. That is, we want [web-root] to become "docroot", not the full path to where docroot happens to be located when the file is generated. Converting from full paths to relative paths turned out to be trivial to do, as paths relative to the cwd still worked.

It turns out that the ReplaceOp already ensures that the parent directory of the destination file exists, so the other half of the responsibilities of ensureLocations was therefore also unnecessary, and the entire method therefore disappeared.

Version: 8.8.x-dev » 8.9.x-dev

Drupal 8.8.0-alpha1 will be released the week of October 14th, 2019, which means new developments and disruptive changes should now be targeted against the 8.9.x-dev branch. (Any changes to 8.9.x will also be committed to 9.0.x in preparation for Drupal 9’s release, but some changes like significant feature additions will be deferred to 9.1.x.). For more information see the Drupal 8 and 9 minor version schedule and the Allowed changes during the Drupal 8 and 9 release cycles.

mbaynton’s picture

Version: 8.9.x-dev » 8.8.x-dev
Status: Needs review » Needs work

I've reviewed the code and extensively tested this. The same path determination and interpolation that's been modified here to meet the needs of template substitution was already being used to determine actual destination paths where scaffolded files should be layed down on the disk, and so my greatest concern was that we might cause regressions with that, but I have been unable to find any issues with the scaffolded files being placed in the wrong location. In particular, composer --working-dir invocations still place scaffold files correctly.

However, I've tested using this with a modified `drupal/core` that includes interpolation instructions for a scaffolded .gitignore, and discovered it does not quite meet our needs for #3082958: Add gitignore(s) to Composer-ready project templates. The problem is that we are defining the location of the web-root to be . for LegacyProject, which gets rendered out to the .gitignore files. Surprisingly, a . anywhere in a gitignore rule breaks it. E.g., git does not treat . in .gitignore paths like Linux/Windows does.

There is no value we can provide for web-root that both causes the scaffold files to write out to the correct location AND generates a working .gitignore.

greg.1.anderson’s picture

Status: Needs work » Needs review
StatusFileSize
new25.04 KB
new11.67 KB

To address #16, I added an interpolatePath method to the Interpolator class. This allows us to adopt the convention that paths are .gitignore-style paths (e.g. [web-root]/index.php produces /index.php), and then fix up that assumption if we are in a context where we know that the substitution is being interpreted as a path (i.e. /index.php is fixed up to ./index.php if it is a path).

The interpolatePath method calls through to the usual interpolate method with $fix_paths = true so that we can in the future, if we desire, add a flag to the replace operation to allow replacements to be done path-style instead of .gitignore-style. I don't think we need this feature today, so I did not provide it, but with this implementation it will be easy to add in the future if needed.

mbaynton’s picture

Status: Needs review » Reviewed & tested by the community

I've reviewed #17. It knowingly does some massaging of the template data that you wouldn't ordinarily expect a template substitution system to do in order to solve the specific problem raised in #16. In my opinion, this is okay, because, like all scaffolding operations, these new template substitutions are configurable. This will allow us to start with the current behavior and if needed add support for more generalized templating in future by requesting it through to-be-added configuration.

Also did an end-to-end test of #17 to verify it solves our "templatize .gitignore" use case.

greg.1.anderson’s picture

Status: Reviewed & tested by the community » Needs review
StatusFileSize
new5.43 KB
new44.43 KB

This updated patch allows us to make substitutions of values such as /[web-root]/index.php in a scaffold file, and still avoid doubling up any directory separators (e.g. the result of replacing [web-root] with "./" will be /index.php rather than /.//index.php or //index.php.

jibran’s picture

Status: Needs review » Reviewed & tested by the community
StatusFileSize
new26.76 KB

New changes in #19 make sense so back to RTBC. Rebased the patch as well.

hussainweb’s picture

+++ b/composer/Plugin/Scaffold/Interpolator.php
@@ -88,16 +88,89 @@ public function addData(array $data) {
+  public function interpolate($message, array $extra = [], $default = '', $fix_paths = false) {

I have been seeing the usage of constants such as `true` and `false` in lower case in past few patches. Have our coding standards changed? The page at https://www.drupal.org/docs/develop/standards/coding-standards still shows that they have to be upper case.

hussainweb’s picture

+++ b/composer/Plugin/Scaffold/Interpolator.php
@@ -88,16 +88,89 @@ public function addData(array $data) {
+  protected function fixDoubleDirectorySeparators($replacements) {
+    $result = [];
+    foreach ($replacements as $path => $value) {

Isn't it better to just str_replace("//", "/", $path)? Is there a use-case where doing all this work is needed?

greg.1.anderson’s picture

Regarding true vs. TRUE, it's true in PSR-2, so I have a bad habit of slipping the wrong one in. The last test has three coding standards violations, for example.

Doing a simple str_replace will not fix the double directory separators, because the double separators do not exist in $path. Here's an example of the scenario.

Scaffold file contents:

# Ignore paths that contain user-generated content.
/[web-root]/sites/*/files
/[web-root]/sites/*/private

If [web-root] is docroot/, then the result of the substitution would be:

# Ignore paths that contain user-generated content.
/docroot//sites/*/files
/docroot//sites/*/private

We cannot simply replace // with / in the resulting template file, because there might be a // somewhere unrelated to any of the paths we substituted. We therefore add another substitution. In addition to '[web-root]' => 'docroot/', we also add '[web-root]/' => 'docroot/'. In other words, we add a trailing slash on the end of the replacement key so that it will consume the extra slash in the scaffold file source. We put this pattern in first so that it will be found / replaced first, and any [web-root] in the file that is not followed by a / will still be replaced with docroot/.

We need to do this work because double slashes in .gitignore files break the ignore rule.

hussainweb’s picture

Ah, got it. We are not parsing the gitignore file, we are just replacing it in the whole file contents.

greg.1.anderson’s picture

StatusFileSize
new26.8 KB
new1.84 KB

Code style.

greg.1.anderson’s picture

Version: 8.8.x-dev » 8.9.x-dev

I imagine we should commit this to 8.9.x-dev first, and then discuss whether it should be backported.

alexpott’s picture

Status: Reviewed & tested by the community » Needs work
  1. +++ b/composer/Plugin/Scaffold/Interpolator.php
    @@ -88,16 +88,89 @@ public function addData(array $data) {
    +   * @param bool $fix_paths
    +   *   TRUE to ensure that [web-root]/index.php is rendered as ./index.php, and
    +   *   not as /index.php.
    ...
    -  public function interpolate($message, array $extra = [], $default = '') {
    +  public function interpolate($message, array $extra = [], $default = '', $fix_paths = FALSE) {
    

    Why is the default FALSE? Or to put this question a different way - why does interpolatePath call interpolate? I think we could copy the interpolate lines into interpolatePath and also put fixDoubleDirectorySeparators() inline and we'd have less API change and less API to support in the future - ie. other code that calls interpolate never has to consider whether it should set $fix_paths to TRUE.

  2. +++ b/composer/Plugin/Scaffold/Operations/OperationData.php
    @@ -90,6 +91,16 @@ public function overwrite() {
    +  /**
    +   * Whether we are going to interpolate replacements in a scaffold file.
    +   *
    +   * @return bool
    +   *   Returns true if interpolation is desired.
    +   */
    +  public function interpolate() {
    

    I was wondering is this should be isInterpolated(). Existing methods on the OperationData class are like

      /**
       * Determines overwrite.
       *
       * @return bool
       *   Returns true if overwrite mode was selected.
       */
      public function overwrite() {
        return !empty($this->data[self::OVERWRITE]);
      }
    
     /**
       * Checks if prepend path exists.
       *
       * @return bool
       *   Returns true if prepend exists.
       */
      public function hasPrepend() {
        return isset($this->data[self::PREPEND]);
      }
    
      /**
       * Gets prepend path.
       *
       * @return string
       *   Path to prepend data
       */
      public function prepend() {
        return $this->data[self::PREPEND];
      }
    

    So I guess this is okay but it's a bit confusing given it's not interpolating.

greg.1.anderson’s picture

Status: Needs work » Needs review
StatusFileSize
new4.35 KB
new32.53 KB

Here's an update for #27

Status: Needs review » Needs work

The last submitted patch, 28: 3086819-28.patch, failed testing. View results
- codesniffer_fixes.patch Interdiff of automated coding standards fixes only.

greg.1.anderson’s picture

Status: Needs work » Needs review

Re-queued tests; they may pass now that the Message Plugin has been reverted.

greg.1.anderson’s picture

core/tests/Drupal/BuildTests/Composer/Template/ComposerProjectTemplatesTest.php does not belong in this patch. That is from another issue, included by mistake (durn patch workflow).

I'll remove it when I'm not in the middle of traveling.

greg.1.anderson’s picture

Status: Needs review » Needs work

#31 should have set this back to 'needs work'

greg.1.anderson’s picture

Status: Needs work » Needs review
StatusFileSize
new26.86 KB
new5.67 KB

Here's the reroll.

jibran’s picture

Status: Needs review » Reviewed & tested by the community

Back to RBTC as #27 is addressed.

greg.1.anderson’s picture

Status: Reviewed & tested by the community » Postponed
greg.1.anderson’s picture

Status: Postponed » Needs work

This issue does not have to be postponed on all of #3092563: Avoid overwriting .htaccess changes during scaffolding > security problem; the blocking portion of that issue was split out into #3103090: Avoid re-scaffolding unchanged files (and printing scaffold file information over and over), which was committed to 8.9.x and 9.0.x. We could therefore re-roll and continue here.

Version: 8.9.x-dev » 9.1.x-dev

Drupal 8.9.0-beta1 was released on March 20, 2020. 8.9.x is the final, long-term support (LTS) minor release of Drupal 8, which means new developments and disruptive changes should now be targeted against the 9.1.x-dev branch. For more information see the Drupal 8 and 9 minor version schedule and the Allowed changes during the Drupal 8 and 9 release cycles.

Version: 9.1.x-dev » 9.2.x-dev

Drupal 9.1.0-alpha1 will be released the week of October 19, 2020, which means new developments and disruptive changes should now be targeted for the 9.2.x-dev branch. For more information see the Drupal 9 minor version schedule and the Allowed changes during the Drupal 9 release cycle.

Version: 9.2.x-dev » 9.3.x-dev

Drupal 9.2.0-alpha1 will be released the week of May 3, 2021, which means new developments and disruptive changes should now be targeted for the 9.3.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

Version: 9.3.x-dev » 9.4.x-dev

Drupal 9.3.0-rc1 was released on November 26, 2021, which means new developments and disruptive changes should now be targeted for the 9.4.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

Version: 9.4.x-dev » 9.5.x-dev

Drupal 9.4.0-alpha1 was released on May 6, 2022, which means new developments and disruptive changes should now be targeted for the 9.5.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

Version: 9.5.x-dev » 10.1.x-dev

Drupal 9.5.0-beta2 and Drupal 10.0.0-beta2 were released on September 29, 2022, which means new developments and disruptive changes should now be targeted for the 10.1.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

Version: 10.1.x-dev » 11.x-dev

Drupal core is moving towards using a “main” branch. As an interim step, a new 11.x branch has been opened, as Drupal.org infrastructure cannot currently fully support a branch named main. New developments and disruptive changes should now be targeted for the 11.x branch, which currently accepts only minor-version allowed changes. For more information, see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

Version: 11.x-dev » main

Drupal core is now using the main branch as the primary development branch. New developments and disruptive changes should now be targeted to the main branch.

Read more in the announcement.