Create a tool to convert a non-Composer-based Drupal to a Composer-based one

My current workflow is `drush generate-makefile example.make` > `drush make-convert example.make --format=composer > composer.json` > copy component's from generated composer.json to list into drupal-project's composer.json + adjustments.

See also: https://github.com/grasmash/composerize-drupal

Agree with @Mike23 in #4

Of course, I'm partial to https://github.com/grasmash/composerize-drupal. I believe that it works for most use cases. It has 94% test coverage and I consider it stable.

Added a proposed resolution and a list of remaining tasks to the issue summary.

As follow-on work, we could consider dropping a Drush 8 policy file into the site to protect it from future pm-update shenanigans. If the user has not converted their contrib modules, they could continue to update them with Drush. In this scenario, after the user has run `composer update` at least once, the policy file could protect the site by forcing the `--no-core` option to be set. Once the user has converted their contrib modules, the policy file could simply prevent `pm-update` and related commands from running on the site at all.

The policy file could either be added by the conversion tool, or we could make it a persistent part of Drupal, and have it decide what action to take at runtime by inspecting the site and seeing if it has been converted.

Note that there are a few features of Composerize Drupal that will no longer be necessary when working with Comoser-ready Drupal sites.

Ah. I had forgotten about the situation where somebody starts with the composer ready tarball, and then installs modules manually, and then needs to become a composer based site.

We'll still need a composerize mechanism to 'convert' those sites for people.

Will the plugin be able to "fix" the situation where a composer based drupal has had a module manually updated?

If that is the case, the plugin will remain useful after initial use.

I am wary as to the safety/reliability of fixing a manually updated module. Are we sure the module was manually updated, or was it patched with cweagans/composer-patches?

I think the "repair" operation is to have the user manually remove the tainted modules, and then run composer install. I don't think we should do this automatically.

Currently, https://github.com/grasmash/composerize-drupal will scan for contrib modules and populate the composer.json require array. It will also scan for patches and populate the patches array.

To answer your question:

Will the plugin be able to "fix" the situation where a composer based drupal has had a module manually updated?

We could easily add an additional command to https://github.com/grasmash/composerize-drupal that *only* scans for contrib modules and updates the require array. The command could be manually run even after an application has been "composerized."

That would be a useful option because I expect us non experts (or sites with multiple levels of contributors) to make such mistakes, so a way to repair the damage would be useful.

We also want to warn people who are using drupal-composer/drupal-scaffold, webflo/drupal-core-strict, webflo/drupal-core-require-dev. They should switch over to drupal/core-composer-scaffold and drupal/core-recommended.

That sounds like it might covers something I was wondering, because many people (myself included) typically setup sites using drupal-composer/drupal-project, so it sounds like this tool would make it simple to convert a site which started with that to using drupal/core-recommended, or at the very least provide documentation to accomplish that?

Yes, switching to the new Composer plugins should be pretty easy. See https://g1a.io/cic, which contains an overview.

We could warn users on old components by submitting a PR to drupal-composer/drupal-scaffold, and just start printing a big warning message every time it runs.

Yes, switching to the new Composer plugins should be pretty easy. See https://g1a.io/cic, which contains an overview.

That's a handy presentation and overview, thanks.

We could warn users on old components by submitting a PR to drupal-composer/drupal-scaffold, and just start printing a big warning message every time it runs.

Sounds like a good idea.

This seems so similar to composer validate that I wonder if we can extend that command with Drupal specific inspections.

I am probably a subset of the target audience for this feature since I've been using tarballs since the 90s :P ... for me, I would need more information on each of these points, for it to be really useful to me and for me not to feel blocked/stuck.

$ composer drupal-status

Drupal Composer Status
======================

Things to do:
 * Rename the project so it is not drupal/drupal.

# give me a command I can copy/paste, please. And please give me a recommendation so I don't need to get stuck in analysis paralysis trying to figure out what the new name should be.

 * Change drupal/core to drupal/core-recommended.

# Where? How?

 * Add drupal/core-dev to the require-dev section.

# Can you point me to docs on this?

 * Determine how to require these extensions: unmanaged_module

# I would love to but... how do I determine how? :)

 * Project specifies patches in extra (patches, patches-file, enable-patching), but does not require the cweagans/composer-patches plugin.

# Uh. What..?! :)

I think that #22 is useful as a step on the road toward implementing the use-case described in the issue summary. As such, I think that its target audience is developers working on this issue.

I think that the best way to address the questions in #25 is to continue the implementation to automate the conversion process.

Ah, ok, fair enough. :) Carry on, then!

so that I could set up an automatic build script that relies on ./vendor/bin/composer running.

Why do you have Composer as a dependency of the site in this scenario? This is unusual.

I'll try to give this a spin pretty soon.

Haven't tested or closely reviewed anything in here. Just flagging that #40 needs work not just for the failing test, but also for CS warnings:

8 coding standards messages
✗ 8 more than branch result

/var/lib/drupalci/workspace/jenkins-drupal_patches-23736/source/composer/Plugin/ComposerConverter/Command/ExtensionReconcileCommand.php ✗ 8 more
line 180	Line indented incorrectly; expected 4 spaces, found 6
181	Line indented incorrectly; expected 4 spaces, found 6
182	Line indented incorrectly; expected 4 spaces, found 6
183	Line indented incorrectly; expected 4 spaces, found 6
184	Line indented incorrectly; expected 4 spaces, found 6
185	Line indented incorrectly; expected 4 spaces, found 6
186	Line indented incorrectly; expected 4 spaces, found 6
187	Line indented incorrectly; expected 4 spaces, found 6

Re: #43: sweet, thanks. I see in the interdiff that the "CS" violations were actually commented-out code. Good move removing it entirely. However, I think we want a follow-up issue for the @todo comment, and add a @see pointing to it.

Hope to get a chance to give this a spin and a more careful review in the coming days...

Cheers,
-Derek

+1 for mile23/core-composer-converter, which just helped me to convert a client site from a tarball into a standalone git repo, with all its dependencies managed by Composer and all its tooling encapsulated by Lando.

There was a lot more to that than just running this tool (and the documentation out there offers many different options!) but this tool provided a valuable service.

I just pulled the repo on a different laptop, unzipped a tarball with the contents of web/sites/default/files from production, ran lando start and lando db-import, and practically jumped for joy when the local copy started up.

Now that I’ve beaten the lando and composer level bosses, the next step is figuring out a Continuous Integration strategy that does not require subscription fees for private repos. (I'm leaning toward GitLab-CI, but open to suggestions.)

Note: once you've got your composer.json the way you want it, you can probably remove this from core since it serves no other function.

#45,

Now that I’ve beaten the lando and composer level bosses, the next step is figuring out a Continuous Integration strategy that does not require subscription fees for private repos. (I'm leaning toward GitLab-CI, but open to suggestions.)

Bitbucket has Pipelines for CI/CD and has unlimited private repos. We haven't used it as we use a different service but it might work for you.

Updating issue summary per discussion with @mile23 on slack. My update didn't go quite as discussed, but I did follow the general goal of rewriting the goals of this issue to do only one thing, and make that one thing the MVP for the Composer conversion tool.

Is it possible to go the other way, from Composer-based Drupal 8.8.2 to tarball?

I ask because I want to migrate a site from Drupal 7 to Drupal 8, which is probably easiest done with Composer, but I want to use Automatic Updates, which only works with non-Composer ("tarball") installations ...

Will it require a tool to convert a Composer-based Drupal 8.8.2 installation (assuming no Composer-requiring modules like Solr are used) to a non-Composer-based one, or can it be done by simply downloading the tarball-based file structure (no "web") copy the contrib modules over and run drush cache:rebuild?

#51: Out of scope for this issue. I'd say the biggest challenge here is that it's more difficult to go back to tarballs if any of your contrib modules have Composer dependencies (at least not easily). There are a number of solutions for managing Composer dependencies without Composer (Ludwig et. al.), so it might be possible to detect-and-repair this.

My motivation for adding the requirement for drupal/core-composer scaffold to be present was so that we could remove the need to have a template composer.json file, and instead just do minor fixups of the composer.json already present in the site. Another way to restate this is that the site must already be on Drupal 8.8.0+ (or perhaps 8.9.0+) before using this extension. While we could consider converting Drupal 8.7.x and even older, there is the additional complication that we have to give the user instructions on adding this tool, etc.

If we instead require the user to `drush pm-update` (or autoupdate) their site to the latest version of Drupal core first, then we know that drupal/core-composer-scaffold is present, composer/installers is present, etc., and anything that is always going to be present is one less thing that we need to do in the patch.

I also think that it would be easiest to have this operate just by running `composer update` (with a prompt before doing anything), although keeping the message-plus-conversion-command is perhaps safer (as we know, running code in a Composer update hook can be dangerous).

Haven't had a chance to look at the actual patch yet. Thanks for pushing this forward.

I agree, and like I stated, going from Composer-based to tarball should ideally only be tried with none-Composer requiring modules.

I just had a go at it, and it went fine. Just copy over modules and settings.php, import the database, run drush cache:rebuild and you're good to go. I don't want to side track this issue, but insert the steps below (using Lando with Drush 8.2.3) for others who are interested in this:

1. Create Drupal installation with Composer, and export database

   cd ~/dev
   composer create-project drupal/recommended-project comp
   cd comp/
   lando init --source cwd --recipe drupal8 --webroot web --name comp
   composer require drush/drush drupal/admin_toolbar
   drush en admin_toolbar admin_toolbar_tools
   drush uli -l http://comp.lndo.site
   lando db-export
   

2. Create Drupal installation with Drush ("tarball") and import modules and database

   cd ~/dev
   mkdir tar && cd tar
   lando init --source cwd --recipe drupal8 --webroot public_html --name tar
   drush dl drupal --drupal-project-rename=public_html
   
   # copy database, modules and settings.php, and import
   cp ~/dev/comp/drupal8.2020-02-17-1581958317.sql.gz ~/dev/tar
   cp -R ~/dev/comp/web/modules/* ~/dev/tar/public_html/modules/
   cp ~/dev/comp/web/sites/default/settings.php ~/dev/tar/public_html/sites/default
   lando db-import drupal8.2020-02-17-1581958317.sql.gz
   cd public_html/
   drush cache:rebuild
   drush uli -l http://tar.lndo.site
   

Re #52:

If we instead require the user to drush pm-update (or autoupdate) their site to the latest version of Drupal core first, then we know that drupal/core-composer-scaffold is present, composer/installers is present, etc., and anything that is always going to be present is one less thing that we need to do in the patch.

Based on my recent experience, I agree. I was unable to composerize 8.7.8 tarball until I used drush pm-update to upgarde core to 8.8.x version, no matter what tool I used. (I also tried to just do it manually by starting with a drupal/recommended-project and using composer require to add my dependencies.)

The only way that worked for me was to update the tarball site to the latest version and then use Mile23's core tool.

As ever, your mileage may vary.

For the record, if anyone did have a strong desire to composerize an 8.7.x version of Drupal, what I would recommend is:

1. `drush pm-updatecode` to get to 8.8.0+ (n.b. do not run updatedb)
2. Composerize the site
3. Downgrade `drupal/core-recommended` to 8.7.8 or desired version (and likewise with any pinned module versions)
4. Run updatedb if needed
5. Undo any pins in composer.json from step 3 if necessary

I can imagine a tool to do that, but I think that's out of scope for this issue.

So, I am reviewing this more (still not all the way through), and I'm noticing that this patch is doing a lot of complex things with Composer internals in order to get a good list of modules to `require`. All of this code is hard to review, and furthermore introduces technical debt in that it would require more work to support Composer 2 in Drupal if we had all of this in core.

I am wondering if it would be reasonable to just scan he filesystem for `modulename.info.yml`, and if there is a `project:` entry injected by d.o infrastructure, then we may presume that `drupal/modulename` exists. We could also build a version constraint from the `version:` entry in the same yml file.

For extensions that do not have injected project information (e.g. see the lightning distro), we could check to see whether the project identified in the composer.json file was registered in Packagist. (This is probably a bad example, as Lightning already requires Composer. I'm not sure if there is anything for us to find in Packagist that doesn't require Composer already, so this particular optimization might not even be necessary.)

Anything that didn't meet one of these two criteria could simply be left in the filesystem / repo as an item to fix by hand.

Does that sound like a viable strategy? We could simplify this code quite a bit if we went this route, and put off the more rigorous attempts at discovery for either the contrib tool, or a least a later version of the patch

A few more review notes (still incomplete):

1. +++ b/composer/Plugin/ComposerConverter/Command/ExtensionReconcileCommand.php
   @@ -0,0 +1,273 @@
   +          $message = 'Adding composer/installers to manage Drupal extensions.';
   

   If we tighten down our validation (require Drupal 8.9.x+), then we can reduce the number of checks here; composer/installers, for example, will always be present.

2. +++ b/composer/Plugin/ComposerConverter/templates/template.composer.json
   @@ -0,0 +1,52 @@
   +    "name": "drupal/converted-project",
   

   I don't think we need to rely on a template. If we validate that we are converting Drupal 8.9.x+, then the project being converted will already have a composer.json that follows this structure. We should read the site's existing composer.json file, validate that it matches our expectations, and then inject the "require" section that we want.

My proposal is that we start with a tool that only does conversion of non-Composer contrib modules into Composer-managed contrib modules for sites that are already Composer-ready. That seems like the right increment of work, and gives most folks a `drush pm-update` -> `composer update` upgrade path. More complicated use-cases can be handled in contrib.

Re #55:

I wonder if my current problem is caused by running updatedb too soon. I might have to do the whole thing again.

Nice. Thanks for capturing all that.

So, I think there was one thing lost in translation, but I think it might be immaterial, and only relevant in the documentation side of things:

🐳If you have a Composer-managed site (using a core template or otherwise), you run the risk of having unmanaged contrib extensions in your filesystem.

The main use case I see for this tool is that loose contract we've had with the community where "You do not have to use composer to manage your site unless you require an extension that requires it."

So, theres a large number of folks who are in the category of 🐮above, they've never used composer, they've used drush to manage their site and they're doing fine that way. And then... they need to install a module that does require composer.

So those users are on a 'composer ready tarball' maybe even 8.9.? when they find they need to "Convert"

In any case, all of the proposed solutions above also solve for this use case, so I dont think this alters any of the proposals. But I *do* think the tool should be geared towards a novice to composer, with super clear 'next steps' etc.

As a bonus, if this tool exists, we can point to it when/if we ever solve: https://www.drupal.org/project/drupal/issues/2494073 -> tell users hey, you have to use composer now. Sorry. Use this tool to get up to speed.

I am kind of -1 on having an "exotic extension registry". The main use-case for keeping this tool installed is to catch mistakes from users who do not fully understand who make a mistake after conversion (e.g. keep using drush dl / drush pm-update). To benefit from this protection, a naive user must:

1. Leave the plugin installed (need to decide how to phrase the "how to remove" documentation, so folks know whether to keep it or not)
2. Understand how to configure the exotic extension registry
3. Bother to configure the exotic extension registry

I think we should simplify further, and only notify about exotic extensions once.

Definitions:

1. Drupal site is not "Composer ready" - drupal/core-composer-scaffold not in composer.json
2. Drupal site is "Composer ready" but not "Composer managed" - using drupal/core-composer-scaffold, but no "drupal/*" contrib extensions in composer.json
3. Drupal site is "Composer managed" - at least one "drupal/*" contrib extension in composer.json.

Definition of a contrib extension:

1. Starts with "drupal/"
2. Is not "drupal/core"
3. Does not start with "drupal/core-"

With those definitions, then, we could:

1. Only warn about exotic extensions when doing the first conversion from "Composer ready" to "Composer managed".
2. Offer to convert sites from "Composer ready" to "Composer managed" on "composer update" / "composer require" (Probably as a follow-on issue, maybe Drupal 9+ only)

As an independent effort, we (ahem, I) could also make Drush dl / pm-update detect "Composer managed" sites and refuse to operate. This would only protect folks who upgrade their Drush, but it would be something.

I don't want Composer to stop people with Drush-managed extensions from using composer require. I might like to go the other way, and break drush dl for sites that have Composer-managed extensions.

We could actually do that without changing Drush 8. We'd just have to write a policy file to the site when we convert extensions to be Composer managed (or maybe even when we detect the use of composer require with a contrib module). This is probably not a good idea, though, as it is unnecessary (and therefore annoying) to folks who don't use Drush 8, and is perhaps too aggressive even for those who would benefit.

Maybe I'll just update Drush to warn the user whenever using dl / pm-update with a module whose composer.json has a non-empty require section.

The list of ones we've already found will be added to the extra when we do the conversion.

This is what I'm calling the exotic extension registry, after the terminology used in #62. I suppose it would work fine to do the interaction steps described in #67, and perhaps warn about exotic extensions the first time they are encountered only (and add them to a list in "extra" after the user is presented with their options).

Re @greg.1.anderson in #66:

Definition of a contrib extension:

1. Starts with "drupal/"
2. Is not "drupal/core"
3. Does not start with "drupal/core-"

Custom extensions might also define a composer.json where the namespace is "drupal" ("drupal/custom-module"), there's nothing which guarantees they won't. Honestly I've done that myself simply because every extension on D.o does so it kind of subconsciously sets a precedent that a Drupal extension has the namespace starting with Drupal, and because the class path would start with Drupal\custom-extension. Whether it's right or wrong is a different discussion, but my point is "drupal/" doesn't guarentee that a contrib project exists for that extension.

Maybe it's pointless to try and make assumptions about the status of an extension based on its name or org. We could get a pretty good notion of which projects are contrib extensions by looking for # Information added by Drupal.org packaging script in the info.yml site. Of course, there's no guarantee that a custom module might have copied that comment from a contrib module and just left it there.

It is still my thought that I'd rather make such heavy use of Composer APIs just to determine if an extension is contrib or not. Maybe we could do three checks:

1. Name starts with "drupal/"
2. .info.yml contains information from Drupal.org packaging
3. The url https://dropa.org/project/foo exists, where "foo" is the part of the name afer "drupal/"

No Composer APIs, relatively quick, determines whether the module is a contrib extension. Maybe this is good enough, for less code?

@AaronMcHale,

Custom extensions might also define a composer.json where the namespace is "drupal" ("drupal/custom-module"),

I'm not sure how this would work. Can you update a custom module with composer update drupal/custom-module?

@imclean The point here is that custom modules should remain in the repository, whereas contrib modules are removed and added to composer.json. So, for example, if a site had only two modules, drupal/ctools and drupal/my-custom-module, both in the modules directory, then ideally we would add drupal/ctools to composer.json (configured to install to modules/contrib) and move my-custom-module to modules/custom.

@greg.1.anderson, I understand, I'm just wondering why custom modules would define a composer.json file at all. Could it be used to manage dependencies even if it can't be installed via composer?

One option could be to provide some interactivity. For example, provide a list of discovered modules and ask to confirm any custom modules which couldn't be automatically detected.

@greg.1.anderson, I understand, I'm just wondering why custom modules would define a composer.json file at all. Could it be used to manage dependencies even if it can't be installed via composer?

One option could be to provide some interactivity. For example, provide a list of discovered modules and ask to confirm any custom modules which couldn't be automatically detected.

Well, yeah a custom module could define a composer.json if it had specific dependencies, that might be an architectural decision, the custom module might be added to the root composer.json by defining it as a path repository. This also applies (and is probably more useful) when using a custom developed Drupal profile.

Additionally, there might be cases where Drupal modules come from a remote repository that isn't Drupal.org, you can easily define Git repositories, or there might be a private Composer server in use.

All of these are technically valid use cases that probably exist in the wild, some of these I've worked with before.

adding "Automatic Updates Initiative" tag because we will probably need something like this for some site to use the new Automatic Updates because it will be composer based.