D10: Run crawler script to update generated.json

I am a little confused by the output in statistics

specifically this part:

  -  (of which ~ 388 generated recommendations overridden by curated recommendations)
  + (of which ~ 311 generated recommendations overridden by curated recommendations)

Did we lose 77 curated recommendations in this process somehow?

❌ pipelines failing!

Array
(
    [package] => drupal/auto_entitylabel
    [constraint] => ^3
    [replaces] => Array
        (
            [name] => auto_nodetitle
        )
    [vetted] => 
)
conflicts with
Array
(
    [package] => drupal/auto_entitylabel
    [constraint] => ^3.0
    [replaces] => Array
        (
            [name] => auto_entitylabel
        )
    [vetted] => 
)

I think it is because ^3 vs ^3.0. Seeing what happens if I change that in curated.json

I think there are probably quite a few of these, another failure:

(
    [package] => drupal/chosen
    [constraint] => ^4
    [replaces] => Array
        (
            [name] => chosen_ajax
        )
    [vetted] => 
)
conflicts with
Array
(
    [package] => drupal/chosen
    [constraint] => ^4.0
    [replaces] => Array
        (
            [name] => chosen
        )
    [vetted] => 
)

I dont want to spam pipelines so maybe I can try to identify which ones differ and fix them all in one go .

I figured out i could run

php scripts/test-unvetted-modules-composer-installability.sh recommendations.json /tmp/test-general-installability-unvetted 10.2.3

to test this locally (to avoid the aforementioned spamming of pipelines 😄). However...i think this is exposing a larger issue. Some of the generated recommendations, for example:

        {
            "package": "drupal/node_revision_delete",
            "constraint": "^2.0",
            "replaces": {
                "name": "node_revision_delete"
            },
            "vetted": false
        },

conflict heavily with the curated.json value, which is:

            "package": "drupal/node_revision_delete",
            "constraint": "^1",
            "install": [
                "node_revision_delete"
            ],
            "replaces": {
                "name": "node_revision_restrict"
            },
            "vetted": false
        },

even though it is vetted:false in curated.json it still feels like something else needs to happen here. Another example is rabbit hole:

Array
(
    [package] => drupal/rabbit_hole
    [constraint] => ^1
    [replaces] => Array
        (
            [name] => rh_file
        )

    [install] => Array
        (
            [0] => rh_file
        )

    [vetted] =>
)
conflicts with
Array
(
    [package] => drupal/rabbit_hole
    [constraint] => ^2.0
    [replaces] => Array
        (
            [name] => rabbit_hole
        )

    [vetted] =>
)

I'm not sure i feel comfortable updating these in curated.json without actually testing them myself locally, but I guess if its not marked as "vetted" that is ok? Updated the MR with updates to the failing packages and an added conditional in the test-unvetted-modules-composer-installability.sh script be more lenient with version checks to start 😊.

I thought the constraints in curated.json would have superseded those in generated.json, but this appears to be throwing an error. My changes have also broken things quite substantially in pipelines 😬 sooo im rethinking this at the moment

Here are the current issues:

  Problem 1
    - Root composer.json requires drupal/ace_editor ^2.0 -> satisfiable by drupal/ace_editor[2.0.0-alpha1, 2.0.0-beta1, 2.0.0-beta2, 2.0.x-dev].
    - drupal/ace_editor[2.0.0-alpha1, ..., 2.0.x-dev] require npm-asset/ace-builds ~1.0 -> could not be found in any version, there may be a typo in the package name.

This one will get addressed with https://www.drupal.org/project/acquia_migrate/issues/3417209.

  Problem 2
    - Root composer.json requires drupal/agls ^1.2 -> satisfiable by drupal/agls[1.2.0, 1.x-dev].
    - drupal/agls[1.2.0, ..., 1.x-dev] require drupal/metatag ^1.0 -> found drupal/metatag[dev-1.x, 1.0.0-beta1, ..., 1.x-dev (alias of dev-1.x)] but it conflicts with your root composer.json require (^2).

I would argue that the recommendation for metatag @ version ^2 supersedes drupal/agls (based on installation usage). So drupal/agls should be moved to curated.json with a note stating that it needs to be manually installed due to the need to downgrade metatag.

  Problem 3
    - Root composer.json requires drupal/codemirror_field ^2.0 -> satisfiable by drupal/codemirror_field[2.0.1].
    - drupal/codemirror_field 2.0.1 requires codemirror/codemirror5 ^5 -> could not be found in any version, there may be a typo in the package name.

This module requires manual installation due to needing to alter the repositories section of composer.json by hand. Should update this in curated.json

  Problem 4
    - Root composer.json requires drupal/commerce_realex ^3.0 -> satisfiable by drupal/commerce_realex[3.0.0, 3.0.x-dev].
    - drupal/commerce_realex[3.0.0, ..., 3.0.x-dev] require annertech/rxp-js 1.3.1.21 -> could not be found in any version, there may be a typo in the package name.

The recommendation from the previous version of recommendations.json still applies to this module.

  Problem 5
    - Root composer.json requires drupal/contextly ^3.0 -> satisfiable by drupal/contextly[3.0.0].
    - drupal/contextly 3.0.0 requires contextly/contextly-kit 5.0.10 -> could not be found in any version, there may be a typo in the package name.

This recommendation still applies to the current module.

  Problem 6
    - Root composer.json requires drupal/feeds_ftp_fetcher ^2.0 -> satisfiable by drupal/feeds_ftp_fetcher[2.0.0-alpha1, 2.x-dev].
    - drupal/feeds_ftp_fetcher[2.0.0-alpha1, ..., 2.x-dev] require ext-ftp * -> it is missing from your system. Install or enable PHP's ftp extension.

This seems to be more an issue with the CI - i think we need to enable a new extension for this to install. Locally I am able to install it.

  Problem 7
    - Root composer.json requires drupal/lightgallery ^1.4 -> satisfiable by drupal/lightgallery[1.4.0, 1.x-dev].
    - drupal/lightgallery[1.4.0, ..., 1.x-dev] require sachinchoolur/lightgallery ^1.2.21 -> could not be found in any version, there may be a typo in the package name.

The same recommendation apples from the previous version.

  Problem 8
    - Root composer.json requires drupal/monster_menus ^9.1 -> satisfiable by drupal/monster_menus[9.1.0-rc1, 9.1.0-rc2, 9.1.0-rc3, 9.1.0].
    - drupal/monster_menus[9.1.0-rc1, ..., 9.1.0] require drupal/core ^9.4 || 10 -> found drupal/core[9.4.0-alpha1, ..., 9.5.x-dev, 10.0.0] but it conflicts with your root composer.json require (10.2.3).

This module might be ok to use (haven't tested) but cannot be installed due to a typo in its composer.json file, here. Based on the module .info file this should be compatible with ^10 which would install on 10.2.3, but this requirement in composer.json prevents installation.

  Problem 9
    - Root composer.json requires drupal/signaturefield ^1.1 -> satisfiable by drupal/signaturefield[1.1.0, 1.x-dev].
    - drupal/signaturefield[1.1.0, ..., 1.x-dev] require npm-asset/signature_pad ^2.3 -> could not be found in any version, there may be a typo in the package name.

The same recommendation applies.

  Problem 10
    - Root composer.json requires drupal/smartmenus ^2.0 -> satisfiable by drupal/smartmenus[2.0.1-beta1, 2.0.1-beta2, 2.0.x-dev].
    - drupal/smartmenus[2.0.1-beta1, ..., 2.0.x-dev] require drmonty/smartmenus 1.1.1 -> found drmonty/smartmenus[dev-master, v0.9.6, v0.9.7, 1.0.0, 1.0.1, 1.1.0] but it does not match the constraint.

This module requires manual installation. See here.

  Problem 11
    - Root composer.json requires drupal/swagger_ui_formatter ^4.0 -> satisfiable by drupal/swagger_ui_formatter[4.0.0].
    - drupal/swagger_ui_formatter 4.0.0 requires php ~8.1.0 -> your php version (8.2.15) does not satisfy that requirement.

This module is not yet compatible with PHP 8.2. It also appears the issue queue is disabled? Odd.

  Problem 12
    - drupal/drupal[8.4.0-alpha1, ..., 8.7.x-dev] require composer/installers ^1.0.24 -> found composer/installers[v1.0.24, ..., 1.x-dev] but it conflicts with your root composer.json require (^2.0).
    - drupal/at_tools[3.6.0, ..., 3.x-dev] require drupal/adaptivetheme ^4.0 -> satisfiable by drupal/adaptivetheme[4.0.0, 4.1.0, 4.x-dev].
    - drupal/adaptivetheme[4.0.0, ..., 4.x-dev] require drupal/core ^8.8 || ^9 -> satisfiable by drupal/drupal[8.4.8, 8.5.15, 8.6.18, 8.7.14].
    - Root composer.json requires drupal/at_tools ^3.6 -> satisfiable by drupal/at_tools[3.6.0, 3.x-dev].

I am a little confused on this one. The project composer.json file says it needs drupal/adaptivetheme: ^8.4 but i dont see that as a realease/tag.. Maybe this tracks to 8.x-4.x? If so, then the issue is that 4.x of drupal/adaptivetheme is incompatible with Drupal 10.. It also appears that at_tools has been abandoned and now the work is being done in at_tool

I made this changes locally and then saw this error:

  Problem 1
    - drupal/drupal[8.4.0-alpha1, ..., 8.7.x-dev] require composer/installers ^1.0.24 -> found composer/installers[v1.0.24, ..., 1.x-dev] but it conflicts with your root composer.json require (^2.0).
    - drupal/basic_cart 7.0.0 requires drupal/easy_install * -> satisfiable by drupal/easy_install[dev-1.x, dev-3.x, dev-4.x, dev-5.x, dev-6.x, dev-7.x, dev-8.x, dev-9.x, dev-10.x, 1.0.0, 1.x-dev, 2.0.0-alpha1, 3.x-dev, 4.0.0-alpha2, 4.x-dev, 5.0.0-alpha3, 5.x-dev, 6.0.0-alpha4, 6.x-dev, 7.0.0-alpha5, 7.x-dev, 8.0.0-alpha6, 8.x-dev, 9.0.0-rc1, 9.x-dev, 10.0.0, ..., 10.x-dev].
    - drupal/easy_install[dev-10.x, 10.3.0, ..., 10.x-dev] require drupal/core ^8 || ^9 -> satisfiable by drupal/drupal[8.4.0-alpha1, ..., 8.7.x-dev].
    - drupal/easy_install[dev-1.x, dev-3.x, dev-4.x, dev-5.x, dev-6.x, dev-7.x, dev-8.x, dev-9.x, 1.0.0, ..., 1.x-dev, 2.0.0-alpha1, 3.x-dev, 4.0.0-alpha2, ..., 4.x-dev, 5.0.0-alpha3, ..., 5.x-dev, 6.0.0-alpha4, ..., 6.x-dev, 7.0.0-alpha5, ..., 7.x-dev, 8.0.0-alpha6, ..., 8.x-dev, 9.0.0-rc1, ..., 9.x-dev, 10.0.0, ..., 10.2.0] require drupal/core ~8.0 -> satisfiable by drupal/drupal[8.4.0-alpha1, ..., 8.7.x-dev].
    - Root composer.json requires drupal/basic_cart ^7.0 -> satisfiable by drupal/basic_cart[7.0.0].

This module does not appear to be installable right now due to an issue with Easy Install being incompatible with Drupal 10. Easy install needs to be patched - see https://www.drupal.org/project/easy_install/issues/3297022.

Then I hit:

  Problem 1
    - drupal/block_country dev-1.x requires drupal/ip2country ^1.9 -> found drupal/ip2country[dev-1.x, 1.9.0, 1.10.0, 1.11.0, 1.x-dev (alias of dev-1.x)] but it conflicts with your root composer.json require (^2).
    - drupal/block_country 1.x-dev is an alias of drupal/block_country dev-1.x and thus requires it to be installed too.
    - Root composer.json requires drupal/block_country 1.x-dev -> satisfiable by drupal/block_country[1.x-dev (alias of dev-1.x)].

This module does not have a stable release. Stopping for now...

Currently we have 171 generated recommendations. After this there are 1423 generated recommendations.

I realized this morning that what I was doing in my previous efforts was just re-vetting all those modules which failed to install cleanly...which is what we want to do anyway but there are probably higher priorities 😄. that being said, i think i should break some of my comments in #7 into tickets as the work (or groundwork) is already done now.

New idea. What if we just omit all the d10-install-breaking-modules from generated.json? This would mean we still get an additional ~1,200 modules which install successfully on D10 and it would unblock this issue, which relies on basically vetting every module which fails to cleanly install via composer.

Locally I reset things back to recommendations-10, re-ran make -B and cofirmed the mass update in generated.json. At this point my statistics showed:

$ cat ./statistics.txt                                                                       
410 recommendations (curated)
  (of which 364 contrib recommendations)
1480 recommendations (generated)
1790 recommendations (total)
  (of which 89 vetted recommendations)
    (destination: targeting 73 composer packages)
    (destination: targeting 121 Drupal modules)
    (source: migrating from 231 Drupal modules)
    (source: obsoleting 63 Drupal modules)
  (of which 63 obsolete recommendations)
  (of which 3 patched recommendations)
  (of which ~ 310 generated recommendations overridden by curated recommendations)
41 unique patches, or 13.7 (avg) per patched recommendation for a total of 41 (36 max)
  (of which 3 patches are not migration-specific, but necessary bugfixes)

Then i added the modules i knew from previous efforts to be breaking to a new file, iterated over it and removed those from generated.json. Then re-ran the CI process locally to make sure things worked. Whenever there was a failure, i removed it from generated.json and tried again. I attached the list of packages which caused failure. The result appears to be a passing CI pipeline now 🎉. I will leave it up for discussion what other implications this has and if it is a good idea or not 😬

I believe it is an improvement 🤞 because the majority of these modules should work on Drupal 10 (considering they have a D10 compatible release tagged). So even though we haven't tested them or vetted migration paths, i believe it makes things easier for the end user because they will come preinstalled (but not enabled) if they had them on Drupal 7.

😁 i did! however I am not sure we are talking about the same issue. I encountered 2 types of issues when working on this:

1. Issues where version constraints differed between curated.json and generated.json -- as you described in your previous comment, these were mostly submodules and easy to fix by manually changing the version.
2. Modules which get picked up by generated.json but are not installable via composer

The second issue above is where this is stuck, none of these modules - https://www.drupal.org/files/issues/2024-02-10/d10-composer-install-brea... - were able to be added to a D10 project via composer, cleanly. For example drupal/field_properties:

Your requirements could not be resolved to an installable set of packages.

  Problem 1
    - Root composer.json requires drupal/field_properties ^1.0@alpha -> satisfiable by drupal/field_properties[1.0.0-alpha2].
    - drupal/field_properties 1.0.0-alpha2 requires drupal/core ^8 -> found drupal/core[8.0.0-beta6, ..., 8.9.x-dev] but the package is fixed to 10.2.3 (lock file version) by a partial update and that version does not match. Make sure you list it as an argument for the update command.

or smartmenus:

[6:53:43] ~/Sites/d7-upgrades/the18-d10 [main] $ lando composer require 'drupal/smartmenus:^2.0@beta'
./composer.json has been updated
Running composer update drupal/smartmenus
Gathering patches for root package.
Loading composer repositories with package information
Updating dependencies
Your requirements could not be resolved to an installable set of packages.

  Problem 1
    - Root composer.json requires drupal/smartmenus ^2.0@beta -> satisfiable by drupal/smartmenus[2.0.1-beta1, 2.0.1-beta2].
    - drupal/smartmenus[2.0.1-beta1, ..., 2.0.1-beta2] require drmonty/smartmenus 1.1.1 -> found drmonty/smartmenus[dev-master, v0.9.6, v0.9.7, 1.0.0, 1.0.1, 1.1.0] but it does not match the constraint.

So for these...i feel like we would have to vet each on against D10 and provide a curated recommendation in order for this to be complete.

To be honest a lot of these packages look as though they are in alpha/beta and dont have a stable release to begin with...which makes me wonder why the crawler script was even picking them up because it should only get D10 packages when:

 * For the given project, try to derive a D7-D10 migration recommendation.
 *
 * Projects are ignored if:
 *
 * - Their owner has declared that it has been replaced.
 * - Their status is anything other than receiving maintenance fixes only or
 *   under active development (e.g. not obsolete).
 * - They do not have a Drupal 7 release.
 * - They do not have a Drupal 10 release.

the above snippet was taken from crawl.php

made an MR to show i am still noodling on this 😬

I re-ran the make -B command, then my process was as follows:

1. Run sh ./scripts/test-unvetted-modules-composer-installability.sh recommendation.json /tmp/test-general-installability-unvetted 10.2.3
2. Go to new directory and run composer install --dry-run
3. Move each failure to curated.json with a note as to why it fails to install
4. Run composer remove <package> then re-run composer install --dry-run

Basically just worked through all the various failures during composer installation, went to the projects to review why it failed and then updated the note with the findings. Then my pipelines jobs started to fail due to PHP 8.2 incompatibility - so I made a few more updates to address that after changing versions and testing again locally

I think i covered all issues, but LMK if i missed something 😎

I also think that I found a bug in the crawler that was stopping crawling ~400 pages in, when there are over 650 pages of results to iterate through. Probably best to keep that in the original parent issue and break out another update once possible...assuming this one is OK.

Made some updates 🥹 - we also had a pipelines failure due to an issue getting resolved - https://www.drupal.org/project/monster_menus/issues/3422059. So the system is working! 🎉

By "original parent issue" you mean #3414727: Refine the crawler that generates `generated.json` to support full Composer constraint expressions?

yep! I have found a lot of "sticking points" in the current crawler and I think there could be a substantial amount of projects still missing from generated.json. I suppose I could keep going in this issue with an update-generated-3 branch ... but since it might change the functions of the crawler i thought perhaps better off in the parent.

I could also change the crawler locally but only commit the updated JSON files (not the script updates)