The patches on #2951072: Provide Scheduler drush commands for Drush 9 modify composer.json and there have been recent enhancements to DrupalCI test procedures where modifying the composer.json via a patch being tested causes a recalculation of the module dependencies.

I think the tests on that issue are failing because the devel_generate and rules modules are loaded because they are included in our two .info.yml, but then removed because they are not specified in our composer.json file.

Comments

jonathan1055 created an issue. See original summary.

jonathan1055’s picture

Status: Active » Needs review
StatusFileSize
new495 bytes

To verify this, here is a patch which alters composer.json but only changes the text and does nothing else.

Status: Needs review » Needs work

The last submitted patch, 2: 2985051-1.dependencies_in_composer_json.patch, failed testing. View results

jonathan1055’s picture

In the console output https://dispatcher.drupalci.org/job/drupal8_contrib_patches/33118/console we get:

Adding testing (require-dev) dependencies.
Adding require-dev
sudo -u www-data /usr/local/bin/composer  require --no-interaction 'drupal/devel_generate:*' 'drupal/rules:*' --prefer-stable --no-progress --no-suggest --working-dir /var/www/html
./composer.json has been updated
Loading composer repositories with package information
Updating dependencies (including require-dev)
Package operations: 4 installs, 0 updates, 0 removals
  - Installing drupal/devel (1.2.0): Loading from cache
  - Installing drupal/typed_data (1.0.0-alpha1): Downloading (100%)
  - Installing drupal/rules (3.0.0-alpha4): Downloading (100%)

Then after the patch has been applied we get (with ... for omitted lines) :

----------------   Starting update_dependencies   ----------------
...
composer.json changed by patch: recalculating depenendices
...
Removing dev dependencies: sudo -u www-data /usr/local/bin/composer remove drupal/devel_generate --no-interaction --working-dir /var/www/html
...
Removing dev dependencies: sudo -u www-data /usr/local/bin/composer remove drupal/rules --no-interaction --working-dir /var/www/html
...
Removing project: sudo -u www-data /usr/local/bin/composer remove drupal/scheduler --no-interaction --working-dir /var/www/html

Then the Scheduler module is re-added, but with no dependencies

  - Installing drupal/scheduler (dev-ancillary-branch): Mirroring from /var/lib/drupalci/workdir/scheduler

So the question is: Do we have to repeat the dependency and test-dependency information in both .info.yml and composer.json? Or should the DrupalCI procedure have re-added the dependencies using what is specified in .info.yml?

jonathan1055’s picture

Status: Needs work » Needs review
StatusFileSize
new433 bytes

Patch to add drupal/devel as a test dependency in composer.json. The rules dependencies have not been changed, so rules tests will still fail.

Status: Needs review » Needs work

The last submitted patch, 5: 2985051-5.dependencies_in_composer_json.patch, failed testing. View results

jonathan1055’s picture

That's good the devel module is re-added:

Adding testing (require-dev) dependencies.
Composer Command: sudo -u www-data /usr/local/bin/composer require 'drupal/devel:>=1.0' --prefer-stable --no-progress --no-suggest --working-dir /var/www/html
Updating dependencies (including require-dev)
  - Installing drupal/devel (1.2.0): Loading from cache

and SchedulerDevelGenerateTest now passes.

jonathan1055’s picture

Status: Needs work » Needs review
StatusFileSize
new942 bytes

This patch adds new composer.json to the scheduler_rules_integration sub-module. Not sure it this is the correct way to go, as adding 'rules' into the require-dev list in the main composer.json would probably fix the problem. However, having a separate composer.json for the sub-module feels the right thing to do. Lots of guessing here, so this probably won't pass first time.

Status: Needs review » Needs work

The last submitted patch, 8: 2985051-8.dependencies_in_composer_json.patch, failed testing. View results

jonathan1055’s picture

Status: Needs work » Needs review
StatusFileSize
new966 bytes

Maybe it should be "drupal/rules" not "rules/rules" as per .info.yml

Status: Needs review » Needs work

The last submitted patch, 10: 2985051-10.dependencies_in_composer_json.patch, failed testing. View results

idebr’s picture

Status: Needs work » Needs review
StatusFileSize
new2.63 KB
new3.68 KB

According to the documentation page Add a composer.json file the dependency should be in the project's composer.json file:

If a module is a contributed module on drupal.org, has dependencies on other modules, and wishes to test changes to those dependencies using drupalci as part of development, they must have a composer.json that expresses those drupal module dependencies (drupalci can only detect changes to dependencies in patches within composer.json, not in .info/info.yml files)

In addition the dependency should be in the info file:

Regardless of whether or not a developer has a composer.json file, their drupal module dependencies must still be expressed in their .info.ymlfiles so that drupal can ensure that the correct modules are enabled.

Also, the composer.json file syntax should use 4 spaces as indentation: #2654894: Use an indent of 4 spaces for composer.json

jonathan1055’s picture

Thanks for the patch. Yes the tests do pass with your patch, but I knew that was going to happen if you add the test dependency to the main composer,json file. The critical thing, however, is that we already have the rules test dependency defined, in the sub-module /scheduler_rules_integration/scheduler_rules_integration.info.yml file. It is only required as a test dependency for that sub-module, so should not be in the main project scheduler.info.yml file or composer.json file. Otherwise you cannot test Scheduler locally if you dont have rules, or rules is not working (there is no full release yet). That is why we split out the rules integration code into a sub-module which can be enabled and/or tested separately.

I have an idea that the tests in #10 failed because there is not currently any /scheduler_rules_integration/composer.json file so the patch to this file did not trigger a re-evaluation of the dependencies in the patched file.

Thanks for the hint about using 4 spaces.

The last submitted patch, 10: 2985051-10.dependencies_in_composer_json.patch, failed testing. View results

  • jonathan1055 committed 98ac675 on 8.x-1.x authored by idebr
    Issue #2985051 by jonathan1055, idebr: Add plain composer.json to...
jonathan1055’s picture

StatusFileSize
new896 bytes

Now that the file /scheduler_rules_integration/composer.json exists we can test the dependencies properly.

jonathan1055’s picture

StatusFileSize
new978 bytes

Try again now that the committed composer.json syntax is correct.

Status: Needs review » Needs work

The last submitted patch, 18: 2985051-18.dependencies_in_composer_json.patch, failed testing. View results

jonathan1055’s picture

Status: Needs work » Needs review
StatusFileSize
new1.01 KB

Now adding require-dev to the rules file

Status: Needs review » Needs work

The last submitted patch, 20: 2985051-20.dependencies_in_composer_json.patch, failed testing. View results

jonathan1055’s picture

Status: Needs work » Needs review
Related issues: +#2692407: Test_dependencies are downloaded before applying patches, rather than after
StatusFileSize
new1009 bytes

OK, so it looks like idebr was right, and this is also confirmed by #2692407-34: Test_dependencies are downloaded before applying patches, rather than after.

When require-dev drupal/rules is added to the main composer.json I can still run local PHPunit non-rules tests if the Rules module is not available. This is good because it does not hinder non-rules development, which is what I was concerned about.

This patch does not add rules:rules to the test dependencies in .info.yml file.

jonathan1055’s picture

StatusFileSize
new1009 bytes

I must get a json syntax checker!

jonathan1055’s picture

StatusFileSize
new1018 bytes

OK, that works. Now to see if changing "drupal/devel" to "drupal/devel_generate" works, as this is what we have in .info.yml

  • jonathan1055 committed 3b3ee29 on 8.x-1.x
    Issue #2985051 by jonathan1055, idebr: Add require-dev devel_generate to...
  • jonathan1055 committed 696b138 on 8.x-1.x
    Issue #2985051 by jonathan1055, idebr: Add require rules and scheduler...
jonathan1055’s picture

StatusFileSize
new480 bytes

Having committed the minimum require-dev to the main module composer.json and minimum require to the sub-module, it will be useful to see if this is sufficient to re-load the dependencies after a dummy patch to the main composer.json. I expect it will not, and this test will fail, but useful to know for certain.

Status: Needs review » Needs work

The last submitted patch, 26: 2985051-26.dummy_change_to_composer_json.patch, failed testing. View results

jonathan1055’s picture

So the sub-modules actual dependencies are removed and not re-added, if the main module's composer.json is patched.

jonathan1055’s picture

As we do not want to add require-dev drupal/rules to the main module's composer file I think we may have to leave it here, and just accept that patches to the main module composer.json will cause the sub-module tests to fail. But that does seem very limiting.


edit: It is too much of a pain to have Rules tests fail on d.o. every time we patch composer.json - so I am going to add require-dev drupal/rules to the main module composer.json
jonathan1055’s picture

Status: Needs work » Needs review
StatusFileSize
new649 bytes

Here's the patch. Added the equivalent test dependency to .info.yml to keep them in sync.

  • jonathan1055 committed 8e0be71 on 8.x-1.x
    Issue #2985051 by jonathan1055, idebr: Add Drupal/Rules as test...
jonathan1055’s picture

Status: Needs review » Fixed

Fixed. Thanks @idebr for your help.

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.