Create Build Tests For Composer and Drupal

Just some thoughts:

It turns out that you can make a packages.json file and specify it for composer create-project. This would let us do an in-place build.

So we'd make a packages.json file for all the packages in the core filesystem. Their repository types would be path. We'd do composer create-project --repository our/packages.json. Then we'd have our criteria for pass/fail.

Moving forward, using only a diff against the tarball isn't going to be enough, but for initial work it might suffice to prove that we got all the files we need.

Eventually we're going to have to run some tests in order to make sure everything is autoloaded and so forth. This might not work so well for composer projects that rearrange the file system a lot, because there are a lot of tests that make assumptions about the file system. For now we should run unit and kernel tests, while we're working on drupal-project-legacy. The goal of legacy is to end up with the same file system.

So:

• If the canonical subtree split file looks like a packages.json file we could use that. If not, add a tool to build one, either from the subtree split file or just from known locations.
• Apply patch.
• create-project in place with packages.json.
• Diff resulting project against repo.
• Run some tests.

This issue and #3031379: Add a new test type to do real update testing have very similar needs from the testing system, but unique special cases.

This issue addresses a testing scenario which we will need in the (hopefully near) future, while #3031379: Add a new test type to do real update testing addresses a current need.

Postponing this issue on that one, since we'll likely develop a new testing framework there which will be used here.

Just for reference, I made a test script based on #4 / #7 and, over time, have been trying to minimize the amount of file fixup and diff ignore files done to get a passing test.

See the test file, test.sh, which includes comments on the differences.

I don't think that we need / should compare the result of composer create-project against the tarball downloads. While that is an interesting question today, in the near future, the download tarballs will themselves be created by composer create-project, making it a somewhat pointless comparison. Even if you go for the double-accounting benefits of testing that the tarball actually did what it was supposed to, it would be a difficult thing to compare them, because the tarball will be the last packaged result, whereas composer create-project will have all of the changes from the latest patch. We want to test the later, not the former (with the exception of update testing).

The current interim DrupalCI test scripts do this:

 info "Installing recommended project composer template"
 composer --working-dir="${SOURCE_DIR}/composer/Template/RecommendedProject" config repositories.scaffold path '../../Plugin/Scaffold'
 composer --working-dir="${SOURCE_DIR}/composer/Template/RecommendedProject" install --no-suggest --no-progress --no-interaction
 info "Recommended project composer template installed successfully."
 assertScaffold "${SOURCE_DIR}/composer/Template/RecommendedProject/web"
 
 info "Installing legacy project composer template"
 composer --working-dir="${SOURCE_DIR}/composer/Template/LegacyProject" config repositories.scaffold path '../../Plugin/Scaffold'
 composer --working-dir="${SOURCE_DIR}/composer/Template/LegacyProject" install --no-suggest --no-progress --no-interaction
 info "Legacy project composer template installed successfully."
 assertScaffold "${SOURCE_DIR}/composer/Template/LegacyProject"

We should make something similar using phpunit tests, perhaps followed up by some tests to perhaps install Drupal, and then start up a web browser and see if it worked.

Regarding upgrade tests, I would imagine that we would want to rely on drupal.org to download an old version of Drupal to upgrade from. For the target site, though, we should be upgrading to the site-under test; Packagist, Drush etc. should not be involved.

I think the process would be something like:

1. Download and untar an old version of Drupal from drupal.org
2. Run site-install
3. Copy the site-under-test code over the unpackaged site
4. Run updatedb
5. Profit

The last step is the tricky one, as we'll need to detect whether anything went wrong with the upgrade. To do a really thorough upgrade test, we'd want to enable all of the core modules and add date to them and then check to see if they still worked. An MVP upgrade test could just check to see if the updates all ran without error, and the home page can still be fetched without a WSOD.

I think this issue should just be focused on composer create-project and the installation of a site. We should make a follow-up for the upgrade test.

I guess another question is, how many of those tests should run on every patch, and how many should run less often?

If we wanted to install Drush 8.x as a fixture, we could configure it to get update information from somewhere other than drupal.org. We might be able to get pm-update to think that there was a new release available from the current SUT. We'd probably have to package the SUT up as a tarball and put it on disk somewhere, and make release XML to point at it. A little complicated, perhaps, but possible.

But I guess we should discuss this on our follow-on issue.

As a reminder, when we add real tests, we'll want to update https://git.drupalcode.org/project/drupal/blob/8.8.x/core/tests/Drupal/T... to search for Build tests.

This seems a duplicate of #3082230: [meta] Convert some tests to Build tests.

New child issue: #3086644: LegacyProject composer templates wrongly reference 8.x + fix test coverage

Fix issue tags and update issue summary

New child: #3096044: Ensure quickstart feature works with drupal/core-recommended