See #1455614: Packaging script doesn't allow distributions to patch core or specify a git revision for the gory details. In short: if you define Drupal core in the drupal-org.make file it doesn't actually work as expected. After much discussion, the distribution maintainer community and myself decided that it's better to define core in a separate file if you want to specify patches, Git hash, etc. Basically, anything more fancy than the latest official release.

So, this issue is about fixing the validation code (both for drush make --drupal-org and drush verify-makefile ) to reflect these proposed changes:

A) Prevent people from defining the 'drupal' project in drupal-org.make itself.

B) Add reasonable validation for the drupal-org-core.make file. This should basically *only* allow defining the 'drupal' project.

It's not clear the best way to refactor the underlying code, since we now need basically two separate validation code paths. I'm also not sure the best way to trigger the different validation code paths via command line args and such. One possibly sane proposal is that if you just invoke drush verify-makefile with no arguments, it searches for the existence of drupal-org.make and drupal-org-core.make and validates each according to their own rules. But, it does seem useful to let people provide a filename themselves and therefore to specify which set of validation rules they want. Re-using --no-core seems a bit wonky. I'm totally open to other suggestions. ;)

Support from Acquia helps fund testing for Drupal Acquia logo

Comments

dww’s picture

Based on a fruitful IRC chat with jhedstrom, we came up with the following for the UI:

  • verify-makefile will have a new --verify option. It will default to 'contrib' if not specified (the current behavior), but you can also use --verify=core if you want to trigger validation of a drupal-org-core.make file.
  • verify-makefile will continue to take a filename argument, but it'll now be optional. If it's not there, we'll just look for drupal-org.make and/or drupal-org-core.make and verify each according to their own rules.
  • The --drupal-org option for make itself will now take a value, and default to 'contrib'. But, you can also use --drupal-org=core if you want it to build a drupal-org-core.make file and enforce the core-only validation. This is what the d.o packaging script will do.

So, mostly, things should be unchanged for distro maintainers. All the existing workflow, tools, and command args should be the same. However, there would be these new ways to incorporate drupal-org-core.make files into the mix and still be able to verify/build them as needed.

In terms of the code itself, it looks like about 1/2 of the transformers in drupalorg_drush_make_validate_info() can be shared, and the other half need to be forked. Seemed easier to just have 3 arrays of transformers (core, contrib, common or something) and just make it the problem of drupalorg_drush_make_validate_info() to pick the right set of transformers to use, instead of trying to teach the transformers themselves about the different validation rules. I'll see how I feel about this once I'm deep into the coding, but for now, that seems right.

dww’s picture

Status: Active » Needs work
FileSize
7.56 KB

Work in progress. Instead of using a separate --verify option for verify-makefile, I'm currently just always using --drupal-org for this. Keeps the code more simple. Probably I should complicate the code a bit to keep things a bit more sane for distro maintainers, but for now, this is basically working.

Got all the filename stuff sorted out as described in #1 for verify-makefile. If there's an arg, we use it. Otherwise, we search for drupal-org.make and drupal-org-core.make and if either/both exists, we try to verify them with the appropriate rules.

The primary validation differences between the 'core' and 'contrib' rules are that if you use 'core' you *only* want the 'drupal' project and you want no libraries at all. If you use the default 'contrib' rules, the only thing is you *don't* want to define 'drupal'. None of the current validation transformers are operating on the keys for the projects array at all. So, instead of trying to wedge this into a transformer, I'm just hard-coding it as we're iterating over the projects and libraries. Seems cleaner this way, but I'm open to suggestions.

There might still be a few details of different transformers we want or don't want in the two different cases, but I think this is pretty close to all we need.

There's one core-only test that's currently failing (which is expected, given the changes in validation). So, I need to update that test to pass --drupal-org=core. Probably would be good to add some other tests here, too.

dww’s picture

Issue tags: +Needs tests
FileSize
8.88 KB

Okay, this cleans up the broken test. There were a few bugs in the test already:

- Although this test case and .make file was called 'core-only', it actually defined both core and view (wtf?)
- drupalorgDrushMakeTestCase::runMakefileTest() was using array_merge() to join the default options with whatever each test specified. However, that prevents you from actually overriding those defaults. Since we were setting --drupal-org by default, we couldn't set it to 'core' for specific tests. We now use + instead of array_merge() to get the expected behavior where the defaults are used only if the test doesn't already define one of the options.

Once those were resolved, adding --drupal-org=core for this particular test got it passing again as expected.

However, we should also add some tests we expect to fail now like a .makefile that defines both core and contrib and then try to build and verify it multiple times with --drupal-org set to 'core' or 'contrib' -- it should fail in all cases.

And, I still need to figure out if there are any transformers that should be different in the two cases and add/fix those... But, it's awfully late now, so I'm going to get some sleep, first. ;)

dww’s picture

Okay, here's a patch with just the test fixes from #3 plus 6 new tests as described there. If you apply this and run it, all 6 tests fail. If you then apply #2, they all pass. So, removing 'Needs tests' since I think we're good on that front now. However, still at needs work to cleanup the transformer lists in case there are any other differences between the validation rules in the two cases. Stay tuned.

dww’s picture

Nearly there. This is a combined patch with the tests from #4, the code from #2, and additional code for the validators... Now I just need to fix the version magic in DrushMakeDo_DownloadGit to handle core git labels. Stay tuned.

dww’s picture

Status: Needs work » Needs review
FileSize
19.3 KB

Okay, this fixes DrushMakeDo_DownloadGit so that we're properly setting 'version' in case you define core via a Git branch or tag (which ensures we get the right release info into the metadata files). I think this is complete now.

The only other special-case validation is that we prevent a 'subdir' project-level attribute for core. There's no test for this, yet. Not sure it's really that important. ;) I *think* subdir is just ignored when building core, anyway.

jhedstrom’s picture

Status: Needs review » Reviewed & tested by the community

This looks good to me. I like the approach of keeping the behavior basically the same, but magically adding support for verifying the core make file.

dww’s picture

Status: Reviewed & tested by the community » Fixed

Thanks for the review and the feedback on the approach, jhedstrom!

Committed, pushed, and deployed.

Status: Fixed » Closed (fixed)

Automatically closed -- issue fixed for 2 weeks with no activity.