Problem/Motivation
Follow-up to #2677532: Move drupal_check_incompatibility() functionality to a new Dependency class and Version component
From @mixologic:
Just to throw this out there again, I had to redo the version parsing when I built the composer facade, and built a test for it so that it would handle *all* of the oddball edge cases that people have, over the years, put into their info and info.yml files that we see on drupal.org.
http://cgit.drupalcode.org/project_composer/tree/project_composer.module...
The test I wrote for that is here: http://cgit.drupalcode.org/project_composer/tree/project_composer.test#n54
Perhaps some of those cases can be adapted into this class?
Im not sure if we do or do not want to introduce things like adding the ~ and ^ operator or not at this juncture. It should still be BC compatible, just with expanded capability.
🚨🚨🚨
It seems like there are major risks for getting this wrong. The one that comes to mind is deployment scripts that enable modules that all of sudden start to not enable modules or enable modules that previously weren't being enabled by the scripts(because they didn't meet dependency requirements before this change but were left in scripts)
🚨🚨🚨
Proposed resolution
Pull the test into core.
Pull in the changes to make the tests pass.
Remaining tasks
User interface changes
API changes
Data model changes
| Comment | File | Size | Author |
|---|---|---|---|
| #40 | interdiff-24-40.txt | 2.04 KB | tedbow |
| #40 | 3004459-40-composer-format-directly.patch | 21.22 KB | tedbow |
| #34 | 3004459-33-with_tests_3066448-8.patch | 22.85 KB | tedbow |
| #31 | 3004459-31-more-test-cases.patch | 16.77 KB | tedbow |
| #31 | 3004459-31-more-test-cases-test-only.patch | 4.81 KB | tedbow |
Comments
Comment #2
MixologicIm going to move the parent issue to the new meta.
Comment #4
larowlanFolds in and includes #3047898: project_composer_parse_dependency() behaviour for <=8.x-1.x is not consistent with composer/semver
Comment #6
larowlanComment #7
larowlanComment #9
larowlanRemove left-over case from debugging
Comment #10
kim.pepperLooks great! Just the code sniffer issues from what I can see.
Comment #11
kim.pepperFixes code sniff issues.
Comment #12
lendudeLooking good already, but might just be me, but the whole flow of the code is a little hard to follow right now. Little too many nested if's and switch stuff going on here in one big method.
So mostly some observations that might make this a little easier to follow.
as 'an' array?
the $p_ prefix for these variables is unclear
Can we turn this into a proper english paragraph?
Can we turn this into an early return instead of one giant nested if(). And update the comment to reflect that?
didnt => didn't
This if() always leads to the return at the end of it. Can we split this logic into a separate method and just make this if return the result of that? Might make this whole thing a little easier to read.
The same as for the 'one constraint' logic, can this just be split into a separate method or something?
edit: fixed some dreditor messes I clicked myself into
Comment #13
lendudeAlso, we seem to be missing test coverage for this?
Comment #14
larowlanAddresses #12 and #13 most of these are as found in project_composer, so tidied up a bit and refactored to get rid of else/elseif - it feels cleaner.
Also made the method public as I found I need it in the BC layer in #3005229: Provide optional support for using composer.json for dependency metadata
Comment #15
phenaproxima@larowlan wrote an amazing patch, over in #3005229: Provide optional support for using composer.json for dependency metadata, to make composer.json the canonical source of module metadata.🤓😲It includes the code from this issue, so this is a blocker for that one.
Comment #16
phenaproximaBrief, shallow review. The actual logic is way too complicated for me to parse, especially this close to bedtime.
Misspelled; should be "compatibility". Also, can the doc comment provide an example of what the value of this property would be?
Nit: "composer" should be "Composer". This doc comment would also benefit from an example, I think.
This could use a comment.
Can this comment expand on what the return value might look like?
Thank you for renaming this variable. :)
And this one!
Nit: Should be "Composer-compatible".
As elsewhere, Composer is a proper name, so it should be capitalized. I'll stop mentioning it now. :)
"op" seems to suggest "operation", as opposed to "operator". So we should probably just say "operator".
This comment can be improved. :)
This expression looks like it's repeated three times in the same method. Maybe it should be a variable?
"constraint" :)
We should use === here.
Also here.
And here, as well as the rest of the method.
There appear to be superfluous parentheses here.
Comment #17
mglamanI see one thing missing from the test coverage: I feel like I have seen constraints as ^2@beta or ^2@alpha.
Found some examples in a project:
We didn't mark them this way, Composer did after
composer require. This is the root project's composer.json and something Drupal won't grok. But I'm assuming these should be added to the test coverage.Comment #18
alexpottDuplicates
Because there's no corollary of this in the composer code I think this might lead to interesting abilities for the composer driven isCompatible().
So if I add
to \Drupal\Tests\Component\Version\ConstraintTest::providerIsCompatible() - HEAD passes... with this patch it fails. Also what is super interesting is last test fails with the patch and I expected only the middle two to fail... huh.
Comment #19
larowlanIn info files? Because the current logic is taken from project_composer which is part of the d.o composer facade which translates info files to composer.json and was based on Mixologic querying all the stuff seen in existing modules
Comment #20
mglaman@larowlan no, not from info files, these are in composer.json. I assumed we needed Drupal core to grok the
@STABILITYtags. Those values are returned from the Composer facade as far as I know.Comment #21
larowlanNo its the other way here mate, we're providing the composer facade in core so that info files are cast to composer.json in a BC layer.
Fixes #18 and #16
Comment #22
mglaman😄cool, I just wanted to check
Comment #23
lendudeAre we doing return type hinting now? And if so, why just these two methods?
Comment #24
larowlanPhpstorm is too smart for it's own good, removed those typehints
Comment #25
lendude@larowlan thanks! Looks great to me now, all feedback has been addressed.
Comment #26
alexpottSo if I add back
'common_test (=8.x2.x, >=2.4)'- is that really correct? It seems we've changed the meaning of(=8.x2.x, >=2.4)from having to satisfy both constraints to be being an OR. Not sure that's right.This doesn't look very broken.
Comment #28
tedbowI see that @larowlan made this method public in #14
but since that issue is postponed on 2 other issue couldn't we just make the method public then? This class is marked as internal so it seems like we are creating an API we don't need too. What if in that issue we want to refactor and ComposerHelper::convertFromDrupalConstraintFormat() as so don't need the current method at all. We couldn't remove it because it is public.
I removed this change and the test still pass so the comma is still enforcing an AND. I don't think we need any changes to VersionTest in this issue.
This all points to a lack of test coverage for the current functionality.
All the new cases pass with any changes to
\Drupal\Component\Version\Constraintin this issue.It seems weird to adding new test coverage for existing functionality and changing how that same functionality works under the hood in the same issue. Right now I don't think this issue has any test-only patches so the only way we can be sure the new test cases aren't actually showing new/changed functionality is if people have been running these locally without the
Constraintchanges.I am uploading a test-only patch.
Comment #29
tedbowAdding those test cases to this patch
Would that be good idea because I feel like
\Drupal\Tests\Component\Version\ConstraintTest::providerIsCompatible()is not very comprehensive right now. We are already adding test coverage for existing functionality in this issue. see #28.3Also since this issue does change functionality in that it fixes #2641658 are we sure it doesn't change other functionality that we don't want? If had better test coverage it would seem safer.
Comment #31
tedbowThis is the problem.
This is not how the current behavior works although it seem logical. Even with an "=" it uses AND.
This the only test case that has an "=" and multiple constraints.
The reason that this passes in the test-only patch in #28 is because
2.4-beta3passes for both constraints so it works whether an AND or OR operator is used.I will add a test case for this.
This assumes only 1 or 2 constraints but currently we support more than 2, always with AND.
We even document this https://www.drupal.org/docs/7/creating-custom-modules/writing-module-inf...
dependencies[] = exampleapi (>1.0, <=3.2, !=3.0)(I tested and works on 8.x too)
I will add a test to prove this fails and pass with changes.
testGetComposerConstraint()so we can patches with exact changes toConstraintTestwith and without changes to Constraint this will show 3 fails with changes.I think we should make test only issue to harden are existing tests for this
Comment #33
tedbowI opened #3066448: Harden test coverage for dependency Constraint class
In addition the problems mentioned in #31 that also shows that we currently support '<>' for not equals which this patch does not.
I am uploading at patch with the changes to
\Drupal\Component\Version\Constraintfrom this issue and the changes to\Drupal\Tests\Component\Version\ConstraintTestfrom the other issueWill set to Needs Review but then I think should postpone this issue. Hopefully we can get the other issue committed soonish since it is only test changes.
Comment #34
tedbowwhoops forgot to rebase my branch 😢
Comment #36
tedbowSo there were 30 test failures plus some errors. Not as bad as seems because a lot of the test cases fail for the same reasons. Still think we should postpone and finish #3066448: Harden test coverage for dependency Constraint class so we can make this change with confidence that we aren't breaking current sites.
It seems like there are major risks for getting this wrong. The one that comes to mind is deployment scripts that enable modules that all of sudden start to not enable modules or enable modules that previously weren't being enabled by the scripts(because they didn't meet dependency requirements before this change but were left in scripts)
Comment #37
wim leersMoved this into the issue summary, because it's worth drawing attention to:
Also, raising priority to based on it.
Comment #38
xjmComment #39
wim leers<>: Wow, excellent finds!All caught up now. @tedbow already fixed most things. And he extracted the test coverage expansion/hardening to a separate issue to ensure this issue only is concerned with what's described in the issue title.
AFAICT the primary thing to still figure out — after #3066448: Harden test coverage for dependency Constraint class lands — is #26.3.
Comment #40
tedbowGot thinking about another potential problem with this approach.
Would
\Drupal\Component\Version\Constraint::getComposerConstraint()actually just pass a composer version string on without changing it? In some cases, yesI think this introduces a problem because then it just passes it to
Semver::satisfies(). I think some new to drupal devs would assume our constraints work with the composer format in .info.yml files and try it. If they happen to try strings that work then they would think this is the way it is suppose to work.I think this will be more complicate if we do #3005229: Provide optional support for using composer.json for dependency metadata also(I think you could do that issue without this one) because I think devs would more likely think you can use composer strings in our info file.
So if we start to have devs using composer strings in the info file then the upgrade proposed in #3005229's
UpgradeInfoFilesCommand.phpbecomes more complicate. Right now it assumes it is always drupal format. Also\Drupal\Component\Version\Constraint::getComposerConstraint()actually transform some composer strings and not others.Here is test to prove the problem. It shows how composer string pass tests. interdiff against #24
Comment #41
tedbowSo here is example of using a composer version constraint. I think people will start to do this.
Do we actually want this to happen? In some cases you could use composer directly and this could be very confusing because I think not in all cases.
At least we don't have test coverage for it.
setting as needs work to figure that out but also it should then be postponed on #3066448: Harden test coverage for dependency Constraint class
Also needs a issue summary update because there is not state reason why we are doing this and what the benefit will be.
#3005229: Provide optional support for using composer.json for dependency metadata is postponed on this issue(UPDATE fixed my type) but I am not sure it needs to be.
Comment #42
wim leers#40: isn't this contradicting what #2313917-92: Core version key in module's .info.yml doesn't respect core semantic versioning is doing? Then again, #40 is 7 days old, #2313917-92 is 1 day old. Does this mean you've changed your mind about #40?
Comment #43
tedbow@Wim Leers re #42
No, right now approach in #2313917-92: Core version key in module's .info.yml doesn't respect core semantic versioning only concerns the
core:key in .info.yml files. It uses\Composer\Semver\Semver::satisfiespretty much directly so it won't support the Drupal constraint strings(except for any that would be the exact same as strings that\Composer\Semver\Semver::satisfieswould understand)This current issue from my understanding only deals with the
\Drupal\Component\Version\Constraintclass which only affects thedependencies:key in .info.yml files. It doesn't actually intentionally make a change so that you can use composer strings independencies:. My point in #40 and #41 is that the way\Drupal\Component\Version\Constraint::parseToComposer()currently works it seems will effectively let you start to use some but not all composer strings.If in #42 contradicting meant that if approach in #2313917-92: Core version key in module's .info.yml doesn't respect core semantic versioning was taken the
core:key would support Composer dependency format anddependencies:would support our current Drupal specific format then yes. But we could discuss that on #2313917I asked for an Issue Summary update in #41 because I don't think it is clear why we are doing this change and what the benefit is.
#3005229: Provide optional support for using composer.json for dependency metadata is postponed on this issue but I am not sure needs to be.(I mistyped in the last line of #41 fixed). see my comment which I will clarify after this.
Also #3005229 is tagged on Needs release manager review, Needs framework manager review. It would be pretty big change meaning for Drupal 9 or a release that supported both 8 & 9 you would need a composer.json file. It #3005229 gets rejected by those reviews do we still want/need this current change. If so again I think it should be clear why.
Comment #44
tedbowClosing as won't fix
See related plan #3069795-14: [meta] Improve dependency management for extensions as laid out by @Mixologic after discussion with @catch/@xjm/@larowlan/@Gábor Hojtsy/@tedbow/@Mixologic
tldr
- drupal:system (>=8.6.3)by instead using thecore: ^8.6.3Comment #45
xjm