git commit -m 'Issue #2337283 by hussainweb, Mile23, alexpott, dawehner, naveenvalecha, aditya_anurag, rakesh.gectcr: [meta] Add a composer.json file to every component (module, later)'
Problem/Motivation
Our extensions are based upon .info.yml
files which works fine and I don't see a reason to change that (at least for now and this issue).
On the other hand though those doesn't provide any metadata you could use in order to use composer properly.
Contrib established now a policy (or people pushed it quite hard) to add composer.json files to every module:
- https://github.com/webflo/micro/blob/8.x/composer.json
- https://github.com/larowlan/default_content/blob/8.x-1.x/composer.json
- http://cgit.drupalcode.org/page_manager/tree/composer.json
With that and some other tricks you can use composer to build your Drupal together with the modules (this is not about the dependencies of each module) (see https://erdfisch.de/node/303)
We should probably start with the Drupal\Component namespace.
Components in the Drupal\Component namespace with composer.json files:
Plugin
ProxyBuilder
Utility
Plugin
is broken, being non-valid JSON.
Utility
is an interesting case because it contains a number of different one-file classes. Other components such as Bridge and DateTime also contain only one class. We should probably move the Utility classes to their own namespace.
Proposed resolution
Add a composer.json to every module sub component in core. This tells people that they should do that as well. On top of that the composer.json format
provides a lot of additional metadata you can use, like the maintainers, which would be a great replacement for MAINTAINERS.txt because the information
is much more in context.
Edit: This issue and sub-tasks coved only subcomponents. Adding composer.json to core modules is still under discussion at
Remaining tasks
- Add composer.json files to the various components, fix existing ones which are broken.
- Make sure they pass coding standards and have accurate information.
- Figure out a way to test whether the composer.json files are actually meaningful.
User interface changes
API changes
Possibly changing the names of some namespaces.
Beta phase evaluation
Unfrozen changes | Unfrozen because it only adds metadata to the project. |
---|---|
Disruption | Minimal disruption to core. |
Comment | File | Size | Author |
---|---|---|---|
#98 | 2337283_88.patch | 332.14 KB | alexpott |
#90 | interdiff-2337283-88-90.txt | 3.58 KB | aleksip |
#90 | 2337283_90.patch | 335.9 KB | aleksip |
#88 | 2337283_88.patch | 332.14 KB | heykarthikwithu |
#85 | 2337283_85.patch | 332.14 KB | Mile23 |
Comments
Comment #1
tim.plunkettAdded a more expansive example (hosted on d.o too!)
Comment #2
dawehnerHere is a start, just to give people some ideas: You can distinct between original authors and current maintainers/branch maintainers/documentation maintainers (whatever you can imagine)
Comment #3
larowlanLove it - should we make this a meta? I could see some components being contentious so split/smaller issues might maintain velocity
Comment #4
dawehnerWell, I guess first we should get a discussion rolling what we want to put into them, and figure out things like whether it is a good to drupal/module in there which is at least atm. a requirement in contrib world.
Comment #5
davidwbarratt CreditAttribution: davidwbarratt commentedComment #6
Mile23Comment #7
dawehnerLet's follow the suggestion of @larowlan and make that a meta.
Comment #8
Mile23Comment #9
dawehner@Mile23
Thank you for adding a better issue summary. Do you want to work on this issue? I would be glad reviewing it!
Comment #10
Mile23This work should only proceed once a decision has been made as to how we're going to manage the tests for these namespaces as dependencies.
That is, if I do this:
composer require drupal/gettext ~8
I would expect to get a package in myvendor/
directory with aREADME.md
and asrc/
directory and atests/
directory so I could run the tests.How will we do that?
If we're not doing that, why are we making
composer.json
files?Comment #11
dawehnerIMHO, they don't matter. If you on your own small PHP based project want to pull in some component, you won't run the uni tests of the vendor directory, especially like symfony takes actually some time. The value is in making the components itself available. By doing that, we also show that we are open to other people
out there in the PHP community.
A readme would be certainly really nice, but I think that its not a requirement to have it, don't you think so?
Comment #12
jhedstromI looked at how Symfony does this, and while there is a Tests directory for each component, there isn't a
src
directory in the subtree split.https://github.com/symfony/Yaml
I don't think moving our unit tests into each component would need to block the first step of adding a
composer.json
file for each one.Comment #13
Mile23My point is that the first step is knowing how we'll do it. Measure twice, cut once, etc.
I realize this makes me a rebel in the Drupal world. :-)
Comment #14
dawehnerSo do you totally disagree with doing it or not? Your last answer kinda opened up multiple ways of interpretation.
Comment #15
jhedstromIs there any reason not to just follow Symfony's example? Specifically,
target-dir
andautoload
directives from the variouscomposer.json
files:(we'd use
psr-4
instead).Comment #16
dawehnerI don't see a reason why not in general.
Comment #17
Crell CreditAttribution: Crell commentedWe've already done this for one component. Do it for the others and move on. If we want to add a README to each component that's fine, but don't worry about the tests. We expect anyone using the components stand-alone to come back to core to submit patches anyway, so if the tests aren't in the split repos for now, meh, who cares?
Let's stop talking and just move forward. We need to do more of that.
Comment #18
Mile23Making template versions.
Comment #19
Mile23This is for Gettext. Presented here for your perusal and criticism.
Let's refine the template and then we can repro it into all the sub-issues.
Also: I updated the issue summary on #1826054: [Meta] Expose Drupal Components outside of Drupal
Comment #20
Mile23Derp.
Comment #21
Mile23Comment #23
dawehnerThank you for posting a first example!
I agree that we should ship with the license file, its just the way how its supposed to work.
Can you define the same key in json twice?
Comment #24
Mile23Changed tests path.
Removed duplicate 'source' key.
Added information about how to test only the component in question.
Comment #25
Crell CreditAttribution: Crell commentedEverything else we have specifies PSR-4. No need to use PSR-0 on this one. For a split repo we would probably want to remove a few layers of directory, too, so the composer file should reflect that.
Comment #26
Mile23Using PSR-4, no target-dir.
Makes me itch to not have
src/
andtest/
:-)Comment #27
dawehnerThis looks pretty sold for me, but I guess someone has to try out whether a subtree split package can actually be done.
Comment #28
Mile23Happening over here for
core/
: #2352091: Create (and maintain) a subtree split of Drupal coreComment #29
dawehnerRight, but you agree we would need one for the plugins as well ...
Comment #30
Mile23Also in the issue summary for the meta: #1826054: [Meta] Expose Drupal Components outside of Drupal
Who wants to volunteer? :-)
Comment #31
davidwbarratt CreditAttribution: davidwbarratt commentedComment #32
Mile23Important info about how to name these things: #2401519: [policy] Decide on Composer Package Names
Comment #33
Mile23This issue adds a test to check the integrity of composer.json files: #2472269: Fix syntax errors in Drupal\Component\Plugin's composer.json, Add test
Comment #35
dawehnerNice.
Yeah I think adding those composer.json files would still be a nice thing, it makes it clearer who maintains what, and enables people to reuse the components,
which you simply can't overestimate in the result.
Comment #36
Fabianx CreditAttribution: Fabianx as a volunteer commentedSo to be clear this is blocked on someone trying to push a subtree-split of a component to GH and see if its installable via composer?
Comment #37
Mile23I think #2472269: Fix syntax errors in Drupal\Component\Plugin's composer.json, Add test gives us tests as to whether the
composer.json
files are valid, so we could proceed per component.Making it so everyone can just use composer to grab these components will require the subtree split, but we can proceed with making the files now. The only difficulty will be determining dependencies and then checking them for patch review.
One clue might be
use
statements. For instance,\Drupal\Component\Annotation\Plugin
saysuse Drupal\Component\Utility\NestedArray;
, which shows us the dependencies.Any errors here will become much more apparent as people start using them. :-)
Comment #38
dawehnerI don't think so. What we want to is agree which information we should put into it and then someone writing a patch to include them.
The subtree splits can be done later independently.
Comment #39
davidwbarratt CreditAttribution: davidwbarratt at Golf Channel commentedSee the subtree split thread for that discussion #2513388: Create (and maintain) a subtree split of each Drupal Component
Comment #40
Mile23Template composer.json in #26.
Comment #41
Mile23Ready for review: #2474903: Add composer.json to \Drupal\Component\Gettext\ component.
That issue is also set up so you can clone it (using dreditor) for the other child issues.
Comment #42
Mile23Added ten kerjillion child issues: #2600718: Add composer.json to \Drupal\Component\Bridge component. #2600720: Add composer.json to \Drupal\Component\Datetime component. #2600722: Add composer.json to \Drupal\Component\Diff component. #2600726: Add composer.json to \Drupal\Component\Discovery component. #2600728: Add composer.json to \Drupal\Component\EventDispatcher component. #2600728: Add composer.json to \Drupal\Component\EventDispatcher component. #2600734: Add composer.json to \Drupal\Component\HttpFoundation component. #2600738: Add composer.json to \Drupal\Component\Render component. #2600740: Add composer.json to \Drupal\Component\Serialization component. #2600742: Add composer.json to \Drupal\Component\Transliteration component. #2600744: Add composer.json to \Drupal\Component\Utility component. #2600746: Add composer.json to \Drupal\Component\Uuid component.
Comment #43
xjmTagging for framework manager review. Thanks for the meta issues.
Comment #44
alexpottDiscussed with @xjm - I don't think that there is a case for adding this to 8.0.x since that is patch release branch and only should include fixes to stable code and all of Drupal/lib is stable.
Note that including composer.json files has been approved by Dries in #2176065-33: Introduce a composer.json for Drupal\Component\Utility
Comment #45
alexpottComment #46
alexpottAll the people mentioned in the above potential commit messages need credit on this issue when it lands
Comment #47
alexpottHere is a start of merging all the patches together...
We need to ensure that all components that are missing a composer.json have one and that all the expected files are present - for example TESTING.txt is missing from UUID. Plus there are components which still need to be addressed...
Comment #48
alexpottFound some more issues to close and merge :)
Comment #49
alexpottMerged those patches in
Comment #50
naveenvalechaThanks for merging all those.
We don't need cd core here.
The group name is Inspector in the test class(InspectorTest.php) Annotation.
$ ./vendor/bin/phpunit -c core --group Inspector
Same as above we don't need cd into core here.
There is no group available "Diff" in any tests. So need to replace it with correct one.
Remove cd core from here.
Also the group name is Symfony in ContainerAwareEventDispatcherTest.php
$ ./vendor/bin/phpunit -c core --group Symfony
Can we update the group name in the tests file in this issue or should we create a follow up later to not disturb this one.
We don't need cd core here.
$ ./vendor/bin/phpunit -c core --group FileCache
Why this change ? irrelevant change
Similar as above.
$ ./vendor/bin/phpunit -c core --group Graph
Remove the cd core here.
The group name is "Routing" in SecuredRedirectResponseTest.php so change & question is as above above.
$ ./vendor/bin/phpunit -c core --group PhpStorage
Take care as above.
the test group name is "utility".so same question and change as above.
$ ./vendor/bin/phpunit -c core --group utility
$ ./vendor/bin/phpunit -c core --group Serialization
Need a new line :)
This Should be in the Uuid compononent ?
This file should be in Uuid component ?
We do need to include Uuid component as well.
Comment #51
dawehnerSounds like we should add a follow up to fix that.
Comment #52
hussainwebRegarding 2, 5 and similar comments, I have made the changes but I think followup is needed to fix the group name. The massive file size difference is because LICENSE.txt was missing in couple of components (DependencyInjection, Plugin, ProxyBuilder).
Comment #53
hussainwebOf course, it helps if I actually upload the file. ;)
Comment #54
naveenvalechaThanks for updated the changes. An interdiff would help a lot here to review the updated changes. interdiff please!
Irrelevant change. Should we take care of it in this or a followup ?
Comment #55
naveenvalechaCreated followups for 50.5 and 50.9 https://www.drupal.org/node/2630986, https://www.drupal.org/node/2630988
Comment #56
hussainwebSorry about the interdiff. I realized too late. I needed to use some git magic but got it.
Comment #57
naveenvalechawe are very close to RTBC! Some more nitpicks! Reviewed the interdiff.
Please remove the trailing "/" from core/ to make consistency over all Testing.txt files.see others components TESTING.txt
Please make it to
$ ./vendor/bin/phpunit -c core --group Gettext
$ ./vendor/bin/phpunit -c core --group Graph
Make it to $ ./vendor/bin/phpunit -c core --group HttpFoundation
Also we need to update the group name in SecureRedirectResponseTest.php that we can do in followup.
same as above.
The group name is proxy_builder in test.so change it to "proxy_builder"
Comment #58
Crell CreditAttribution: Crell at Palantir.net commentedThe autoloaders should be defined to use PSR-4, not PSR-0. PSR-4 can handle full-path without a directory prefix just fine; but PSR-0 and PSR-4 have slightly different parsing rules. Better to just use PSR-4 everywhere.
Comment #59
hussainwebI think it was psr-0 because we copied it from one of the components which already had a composer.json. Functionally, there will be no difference if we just change it. But should we do it in a follow up to keep a cleaner history?
Comment #60
hussainwebFixing for comments in #57. Thanks for the review and getting the group names, @naveenvalecha.
I am still not changing it to psr-4 yet as @Crell pointed out. I am not sure if we should do that here or in a follow-up.
Comment #61
naveenvalechasounds we should add a follow up to fix that.
Rest looks good, we can initially ship with this code.
Comment #62
hussainwebCreated #2631582: Change autoloading standard in composer.json in Drupal components.
Comment #63
alexpottMissing -c core.
I think we should make the change to PSR-4 as Crell suggests since doing that in a followup is pointless.
Comment #64
hussainwebThe idea was to keep a cleaner history if possible. But yes, the changes are not functional, so it's fine.
Comment #65
naveenvalecha@alexpott,
Should we also update the tests group name in this as well b/c those are not functional changes, is that fine if we also keep that as well instead of the followups #2630986: Update the test group name in InspectorTest.php to Assertion,#2630988: Update the test group name in ContainerAwareEventDispatcherTest.php Annotation to EventDispatcher
Shouldn't we change the group name of the test as well b/c there are lots of tests that fall under Symfony groups in core tests ?
Irrelevant change. Please revert it.
The group name is "Routing" in test files.Have question regarding same in the comment above.
Comment #66
alexpott@naveenvalecha let's not make test changes in this issue.
Comment #67
naveenvalecha#66, Thanks for confirming!
From #66, Please ignore #65.1
In #65.3 Please do change to
$ ./vendor/bin/phpunit -c core --group Routing
we will do the changes of group names in followups.Comment #68
hussainwebChanged as per #65 and #67. Thanks!
Comment #69
hussainwebComment #70
naveenvalechaBack to RTBC as all mine & alex suggested changes has been addressed.
Added proposed commit message as well from #2337283-48: Add a composer.json file to every component
Comment #71
naveenvalechaComment #72
Mile23Hmm....
Suggest
~8
instead. This would be in all places where we declare an interdependency such as this.Inspector is the group name of the test, but it should be Assertion.
Should be a follow-up. EDIT: Ahh, there it is: #2630986: Update the test group name in InspectorTest.php to Assertion
Some of these packages don't actually require PHP 5.5.9, but we'd prefer they have the same requirement as D8 core. So no change needed.
Comment #73
hussainweb#2630986: Update the test group name in InspectorTest.php to Assertion is in and I have changed the test group for that in
Assertion\TESTING.txt
. It will be a good idea to get the other issue in as well - #2630988: Update the test group name in ContainerAwareEventDispatcherTest.php Annotation to EventDispatcher , which is already RTBC.I also changed all instances of
>=8.0.0
to~8.0
as per #72. While it's not strictly in scope of the issue, it is similar to the change we did in #64 to change everything to psr-4. It is not functional yet so we might as well change it here.Comment #74
Fabianx CreditAttribution: Fabianx as a volunteer commentedBack to RTBC - I guess.
Comment #76
hussainwebIt's just a reroll due to #2550519: Crypt::randomBytes()/drupal_random_bytes() doesn't actually return cryptographically secure random bytes. Setting back to RTBC.
This is the only difference between the two patches:
Comment #77
alexpottWe need to make this change - I would have fixed on commit but I've contributed to it so I can't commit.
Comment #78
hussainwebChanging as per #77. If that was the only issue, it is good to go to RTBC but keeping it as "Needs Review" just in case.
Comment #79
alexpottHas someone checked that all the dependencies are correct as this is where mistakes are most likely to be made.
Comment #80
Mile23I looked at the Diff dependencies for this, and they seem correct: #2472633: Expand PHPUnit test coverage of the Diff component
Comment #81
Mile23That's why they were separate issues. :-)
Here's an easy review. I'm poking through the dependencies now, and will add another review.
Should be ~8.1
Should be ~8.1
Should be ~8.1
Should be ~8.1
Should be ~8.1
Should be ~8.1
Comment #82
Mile23I searched each component for
use
statements and then compared those to the composer.json file dependencies.Annotation: good.
Assertion: good.
Bridge: Needs symfony/dependency-injection, uses Zend\Feed, which I believe is in zendframework/zend-feed (there are a couple of different zend feed packages on packagist, which is why I'm not sure.)
Datetime: good.
DependencyInjection: Needs symfony/dependency-injection, symfony/expression-language
Diff: good (see above).
Discovery: Needs drupal/core-filecache, doesn't use core-utility.
EventDispatcher: good.
FileCache: good
Gettext: Needs drupal/core-utility.
Graph: good.
HttpFoundation: good.
PhpStorage: good.
Plugin: Needs symfony/validator.
ProxyBuilder: Doesn't need symfony/dependency-injection afaict. Might need more investigation.
Render: good.
Serialization: good.
Transliteration: good.
Utility: Needs drupal/core-render.
Uuid: good.
Comment #83
Mile23Comment #84
Mile23Added changes from #81, #82.
Comment #85
Mile23Needed a reroll.
Comment #86
tstoecklerNot on my watch will the British invade Drupal core...
should be
(Note that the component (and the namespace) is already named "correctly".)
Comment #87
heykarthikwithuworking on this.
Comment #88
heykarthikwithuChanges as per #86
Comment #90
aleksipBumped ~8.1 to ~8.2. Really hope to see this issue land in 8.2!
Comment #91
Crell CreditAttribution: Crell at Palantir.net commentedThe composer.json files don't have any bug-impact. Why can't this still be done in 8.1?
(Looks like both patches would need to be applied, as they do differ in version numbers but are otherwise the same I believe.)
Comment #92
xjmSo the current patch on this issue is only for 8.2.x. Either it is not for 8.1.x, or it is not RTBC.
The issue was made a minor version target by @alexpott back in #44. We discussed whether or not it should be today, but personally, I would be fine with this going into 8.2.x now or whenever.
Comment #93
Crell CreditAttribution: Crell at Palantir.net commentedxjm: The patch in #88 is for 8.1. The patch in #90 is for 8.2. Other than changing that reference for component-dependencies, there's no difference between them. Thus, this issue contains an RTBC patch for both 8.1 and for 8.2, either or both of which can (IMO) be committed. (I would favor both.)
Comment #94
xjmBTW, I belatedly remembered the exact reasoning for this being a minor version target. It was not about any risk particularly, but that it is equivalent to a coding standards change, which we only do in minors.
So marking against 8.2.x for now. Let's get the committer review and potentially a commit there while it is up to date, since 8.2.x is not frozen (8.1.x is in commit freeze and about to go into RC). Then, if there are strong feelings that it should go into 8.1.x as well that could be discussed.
Comment #95
aleksipPersonally I'd love to see this in 8.1 already! Making these components available to the larger PHP community is an important part of 'getting off the island'. I'm involved in a Drupal/Pattern Lab integration project and we need to use several Drupal Components. October and 8.2 seem such a long way away... :)
If this is considered for 8.1 there is also #2664570: Move Attribute classes under Drupal\Component where the 8.2 patch applies cleanly on 8.1 as well (the patch only moves the classes without any functional changes).
Comment #96
alexpottCommitted 28c46c2 and pushed to 8.2.x. Thanks! I've creditted everyone who commented on the issue because the rolling the patches was the simple part of this one.
Personally, I think given @xjm's explanation that this is equivalent to a coding standards change then actually RC is the perfect time to commit this as that is when we target large coding standards changes. So I'm moving this issue back to 8.1.x and will wait for @xjm's opinion as release manager. But I can see no good reason for not committing this to 8.1.x.
Comment #98
alexpottRe-uploading #88 so the 8.1.x patch is last on the issue.
Comment #99
xjm@alexpott has a good point about RC and big coding standards changes. (E.g. we just removed
@file
from a zillion classes during RC for the same reason.) I can see the case for it being an RC target; I feel far more comfortable with that than with a patch release. I want to think about it overnight, but adding to the RC target triage queue to make sure we don't miss the window.Comment #100
xjmI see there is already a change record, but is there anywhere in the handbook or API documentation that we should also document this best practice?
Comment #101
Mile23I think there'll be more documentation needed, and more opportunities to add it, as we move on with the parent #1826054: [Meta] Expose Drupal Components outside of Drupal
Unless there was a specific best practice you had in mind...
Comment #102
xjm@Mile23, thanks! I think some overall dev docs make sense to add as a step in the meta but not needed here. I also just checked and it actually looks like the README for
Drupal\Component
does cover it already:So I think my comment in #100 is already addressed. Tagging as an RC target so we can ship this in 8.1.x too, and also marking for the release notes so I remember to include it in the section about DX and coding standards improvements for the minor. I'll leave for @alexpott to commit though since he already reviewed it for 8.2.x.
Comment #103
alexpottCommitted 68b6639 and pushed to 8.1.x. Thanks!
Comment #106
fgmComment #107
fgmComment #108
vijaycs85