Distribution Modernization Initiative meeting on October 26, 2021 [#3245841]

This meeting:
➤ Is for distribution developers, initiative contributors, the Drupal Association and anyone interested in the initiative.
➤ Usually happens every other Tuesday at 1400 UTC.
➤ Is done over chat.
➤ Happens in threads, which you can follow to be notified of new replies even if you don’t comment in the thread. You may also join the meeting later and participate asynchronously!
➤ Has a public agenda anyone can add to.
➤ *Transcript will be exported and posted* to the agenda issue. For anonymous comments, start with a :bust_in_silhouette: emoji. To take a comment or thread off the record, start with a :no_entry_sign: emoji.

:zero: Who is here today? Comment in the thread below to introduce yourself and tell us why you are joining us.

:one: Do you have any topics to propose for the meeting today? Feel free to propose them in this thread, and then I will give them their own unique threads for discussion. Conversation moving slow? Go ahead and open your own thread in the next numeric order.

0️⃣ Who is here today? Comment in the thread below to introduce yourself and tell us either a) What distribution you maintain or regular work with or b) what your other interest in this initiative is! :slightly_smiling_face:

hestenet (he/him)	Tim from the Drupal Association - my interest is in how we update Drupal.org and our procedures to better support distributions in the D9/Composer era
mixologic	Heyo, Ryan here. Im here because we're going to empower distributions with composer, and I have a lot of thoughts etc.
Rajab Natshah	Hi,Rajab Natshah, Technical Product Lead at Vardot. A maintainer of the Varbase distribution (edited)
rszrama	Other Ryan here; Centarro is reviving Commerce Kickstart as a distribution to be used as the foundation for building Drupal Commerce sites with an optional full demo mode reproducing the current “Commerce Demo” store.
dwkitchen	David here working with @rszrama on Commerce Kickstart
Mohammed Razem	Razem here, Product Owner of Varbase distro.
shawndearmond	Shawn DeArmond. My team builds and maintains a distribution for web sites at the University of California Davis. We have over 750 live sites and counting. https://sitefarm.ucdavis.edu
dimkritsotakis	Dimitris here, building a D9 distribution for our local and national members of our organisation, ESN (esn.org)
dww	Derek. User of a distro for a personal site. Builder of the bespoke D7 distro packaging mess (you’re welcome!) :joy: long term interest in distros, community plumbing, etc
smccabe	Shawn McCabe, mostly interested in Commerce Kickstart and the ability to add commerce into other distros
ronaldtebrake	Ronald here, maintainer of Open Social
DyanneNova	Em here, long term interest in Distros, mostly from COD and Lightning, curious to see where things are headed
Tiago Barreiro de Siqueira	Tiago here, maintainer of Open Social, also working with @ronaldtebrake
bircher	Fabian Bircher, long term interest, previously built and maintained the Drupal 7 distro @dimkritsotakis is working on.

1️⃣ Do you have any topics to propose for the meeting today? Feel free to propose them in this thread, and then I will give them their own unique threads for discussion.Later on, if the conversation is moving slow? Go ahead and open your own thread in the next numeric order.

mixologic	Lets talk about what we want composer to be able to do in the context of a distribution.
Mohammed Razem	D.o listing of distros that are compatible with d9 is not properly working. Also, potentially have a better way to list/promote distros on d.o.https://www.drupal.org/project/project_distribution?f%5B0%5D=&f%5B1%5D=&...…]Afull&f%5B4%5D=&f%5B5%5D=&text=&solrsort=score+desc&op=Search
Rajab Natshah	Many of the current distributions are not up to date with Drupal 9 standard and way.Some of them still use functions from D7 or D8.
Mohammed Razem	Installation profile inheritance. Will this “realistically” ever be added to Drupal core? Should distros continue to build on top of this patch and/or concept of extending profile?#1356276: Allow profiles to define a base/parent profile (edited)
dimkritsotakis	If not considered already, can the automatic updates module be adapted to distributions? And how? We should consider some basic ideas in order to adapt the automatic_updates code to allow updating a distro core. Unless it's too early to discuss something like that.
hestenet (he/him)	@dimkritsotakis There was actually a recent question put out by the autoupdates initiative themselves about that - I'll find the link...
hestenet (he/him)	https://drupal.slack.com/archives/C7QJNEY3E/p1634664247264500
dimkritsotakis	@hestenet (he/him) I understand that's to upgrade core while in a distribution. But I'm talking on updating the distribution files through that module
hestenet (he/him)	@dimkritsotakis Oh - I follow you... Right now they haven't even started on any contrib support at all (even for regular modules) it's just core - but that is part of their long term plan.

2️⃣ For those of you who maintain distributions - please use this thread and let us know - what are the current workaround solutions you use to do that?

Grzegorz Pietrzak	For us (Droopler distribution) the biggest issue now is that drupal.org packagist server produces distribution's composer.json files without any dependencies - so we can't encourage users to use drupal/droopler composer lib, but we have to utilise github and "real" packagist.org instead.
Rajab Natshah	Maintaining distributions needs a bit more work and time.I'm using the Visual Distribution Operator (VDO)So many bash files which save big time.My wish is that Drupal.org could have a VDO tool to manage distributions
ronaldtebrake	Just a cross reference: https://drupal.slack.com/archives/C51GNJG91/p1616774575037500Loved this conversation in the thread which basically outlines the same as Droopler encounters and what we encountered with Open Social.
shawndearmond	The distribution that we manage is ONLY used within our organization… however, we’ve done some cool things, and it would be neat to share best practices / code with other organizations trying to do the same.
Pasqualle	It seems public distributions are using packagist.orghttps://packagist.org/packages/degov/

3️⃣ We should get clear on what a modern distribution is and what it isn't as we talk about the updates to the infrastructure.(For example, whether a distribution is more commonly treated as a starter kit vs. an off the shelf- complete product vs. something else)

rszrama	Interesting question … we definitely want people to be able to use Commerce Kickstart with no modifications, build their store, and go live as is. If the feature set works for them, awesome! But given the nature of Drupal as a platform … it’s not likely most users will use it as a complete product. @dwkitchen has some thoughts on how we can still permit people to customize things while keeping a base feature set up to date with each release.
dwkitchen	I think there are both use cases; the problem comes when mixing up the two.There is a requirement for a starter-kit where configuration is installed and then the site build is free to change how they wish and not care where they started from.Then there is a second requirement to product that is maintained, the site instance has very few configuration differences from each other and the distribution is used to maintain all the sites
mixologic	I think the difficult part to reconcile is when you have a distro that ships with, say, a preconfigured view, and the end user customizes that view (adds a field etc), and then the distro also adds something upstream. Is there an upgrade path for those users?
njim	This is the main reason I'm joining this conversation today. I'm working with a university to develop a Drupal as a service type of product. In the past I've largely used distributions as starter kits and it's been great. But I see a need to have a product roadmap. The challenge is pushing updates to thousands of sites while hopefully allowing the downstream sites to maintain some variation.
dwkitchen	I have previously worked at Publishing Company and Pharma Company, where they also have many sites running on the same distribution, were the only “configuration” difference is the site name; it is just all the content that is different
dwkitchen	The use of the Distribution is to be able to roll out updates and new features to all the sites.
mixologic	So for the university/pharma use cases, those are similar to custom modules. i.e. not something we'd host on drupal.org, but those orgs will still probably want to use composer, so we should still look at what the best practice would look like for that scenario.
dwkitchen	Yes, so the distro and custom modules where all hosted privately with private packagest for composer
dwkitchen	But back to question 3️⃣ I think there is a need to have “install profiles” but these are different from “profiles”
dww	This is a really tricky problem. I used a distro for a D7 personal site. But I needed to do a lot of customization, and then updating it was a mess. Would have been way better if it was structured and billed as a starter-kit.I can also see the use cases for what y’all are talking about with a distro that handles the updates for mass deployments.It’s not always obvious, in advance, which path a given site is gonna take.Ideally, we’d have structure / plumbing / best practice that handles both well.
ronaldtebrake	Fully agree with what @rszrama said.We want people to be able to use our Open Social distribution as is.We even have it at the heart of our SaaS and Enterprise offering so we also need to worry about our own upstream work in that sense so we need to take that in to account.We also noticed quite some end-users don’t really know anything about Drupal when they start using Open Social. A startkit or distribution is also an entry point to Drupal (and all other tools) behind it in that sense.So we are responsible for quite some personas and we try to balance that. I mean even if we position ourselves as a full featured distribution, we use Drupal and we know how amazing it is to use it and its flexibility and powers. So if they decide to use it as starter kit, we want to try and accommodate that. (edited)
ronaldtebrake	One of the things that helps us is CMI and the ecosystem in contrib. Using https://www.drupal.org/project/update_helperIt will only update if the expected config matches.For examplehttps://github.com/goalgorilla/open_social/blob/main/modules/social_feat... core.entity_view_display.node.event.small_teaser: expected_config: content: groups_type_public_group: label: above region: content settings: link: true third_party_settings: { } type: entity_reference_label weight: -5 update_actions: add: hidden: groups_type_public_group: true delete: content: groups_type_public_group: { }That allows for people staying close to Open Social to benefit from all our changes. However those who change the configuration can do so, but it comes at a cost, it means more work in following the update path.Still believe there is work for us on that front, but there are options :slightly_smiling_face: (edited)
hestenet (he/him)	That is a helpful example of accommodating both scenarios @ronaldtebrake - I was not familiar with update_helper before now..
Mohammed Razem	We use update_helper in Varbase. We would like to maintain the distro as a “starterkit”, yet inform the webmasters/developers who use Varbase about beneficial updates, whether in config or not.Because of update_helper module (and it’s integration with Checklist module to provide a UI of what has been updated), we opted to categorize changes/updates of the distro releases into 4 categories:Forced Update: were have a new module that became a dependency, a configuration “fix”, or a database table alter. This is forced and will always be executed through hook_update.Forced Update if Unchanged: Mostly a configuration change. Using update_helper config, in the hook_update we check if this setting remained the same before we do it so we don’t override user’s setting. If the user has overriden it, it becomes an “Optional Update” - see #3 below.Optional Update: A nice enhancement that we recommend users to use or an update that failed to apply (see #2). There’s no hook_update for this. Instead it is communicated through update_helper checklist integration in Drupal UI. The UI would give instructions to the users how to apply this update (mostly using a Drush command that reads update_helper config) or manually by liking to release notes or help articles.No Update: Only new installs would get this change.update_helper allows us to inject config changes for #2 and #3.See https://docs.varbase.vardot.com/dev-docs/updating-varbase/handling-confi... (edited)
Mohammed Razem	There’s a messy patch that follows this logic #3024165: [PATCH] Enhancements, Better messages for the unable to apply updates, drush command - would be great to meet with the maintainers to consolidate the work together.
Pasqualle	I have worked on an inhouse decoupled Drupal distribution where multiple different websites were server from the same codebase. All the manual changes made in Drupal admin were committed back to the codebase.Having 1 codebase had a benefit of easy scaling with kubernetes pods. We could use 1 pod for all websites, or spin up additional pods for given website in case of high traffic.There is an interesting module to get user changes back to the codebasehttps://www.drupal.org/project/config_auto_exportThe module was also presented athttps://www.talkingdrupal.com/236

4️⃣ What is the scope of how Composer should be used in the context of a distribution? Are we always 'composer create-project'-ing with distributions? Are we ever composer 'requiring' distributions? @mixologic has some thoughts here.There are multiple components that all might be part of making a distribution - here are a few of them:A composer profileDrupal install profileThe primary distribution 'monorepo' The additional contrib modules it requiresThe collection of patches that may be required How should we understand how all these should relate to each other in a modernized version of distribution support? (edited)

mixologic	This probably covers #4 above
mixologic	So, first thing we'd like to introduce is templates on drupal.org. Much like we already have drupal/legacy-project and drupal/recommended-project , we'd like to make it possible for distribution maintainers to have their distro template in the drupal namespace.
mixologic	This will also allow for things like 'opinionated templates' that do extra things like come with cweagans/composer-patches and drush/drush etc.
mixologic	Secondly, we should figure out how to sort out an "installation profile" from a "distribution"
mixologic	Should be be able to composer require a .profile separately from the whole distro? or is all the config and modules a requirement and they must go together?
mixologic	What should the templates have in their require statements to build a distro, in other words?
Rajab Natshah	A generic distro profile template and a project template proposed by drupal like drupal/recommended-project .. Lightning was a good hint on what we can do. or what is best to work with like cweagans/composer-patches we could have something like drupal/composer-patches it will be used for sure. Now we add that in almost every project. as patches are needed in the long run of projects

5️⃣ Listings of Distributions on Drupal.org, especially when filtered by major version compatibilty (e.g: compatible with D9) is not working properly - what might be needed to fix this, and what might be a better way to list or promote distributions?

hestenet (he/him)	@Mohammed Razem raised this topic
hestenet (he/him)	This issue might be deferred until we know how the distribution content type on drupal.org might change as part of this initiative - see item 4️⃣ - however - in the long run, we do need to take a pass at the user experience of finding a distribution to use once we have the distributions updated.
Mohammed Razem	Makes sense. However, since the current search on drupal.org is broken and probably people might think there are NO distributions available on Drupal 9, I think it makes sense to push for drupal.org search to be hotfixed to list distros supporting D9 properly.There’s an issue here: #3134858: Distribution list erroneously shows 0 projects Drupal 9 / 10 compatible
hestenet (he/him)	Gotcha - reading up on that issue.
hestenet (he/him)	That may be relatively straightforward.

6️⃣ Once we have more fully defined the D9 'standard' for building distributions - how do we help the many existing distributions that are not up to date for Drupal 9 yet? Some that still have d7 or d8 functions still, etc.

hestenet (he/him)	@Rajab Natshah raised this topic
Rajab Natshah	Last week I had a general check on all distributions ( build + install )only around 27 profiles work with Drupal 9
Rajab Natshah	Reporting that they do have so many errors, notices, and warnings related to:Drupal 9 readiness, Depreciation .. not using Drupal Standard and practice No good support for PHP7.4 yet. ( Array, values, is countable. and moreNot using the Drupal Config Factory standard way for configs ( some mishmash arrays saved as YAML in active configs ( no schema )Not the right way to use composer ( static calls )Not the right way to require libraries Trying to do some changes to files on the serversAlmost like a Drupal 7 developer is learning to work with Drupal 9
njim	When someone audits projects, do we create issues about the distro not being Drupal 9 ready? I remember my team swarming on modules as things were being migrated to D8. I believe we used some tags to filter which ones needed attention and just chipped away.
rszrama	I’m not sure it’s gonna be worth the effort to try to help a D7 distro make the leap. :sweat_smile:
njim	Agreed. I was mostly thinking about D8 deprecations. These are usually great tasks for folks new to open source.
Rajab Natshah	Me too only Drupal 8 to Drupal 9
Rajab Natshah	But only 27 are working under 9my list for Drupal 8 distros is at workspace.profiles.settings.yml
Rajab Natshah	Having a list of bash files for distros to build them and test install them toohttps://github.com/webship/vdo-project/tree/9.0.x/profiles

7️⃣ Installation profile inheritance - do we anticipate this core issue actually being committed, and what impact would it have on distributions?#1356276: Allow profiles to define a base/parent profile

dwkitchen	I have been a regular user of this patch for at least 8 years
hestenet (he/him)	This was also raised by @Mohammed Razem
dwkitchen	Again as with the discussion on 3️⃣ the use case is in private distros that are not going to get published on d.o
dwkitchen	In my previous organisation we had :a: the general distro, we then had multiple brand based distros that inherited from :a:; Brand 1 :large_green_circle: Brand 2 :large_blue_circle: Brand 3 :large_yellow_circle:, etc. Then each brand distro would be used for the site in each territory :large_green_circle: :uk: , :large_green_circle: :us: , :large_blue_circle: :fr: , :large_blue_circle: :de: , etc
dwkitchen	I also see the potential for this with Commerce Kickstart as a base profile, that is sub-profiled by a organisation with multiple brands or teritories
njim	Agreed that the use case is likely not to be published to drupal. This is likely about an organization that wants to extend an existing profile or maintain a series of related profiles. In both cases they are likely not putting them on drupa.org.But I don't understand why this matters. Drupal is made to be extensible. While I want to contribute code whenever possible, the fact that I can write custom modules is just as important.
dwkitchen	I think I have seen it as an argument that there is no demand for [the patch] as there are almost no distros on d.o using it (edited)
smccabe	i gave a talk at dc a few years ago related to dynamically building profiles w/ composer and this was the #1 complaint by everyone, they had essentially the same use case @dwkitchen describes. I also know this was the blocker for openedu as they were based off of lightning
dwkitchen	https://youtu.be/6yiK9DIJQvo?t=2314
dwkitchen	There is a discussion about the patch here, and I ask question and give the use case.
smccabe	ah yes, i should clarify, i mean entirely on the module side of things, if we're talking about default content from install profiles I don't care one wink. In that case, shouldn't it really be 3 things. Modules, Install flow changes and default content/settings?
bircher	When I started following that issue it had only one page of comments and I originally though that this was a neat idea. I also remember the drupalcon Vienna session and I agreed with dawehner, phenaproxima and the other speakers already then. Now I am even more convinced that profile inheritance is not a good idea.I don't mean to start a flame war with the people who are heavily invested in this.I just think composing is better than inheriting.instead of inheriting things from a profile you should script copying the things you want from the profile to a new one. Ideally the profile is as empty as possible with just some default configuration and some install steps with pretty forms. The rest of the logic and default config you intend to maintain should be in modules that you then require with composer. I think we should go a step further and stop loading modules form install profiles sub folder and in general treat profiles less like modules.I have worked a lot with config management and config updates and consulted on the development of big install profiles and the leaner a profile, the more features are well defined in their module the smoother the process.Inheriting profiles when you maintain both the parent and the child profile is still somewhat doable and I guess that is why people still use that patch. But if it were in core then anyone could inherit from any other install profile which would make the maintenance a nightmare either for the parent profile maintainer if they want to provide some backwards compatibility for possible unknown sub profiles. or the maintainers of the sub profiles when inevitably the parent profile changes things. In that case you are just better off making a clone of the profile and porting updates that are relevant at your own leisure. Of course the tooling for this could be standardised etc an have a recommended way forward.There is https://www.drupal.org/project/install_profile_generator which could help people off their inheritance sites.

8️⃣ We'll be working to gather insight from this conversation and from some of our internal planning about distribution support, and preparing a meta/plan issue on drupal.org to outline some steps. We'll hope to share before the next asynchronous slack call in 2 weeks.

Participants:

hestenet, Mixologic, Rajab Natshah, rszrama, dwkitchen, Mohammed Razem, shawndearmond, dimkritsotakis, dww, smccabe, ronaldtebrake, DyanneNova, Tiago Barreiro de Siqueira, bircher, Grzegorz Pietrzak, Pasqualle, njim