[Meta] Discuss how to merge new modules into core

I would like to see a distribution approach to drupal core. Where modules are in separate repos and a drush make file is what determines what is in drupal core. Would certainly be easier to see who is the maintainer of what module, but the problem people have is now instead of 1 issue queue there are 20+. This might be able to be mitigated by better tools to help combine issue queues.

I think I remember this topic being discussed in the past though. I'll update if I can find the issue.

agreeing with @redndahead, that would be pretty cool

First of all while I have opinions on this, it feels more of a Dries issue to me since it's similar to adding extra core maintainers, and comments below are definitely 'personal capacity' rather than 8.x maintainer.

I'd like us to look at eventually moving things like the database and cache subsystems out of core (especially when they're at the point that they can be packaged as standalone libraries for use in other PSR-0 projects) then pulling them back in again - basically anything in 'components' this could arguably be done for. Whether that's practical or not I'm not sure, but it's a reason to try to move core subsystems to components so they're sufficiently decoupled it'd be possible, we'd essentially then treat those components as third party libraries.

Core modules don't have the same potential for being used outside Drupal or independently of Drupal major version, and they also by definition have a lot more dependencies on other parts of core, but they're also potentially decoupled enough to be packaged separately in the sense that thousands of contrib modules already are.

More specifically.

but the problem people have is now instead of 1 issue queue there are 20+.

Yeah packing contrib modules with core has been brought up a lot in the past, and this has been one of my main concerns with it when it's discussed. We'd have to come up with a plan to resolve that and I've got no idea how that looks - there's already serious issues with the core queue in terms of duplicate issues all over the place and I could see issues being posted against both projects adding to this.

There's also issues like API changes in core that modules would rely on. For example when writing a core patch that changes APIs, I'm required to update all core modules to the new API (either in the initial patch, or before the backwards compatibility layer. If modules are packaged separately, then to keep them in sync either means x number of maintainers committing x number of patches in the various issue queues, or allowing things to get out of sync. Now I for one dread the moment when I have to open up poll module or tracker to update something to a new API, so this could be a good thing or bad depending on your viewpoint.

For 2a and 2b could we maybe open a new issue postponed on this one? They're two separate discussions IMO and I can see this issue being very, very long just discussing #1.

One of the other benefits of decoupling is that it will allow other distributions to slim down. They would only have to package the modules they need and not all of drupal core. Other cool possibilities are available also like having a drupal core builder ala jQuery UI, where you choose only the modules or functionality you want and when you download it that's all you get.

But the multiple issue queue is a huge one. What would it look like for a core api change? Someone submits a patch to core (not including the modules), you put an issue into each project to notify them of the possible change, after a period of time that gets accepted and then those issues turn into now implement this in this module.

How would testing work? Once you put a patch in core you wouldn't be testing the rest of the modules because they would fail until they are patched. You would almost need to be able to test multiple patches at one time.

I am sure there are many other things to consider.

I'd like us to look at eventually moving things like the database and cache subsystems out of core (especially when they're at the point that they can be packaged as standalone libraries for use in other PSR-0 projects) then pulling them back in again - basically anything in 'components' this could arguably be done for. Whether that's practical or not I'm not sure, but it's a reason to try to move core subsystems to components so they're sufficiently decoupled it'd be possible, we'd essentially then treat those components as third party libraries.

The Component namespace was added precisely to support that effort, so I am unsurprisingly in complete agreement here. :-)

The biggest challenge with a Lieutenant model for modules is, as catch notes, the ever shifting API sands of core development. Even a seemingly small patch can require a change to, say, everyone's hook_init(), which in turn means any "core but not" modules are immediately broken. Unless the API is more stable, which I don't expect it to be any time soon, I don't know how viable that would be. (This same challenge does apply to Components, too, although to a lesser degree.)

The other problem, as catch also notes, is the idea that "core" modules are in some way special. Any core module that is core by virtue of anything more than "we happened to put it in the download by default" is a bug. Yes, that means filter.module needs to be refactored to be optional; the engine parts of it should be a component.

Right I'd like to get core to the point where every module is optional, and if you have your db connection (and now config directory) set up, then you'd not need to actually run any installer, and could just immediately start enabling specific modules via drush and expect things to work - i.e. no more drush si, no more 'testing profile' since it would have no modules in it at all. We're a long way from that point still though, but CMI and changing bootstrap etc. is slowly moving us there.

The technical aspect of decoupling things so that this is viable though is different from the social issue of how various components get maintained, but that social issue is likely to be similar for subsystems as for modules.

Adding note about sandboxes.