[meta] Smarter UI/API separation for modules

Totally the opposite. API provider modules have nothing to do with UIs and if you mix them up you get the horror called block module which is block, block_ui and block_custom stirred, mixed and shaken until your ears bleed.

Separation via file? Maaaaaaaaaaaaybe. But then you need to make sure that hook implementations can live in files. Which they currently can't so mixup occurs.

So let's approach Registry once again for D8 ;)

Does Views UI take a lot of Ram merely by being turned on? I think the Ram used for fancy/complex admin UI screens may be part of the rationale?

Also because more people want to have an API-like Swiss Army knife quality to their app, detaching the UI from the core stuff may serve that kind of design pattern. Eliminating the complexity makes sense to me tho :)

@webchick,

maybe what you are really against is: doing it just for the sake of doing it, without understanding benefits/implications.

If a module is separating API and UI into two different modules, to make UI pluggable ("I think your UI sucks, I will use my own") then it's a perfect example of a separation of concerns, if it is done because "other cool modules do it" then we have a problem :)

There's also an issue of scalability, which is more general and goes beyond foo and foo_ui issue.

Creating many modules vs one big one is a constant struggle between two forces:
1. Usability (one big module is more usable, like you noted).
2. Scalability. Drupal loads [practically] all enabled code at each request, which has significant, adverse affects on scalability. Ability to disable unnecessary modules in production is an important benefit. For instance, most sites do not need views_ui in production.

If Drupal ever gets solid on-demand loading of code, the importance of "chopping up" modules will decrease significantly. It is annoying, for end-users, though - I will give you that much :)

"That's why the good lord invented hook_menu/form_alter()"

You sure it was good lord? :) More seriously, though: just disabling "default" UI module leads to much less fight-the-default-behavior code. If you recall, one of the main developer complaints when developing with Drupal (esp. from developers trying to use it as a framework) is that they spend most time "fighting" default Drupal behavior and not enough - developing useful code.

P.S. To clarify: I am playing a "devil's advocate" to try provide most possible arguments in defense of the original pattern, but I am very happy this discussion was started and would wholeheartedly welcome an alternative solution that satisfies all original requirements.

OK, but one requirement missed in the solution in #9 is if you want the module to be active on dev/staging/integration, but don't want it in production, which could be the case for views_ui. It's very easy to have varying set of enabled/disabled modules in prod/elsewhere, but it would take quite the amount of code to add that check to the three-liner, eh?

I've thought several other modules could have used a separation of UI and API, most notably Features. IMHO if the main module file can be kept bare enough, without a stupid include('mostofthemodulecode.inc') in the top of the module, and there are sufficient permissions to stop people shooting off their feet on a production site, then I'm all for keeping them as one. I think a step towards this is to stop using modules like adminrole and secure_permissions to automatically assign all admin permissions to specific users for the production site so people aren't tempted to play around.

One benefit of separations is *forcing* developers (generally ourselves) to expose APIs for all operations. Look at the mess of copy and paste snippets that infests so many `.profile`, `.install`, etc. that resort to poking at core tables because the only alternative -- if there is one -- is programmatically submitting forms.

Writing an API and then coding against it means that anyone who needs to interact with our code can do so without having to duplicate bits and pray the schema doesn't change and that the API and UI are both first class citizens.

BTW there is an effort underway to add a hook for defining variables (#870040: Provide hook to allow modules to define what variables they own) as a patch for Strongarm (if I can ever get back to working on it), which is kinda like hook_settings (I never used 4.7 so ICBW)?

The issue @thsutton and @DamienMcKenna raise is an important, related one that should be considered as part of the final solution. Modules often implement logic in form_submit() and/or finish their "API" functions with drupal_goto(). Both of these create significant problems down the road.

Not sure how much of these can Core enforce and how much should just be captured as a set of Drupal Design Patterns for module developers?

I'm with chx here, mostly.

The reason this is done is, as thsutton said, to force an API upon developers. However, a model where the API is not forced upon developers, as again thsutton pointed out, is what most of core is. Because core doesn't always have a clear API/UI separation, we end up having to execute sql queries ourselves and do all sorts of messy things to get around that very fact.

I don't think token is such a great example. For token, nobody installs just token module itself, because token does nothing (just like views.module). Yes, it has a theme function, but I would argue that that's actually part of the API, providing an underlying layer so others (like pathauto) can show their token forms.

irakli, I don't think core can or will ever be able to enforce things, and it could be argued that that's a good thing - for example, if you are on a really quick timeline and just need to push a site out fast, we've always, and should always, leave the doors wide open for doing whatever one wants (for example, not enforcing a split).

I do agree with webchick, however, that sometimes the split can be excessive (i.e., I'm not sure it's necessary to have completely separate modules), but I can't really think of a better way of having two pieces of code completely separate (API and module). One possibility, which is something we could experiment with in core, is a .api.inc file, or for more complex use cases, like views, an api/folder.

But yes, with proper code-splitting-out (hook_hook_info, for example), it shouldn't be too hard on memory to have it in one module. However, I don't think "hiding the interface" is most people's problem. The interface is easier to hide. Harder problems to solve are API/UI separation and memory.

Um, separate files and separate modules are not the same thing. When you separate into a UI module you can choose to remove the UI module and then that code never reaches your production server. If it never reaches production, it does not have to be audited for exploits, come up in grep results, and so on. Restricting by permission is hardly an adequate substitute for code not being present all.

And the sea of checkboxes can be cured by a decision of hiding API modules and letting the dependency system install them. This idea certainly needs more work but i think it can be worked out. After all, if you only use the UI then just enabling the API module is pointless. If you are a developer then your install profile, drush en and so on can install it without the UI.

More, if you really want I can show how your install file can hide itself from the module page even in Drupal 7. But I warn you: that code is FUUUUUU not pretty.

I almost started to agree with webchick.

And then I remembered that this is merely a side effect of the modules UI sucking, and eliminating a view extra "foo UI" modules won't make it suck less. It is clear that I really need to get off my butt and spend some time writing a proof-of-concept UI for the modules page. Particularly if I can team with one of the UX folk to see if I can get someone to visualize what I want to accomplish.

I've always thought that if there's a good chance that people will want to use the module without the UI, either because the configuration is completely features-driven or because they provide their own, the split of UI and core functionality is entirely justified.

For the oauth module I actually did a double split: provider UI and consumer UI. As there are different needs based on whether the site is acting as a provider or consumer (or both). In many cases modules using oauth roll their own UI to create consumers etc and initiate authorisation processes.

I definitely think it (separating UI/API via separate modules) is a good idea (even though we can all agree that every good idea can be taken too far).
You've mentioned Drupal Commerce as an example, and the reasoning behind it is sound, previously there was a need to develop whole alternate UIs for Ubercart (I believe someone from Commerce Guys knows that story better than me), and writing a new UI module is much nicer to me than doing three thousand overrides.
As others have said, this forces the developers to think about having an API, and makes the resulting code clearer (I need to understand a codebase, I start from the API module and go from there).

In any case, it's an emerging best practice. Having a discussion over best practices and writing the conclusion down would certainly benefit new and existing developers, since people are still figuring out how to organize their code (which files, folders, modules, etc). This figuring out mostly comes down to copying successful examples (for example, my gsoc project had a similar folder stricture to Drupal Commerce, which then again takes after CCK...)

And yeah, I agree that the modules page UI sucks. I see that we are slowly identifying that as an underlying concern.
And huge module packages (like Ubercart, or Drupal Commerce, or whatever) really show that problem very well.

Totally agree with webchick. If each module start to make its own UI module, the module list become too long and many disagrements are coming ...

I like the idea that UI is separated from logical code. But doesn't make sens for me to have an "image_cache" module and an "imagecache_ui" : they are both imagecache, the same entitie, so the same module conceptually.

Ui should be separated from logical code, but should not be another module with another name...
But maybe a new way of creating an UI for modules could be imagined, in order that we can more easily plug / unplugged an UI, in a way that would be the same for all modules ?

I mean : if you make a module_ui just in order than we can uncheck a checkbox to desactivate UI, it's surely possible to code this checkbox WITHOUT creating a new module, because of all negative points that webchick is talking about.

@merlinofchaos. I agree that the source problem here is the module page UI.
What about module_filter ? It's a MUST for me since I discovered it. Simple and functional.

I think the heart of the issue is really bojanz' closing remark:

And yeah, I agree that the modules page UI sucks. I see that we are slowly identifying that as an underlying concern.
And huge module packages (like Ubercart, or Drupal Commerce, or whatever) really show that problem very well.

IMHO there are enough compelling points to steer the discussion away from 'separating UI and API modules: yay or nay?', and head towards what we can do to improve the module selection page's User Experience.

anything that helps strengthen the API is a good thing so at least splitting out API and UI code is a plus

Moshe makes a very good point in #19 - removing UI code removes a lot of potential vulnerabilities so separate modules has a strong benefit

I agree that the module page UI isn't perfect, but I don't think it's the source of the problem. With the increasing popularity of Drush, Drush Make, distributions etc, the module page is not the only way to enable modules anymore. I see the module page less and less, but also with Drush, the separation between foo and foo_ui can be annoying. When I want to try out foo module, I can do:

$ drush dl foo
$ drush en foo

...only to find out that nothing changed on the surface. I'm not sure which method of separation is the best, but some kind of standardization would allow Drush to enable the ui automatically, or notify the user that there is a UI component to be enabled separately.

It would be good to start early with redesigning the modules page for D8. With the blocks admin ui it's one of the big remaining UX stumbling blocks in core. There's quite some prior art around for it and I even remember an outline that matched merlinofchaos's ideas quite nicely (at the time :-)

Drush should then have an option to enable a group of modules, no?
Even if you had half the number of modules than listed above, it's still painful, and you should be able to enable them as a group.

The API/UI separation issue is interesting and has validity, but I don't think we should argue based on the fact that drush is missing a feature.

But it does nothing to stem the tide of 'Let's recommend as a best practice doubling number of modules a site has enabled at any given time."

This, however, is a good point.

@webchick The latest drush HEAD has a cool new feature: you say drush dl views and it tells you Project views contains 3 modules: views_export, views, views_ui. .

There are a lot of very legitimate concerns here, but basically a bogus conclusion.

The legitimate concerns are:

- The module page doesn't scale to the number of modules
- It hard for a user / developer to identify the set of related modules, both on the module page and with Drush
- When you enable an API module, nothing tells you that there is an UI module that you might want to enable

The bogus conclusion is that we need to merge UI and API modules. We don't want that. It would be horrible.

When you merge everything in one place, you get Ubercart. The same type of flawed reasoning lead to Ubercart being just a monster project: "users have trouble downloading several modules, conclusion: we need to limit the external dependencies and reinvent the wheel as far as we can". The real conclusion in that case would be "we need to make it easier for users to download several related modules, either by promoting distributions or by having a nice UI from where you can download and install a module". Both of them has been done.

The real conclusions we can make here are:

- We need to redesign the module page so that it scales better with the number of modules, which is going to be even larger then it already is (especially because there are no performance concerns anymore with the number of modules, thanks to the hook implementation cache)
- We need a way to identify "related" modules (#328932: Modules and partial dependencies - enhances[] and enhancedby[] field in modules' .info.yml files has some background and discussion)
- We probably should indicate to the user that there UI modules are available if he only enables the API module

For UX the default behavior would be that 'install Views' means installing and enabling both the API and UI parts. That there even is a distinction between the two is 'advanced' in the sense that it becomes only relevant to know about it further down the road in your usage. Anyway, #538904: D8UX: Redesign Modules Page says hi.

API and UI should be separated. I think it will open the door for more creative administration pages, by allowing other developers to design what they feel is a better UI for any given module. If one of the major concerns is the module admin page getting too cluttered then I feel a solution to this would be to use a JS show/hide after checking to enable the module expand to show a list of available UIs for that module. This makes sense to me because for a UI to be enabled, it must have a dependency, so there's no need to see this option until that parent module is marked to be enabled.

To developers APIs should be separated from UIs. To point and click users Drupal doesn't make sernse without a UI. Some APIs need specific UIs to function (Views versus Views UI or Simpleviews). Some APIs are just required by other modules that offer features. Next to that it's not always clear whether a module is an API or not.

There are also reasons for separating API and UI within a single project:
- It helps keep the API a real API. Too many form submit handlers contain code that belongs in API functions.
- If a module is used as an API by another module, its own interface won't make the interface unnecessarily cluttered.

The added bonuses are less menu items and less hook invocations, for instance.

so yeah, 2 things here.

First and foremost (as others have mentioned) the modules page in unwieldy and has been so for a long time despite efforts to the contrary. A quick fix against this for D8 could be to provide a "projects" page as the default (allowing individual modules to still be selected on a different page) and this projects page would simply enable EVERYTHING the module developer felt was relevant to the project (i.e. if merlinofchaos so chose, he could enable views and view_ui if a user selected to turn on the views project). Down this path be some dragons though because of "configuration" i.e. in the case of a system like Drupal Commerce, it might be nice to be able to have different project configurations that one could just click and be done (i.e. you can only pick one of several configs for a project... further customization would require visits to the actual module page and a knowledge of what you're doing).

This leads into my second point nicely which is install profiles and the matter of a core size I will not specify here. I'm not attempting to turn this into THAT conversation, but it's worth mentioning that from a "distro" view point, this is rather trivial to fix. And while that doesn't solve the problem of developers trying to learn and use core drupal+contrib, I'm questioning whether that's something that can be fixed. I personally view *_ui modules as a bit of a godsend because when I look at the core module, I don't need to mentally apply a filter against everything that's UI, and if I look at the *_ui module (very rare) I get to see a proper implementation of the api. YES this is what I should see if they were all one module, but I'm afraid the UX "wtf" is worth the DX "thank God", in my opinion.

Eclipse

I've never been referred to as the good lord before, at least not in public. :-)

Eliminating UI modules was one of the reasons I originally pushed for file split in page callbacks. It wasn't a major impetus behind hook_hook_info(), but I agree we should be using it FAR more than we are now. (I find it amusing, actually, that webchick is endorsing it here as she originally was against it on DX WTF grounds. :-) )

To be sure, there are valid use cases for splitting up modules. I've some custom modules for client sites that really should have been split up more than they were. However, in most cases I've seen the UI/API split in modules becomes quite arbitrary. For instance, I cannot immediately grok why some Field API hooks are part of field.module and others part of field_ui.module. I'm sure there's logic to it, but not logic that I can really follow. The result is that the Field API is split between two modules (since the UI is a key part of the Fields API, for better or worse, with all of the widgets and settings it uses).

The performance and memory concerns of bigger modules can be solved by proper use of hook_hook_info(), the "file" key for menu and theme hooks, and class autoloading. There's no longer any need to split modules up just for that, certainly not enough to call it an even remotely "best practice".

The code organization concerns I'm iffy on. When you can separate code into files within a module without having to add more modules, this argument is considerably under cut. OTOH, it makes sense if you're leveraging an API. Eg, if you're defining a new entity type and 3 bundles for it, it could make sense to have 4 modules: One for each. Or it may not; that depends on just how big and complex each of those are. The deciding factor for me, really, would be whether I'd have to put different code for each bundle into the same hook_node_load() (or whatever) with a switch statement. That to me is the code smell that I may want to split the code into separate modules, when hook implementations become uncomfortably comingled otherwise. If they're all effectively the same, then leave them all in one module.

The "force me to make an API" argument I don't fully buy. If someone is disciplined enough of a developer that they will make that clear UI/API distinction, they can do it in one module namespace just as well as in several. If they're not, then an API/UI split is going to get violated sooner rather than later. To be sure you always want a separate API from the UI, but I don't think that crutch is very useful in encouraging that.

The "make the UI swappable" argument is, I think, valid... if and only if the UI truly is swappable. That's very often not the case. I can only think of one "alternate UI" off the top of my head anyway, which is Simple Views. (There may be others I don't know about, of course.) In practice I think this is extremely rare, so I don't know that it's a valid argument in most cases.

So will a better module page fix this? No. It definitely helps some problems. But it does nothing to stem the tide of "Let's recommend as a best practice doubling number of modules a site has enabled at any given time." Seriously, ugh. :(

I disagree with this. A better modules page is not just about eliminating checkboxes. A better module's page is about coming up with a new workflow that will allow us to manage our modules in a better way. In a sense, we can learn from some of the things drush is doing, and we can learn from the problems you have.

One obvious problem is that we need to make it simple to activate a bundle or multiple bundles of modules. Now, across projects this may not be so easy (beyond dependencies) but within a single project, it seems like I should be able to set up one or more 'bundles' of modules. i.e, in CTools, "activate CTools" and "activate Page Manager" and "activate Views" will turn on the modules that I most commonly assume will be turned on. We'd need to do that in a way that's not too bad to change the defaults, but if you're happy to go with the defaults, click a couple of times and go.

It seems like we could note this in a .info file somewhere or something along those lines. Drupal understands projects -- I think we can expand on that.

The "force me to make an API" argument I don't fully buy. If someone is disciplined enough of a developer that they will make that clear UI/API distinction, they can do it in one module namespace just as well as in several. If they're not, then an API/UI split is going to get violated sooner rather than later. To be sure you always want a separate API from the UI, but I don't think that crutch is very useful in encouraging that.

Yes, you always want to separate API from UI. However, if there is the possibility to turn on a module without UI support, developers will be more likely (I think) to separate the two. We shouldn't separate API and UI modules because of this, but it is a consequence of such a separation. A consequence that will benefit code quality.

Yes, we can do a lot more with .info files. And we do not have to wait for D8 to come up with something. Some time ago, Dave Reid and I already wanted to do #624848: Allow to retrieve a list of modules defining a certain .info file property for D7, because Dave has a nice idea in http://drupal.org/project/module_supports, and http://drupal.org/project/admin_menu similarly implements a feature that allows to disable "developer modules" in one feel swoop. Right now, that feature is parsing the module list for an "_ui" suffix in module names and auto-suggests those (however, still configurable). Short-term, I'll replace that with scanning for the .info file property "tags", as discussed in #624848.

BTW, the argument that worked for me was the "I don't have to have this code on my server, meaning I don't have to do security sweeps on it" was the one I bought when I kept the Views UI separation in Views 2. My original plan was to remove the separation.

I don't feel I have much to offer by way of suggestions other than to +1 Earl's comments about simplifying installation by allowing full project installation instead of requiring module-by-module decision. I only have some bit of our thought process that may inform any possibilities moving forward...

As others mentioned above, we wanted to allow a fully pluggable UI, whether it gets used or not. We have a very strong suspicion this will happen, because we've made the intentional decision to privilege developers and to depend on distributions / Features. Our goal, then, is to have a palette of modules that can be mix-n-matched, discarded, and replaced to develop tailor-made Commerce installation profiles / Features. We could bake the default UI for API modules into them, but requiring developers to stay on top of any changes to the default UI so they can overwrite / rewrite where necessary doesn't privilege them. This creates the UX problems outlined above, but we'll privilege site builders / n00bs at the installation / Features level.

I get Xano's point in #43, but at the same time I'd say we never would've constructed things the way we have if we didn't start with this mindset. i.e. we have entity forms that can be dropped anywhere and no module has to work against redirection assumptions, override buttons / submit handlers, etc. This is because our forms go in the API module and are instantiated by the UI module using a specific form ID that is itself then used to add buttons and redirection as necessary so the form plugs into the default UI's IA just fine. Now these forms can be embedded elsewhere, like if we want to use a modal to allow for product creation from a node form, or if we need to instantiate the form at different locations for different types of users (think peer-to-peer sales vs. store-to-customer sales in the same site). We could reverse engineer this and combine everything into one module now that we've learned our lesson, but then we lose the simple pluggability that privileges the developer at the module level.

[This paragraph doesn't make as much sense as I would've liked:]

I guess I'm not totally grokking webchick's comments regarding performance. If checking for a hook implementation is cacheable and is as simple as a function_exists() when it's not cached, I don't know where the performance hit is [ahh, I'm guessing it's in including files]. Is it never truly cached? We are definitely using files to break apart the core module file, and the default UI modules tend to just be page callbacks, form alters, and default Views anyways... I will say that I didn't use hook_hook_info() on purpose because it didn't allow me to specify a directory that contained my include files. (This was due to another of our principles involving a rigid module file structure.) I don't think that really affects us, though... we're using it properly for the core hooks and Rules hooks that allow it, and in our hooks that could leverage it we simply added a hook_menu() style file key. So... I'm interested in working out the fix here... I just don't really understand where the performance hit is. : )

So... I want to maintain my ability as a module developer to privilege the developers who have to interact with my modules. Right now our only options for privileging users are to make tidy installation profiles and Features modules, but neither of those solutions are totally satisfactory. I think someone mentioned this above, but giving projects the ability to specify configurations of component modules that make sense would be one thing... a little better than just enabling every module in a project, perhaps there's just a key in a module's .info file that defines it as one of the default modules that should be enabled if the whole project is enabled.

And now refreshing the page before posting, I see that sun just pointed out the same thing. Hope you enjoyed my long +1.

I had a really huge reply, and decided ultimately it wasn't worth actually finishing. I'm passionate about this issue, but mostly as it related to drush_make/profiles. I honestly believe that's the solution for our common user, and beyond stating that, I'm going to leave that portion of the topic alone.

@webchick:

You seem to want core to provide some sort of solution that can help here, my BEST suggestion in this regard is to take a look at ctools export_ui which provides a simple methodology for implementing a ctools created interface for your module (which includes all the nice import/export goodies etc). It's OOP and allows you to override the UI that it provides as necessary by providing your own class that extends its base class. It's very savvy and I'm personally in love with it. On that note, it means that if you DO separate your ui into another module that modules ends up fairly small compared to what most UI modules would probably look like w/o it... which may be an argument for making them one module, but whatever... Your performance concerns are mostly targeted at D6 which has a limited life-expectancy at this point, and probably won't end up with many of the new modules that will embrace this backported (drupal commerce for example, has NO intention of backporting). Furthermore, developers really don't have a file bloat issue because, properly separated, they should end up with about the same number of files in any event, and the separation can actually be helpful in narrowing where they need to look if they have any sort of guess about what they're looking for.

Eclipse

If you drush en -y views_ui it will pull in views. That's what I said before. You do not need to enable fifty modules. One or two is enough. We can even do what Debian does and have empty modules that are used merely to enable the rest via dependencies. yes, there is a problem but I think we can solve it without nuking the nice separation in UI and API modules.

We can even do what Debian does and have empty modules that are used merely to enable the rest via dependencies.

There goes my idea for config profiles... :-P

Can't we add some sort of state to modules? Like "present", "active", "active & UI"? "present" would mean Drupal has confirmed a module is there, so it can be used by other modules as an API. "active" would mean its hooks are invoked, except for hooks that add interface elements (hook_menu(), hook_block(), ...). "active & UI" means all hooks of that module are invoked.

Is it too simplistic (or confusing) to use a tripartite breakdown (Drupal's tri-force?):

1. At the Drupal module level, we privilege developers.
2. At the Drupal-as-a-product level, we privilege technical users.
3. At the Drupal distribution level, we privilege users who acquired Drupal as a solution to a specific problem.

To assist those in category 1, we allow modules to go as granular as they need to be. If there's a resulting performance issue, we can find a technical solution... but these people probably aren't the best to fix the usability issues.

However, module developers in category 1 (especially those managing projects comprised of multiple modules) would still have patterns to follow that keep those in category 2 in mind. If I can define patterns of use for the modules in my project that are little more than instructions on which modules to enable for various use cases, then I have something as a module developer that lets me simplify things for people in category 2 without needing a full-blown core Features solution. This would let me download the Views module and enable all components of the Views project using the "Default" configuration - Views and Views UI. The problem is where does this metadata reside? I need it before the module is installed, so .info files seems the ideal solution, but are we then going to have to define miniature configurations much like we do for packages and trust that no one else uses the same configuration name as my project? The alternative is as chx mentioned above to just include a module in my project that installs the various component modules and performs any necessary initial configuration... but then it's just a Features module.

When I consider those in category 3, it helps developers of said solutions to have a more granular / pluggable set of modules. It doesn't really matter whether or not core has some simple pattern for enabling sets of modules, as this is going to happen via the installer anyways.

Concluding thought... it seems if we want to help casual users who expect to grab a download from d.o and go, we need a core method of enabling a predefined set of modules within a project that contains many different modules. I think this point was raised above several times, I just don't know how much power you'd want to pack into this feature before you just roll Features into core, and I don't know where the metadata for these module configurations should reside. At its simplest, we'd need project level awareness for projects that contain multiple files, but this would need to be separate from package level awareness... do you have a separate project .info that lists a group of modules to enable? Or do you just add a configurations array and let each module specify that it's a part of a particular project's configuration? I can't even see that working without being too confusing unless there's a way to identify all the modules that come from a particular project... and now I'm talking in circles.

I think you're right, webchick, that is something we should be concerned about. The four UX principles from www.d7ux.org come to mind:

1. Make the most frequent tasks easy and less frequent tasks achievable.
2. Design for the 80%
3. Privilege the Content Creator
4. Make the default settings smart

This issue seems to be about 'separate UI modules' vs. 'usability for non-technical users', but I think that there really isn't a "vs." here, and those two can be combined.

Non-technical users don't care about the technical solutions under the hood. As far as they're concerned, we might as well create a separate module for every single function - as long as it works. So if module developers and advanced users choose separate ui modules (and they have good reasons to), all we need is a non-technical way to interact with that separation.

I think that the concept of shipping a project with a couple of sensible defaults (eg. Views project = views: on, views_ui: on, views_export: off) is the right way to solve this. The most frequent tasks would be easy, for both non-technical users (the 80% who use the ui to enable the "smart default settings") and for advanced users (who can enable/disable/replace/remove modules one-by-one using the ui or drush).

What @marcvangend said could be easily expressed differently:

enable[views][] = views
enable[views][] = views_ui
enable[features][] = views_export
enable[views_export][] = features
...

As visible, this quickly starts to make no sense. So, let's have another look at the already proposed attempt:

views.info:
tags[] = views
tags[] = api

views_ui.info:
tags[] = views
tags[] = ui

views_export.info:
tags[] = export
tags[] = ui

features.info:
tags[] = export

Now, go ahead and disable all "ui" modules. Want to export stuff into code? Enable all "export" modules. Just want Views to work? Enable all "views" modules.

But where does the metadata come from for these tags before modules are actually installed? Do we really want tags to be cross-project? Backup & Migrate very well might use an export tag but mean something completely different. We have to have some translatable title / bit of descriptive text that we can use to built a usable UI for these... and that info would normally come from some module's hook.

Sun, afaic, you're already making it too complicated for the non-technical user. You're turning my example of single-project sensible defaults into a poor man's features; that's not what I meant. To answer Ryan's question: I think this does not need to be cross-project - we have features, install profiles and drush_make to do the fancy stuff.

IMHO, usability is all about presenting just enough choices at the right moment. Let's think of this as something for your 17 year old niece. She wants to build a basic site and reads that she needs this thing called "Views". So she downloads it, FTP's it to her free webhost (these steps alone are already more difficult than installing a plugin on WP!) and wants to turn it on. Now the question is: What does she see on the module page? Which choices will Drupal present to her? I think the module page should tell her: "Here is vanilla Views, just click to enable. Oh, by the way, you can fine-tune the separate modules over here." Presenting other packaged presets ('all export', 'all ui') at that moment would only make things more complicated.

Here is vanilla Views, just click to enable. Oh, by the way, you can fine-tune the separate modules over here.

Well put. Lets just keep in mind that the way things look doesn't have to be the same as the way things really actually work. I mean, in OS X, you double click an application icon to launch the program. Control-click it and you get an option to 'Show package contents', and oh look! Even a Mac application consists of a buch of text files in multiple folders, mashed together :-)

So, I'd consider it a bad path for core to leave all the UI glueing up to distributions etc. for the reasons webchick outlined in #51. Code-wise, make things as granular, abstract or generic as you want (I have no say in this of course, but that's my understanding of a developer's idea of 'making things better'). For usability, the more complex the application, the simpler the interface has to be. (queue Dries's "I think we can do both" :-)

Recently I was thinking about wizard configurations, especially for large projects like Ubercart and DC.
E.g. on the Modules page you have only one line for 'Ubercart' and corresponding 'Install' button. You hit 'Install' button and it redirects you to 'Ubercart Installation Wizard' which will guide you through all the configuration settings. The great benefit of this approach is that the 'Wizard' can handle all configuration in one place. Furthermore, for the experienced users and developers we can provide 'Skip wizard for manual configuration' checkbox.

If module didn't implement wizard (in hook_wizard() or something like that) then hitting 'Install' button will just install 'default configuration' in the regular way.

However, this requires users to manually press 'Install' for each project they want to install, but this is just concept and needs more understanding.

UPD: This is also opportunity for module developers to organize structure of their projects. Once again example with e-commerce: each page of the Wizard can represent one structural block of the project. First page is for products configuration (prices, attributes, stock levels, etc), second for checkout, third for payment gateways and so on.

Also redesigned Modules page may utilize something like Vertical Tabs for module groups. So in 'Ubercart' group we'll have Ubercart itself and any other contributed project extending Ubercart.

So from this point of view redesigned Modules page must be more about 'projects' rather than 'modules'.

an.droid: I believe Dmitri once proposed something along similar lines. His idea was related to install profiles becoming modules, and then the setup screen for that install profile is essentially what you're describing for hook_wizard(). The first part of that happened, but not the second. :-) There may be something worth looking into there.

Anyway, solution I described in #58 is mostly view from the user perspective. :-)
Developers requires additional tools. I like the idea of Debian meta-packages mentioned by chx in #48. BTW, install profiles (or wizards) can be considered as some kind of meta-packages.

About separating API and UI without separating modules.
AFAICS, separating back-end and front-end paths in different hooks in addition to hook_hook_info() and 'file' attribute makes it possible to move all UI code in include files.
So I think that hook_settings() with 'file' attribute like in hook_menu() + hook_hook_info() should do the job (if I correctly understanded description in #17).

My thoughts are pretty similar to marcvangend. I think there have been a few good reasons for separating the UI (security: since views UI isn't needed in production; pluggability: swappable UIs in commerce), and the solution is to find a way to maintain the flexibility for developers to decide the implementation of the module but hide the complexity of the module relationships to the users who won't care (views_ui is probably most commonly still enabled on sites, many users will never create a new UI for commerce).

Currently every module seems to be treated at the same level by core, when that doesn't seem to be exactly true with modules like _ui / _export / etc which only exist as interfaces to an api module. The main module could specify a set of recommended extension modules, and then a simplified interface would be possible for novice users that would operate with sensible defaults (you're enabling views? you probably want views_ui too, but let me know if you don't). The picture in my mind is that modules that extend others are listed in a collapsed fieldset below the main module; if the module only extends others and provides no real functionality of its own it would no longer be available as a top-level item in the module list. Advanced users could view the modules list as a flat listing without recommendations if they choose.

I'm not sure of the best way to successfully leverage this pattern within drush. My current thought is to have drush ask the user if they would like to install the default related modules, with an option to auto accept/deny those suggestions.

I don't want to get too far off topic on wizard interfaces, but I think there is potential to make it workable for both new and advanced users. I would propose a slightly different workflow, wherein the wizards are presented in a list after enabling the module. A new user could then progress through each wizard in sequence while an advanced user could choose to skip some/all of the wizards or rearrange the sequence to complete the wizards in. This could make initial site configuration for a new user substantially easier than having to find each modules settings pages individually after install (the wizard could simply be the module's configuration page), but advanced users who know where to go aren't required to change their current methods (there's no 'use wizard' checkbox to remember to de-select, they don't have to enable each module individually).

(It's also somewhat unintuitive, but if the module breakdown was foo / foo_api (instead of foo / foo_ui), enabling foo (the ui module) would go "hey, you need to install foo_api for foo to do anything!". This would stop having to remember that there is a foo_ui module that needs to be enabled too, but could be a huge pain to convert to and confuse many more people due to the current foo/foo_ui precedent)

Fact: Drupal beginners identify a module with the name of the project (even I do) AND expect to see the UI when is enabled.

What if we re-think the name of the modules, and we use the dependency mechanism that we have already?

If

Then

I am aware that can be confusing at the beginning, but... isn't it more intuitive? And it will work for D6 and D7, and drush in the meanwhile we think for the *perfect* solution for D8.

This only would solve one point of the webchick's list. But it's the main point because affects to all users. The rest of points are more for developers/advanced users. (I will say enabling Drupal Commerce is not a "frequent task").

I said Views.. but same happens when enabling "Context", "ImageCache", "Rules", etc. When you enable any of these modules DON'T expect to see the interface... you need to enable "context UI, "Imagecache UI", "rules UI".

EDIT. Sorry, I didn't read it but it's true that #61, gapple's last paragraph shows the same idea. Good that we agree on this. :)

@corbacho - thats a very good idea IMO. We just have to figure out how to handle the few modules that already split using the 'UI' suffix. Seems non trivial to change names.

I agree with #63, but let's give @gapple some credit too - I believe he meant exactly the same thing in his last paragraph.

If we get to consensus on the idea that the sensible default should be 'everything on', especially for the (initial) ux on the modules page then we have ourselves a actionable design principle we can start exploring. @corbacho et al: very good idea indeed.

I feel a summary of the discussion so far would be good to have now, anyone up to do that? Good stuff in here…

Yoroy, I don't think that the sensible default always equals 'everything on'. Take the D6 version of CCK for example; you don't want to enable everything by default. Just content, text and number is a good start, and you definitely don't want to confuse a new user with missing output (and the sea of checkboxes) caused by content_permission.

Yeah, I generalized too much. "Enable the API, UI and the 80% feature set by default". Better?

Yeah, much better. It's just a couple of words, but a major difference in code. If we go down this road, we'll need a way for project maintainers to specify which modules should and should not be enabled by default. That's an API change instead of a "enable all modules in this project" button in the UI.

Changing which module has the extra word isn't a real solution... you could split the modules up as much as you want, and if you have a place in the UI to enable a default configuration on a per-project basis, it doesn't matter what they're called. We're just trading one confusing step for another if we just swap API / UI, as users will still have to figure out which ones to click on, may get a message that says something about not enabling enough modules, and still can enable the wrong one. "Oh, I like the sound of API... I'll get that one and install the basic one later if I need it."

As was said above, the d.o project is what most people expect to be downloading and installing, whether they know the single project contains 1 module, 5 or 20. If my project is going to have to specify which modules to enable for the default project configuration (.info file?), the names of the modules are irrelevant and the better UX would seem to be a project level enable.

#70: agreed. And that 0.01% of multi-module projects could still use chx's suggestion from #48: shipping with a package module which contains nothing but dependencies. cck.module, anyone?

But we still wouldn't be solving the problem of "Ok, which modules in this package should I start with?" for something like CCK (or even parts of Commerce like Cart / Checkout) where the module split isn't just on the basis of API / UI. In other words, it might help a bit for some modules to switch module names around, but it's not a full solution... and in pressing for a more robust solution, I think the names of the actual modules becomes a moot point. I'd still push for a "Features-lite" approach at least to enable all the recommended default modules for a project at once... we hide the complexity of multiple modules within a project on the Update Status page, why not put it a level deeper on the Modules page?

Also, just throwing this out there for feedback, it seems like I'd then be expected to change every function like commerce_product_load() to commerce_product_api_product_load() - I suppose I could just not change the function names at all (except for hook implementations), but that's not really the general naming pattern for module API functions (i.e. [module]_[object]_[verb]()).

I'm not sure I agree with reversing the pattern, actually. :-) That's a lot of work, and upgrade headache, to just whitewash over the problem.

I'd counter-propose what I think someone else hinted at above. We've for a long time wanted to have more robust dependency handling than just required/not required. Things like required/recommends/suggests. (Most Linux distros have a well-evolved set of definitions here.) If, say, views.module recommends[] = views_ui, then you can totally enable views without the UI if you want to, and it works fine, but the UI (either Drush or the modules page) can easily say:

"Oh, you're enabling views. You probably want view_ui and advanced_help, too. Want to enable those as well? Y/n"

(Without the obvious Clippy overtones it would be easy to introduce, of course...)

That would be very helpful as well for modules that, say, benefit from but don't require getID3, token, advanced_help, etc. The relationship there is similar to that for UI modules. "You can run it without these, but you probably want them, too."

Take a look at what Facebook or Flickr (or other OAuth-like sites) do when you enable a new Application.

[Application] would like your permission to:

* Send updates
* Look at your photos
* Look at your list of friends
* Steal your bank account number

Do you want to do this?

I even remember a time in Facebook's past when these options had checkboxes next to them. Now it is more of an all-or nothing.

This goes a huge way towards UX if modules did something even a little bit like this.

Views is a module that [rocks my world]. With Views, you can:
* Enable other modules to use views' features. You probably don't want to enable this by itself unless you know what you're doing.

Views also contains a module called Views UI, which does the following:
* Create lists of nodes, users, comments and lots more stuff in Drupal
* Override Drupal's displays, like the home page and Taxonomy Term pages
* Start with a set of Default Views to build commonly used views, and override them
[x] Check here to turn on Views UI

Views also contains a module called Views Export, which does the following:
* Allow you to move Views definitions between sites
* Allow developers to store Views definitions in code
[x] Check here to turn on Views Export

Just wanted to point out some issues and projects that relates to what Crell pointed out in #75.

#328932: Modules and partial dependencies - enhances[] and enhancedby[] field in modules' .info.yml files discussed that Drupal should have support for Debian-like recommends, suggests, enhances and maybe even conflicts - but it was said to be a pure Drupal.org problem and have nothing to do with Drupal itself - something this issue might prove to be wrong.

There's also a module for this which has adapted its own keywords: http://drupal.org/project/module_supports In its issue queue there's a related issue to the above issue: #617068: Give clear semantic meaning for each directive

I really do feel that Drupal should support these type of keywords in the core module system.

Date is an example of the other way: A new user is more likely to think "I would like CCK fields and widgets" (date) than "I would like an API that can be used by other modules" (date_api). I think it has a bit of a different logical structure than views & commerce though; Date API has it's own interface for defining formats, and CCK fields aren't quite the default interface for Date API. You also certainly aren't going to disable date in production, or swap it out for a different implementation.

I wouldn't want to force a foo/foo_ui to foo_api/foo conversion upon developers; as Crell reiterated it could be a lot of work, and I think it could be confusing in its own way. foo / foo_ui makes plenty of sense logically, it's just not very intuitive for a user with the "I want to enable foo" mindset. I think we should be able to make a solution for D8 that allows the module developer to make the best logical decision on structure, with an interface that is intuitive for the user no matter what the developer chose. @Voxpelli mentioned module_supports, which looks like a more extensive solution than just a 'recommends' array as I was thinking of, that could make this interface possible.

Re-reading @corbacho's post, it doesn't quite sound like it was actually the suggestion, but I first read it as "Keep foo/foo_ui, but tell the user they are Foo Core and Foo". This would allow the developer to keep whichever machine name makes sense, but suggest to a user browsing the modules page that they need the UI module over the API module. The challenge is not making the 20% always go "WTF? Why are these named backwards?" when browsing the modules page or looking at the code just so that the 80% check the right box the first time before they actually understand the modules.
I think a good test here is as @webchick described: if someone in a class was distracted by their email when being told to enable foo & foo UI, would they make the right choice when trying to catch up? Could another student tell them "you need to enable Foo", and they then make the right choice?
While it doesn't change the challenge in Drush, I don't think that's a concern for the 80%. While there will still be "WTF? Where is the config page?" issues, I think people using Drush are capable of dealing with there being a separate UI module (and can maybe improve Drush too if it's a frequent pain).

Agreed with @gapple - this is a pure UI issue and should be kept and treated that way. No need to rename or change any .module code and actual functions. Technically, Views module provides the Views API, so the "views" namespace is correct. Merely user-interface-wise, it doesn't (always) make sense to enable the API only. Thus, merely changing .info file strings from "Views" => "Views API" and "Views UI" => "Views" would already help. That said, I constantly had to tackle support requests of users that already understand that an "API" thing is not for them. So renaming "Views UI" to "Views" might be unnecessary overhead. E.g., when "Wysiwyg" was still called "Wysiwyg API", *lots of* users did not even think about even remotely consider to look at and read the project page. Thus, the project title had to be renamed to "Wysiwyg", even though the "API" suffix was technically more correct.

I really dig the 'suggested' keyword (or something similar) in a module, so that when you activate a package, it activates all of the suggested modules automatically.

Re #80: I like that idea too, but the thing that a 'suggested' keyword does not answer, is what "activating a package" means. Packages do not have their own .info file. We could build in the assumption that the module that carries the same name as the package, is the 'representative', but in case of CCK for instance that wouldn't be true either.

Re #81: I don't think the 'suggested' keyword should be limited to one module in a package, but it would probably make the most sense on the root module in the majority of cases. I see it less of 'activating a package' and more of activating soft dependencies to help new users; this might mean that foo_sub1 suggests foo_sub2.

--

A small step to start with for improving the new user experience could be having API modules do a check for the UI module when they're enabled. If not, they can post a message "Foo module requires the Foo UI module to be enabled in order to be configured." It's not great ("Well then why didn't you enable it for me?"), but it's better than not telling the user and expecting them to figure it out for themselves.
Having post-enable wizards in the future could make this process more straightforward; enabling the UI could be a step in the API's wizard if the user hadn't enabled the UI too, and the API could ask other modules if they implement a UI to provide the user with an option if necessary.

With a 'suggested' keyword for modules, core could take care of these messages ("There are suggested modules that you have not enabled. Select any of the following modules you would additionally like to enable"), but I'd probably prefer it be done automatically with the option to quickly disable them after ("These suggested modules were also enabled. If you would like to disable any of them, select them and click 'disable'); advanced users could disable the 'auto-enable suggested modules' feature if desired.

Just wanted to note that the recent discussion of 'suggested' modules circles back into:

tags[] = views

It's not about auto-enabling modules. Much more about filtering modules by tag on the module administration page.

That is, because the current usage of package in .info files throughout contrib does not follow any standard or actually defined rule. Already right now, many people highly object to most package names being used in contrib. Originally, as far as I understand, packages were only meant for... - ? - well, even after spending years with Drupal modules, I still don't understand the intended usage.

But regardless of that, while we could certainly find ways to enforce a "proper" usage of package, the real issue we're facing is that

• modules happen to be grouped in some way
• modules start to use a new package name
• other modules might follow, or not, or only sub-modules of other modules follow (often seen in the case of Views-support modules)
• some modules might make sense for multiple packages
• some modules might make sense for the casual user to be enabled when the user asks for "some-thing" (e.g., "views")
• multiple modules might make sense to be grouped in custom categories; such as "payment gateways"

Elsewhere, we simply call this taxonomy.

On earlier comments:

1. Comparing an .info file tag proposal with Features is plain wrong. What are features? Features are configuration exported into code. This is about modules. Not configuration. Therefore, that entire argument is moot.
2. Let's also keep drush out of this discussion. If you use drush, then you should know what you are doing. After all, drush is a command line tool for advanced users. Does rm ./drupal ask you Do you really want to delete ./drupal? No, it does not. It it is invalid to apply web user interface concepts to a shell/command line tool's interface.

The difference is that tags would be multiples and filterable. Right now package just is, and you can't change it.

As it is, right now we really need the modules page to just list projects and allow filtering by tags, searching and a host of other features to make it easier to find what you want and its status. Which means we probably do need some way to increase the ability to set things on a project, so a project info file might be valuable. That said, we have other ways of getting project info for now so I wouldn't want to require that.

I do agree that tags[] is a valuable addition. However, IMO, it is not the solution for figuring out which modules should be activated by default when you turn on 'views'.

Another approach:
1. Allow module developers to define 'default configuration', which is 'API + UI + 80% of features' (in project .info file)
2. Define two modes of Modules page: Default (simplified), Advanced.
3. Default mode will group all modules by project. So for 'Views' it will be only one row with checkbox. Something like:

[+] Views
     This package will also install: Views UI, Views Export

In this 'Default mode' project will be installed with the 'default configuration'. This 'default configuration' can be used by Drush users.
4. In 'Advanced mode' each project is expanded and user can select desired modules.

However, there is a huge problem with modules grouping.
1. Modules relationships are to complex. We have projects which implements new features (Views, DC, etc), projects which extends features of other projects, and even projects with both new features and support for other modules (Date, DC). Also there are projects with no UI at all (e.g. Transliteration).
2. Current grouping by 'Package' is totally unusable.

I think that core must define several built-in groups for the projects. I like current organization of administration menu (separation into Content, Structure, Configuration, etc). So Modules page can be organized in the similar way. Visually, think about this groups as tabs. As candidates for built-in groups I can suggest:
1. Structure (modules implementing entities, fields, node types, etc)
2. Appearance (modules which affects visual representation of content: all *box projects, any galleries and such)
3. Internationalization

This is not a complete list. The idea is to provide list of groups which is usable by most of the contrib projects. Modules should not modify this list in most of cases. Exceptions are the projects which affects several groups, like Views (which is structure+appearance) or Ubercart.

As the result of using modes for Modules page and this built-in groups, we can imagine how 'Structure' group of the Modules page in default mode will looks like: it will have one row per project with sensitive description about what will be actually installed. If user wants to switch to advanced mode, it's possible at any time.

Also it will be good to allow modules to define 'sub-groups', so if we are in advanced mode in 'Structure' group, there can be 'Date' sub-group for all Date-related modules. This 'sub-groups' can be visually represented as fieldsets in 'group' tab.

I like the suggestion by @an.droid in #86. Keep the module page simple until user clicks to enable the module, then provide them with the defaults enabled within a fieldsets and let them override these defaults at that time. Having a basic with limited or no options and advanced mode is also a good idea.

Drush users don't seem like a target audience that wants to install groups of modules because maintainers said they belong together. I think they are more likely to tailer everything to their needs.

Tags offer a lot of flexibility, which is a good thing, but this flexibility can also be misused, much like packages are being misused now. Tags can be great for filtering modules, but it doesn't look like they can solve the API/UI separation problem.

The only 'proper' solution I have come across so far is rename "Views" to "Views API" and "Views UI" to "Views". However, like packages maintainers can easily ignore this and if that happens nothing will be solved.

AFAICS this issue consists of two separate ones:
1) How do we technically separate API and UI modules?
2) How do we inform the user of this separation through the interface and how do we let him enable the modules he needs?

Some prior art:
http://groups.drupal.org/node/10919
http://www.yoroy.com/elders/drupal/modulepage-sketches.png
http://www.yoroy.com/elders/drupal/module-accordions.png
http://drupal.org/project/module_table

Are we the first system ever experiencing this problem? No. Why not learn from how other communities solve similar problems like Debian, which was suggested in #328932: Modules and partial dependencies - enhances[] and enhancedby[] field in modules' .info.yml files?

Instead of "tags" or something like that use one of Debian's keywords "recommends" or "suggests" or even use the "enhances" one.

The different keywords added to Drupal would then be:

Depends -> MUST be enabled
(Pre-depends -> MUST already be enabled)
Recommends -> SHOULD be enabled
Suggests -> MAY be enabled
Enhances -> MAY be enabled
(Conflicts -> MUST NOT be enabled)

This would enable a lot smarter UI as well since the relations would actually mean something and not just be a relationship between two modules of any kind. By building upon the current dependency system it would also be possible to add version numbers to these keywords which would be great.

Please take into consideration the work that has already been made around this in the other issues I referenced earlier.

A Debian-model for module dependency is a generally +1 thing for a variety of reasons. We want that, no question. The question is whether or not it is also an adequate solution for the API vs. UI question. At the moment, my position is "maybe". :-)

I'm going to backup to the packaged installation issue here for a moment. As mentioned above, no solution REALLY fixes this, but I think a little info file parsing and a new administration page could go a long way here. We've attempted to provide some pretty significant hand-holding in D7 for various tasks (the modules page NOT being amongst them) and this is essentially a proposal to help there.

Info files could provide something like so:

package[group_name][package_name] = Commerce With UI
package[group_name][package_name][modules] = commerce, commerce_ui, commerce_cart, commerce_checkout, commerce_order, commerce_order_ui

package[group_name][package2_name] = Commerce Without UI
package[group_name][package2_name][modules] = commerce, commerce_cart, commerce_checkout, commerce_order

Our packages interface would then group all packages by group name (allowing to to provide multiple groupings if desired) and only one package per group would be enable able. Submit of this page would just enable all modules in all packages selected. This should be relatively simple to build, and could easily be tested out in contrib.

Ok, proceed to shoot down ;-)

Eclipse

Oh, also, I'm not proposing we replace the modules page, just demote it so that something like this is what the average user sees. Drupal devs could still manually enable individual modules, I just don't think that page should be the default. (or, this package management page should be at the same level as the module page at the very least)

Re #92: thanks for the proposal.
"Info files could provide something like so"
That leaves one question open: Which info file? I see three options:
1) The info file of one of the modules in a package (but which one? see also #81)
2) The info files of all of the modules in a package (means duplication of code)
3) The info file of the package itself (API change)

That or we could put together a new flat text file type specifically for this. 1 per tarball, so a commerce.package or something along those lines. rszrama brought this same issue to my attention in irc. This sort of ends up being an info file for the package instead of for a given module. It would be non-required, so you don't have to have it.

EclipseGc: Yes, that's what I meant with option 3). I agree that having a separate, optional info file per package is the nicest solution. Strictly speaking that would be an API addition rather than an API change, so I guess we could start using in D7 contrib without hurting core.

I'm not sure if we need the grouping proposed in #92; it would be another layer of complexity. As said before, I think we shouldn't try to turn this into a poor man's features. In fact, I'm not even sure if the package file should provide multiple packages. If it leaves the less experienced user in doubt which package he should enable, we have reached close to nothing. If we do allow multiple packages, let's at least make it required to mark one of the packages as default. We need a way for Verity and Jeremy to just enable the vanilla Commerce package without making them think.

So my proposal for commerce.package would be

package[package_name] = Commerce With UI
package[package_name][default] = true
package[package_name][modules] = commerce, commerce_ui, commerce_cart, commerce_checkout, commerce_order, commerce_order_ui

package[package2_name] = Commerce Without UI
package[package2_name][modules] = commerce, commerce_cart, commerce_checkout, commerce_order

One more thought: if we don't want to bother developers with yet another file extension, we could consider a naming scheme like [projectname].package.info.

Backing up to my original grouping point, Drupal Commerce is a really good example of WHY I want this. With any commerce based system you won't necessarily utilize all the features. By the same token certain feature groups are not mutually exclusive. Therefore, grouping allows us to define configurations which ARE mutually exclusive while at the same time defining configurations which are not. This could be ESPECIALLY useful if the package_names (which SHOULD be unique) happen to have name specific hooks in the associated .install files that allow further configuration up-front.

Worth noting that none of these items are particularly insane on their own, additionally they're fairly easy to implement and try out in contrib, and they further the goals of install profiles by removing a lot of the work install profiles have to do, instead favoring a reliance on the individual expertise of the module maintainers (while not removing the ability of profile creators to still do their own thing).

Eclipse

oh, and I like commerce.package.info or commerce.pkg.info. Either are ++ in my books, but I don't know what the impact is from an implementation standpoint.

Call me a curmedgeon, but the issue queue doesn't need headline writing.

I think the discussion has split into a couple areas, so I'll try and summarize the solutions so that they aren't lost.

What can we do now?

• Change module names as displayed on the modules page to emphasize the UI module for new users (e.g. Foo / Foo UI -> Foo API / Foo UI)
• Have API modules, after being enabled, notify users if they didn't enable the UI as well
• Meta-modules, which enable other modules through existing dependency system
• Rename modules from foo/foo_ui to foo_api/foo

What should be done in the future?

• Hide API Modules from new users by default
• Debian-style module relationships (e.g. depends, recommends, suggests, conflicts, etc; see http://drupal.org/project/module_supports)
• Improve the Modules page (see #538904: D8UX: Redesign Modules Page for prior work)
• Group modules in some fashion
  • Modules 'suggest' others, and are automatically enabled for new users
  • Module tags
  • Packages info file, defining modules which would be enabled together for new users

@gapple,
Thanks for the summary of this thread. It was needed at this point.

@everyone,
As a proponent of the packaging solution, I would mention it removes the need for painful renaming of existing module structures and meta modules for multi-module enablement.

That being said, I think the "suggests" and "tags" are both potentially awesome upgrades to our existing system for D8, as well as built in notifications for any suggested modules that weren't also enabled during the enable/install process, but the packaging solutions would be a rather separate system that could easily be approached from contrib right now in D7 (or D6)... I guess I'm just getting another plug in here, but I do really think this method could be really powerful, and lend some forward movement to other systems we'd all like to see improved. I'll try to spend some time over the next few days defining what I think this system would/should/could look like and post a follow up here. Questions and concerns are greatly appreciated.

Eclipse

This may be a strech but... Maybe Drush should just support a command like:

drush en packagename - which would enable all the modules that are part of a package.

Of course packagename being stored in a serialized field, in the database does not help at all, if we want this though :(

I think it's important that what can be done right now in D6/7 shouldn't be forgotten. At the very least the concerns about API/UI separation should be documented for new module maintainers to consider, and an explanation of the available alternatives from different perspectives (API code separation, ability to override UI with hook_menu_alter). Several people have said something similar to "separate modules for the sake of separate modules is a bad idea, but there are legitimate uses", so we should make people think twice about the decision so that it is done with reason (and maybe foo / foo_api will make sense instead). Having some consensus and documentation will also give a precedent for people to ask new module maintainers if their API / UI split is really necessary (and potentially remove it).

Changing module display names and having API modules notify users who haven't enabled their UI are both simple changes that can be implemented right now. While changing names may cause some small, short term confusion for people familiar with the projects I think the win for new users outweighs it, and as @sun mentioned merely adding "API" may be enough to get new users to decide "Foo UI" is the right module to enable.

I don't really like the meta-modules idea for D6/7; it seems too easy to abuse, and I can foresee numerous useless meta-modules taking up project namespace when they would be better suited for something like install profiles or Features. As well, it would require the same changes as renaming foo/foo_ui to foo_api/foo, except with potentially more confusion once implemented. While meta-packages work well for Debian, it doesn't change how the programs themselves are coded as it would in Drupal.

on to the future,

I really like the idea of a module.package.info file as @EclipseGc and @marcvangend have elaborated on in the last few comments; it seems like a simple way to hide the complexity of multi-module packages from new users. I think a good restriction would be that only modules within the project can be listed within the a package, or at least the default package if only those packages that the user has all available modules for are shown.

The Debian-style relationships would be good, but I now think maybe too much for this issue. It also isn't mutually exclusive with project packages, and I think each benefits new users and more advanced users separately.
I think automatically enabling packages with 'suggested' relationships as I said earlier would be too much magic. It could potentially chain to quite a few modules, and when your site breaks after enabling a module the culprit could be very hard to track down through the chain (if a new user even bothers).

and tags would be a good improvement, especially for the search utility discussed in the Redesign Module Pages issue, but wouldn't make the decision of what to enable easier for new users.

subscribe

Gapple, thanks for bringing up these issues.

I can't agree that a package should only include modules in the same project download. This is why we have the dependencies in info files now. Also, look at what the Drupal.org site is doing with install profiles and drush makefiles. If we can package up a series of modules for download (with or without core), we should be able to update a package by downloading one tarball and extracting it to our modules folder (assuming no version control). The drupal.org packaging system (in a perfect world) should be able to put all the right files in one place for an install or update.

+1 that we don't need meta-modules - I agree the project namespace problem would be overwhelming.

I agree with gapple; IMHO a package should only be about modules in that project. A package merely describes sensible defaults for that project, to get you started a little quicker and prevent "WTF where is my UI" experiences. For all awesomeness beyond that, we already have drush make and features.

Also going to +1 gapple and marcvangend's comments here. We don't need to reinvent profiles, we just want to give module maintainers the ability to provide sensible defaults for those who wish to utilize them (and alternate setup/configuration for non-intuitive capabilities).

Eclipse

Instead of a separate info file, what about a recommended[] = views_ui property in modules' info files? Maintainers would have a bit more flexibility: if A is enabled, the user might need B and C, but if he only enables C, B, but not A may need to be enabled as well.

Xano,

I think we all agree that would be very useful, but I think the point here is to settle on some preset abilities a given package of modules might have. Again ubercart/drupal commerce are two great example of module packages that have an awful lot of potential variety in how they could be setup. A simple recommendation system won't really help them much because you can make intelligent recommendations if you don't know what the user is wanting to build. Are we selling subscriptions to the site? Are we selling products? Are we gathering information for event tickets? taking donations? etc etc etc. This is part of the thinking here.

Eclipse

I agree that is an issue as well, but that looks like a problem we should solve with configuration profiles, rather than with groupings of modules, because modules do not always have a direct relationship with certain end user features. Views, for instance, can be used to create a wide variety of features, depending on how you configure it. Or Ubercart. By default it's a webshop, but it could possibly be configured to act as a cash register.

Bottom line, afaik modules are too low level to group them based on what users want to build. If we create package info files, those will be some sort of dumbed down config profiles.

Here is an interesting one: Hierarchical Select. If you don't have Book turned on, then HS Book doesn't make any sense. Sure, this is in core, but there are also several CCK field widgets, like Content taxonomy, nodereference, etc. These would be dependencies, but also not a bad candidate for packages, but not all downloaded in the same project.

DraggableViews is another example, and I'm sure there are several more that are no good by themselves, but that have a default package.

The Ubercart + Event registration example certainly requires modules that are not in UC core.

I think it's important to separate:
Install Profiles - "I want to start a site that does X", "I want to create a community blog site"
Features - "I want to add X functionality to my site", "I want to add an employee blog to my company website"
Project Packages - "I want to enable X module", "I want to enable and configure a custom view"

I think cross-project dependencies and functionality that requires multiple modules are much better serviced by Install Profiles and Features which are built to contain the extra configuration necessary to make them work. Project Packages are just saying "If you want to configure a view, you need views_ui enabled too" - no configuration.

@xano
I think with just a simple recommends[] we're lacking any context that a new user requires, and opening up the feature to misuse by module developers. A module can recommend another, but work fine without it. I think there is potential for more harm than good as well; how can recommendations be used to simplify the interface or improve the user experience, and not just add more information for a user to parse? This issue is primarily trying to solve the UI/API separation for a new user, where an API module doesn't do them any good without the UI module too.

@liberatr
For Hierarchical Select, I think packages of HS + hs_book/taxonomy/content_taxonomy would be sufficient. They all require the modules that they enhance, and here the dependency system can do what it is designed for. Packages would say "You want to enable Hierarchical Select for better taxonomy selection" and dependencies would go "Hey, you need to enable Taxonomy to do that". Since several relatively equal options are available (HS doesn't appear to do anything significant on its own), there wouldn't be a default package.
Similar with Draggable Views, except that DV would have a default option of just the main module since it does something useful on its own and Views is specified in the dependencies.

#116 +1

Regarding the Hierarchical Select example: I agree that that is how it should work. The package mentions modules from its own project and dependencies handle the rest. However in this specific example, I don't think the package should include HS_book (which is only available in D5 by the way).
IMO a package (or at least the default package, if we allow multiple packages in the .pkg.info) should only include the bare minimum to make the project usable and understandable for someone who has never used it. As a module maintainer, I would not want to convey the message that HS only works (or works best) when book.module is enabled, or that you need to download a contrib module like content_taxonomy first. Just HS + hs_taxonomy should do I think.

@#116: And how are package info files better protected from misuse than a recommended[] property in existing info files?

I've only read 50% into this, but there are several assumptions here on memory usage/performance/scalability that are completely wrong, so I'm going to respond to those first:

1. The file keys in hook_menu(), hook_theme() mean that memory is not a problem.

This is completely false, the theme, menu, field API and many other systems load the entire theme registry, menu router table, field info cache (or whatever) into memory on every, single, page request to core.

It is quite possible on a Drupal 6 or 7 site for these caches to take around 10mb+ of memory on each request. This was something I only really noticed recently, since I have mainly been profiling Drupal 7 core with reasonably minimal installs up until the past few months.

So in this case, having modules which are defining a lot of menu paths, router paths, implementing hooks etc. that you can disable on production would help to remove some of that bloat. As would refactoring those systems to only pre-load caches for things that are actually in active use (see the memory tag for some patches which start work on this).

2. The number of modules in itself doesn't affect performance.

It does, every time we rebuild module data in Drupal 7, we currently have to find and process all the .info files for test modules that will never, ever be enabled on a real Drupal install, and doing this is a big CPU and memory hit that sends certain pages on certain systems through the memory limit.

We also have to load information about all enabled modules in the system info cache, the more modules are installed, the bigger that gets.

And we also have to include_once every .module file that's in use - if you strace large Drupal installs with 100+ modules enabled, you will see massive sections in module_load_all() that will give you a big headache.

In terms of the overall argument:

# It is a good goal to be able to completely remove code from production installs.
# It is a good goal to be able to do drush en -y commerce and/or not have three pages of scrolling on the modules page. I think we could accomplish quite easily this with meta-modules that only define dependencies, and using hidden[] = TRUE a lot, without needing a complete refactoring of the modules page. This doesn't stop people using drush or whatever to enable modules one by one. When I install or enable packages in linux, there is often a long list of dependencies that I've never heard of that get enabled too, works fine (both from apt and presumably for the UI front ends to apt too).
# Fixing memory bloat is not only about stopping code from being loaded, it is also about stopping useless meta-data from being loaded - this includes having thousands of disabled module .info modules to be parsed, as well as have huge info-ish registries that have to be loaded into RAM on every request. So any real proposals need to look at all these issues. For those who care about shared hosting, we need to reduce the footprint of rebuilds so they don't lead to WSOD or pages that take 15+ seconds to load. For those running high performance sites, we need to reduce the footprint of rebuilds so they don't bring sites down due to race conditions or locks on global caches.

I agree with @catch; it seems like we could get a lot of mileage out of hidden = TRUE here. Especially if we made it so that hidden module dependencies get enabled automatically via the UI (without a confirmation form), and also made some changes to other parts of the admin interface to better deal with the possibility that some of the site's enabled modules may actually be hidden from view. This wouldn't address all of @webchick's original concerns, but I think it would help with a lot of them.

For Drupal 7, I actually posted a contrib module earlier today that does some of that: Simplified Modules.

When I install or enable packages in linux, there is often a long list of dependencies that I've never heard of that get enabled too, works fine (both from apt and presumably for the UI front ends to apt too).

On Maemo (which is Debian based), you install an app and you never even see the bazillions of dependencies it installs to get that done. It's not until you root it (which of course is laughingly easy -- you install the rootsh app and type root. heh.) you see those packages.

Talking about this with chx and jthorson in irc, I remembered http://drupal.org/node/293223#comment-3103432

So there are two goals, that with the current module administration screen conflict a bit:

- hiding clutter from people who don't want to see it.
- letting people see how things work so they can figure it out properly.

When it comes to fundamental concepts like modules, I tend towards the latter when it comes down to it although obviously it's great if we can reconcile them.

Also thought about views_api vs. views_ui etc., I'm not sure switching those makes any sense.

views the module, provides some user interface features (specifically default views and loads of other stuff). Views UI provides an administration interface - that is different from simply a 'ui'.

jthorson also suggested views_admin.

If we think about 'Field administration' 'Views administration' (node, users etc.), that seems like a sensible separation to me, and the bits that people most commonly want to swap out entirely - i.e. admin/content only makes sense on particular kinds of sites and is pretty limited, whereas node/add is rarely completely replaced.

So what we really want, is not necessarily a separation of ui and API, but actually a separation of administration interfaces and the other features that modules provide. For the same reason we have module.admin.inc, this is different from say, module.ui.inc. Some modules you can install just fine that have no administration interface - say if other modules are depending on them (this will be the case for entity module in D8 if and when that patch is committed).

a separation of administration interfaces and the other features that modules provide

but even a pure API module might have an administration screen to input API keys.

This is in fact the usual question: who is the user ? What is the purpose of the user interface? I can feel quite some difference between the API-key-entering admin interface and the Views UI -- and not just because the latter is more complex. The API-key-entering admin interface is a set-once screen while the Views UI is a use-many-times screen.

I think that it might be good for "set once screens" like an API key field page, that could be combined into a "system APIs page".... why should all these form elements that rarely need to get changed all be sprinkled around on separate pages? Maybe it would be better to associate them with 'likely frequency of use' and 'area of the site' (i.e. third party APIs, local user settings, etc) -- combining the elements into fewer pages.

Anything to get people out of the Labyrinth of minor admin pages should be considered I think.

Let's continue to use Views as an example. I've seen three types of sites:

1. I can't trust anyone there to create Views.
2. I can trust a few users with creating Views.
3. Only development staff can create Views.

So, on sites in category #1, Views UI is not needed at all. Either I will create a View and import it from my dev site, or only modules containing canned Views will be used.

The same should be true for #3. We all know (and probably violate) that we should be testing in a test environment. Separating the UI and not installing (as in removing from the distribution package) it on the production server helps to enforce this practice.

That leaves #2 and they should be doing the same as #3. In this particular case, permissions don't force testing, they only limit who can mess the site up.

Views is an extreme example of this separation. I agree with Angie in some respects. The developer must exercise some common sense in when and how to separate the admin stuff. A simple settings page (like hook_settings was) should not be a separate module. Permissions and menu "file" are more than adequate. However, significant site security issues do raise the need for a separate UI module that can be eliminated from a production site. Somewhere in between there may be value to wizards, although I'd rather see those limited to Features installs.

So... it's probably time to start discussing this issue in light of the core initiatives, specifically wscci. I think it's important that we re-evaluate our positions, and I hope to outline what I think is both a good answer to this question as well as being a sane way forward for core in general.

We have a few problems administratively, and I mean that both in terms of a.) where/how administration is located, b.) when/who we want to use it, and finally c.) how that administration is actually built.

Where/How Administration is located

Part of the wscci/cmi initiatives is essentially this notion of dependency injecting our pages. This applies for administration as well. We currently just have a module build a menu item and do a drupal_get_form as the page callback and pass it an argument for what form we want loaded. This is nice, but form and form_state are generated by the system and handed off from there, and truly, many (i.e. most) of our forms could benefit from a greater level of dependency injection at this point. For example... if I were to develop a custom administrative experience for my users (something I do regularly) I might want to specify a different redirect for my users on save. I'm implying here that I would have relocated that administration to a completely different url and that redirecting from there seems quite logical.

In the same way, when building a custom administration, we might take a taxonomy vocabulary term list as a good example. Every term links to taxonomy/term/%term and has a corresponding edit link as well. The form itself does the db query to lookup the terms it's going to render, and none of this stuff is really relevant to the fact that we might have a lot more options if we injected the form with formats for the links we want, and the terms we'll display (opening up the way for far more interesting administration options without significantly increasing the complexity... in fact the forms themselves could conceivably become simpler since they don't have any logic, they're just looking for preset configuration in the form_state... or something similar).

When/Who we want using our Administration

Currently 99% of this is controlled through user_access defaults in menu items with a permission passed to them. In drupal 6 this practice was actually pretty destructive from a distribution maintenance standpoint. We're getting smarter about how we handle this stuff and that's great, but administrations can be relocated, contextualized, and opened to specific user groups in specific contexts. I do this regularly, and have for quite some time. In these cases, hard coded user_access checks IN our forms are harmful. Relocating things with the menu system is certainly possible, but we need to stop designing with the expectation that our single simple permission based use case is the only use case.

How we build Administration

Up until this point most people can probably agree with me, the contentious part is this though... how do we build administration? For D8 I would propose a little more stringent of an approach where by we a.) force developers to create self contained page output systems (I prefer a plugin solution for this which would be fairly straight forward) and b.) we don't force a set of menu items on a user at install time. In this way I would prefer we provide default menu items which can be accepted as a group, or rejected, in which case we assume the user is choosing a more "advanced" mode of operation where they will pick a choose how the administration is deployed through some other mechanism (a lot of talk about a core page_manager style solution has been happening lately, and this is the logical place for this stuff to fall).

What does this buy us?

Well, if we used a plugin style architecture, we'd get the benefit of removing all output creating tools from the loaded code unless a.) they were enabled appropriately, and b.) we were actually visiting them. In addition to this, drupal distributions/install profiles would become significantly more customizable as any output provided by any thing (core or contrib) would have to be properly set to consume dependency injection, and the result of this is that the entire administration system essentially becomes plastic. In this way, the default drupal install profile would simply accept all menu item suggestions from any module, but this sort of thing becomes a decision for the install profile you're currently running, and that really gives distro maintainers a LOT of flexibility to decide what their user experience is going to be.

I'm happy to discuss this at length, but I've put a lot of thought into this, I have a lot of practice doing similar things in both D6 and D7, so please ask questions if you have them, shoot holes where you find them, but I think that this can be a way forward both for this discussion and for drupal core itself.

Eclipse

API clean up might relate to #1224666: [meta] Unofficial Drupal 8 Framework initiative.