There are an issue with the way "projects" (as defined in http://drupal.org/project) relate to actual modules that is blocking other issues like #538660: Move update manager upgrade process into new authorize.php file (and make it actually work) and the module page refactoring.

Right now, a project doesn't say anything about the modules contained within the tarball. That makes the project listing page overly long (modules could be grouped by projects like in the update_status page) and makes it impossible for the plugin manager to figure out which modules to enable when installing and upgrading a project.

After discussions with Angie, JacobS and Adrian, we figured it was time to implement project-wide meta-data in the form of a project-level .info file that would contain the required modules in a project. The file would be completely optional for most modules: if it is not present, the project would be expected to provide a module named the same name as the project (say like the audio module/project).

This would be very useful for projects like CCK (that provides the 'content' module which should generally be enabled) or the Ubercart project (that provides the 'product' module which should be enabled). It would also help in making the module page cleaner and finishing the plugin manager.

Support from Acquia helps fund testing for Drupal Acquia logo

Comments

skilip’s picture

subscribing

skilip’s picture

This is a really good idea. This way the module page would have only extra grouping when needed as well. I mean, in the latest mockup for the new module page, we introduced the 'package' groups inside the 'new', 'enabled', 'disabled' groups. I got a lot of bad responses about that, and this patch would give us the possibility to only group the parts that *really* need to be grouped.

Are there user cases in which there's 'projects' don't have a 'main' module on which the other ones are depending? If this is never the case, why not just use the .info file for it? Just throwing thoughts here....

lut4rp’s picture

Plus one :-)

JacobSingh’s picture

FileSize
1.33 KB

Okay, this patch doesn't really do anything, simply copied the code for loading module info files into one for loading projects. I think the implementation of this will depend a lot on the new UI of the modules page. On the plugin manager, it is more clear.

I've called the new file project.info. Here is my reasoning:

You should be able to ascertain that a given directory contains a Drupal project, what that project is called and what type of project it is.

For the sake of consistently, and because many modules do not want to have a module with the same name as the project, I thought it made sense to not try to cram everything into $module.info. These two files (project.info and coder.info) have different purposes, and different schema.

If we want, we can call it coder.project or something, but I thought using a semi-standard naming convention would save effort.

Here is an example project.info for coder:

;// $Id$
name = Code Review
type = Module
description = Developer Module that assists with code review and version upgrade
default_modules[] = coder
default_modules[] = coder_review
JacobSingh’s picture

Status: Active » Needs review

Status: Needs review » Needs work

The last submitted patch failed testing.

cmcintosh’s picture

I would much rather have the project follow the sort of layout as below:

File Struct:
Project[i] -> modules[ii]->normal layout that exists now

In organizationally i think something like allowing defined types

name = developer suite
type = Project
description = Developer Project that assists with code review and version upgrade
modules[required] = coder
modules[required] = coder_review
modules[required] = devel
modules[themer] = themer
modules[themer] = 'other optional theme dev modules'

The ideal here is you can combine several different sets of modules together in what some would normally do already then inside of the module page you can click either just required, or Required and Themes or Required Themes and Modules, etc. Enabling the corresponding modules. These modules can be listed on the project's entry on the normal modules list page. For more detailed control we could have the project title as a link to a Project Config page, which would allow you to then override the defaults and enable/disable modules per module basis.

JacobSingh’s picture

Status: Needs work » Needs review

Chris and I talked. For now, I recommend we keep it simple, although his approach is very cool, he was thinking more in lines of mini-install profiles. He's actually looking for something more like "Features", which I sent him towards.

I'm very open to changing the conf file schema if people are interested, but prefer to not overcomplicate or "build for the future" at this moment.

webchick’s picture

Yeah, definitely agree that the scope of this should remain basically just informing Drupal core about stuff Drupal.org knows, and enables the plugin manager/modules page to make intelligent decisions about which modules to enable by default and some other stuff. It kinda sucks though that module developers have to type this ... one .info file is annoying enough, let alone two. :P

We've kicked around a variety of approaches...

1. Using Update Status module to provide this information as it does on admin/reports/update. Relies on connectivity to Drupal.org, and doesn't work for custom modules. Bleh.

2. Use the packaging system to either programmatically create the .info file, or move some of these properties into the existing .info file, so the only thing the module maintainer needs to define are the default_modules[] parts that require a human being to make a decision. Problem is not all of these things are able to be programmatically derived.

3. I forget now. Sorry, I'm really tired. ;P

So I think this needs more discussion, preferably from people like Derek Wright, Earl Miles, etc.

merlinofchaos’s picture

1) This will mess up with project.info in project module. Suggest projectname.project.info instead.
2) This should tie into the idea of projects that update.module is already using. It knows what project modules belong to. This meta data can provide more information about those projects. That could and absolutely should tie into update.module, at a minimum, and almost certainly the modules page because the current UX modeling really wants to be able to do more module <--> project defining. This is a good thing. Implementing 1) above can make views.project.info properly match the actual project name.

JacobSingh’s picture

FileSize
1.28 KB

Agree w/ Merlin's points. Here is an updated patch

Status: Needs review » Needs work

The last submitted patch failed testing.

anarcat’s picture

Status: Needs work » Needs review
FileSize
1.67 KB

I think it should be the reverse: Drupal.org should use more of the project's .info files.

Furthermore, I don't see why this is an helper function. It seems to have been based on _system_get_module_data(), which is an helper for system_get_module_data(), so I feel this should be a real, well-defined function. I also moved it in a proper place in the file.

I am not sure we should go back into the "category" nightmare that we want to get rid of in the "packages" stuff, by setting special tasks as described in #7. I would rather see "required", "suggested", etc.

One issue there is with the current implementation also is that it doesn't list all the projects but only the ones that have .info files. I also note that drupal_parse_info_files() gives the spec for the module's info files while it is actually a more generic function, so this should likely also be fixed if we are to reuse the function.

I think that that patch can be complete once it passes testing, fixes the above issues and that we should open a separate issue for integration in the module's list.

For those interested in improving the patch through git, it's available on the branch named "568288" on git://git.koumbit.net/drupal/core/ (see also http://git.koumbit.net/).

anarcat’s picture

FileSize
1.67 KB

Another reroll: use devel's descriptions instead of an arbitrary module to clarify what this does.

I also want to clarify here what I mean by "required" and "suggested". In the module listing (which we should call project listing now, i guess?), we would have the following UI:

Developer suite (*) enable ( ) suggested ( ) all
Ubercart        ( ) enable (*) suggested ( ) all

Those are radio buttons: clicking "enable" would enable only the required modules, "suggested" a bit more, and "all" would enable all. Not sure of the wording here, but the point is that we should avoid letting people use arbitrary strings unless we want this thing to become a total usability nightmare.

Required + suggested or just required is my vote.

The UI issues should be debated elsewhere, as this concerns only the backend.

merlinofchaos’s picture

I'm confused. The doxygen in the latest patch does not match the defaults.

anarcat’s picture

FileSize
1.67 KB

I'm not sure what you mean by that: we currently have defaults? At any rate, I followed what you have suggested above (that coder_review is enabled by default).

mcrittenden’s picture

Subscribe.

skilip’s picture

I do not agree we should use drupal.org information for project descriptions. If we go that way, it is an easy step for using that information for all modules descriptions in .info files, which is a complete other issue. This issue is about providing meta information for projects, thus bundles of modules, which have or have not a main module. It's a pity each module is called a project as well on d.o., so it's a bit confusing.

Is it possible to require project maintainers to rewrite their project structure in the following way?

/project_foo/
/project_foo/project_foo.info <--The project info file
/project_foo/project_foo/
/project_foo/project_foo/project_foo.info <-- The module info file
/project_foo/project_foo/project_foo.module
/project_foo/project_foo/project_foo.install
/project_foo/module_bar/
/project_foo/module_bar/module_bar.info
/project_foo/module_bar/module_bar.module
/project_foo/module_bar/module_bar.install

Or is this a complete CVS nightmare?

moshe weitzman’s picture

Subscribe.

How do we represent that some modules live outside of drupal.org? update module already supports this.

Even though it does cause some cvs headache, i do think it is OK to require proper filesystem hierarchy for submodules. If the benefits are great.

anarcat’s picture

re #18 and #19: i do not think relying on drupal.org is a good idea either: we can get all we want from the .project.info file.

However, I don't see why we need to reorganise the module file hierarchy if we have a proper suffix (.project.info is unambiguous). It's hard enough to have proper naming conventions I don't see why we should go creating another convention, it doesn't add or solve anything. The proposal, as it stands in #16, is totally optional and doesn't change anything to the current documentation or the way we create modules. It just improves the user experience for those more complex modules like CCK and ubercart.

I have yet to see a proper objection the what I am proposing in #16. It seems to me complete and sound (and #538660: Move update manager upgrade process into new authorize.php file (and make it actually work) actually depends on it now).

JacobSingh’s picture

I'm fairly ambivalent about the module subdir thing, although it does seem like a logical improvement to readability, I can't see any good reason not to do it. Especially, if we make project.info obligatory and everyone has to mess with their modules anyways.

A slight correction: #538660: Move update manager upgrade process into new authorize.php file (and make it actually work) doesn't actually require project.info, just looks for it, and uses it if it exists.

Best,
Jacob

merlinofchaos’s picture

I do not like the forced organization. I do not believe that it is at all necessary.

anarcat’s picture

After a short discussion with moshe, I'm not convinced we actually need a metadata file anymore. We could instead present the user with a list of modules to enable *if* the project doesn't package a module with the same name. That could work both in the plugin manager and the module listing.

Basically, the idea is that a user generally knows what he's looking for when he's trying out a project, and if he isn't, well the module listing and descriptions should guide him, so we shouldn't bother too much in listing those defaults, but provide a proper UI for the user to choose from.

Another issue that was raised by moshe with me was that some tools will require "module to project mapping" to resolve some dependencies. For example, you will need to know that it's the "CCK" project that has the "content.module". Right now, tools like drush already depend on Drupal.org to find the proper project paths and things like that, through some XML API i am not too familiar with. I would expect this API to be able to provide that mapping if necessary, as Drupal.org is the place where all those modules get uploaded. This requires some work in the project tools on drupal.org, which is in #102102: Parse project .info files: present module list and dependency information.

This seems to cover it all: basically, I think the UI refactoring doesn't require a metadata file after all and I think we can find alternate solutions.

dww’s picture

Status: Needs review » Needs work

I've just recently returned from being offline for two weeks and trying to catch up on a huge pile of emails and issues. I've only briefly skimmed this issue and patch. On first glance, I'm confused.

Each module's .info file already says the project that module belongs to (either via a packaging script, or via CVS Deploy). This is how update.module can do the grouping. If that's the case, what do we *really* need a separate project-wide metadata file for?

If you want to clean up the modules page in the admin UI, do the same thing update.module does. This does *NOT* require connectivity with drupal.org. update.module just looks at data that's in the .info files already (or data that's altered and put there by cvs_deploy).

If you want to know what project to download to satisfy a given dependency, you need all the metadata for all projects anyway, and in that case, you probably need to be using Adrian's YAML stuff -- you need a full DB of all modules and projects. I don't see how a per-project metadata file (on its own) helps this problem.

All this recommended/suggested/required/optional stuff is misplaced. That belongs in each module's own .info files via #328932: Modules and partial dependencies - enhances[] and enhancedby[] field in modules' .info.yml files #188038: module info file: recommended modules et al.

My inclination is to call this "won't fix", but I'm willing to hear counter arguments first.

However, if a patch like this is committed for some reason, I think it's confusing and weird that the project-metadata alter hook is also called hook_system_info_alter. That seems like a major DX WTF to me. How do you know if you're altering module or project metadata (especially if the key is the same in most cases)? Ugh.

p.s. Requiring a specific directory layout seems like a *terrible* idea with *no* benefit. ;) Please let us not waste any more time considering that option...

JacobSingh’s picture

I'm on vacation, so this will be my last post here most likely. I'm fairly ambivalent, however, a few things to consider:

1. I already see this post getting emotional. There isn't time for that, let's please try to back up statements with reasons and not just say it is terrible, awful, stupid, useless, etc.

2. The modules page is broken because user's cannot understand the module descriptions and their relations. Modules are split mainly for technical reasons, which non-technical people don't get. There is a lot of value in having a "Check one box and a get a product" type approach. Features is doing something similar, but even with the modules page, this is the only way to fix it AFAICT. Offering them a list will not help that problem and will force us to write more UI. I don't see how this solves the problem.

3. For the Plugin Manager, see #2. My original suggestion (and what the PM currently supports) is an option project spec file, and if you don't provide one, it will just try to enable the default module. This is still okay I suppose.

4. For the future of Drupal and modules, we need to support everything we need without hitting d.o. This is crucial of all packaging formats. It cannot be dependant on a single repo. I think we can all agree on that.

5. rpm, deb, etc all have a package spec file. I'm not saying 100% that we need one, but hey, just saying... They obviously see some value in it.

This issue calls into question a big debate in the drupal community: Are modules libraries or products, for that matter, is Drupal a library or a product. I would like to think that modules are libraries for the most part, and become a product when in a package. Further, we aren't just talking about modules, and it is important to remember that. Until something like Features is part of core, if we expect core to be usable on its own, we need to shorten the modules page and make it more end user friendly. Does anyone have any other ideas on how to do this?

Best,
Jacob

anarcat’s picture

Status: Needs work » Postponed

So I agree with Derek: having a project-level metadata file is not (currently) necessary. The mapping is done in the modules .info files themselves right now and it's sufficient for all needs *but* one: the idea of "default modules" to be enabled in multi-module projects like ubercart and CCK (and especially projects that do not have modules matching the project name).

However, that problem can be resolved without a .info file. Other solutions were proposed, and there could be more alternatives.

This applies both to the project manager and the module listing. I think that just providing a metadata file isn't necessary, it's not worth the trouble yet. Maybe we'll need it later, but right now, as one said "it's a solution looking for a problem".

I have therefore opened a separate issue specifically for the problem we are trying to resolve here that is not bound to a technical solution. I have expanded on other potential alternatives there. See #571664: Add a way to specify modules to be enabled by default when a project is installed.

This issue can eventually be reopened once we have other use-cases for a project-level metadata file. Right now, I can't think of any, and no one found any that i know of.