Make it easier to identify and discover ecosystem modules

I think this is an excellent idea. It could go a long way toward making finding contrib modules easier.

I find if we make a more conscious effort of hijacking body tags like what we did with #D7CX, it might become easier to find similar modules. Solr is quite amazing, we should take advantage of it. You can effectively tag your own project with your own custom eco-system tag by just sending them to...
http://drupal.org/project/modules?solrsort=sort_title%20asc&display=tabl...

We now have http://drupal.org/project/project_dependency on Drupal.org, which keeps a table of dependencies like

mysql> SELECT * FROM project_dependency_component WHERE component_id = 24;
+--------------+-------------+--------+--------+----------------------------------------------------------------------------------------+
| component_id | release_nid | name   | title  | description                                                                            |
+--------------+-------------+--------+--------+----------------------------------------------------------------------------------------+
|           24 |     1139480 | a_hole | A-Hole | See if your JavaScript follows good code isolation and dependency injection practices. | 
+--------------+-------------+--------+--------+----------------------------------------------------------------------------------------+
1 row in set (0.00 sec)

mysql> SELECT * FROM project_dependency_dependency WHERE component_id = 24;
+---------------+--------------+---------------+----------+----------+----------------------+-----------------+
| dependency_id | component_id | dependency    | core_api | external | external_release_nid | dependency_type |
+---------------+--------------+---------------+----------+----------+----------------------+-----------------+
|            20 |           24 | bad_judgement | 7.x      |        1 |                    0 |               0 | 
+---------------+--------------+---------------+----------+----------+----------------------+-----------------+
1 row in set (0.00 sec)

I think this goal might be achieved by exposing this data on module pages, #1299240: Expose the dependencies on the project page. That would avoid additional info for project maintainers to update and remove the need for manually-updated lists often seen in the project description. Here is a quick example, our top 10 depended-on modules:

mysql> SELECT dependency, count(*) c FROM project_dependency_dependency GROUP BY dependency ORDER BY c DESC LIMIT 10;
+------------+------+
| dependency | c    |
+------------+------+
| views      | 4786 | 
| taxonomy   | 3387 | 
| content    | 3043 | 
| ctools     | 2220 | 
| features   | 1896 | 
| menu       | 1193 | 
| strongarm  | 1153 | 
| token      | 1140 | 
| search     | 1069 | 
| comment    | 1044 | 
+------------+------+
10 rows in set (0.57 sec)

I definitely think "Ecosystem" is probably not the right word. :) That does not indicate to me what the relationship is at all, and in fact makes me think these are modules that have something to do with literal ecosystems. (Actually the metaphor doesn't even fit very well.) We want something that clearly indicates the relationship between these modules, and if it's not "modules that extend this module" or "modules that depend on this module" (as joachim implies in #7), then what is it? I don't even really understand, even having read this issue.

Also, it seems to me that it would make sense to include the sidebar on the "child" module as well. Finally, what happens when the ecosystem is overly large? Do you display only the most popular modules, maybe with a "more" link?

Our project pages are too cluttered and overcrowded with "related" information today already.

Why don't we introduce a new tab "%project extensions", which contains the full list of dependent projects, and which thus has the possibility to show more information about each dependent project?

As you know, the project titles are not always really clear, and you need to at least read the teaser/summary to understand what it really does.

A separate view also allows to filter by

- core version
- possibly even major version of the parent project
- project category
- sandbox/full-projects
- etc.

Frankly, I think this goes beyond "Ecosystem" to just "exploring / finding modules" in general, as in this discussion.

I have a couple of comments:

• The existing taxonomy filters available at /project/modules are just too simplistic. Selecting any one of the terms often still makes me review hundreds, if not thousands of modules.
• I often want > 1 term to filter on, e.g. "User access & authentication" AND "Third-party integration" AND Utility. Using multiple terms with booleans would narrow my search dramatically.
• I want a better manner to review listings (as discussed in the link above).
• Folksonomy/tagging - vs. formal taxonomy - seems untapped. Social tagging would best leverage the "Drupal way", and get community participation to add value (to classification). And a richer, wider set of tags seems like it would add value. (It could, for example, solve the "ecosystem" problem being referred to in this thread.) Yet the potential for lots of almost-identical tags remains; though one solution might be to invent some kind of workflow that routes new tags through maintainers before making them active.

This problem seems to beg search-based navigation using Solr; it would not only provide a sophisticated way to handle the filtering described here, it could also bias results based on # installs, etc. My favorite example of search is still the search results page at Kayak; I'd kill to have a metaphorically-similar interface for finding modules.

Re: #15: The project browsing pages are entirely solr-based search. I'm not sure what you want that you don't have in that regard. It already does bias results based on # of installs (that's the default sort order). Maybe we did too good of a job making it not look like Solr so you didn't even recognize it. ;)

But honestly, most of what you're talking about is off topic for this thread. We don't need yet another generic "make it easier to find modules" discussion here. joachim has a specific proposal to address a subset of the problem. Let's use this issue to discuss the merits of that proposal, instead of dragging the entire debate back in to each discrete issue. We can't make actionable progress if each time someone proposes something concrete we use it as an opportunity to discuss everything all over again. We need achievable things to implement, or we're just talking in circles. Yes, we need to step back and look at the big picture, too, but that's what venues like g.d.o are for. This specific issue is a poor place to discuss the whole picture. So let's not. ;)

Back to the specific proposal here -- we do have the project_dependency data now, which parses all the .info files of everything packaged on d.o to record dependencies between projects. There's already a proposal for displaying that info on project pages over at #760890: Display module dependencies on project pages. So, if that's all we're going to gain from this, we should just close this as a duplicate. However, I get the sense joachim wants something more than that. Yet, my concern is that if we do something other than the auto-populated dependency listings, it's going to require human intervention and we've already got resource problems with not enough volunteers to deal with stuff like this. I'm reluctant to put energy into a system that depends on lots of human effort.

So, joachim, can you clarify what's different about this issue from the info we'd gain automatically via #760890: Display module dependencies on project pages? Is the difference worth the effort? Should we just close this and focus on #760890 instead?

Thanks,
-Derek

Alternate proposed situation:

Let a robot fill in these "noderef" fields automatically.
-- 1-time human effort to teach the robot what to do, then 0 effort ongoing.
-- Equivalent or possibly less 1-time human cost than the proposal here.

I don't really understand why making humans do it provides "more refined info" other than the fact that in practice, only a small % of "ecosystem module maintainers" will actually use this feature and provide any data at all.

Maybe you're worried about the fact that 1000 modules depend on views, so you don't want an automated list. Fair enough. But "more refined" is a fancy (and IMHO misleading) way of saying "incomplete." In principle, nothing would stop all 1000 dependencies of views to use this feature. So AFAICT, the only difference between letting robots do this automatically and correctly, and asking 1000s of humans to take action, is that we know the robots will do a more thorough and complete job.

I'm not trying to discourage your initiative here, sorry if that's the result. Please don't give up just because I'm asking questions and pushing back a bit. I'm only trying to figure out why we should invest development time in this feature instead of the automated one. I understand people do this already via manual effort. I do it myself at the signup module project page. That's not what I need to be convinced about. ;) I need to be convinced why partial results that have to be manually provided by maintainers "in the know" enough to actually use the new feature are better than automated results. If the only answer is "scale", then I say "provide a useful default sort on the automated results". For example, order those by # of site installs using the module. So, you still would see (via a pager) all 1000 things that depend on views, but the top of the list would be the more popular (and presumably, therefore useful/working) dependencies.

Thoughts?

Thanks,
-Derek

p.s. Jumping off from what both sun and batsonjay have said, maybe this /project/[whatever]/extensions page is just a pre-filtered Solr search interface, much like the module browsing page, except restricted to only display projects that include a module that depends on the [whatever] project. So, we'd just have to inject dependency info into the solr doc for each project node over at #760890: Display module dependencies on project pages. Then you could have all the same fancy sorts/filters/search goodness, but only on the subset of projects related to [whatever]. That seems like it'd be pretty useful, and would require very little work (since we've already got all the plumbing in place for this stuff via the project solr module).

If that's a better solution to what's proposed here, let's mark this duplicate, copy this proposal over to #760890, and start actually making progress on that.

If what's proposed here is better, I still need to be convinced. ;)

Thanks!
-Derek

@joachim: You totally convinced me. :) Was merely unhappy about the location of where it gets output in the mockups.

@dww: I don't think that module dependencies deliver the picture that @joachim is aiming for here. What about optional module support, for example? (i.e., there's no dependencies[] in any .info file at all. Perhaps there's a module_exists(), but not necessarily. Many help enhancements and integrations can be achieved by implementing hooks or alter hooks only and those use magic naming as you know.)

That's not to say that machines couldn't be used to provide suggestions based on automatically parsed stuff.

But, as a project maintainer, I definitely know that there can be other projects/modules, which you explicitly want to "discourage" and not recommend to anyone. Because they might be deprecated approaches (but still popular), or because that other projects' functionality has been merged into your project. Or whatever.

I definitely like the bi-directional concept that @joachim came up with here, and I like the human curated part of it.

I don't quickly grok the schema from project_dependency. Seems like a fairly nasty query to answer your question, and I don't have time now to sort that out. Feel free to ping me in a few days if I haven't replied here. You might also have luck asking over at #760890: Display module dependencies on project pages since in theory the project_dependency maintainers would see it there and they might more quickly be able to produce such a list.

I'm very interested in helping tackle this problem. As a proven non-engineer architect, site builder, content strategist, and former Drupal shop owner, I've struggled with this part of Drupal since 4.7.

We can anticipate over 20,000 modules in the near future. We may find Drupal adoption dropping because module usability is very unwieldy when checking out Drupal functionality for the first time (new customers). Module search for customers, is too daunting to comprehend. For developers, it's too time-consuming. You could have built the module before finding, reading, downloading and testing 15 variants. And who has time to keep up?

The search capabilities can certainly leverage taxonomy / solr, etc. but it has to be a multi-dimensional matrix. It has to take into account all types of users, all types of modules, all the Drupal variants (5,6,7,7), the status of the module (active, not active), complexity, configuration, dependencies, usage, critical bugs, etc.

I definitely agree that free tagging should not be used, the variants on terms will balloon and make more of a mess. This is a library of 20,000+ items and good classification, cross-referencing with an eye to all the stakeholders will be very useful.

Of course this ties back to a bigger conversation too about the heavy-lifting module maintainers take on to make Drupal better. I bring that up just to say thank you to those brave and brilliant folks. Hope this is helpful, thanks to everyone for their passion for making module search better.

+1 to flag eco-projects and also +1 to relations for projects

+1

At this point, this is probably something that will have to wait until after the Drupal 7 upgrade.

Also, +1 to what joachim is suggesting. Would be a very nice thing to have.

mentioning Flag and Zen, change use of flag to a boolean field

So the Drupal 7 upgrade has happened. I definitely agree with @joachim's suggestion!

Marked #1321434: Make it easier to identify sub-modules as a duplicate.

comment deleted

comment deleted

It's re-opened. Sorry for the confusion.

I applied the patch here: https://search_api-drupal.redesign.devdrupal.org/

Went to enable the feature:
/admin/structure/features

I can then go to https://search_api-drupal.redesign.devdrupal.org/ecosystem-projects/path...

But I can't find the ecosystem links on https://search_api-drupal.redesign.devdrupal.org/project/libraries

I'm not sure what I'm missing.

I've attached a patch which works better (I think) with the devww environment. It should be identical though to what you've provided.

[mgifford@devwww drupalorg_user]$ drush features-update drupalorg_ecosystem
Module appears to already exist in sites/all/modules/drupalorg/features/drupalorg_ecosystem
Do you really want to continue? (y/n): y
Created module: drupalorg_ecosystem in sites/all/modules/drupalorg/features/drupalorg_ecosystem [ok]

Well, that didn't seem to be the issue. I'll try and reach out on IRC.

This could be a partial alternative to #480356: Implement Community Tags for d.o projects, so a late +1 from me.

The links to the dev sites are outdated... any chance of getting a fresh dev site for this?

The content type for project has been split ... I've adjusted to add the ecosystem field to project_module and project_theme.
Updated the patch, I'm working in a sandbox ... https://www.drupal.org/sandbox/helmo/2771139 on the feature/ecosystem-1323826-HEAD branch.

Test plan: (executed on https://eco-drupal.dev.devdrupal.org/)
- apply a patch from this issue
- enable the feature 'drupalorg_ecosystem'

I added some demo content:
- https://eco-drupal.dev.devdrupal.org/project/hosting_git
- Edit
- add "Hostmaster (Aegir) (195997)" in the Ecosystem field

- https://eco-drupal.dev.devdrupal.org/project/eldir
- Edit
- add "Hostmaster (Aegir) (195997)" in the Ecosystem field

TEST:
- View https://eco-drupal.dev.devdrupal.org/project/hostmaster to see the extra resource link
- View the new page: https://eco-drupal.dev.devdrupal.org/ecosystem-projects/hostmaster

See https://eco-drupal.dev.devdrupal.org/ecosystem-projects/hostmaster and https://eco-drupal.dev.devdrupal.org/project/hostmaster for a demo

I'm concerned about introducing the "ecosystem" terminology here. Does that broadly communicate the concept to Drupal developers? To Drupal developers from many language backgrounds?

I asked in #infrastructure about doing something like this and found this issue after discussion with Mixologic, I think the discussion is worth pasting here:

realityloop: is there any way to search modules on d.o based on their dependencies?
Mixologic_: what sort of search? do you want to know what depends on a module, or what a module depends on?
realityloop: so essentially I'd like an easy way to search for all modules that depend on context for instance
Mixologic_: wonders what that query would entail
realityloop: it would have to inspect info files I imagine.. possibly not something thats possible..
Mixologic_: no, its possible. project_dependency already does that so we can pull in dependencies for the testbots
Mixologic_: so the query is made more difficult by the fact that that sql is a bummer for storing bidirectional relationships
Mixologic_: we've got "what does this module, in this particular release, depend on" in the database
Mixologic_: so to get the inverse, which is "which modules depend on a module" means we need to figure out if we just want the most recent release of module, or if it has ever relied on context.
Mixologic_: and to top it off, you probably want to know which *projects* most recent release contain a module that depends upon context..
realityloop: would be nice :)
realityloop: would be awesome to have a view that showed all dependent modules on a projects page. would help a lot with visibility of modules you might want to use in a project..
Mixologic_: the other challenge is the conflict in the data model between projects and modules. Should we show other projects that contain a module that is dependent upon, say, context_ui, if it existed?
realityloop: it becomes a possible rabbit hole.. but gut feeling would be yes if it was possible. I'd be happy to help sprint on it in the afternoon at Dublin sprints, I'm running the first time sprinters workshop in the morning
Mixologic_: So how about "show project names that have a release containing a module that is dependent upon any module in the project being displayed" ?
realityloop: that seems like what would be ideal
Mixologic_: is hacking together a query
Mixologic_: couple of other things to think about. We've got drupal modules, and we've got composer dependencies as well, so we both want to display "what this project/module depends on" as well as "what depends on this project module"
Mixologic_: https://gist.github.com/anonymous/73445728b98ae7c2469d327041402e7f save this gist as a starting point to get the necessary data
realityloop: thanks.. I'll create an issue about it too, do you mind me pasting copy of log here into the issue?
Mixologic_: no problem
Mixologic_: one thing to consider is a UX for popular modules. i.e. Context has 187 modules that list it as a dependency: https://gist.github.com/ryanaslett/d6c0bb46006630ed4b48b1453a77af27
realityloop: might want to show top [n] based on usage, with link for more
Mixologic_: heh. I know just the joins to make that happen.
Mixologic_: there may also already be an issue. I know that *certain* users can see the project dependency data on release nodes.
realityloop: this looks like it's close to my thinking one it's utility https://www.drupal.org/node/1323826
Mixologic_: speaking of that one, project_dependency historically has only kept parts of the .info files. I recently had us add the "package" line into them
Mixologic_: So the "package" line that allows modules on the module page to group together, we have that data now in our database, so we can start to use that too.
Mixologic_: so restricting to just things that call themselves part of the "Context" package, we end up with only four projects: context, context_profile_role, context_flag, context_panels_condition
Mixologic_: but that has a lot to do with us not backdating that data.
realityloop: yeah I wasn't after ones that call themselves part of the project.. because that depends on developers selecting that when they make module.. I think dependencies is better as much more likely to be set in a useful way

Thanks @realityloop,

I like the idea of having this automated through dependencies. The new ecosystem field this issue is trying to add would also depend on developers filling it, just like the "package" line in info file.

Another downside of using 'just' package information is that a module could be part of multiple ecosystems.

We lost the dev server to the cloud and I've rebuilt the dev site at https://eco-drupal.dev.devdrupal.org/project/panelizer. The patch also needed a re-roll, which is attached.

I should also mention that the actual deployment of this will have to happen at least after DrupalCon. The way Features gets fields configured involves a lot of field cache clearing one after another. Unchecked, I've seen this take Drupal.org offline for over 30 minutes. That said, it is doable to get this deployed a less-hectic week.

Maybe we can use the dependency information we have to pre-fill this field for existing projects.

We could query the most recent stable release for a project, and all the projects that depend on that.
@realityloop already made such a query for #760890-32: Display module dependencies on project pages
This would solve the problem of it not being very useful until many maintainers have added ecosystem information, while preserving the freedom tu finetune it for maintainers.

Another nice follow-up would be to add this as a facet on https://www.drupal.org/project/project_module

Bump.

Yes please.

It seems so obvious. What can the problem be?

This would a great start to find a way to identify ecosystems. RTBC++

I re-deployed this on a fresh dev site, https://eco-drupal.dev.devdrupal.org/, to make sure everything was still working well.

I’ve committed some fixes, including:

• Re-roll for upstream changes.
• The field instance configuration was inconsistent between module & theme projects. Form weight was missing from module. Display wasn’t hidden on theme.
• The View’s empty message wasn’t displaying because it seems it might have been developed outside of a Drupal.org dev site, it had the more modern filtered_html input format (Drupal.org has been upgraded from previous versions, so many of our input format names are numeric).
• Made the View path more-consistent with the site, project/…/ecosystem
• Finished moving the code to assign $machine_name
• Only add the “Projects that extend this” link if there are projects that extend it.
• Replace a loop since we can use array_column() now.

Still to do:

• The View’s header is also missing because of input format configuration.
• This field is only added to module and theme projects. Is this intentional? This could be useful for Community projects as well. In general, we keep all the project content types in sync, except for project_translation, which is deprecated.
• Write a snippet of PHP to put the new field instances into the “Project classification” field group, which isn’t currently exported into a Feature.

Write a snippet of PHP to put the new field instances into the “Project classification” field group, which isn’t currently exported into a Feature.

foreach (['project_module', 'project_theme'] as $project_type) {
  $groups = field_group_info_groups('node', $project_type, 'form', TRUE);
  if (isset($groups['group_project_classification'])) {
    $groups['group_project_classification']->children[] = 'field_project_ecosystem';
    ctools_export_crud_save('field_group', $groups['group_project_classification']);
  }
}

This has been deployed.

Tested this out, and had some issues.

I went to http://www.drupal.org/project/views_field_view and added it to the Views ecosystem.
After saving I see Ecosystem: Views on the project page

On the Views project page, in the right sidebar, I see Projects that extend this
Clicking takes me to https://www.drupal.org/project/views/ecosystem

Which lists Views as the only module.

Adding another module (https://www.drupal.org/project/views_filters_summary) does not change the ecosystem page, it only lists Views.

I cleared the Views cache and this seems to be fixed.

Confirmed that it now works as expected, thanks!

I tried adding my module https://www.drupal.org/project/facets_block to the Facets ecosystem, but I can't find the Facets module in the Ecosystem autocomplete field. Am I doing something wrong?

Confirmed what @gnikolovski mentioned, the autocomplete likes to show other suggestions even when one matches exactly what you typed.

The same happens for Devel ecosystem.

Firsty, It would be good to see this issue reopened by a maintainer as it is of ongoing interested to the community.

Secondly, I'd like to add a feature request for some filters to be added to the ecosystem listings so that it is possible separate results by major drupal version.

Thirdly, the ecosystem lists do not seem to be complete (as mentioned by others). Take Commerce for example. The results suggest that there are approximately 80 related projects, when in fact there are many more. Commerce for Drupal 8 has more than this number of payment service providers alone.