[meta] Index all of contrib modules at api.drupal.org

It would be another branch, so it'd be head, 5, 4.7, contrib-head, contrib-5, etc. All that'd have to be done is to set up the script to check out the new branches and add 'em to api.module config.

Oh and then 95% of contrib would basically be unsuitable because at the moment very few people document for api.module. I do in bits and pieces, but not completely. And even then there still need to be topics.

TO do contrib right we'd need to make it easy to get right to the documentation for just one module; I'm not sure it'd be good to have api.module mixing all of contrib together. It'll be very difficult to find stuff.

So maybe what we need to do is make contrib.api.drupal.org and create some way for module owners to create the appropriate branches for their module? That way the documentation can be properly contained.

I totally agree that managing branching is hard, exponentially harder when multiplied by contrib. I'd suggest just targeting contrib HEAD in the first instance, to get this rolling.

Yes, a stand-alone subdomain would be appropriate, not mixing it with core.

I first mooted this idea to illustrate that many contrib modules are incompletely documented. My thoughts are that if a system published a modules dirty laundry, it would be a great step forward in bringing the maintainers attention to this side of their code.

For folk Like Tony who are volunteering somewhat to improve things, I'd note that

Contributed patches for doc-blocks are also welcome for submission in project issues

and can probably be folded in easily as they are simply comments which should not require testing.

A cross-reference to the brilliant (if pedantic) coder.module should also be published.

I'm aware, however that contributed patches of code-style clean-ups are much harder to review and should probably be done in the first case by the real module maintainers. Wasn't there a 'code review' script that used to log lots of complaints against the contrib codebase?

To answer a few of Merlins points, I personally think that the odds are better than 95% against WRT existing modules. My experience with picking up and looking at old projects has been good.
95% of them need tidy-up work but even then, they are mostly OK.

WRT topics, I wonder if api.module (and phpdoc) don't already support that already. Perhaps documenting the usage of the @group function at the top of modules (in the @file) block would add module-name topics to module docblocks. I've not tested, but I think that's already available.

As for management ... I don't want to think about branching, but maybe an opt-in button on the project page (like the current 'link to CVS') would be a smooth start.
If that has to migrate to the 'release' definition to really work right, OK, but not yet, Mkay? Already I have trouble keeping track of which doc version I'm looking at on api.drupal.org. (tho having the history available is enlightening)

I'll go play with api.module again... see if I can add some more guidelines

Unless I miss my guess, the two most visited documentation pages would be for Views and CCK; and for Views, without branching the documentation would be pretty bad. So I'm not sure launching without branching for contrib would be a good idea.

This is a good idea.

But, as mentioned, in contrib there are many branches.

In many cases, many of them are actively maintained.

Indexing HEAD only is problematic, since many module maintainers don't bother with HEAD until a new Drupal version is out, then they sync HEAD to the latest version they were working on, then go from there. So for the longest of time HEAD would be stale/unstable.

So, do we index DRUPAL-5--1, or DRUPAL-5--2, or --3, ...etc.?
Or do we index all of them?

If all of them, then how will the interface look like for selecting the branch?

dww's new proposal is for branches to be just DRUPAL-X--N, then M, O, P, ...etc. with the number incrementing. So, I hope there are only (say) a max of 10 branches per Drupal version.

A list box? A tree?

This requires 2 things:
- API module CVS version is currently about 90% done with a menu rewrite. That needs to finish and be running.
- The server performance and hardware situation needs to be firmed up a bit.

In parallel, I will help make API module good for indexing plenty of modules. This is good for everyone, not just Drupal.org, since Drupal shops tend to accumulate modules and can maintain an internal documentation server. Please send your submissions in over at the API module project page.

The core 5.x patch queue is down to 42 patches needing review today, so I will focus a bit more time on API module.

I've also thought about this some time ago, but after looking at the code of API module I came to the conclusion that API module needs to be refactored to support zillion projects, branches and versions like in contributions. Otherwise, you won't be able to administer the API documentation.

Another thought regarding this was to develop an integration between API and Project module. Module maintainers should be able to control which branches and versions should be indexed. None by default. And there may be branches or versions that should not be published anywhere. Since each project defines its own branches and versions, API needs to query this information from project module anyway to provide a custom menu for each project. Following this, an integrated API documentation, available on f.e. http://drupal.org/project/admin_menu/api, should be the goal.

Until we get our act together, Ax's doxygen powered docs are up again for Contrib. Lots of juicy info there: http://drupal.kollm.org/doxygen/_contrib/drupal-contrib-phpdoc/modules.html

I removed the big blockers:
* No more use of the preg_*() 'e' modifier. I didn't find any specific exploits, but decided that removal of all generated executed code is best.
* The menu rewrite is working well. There are some rough edges as noted in the API module issue queue.

Still need to figure out:
* What sort of hardware resources would this take?
* Add a concept of 'projects' to the API module, possibly integrate directly with project and/or CVS modules. I don't think parsing everything at once with the existing API module would be good for the UI.

I think this would be a very helpful feature, although I don't think that contributed modules should go into api.drupal.org. Mixing cleanly documented core API with ugly contributed API documentation is a bad idea... Maybe a contrib.drupal.org or something?

I think contrib.drupal.org would be a better name indeed.

+1 for contrib.drupal.org in #9.

I think it's easiest to let the user search for a module first (Views, for instance) and that at the next page the interface is the same as at api.drupal.org, including how all branches are shown. Perhaps it's best not to show certain older branches; hide branches 5.x-1.0 till 5.x-1.3 by default if 5.x-1.4 is the latest release. Hiding development branches might not be such a good idea, because it allows for a nice and quick peek for developers. Now I'm writing this I come to think showing branches the same way as at api.drupal.org may take up too much space. A vertical list in a block would offer a better overview.

I think only branches should be indexed as there are millions of different tags that could be available. Here are the branches that I think should be indexed:

HEAD

DRUPAL-5
DRUPAL-5--2
DRUPAL-5--3
DRUPAL-5--4
DRUPAL-5--5
DRUPAL-5--6
DRUPAL-5--7
DRUPAL-5--8
DRUPAL-5--9

DRUPAL-6--1
DRUPAL-6--2
DRUPAL-6--3
DRUPAL-6--4
DRUPAL-6--5
DRUPAL-6--6
DRUPAL-6--7
DRUPAL-6--8
DRUPAL-6--9

Although only few modules actually make it to a version 4 branch, it would certainly cover all traditional Drupal version naming....

The update status XML tells us exactly what to tags index for each project and how to label them.

I also think contrib should be indexed (contrib.drupal.org or contrib.api.drupal.org, i don't care). However, I don't think every module out there should be in there: only modules which explicitely request it (through a release node? through their project configuration?) should be allowed in. Since indeed a lot of people don't care to document their code properly, this will make sure that only people that do (or that pretend to) will show up there.

Plus, that would be an easy kill switch for people that don't actually provide useful API documentation. Finally, having a configuration dialog would also allow to specify which *branches* are to be indexed (and this could be used for the drupal core too... want to deprecate 4.7? just edit the drupal project...).

I don't know how to scale the API module however. It's not designed to deal with multiple projects... maybe create a site per module (with Aegir ;)?

It would be super awesome if we could just have the documentation as a tab on the actual project page.

#14: see #13. We can start with indexing the most recent supported version for 5.x, 6.x, 7.x, etc. They all go on one site, but #218306: Allow multiple, partitioned paths to be indexed needs to be done first.

#15: The API documentation is one page per file, function, group/topic, constant, global, and class. It doesn't really fit into a tab, and will be on a separate site, api.drupal.org. It will be a link toward the top of the sidebar on a page that looks something like this prototype: http://infrastructure.drupal.org/drupal.org-style-guide/prototype/module...

I registered drupalapi.org last year and never did anything with it. Last year, I set it up to index my sites/all/modules directories for both Drupal 5 and Drupal 6 branches. The result was what you see at http://drupalapi.org . As you can see, there is some header documentation that would be nice to have depicting which modules are available.

Another nice feature would be documentation for directories? So you could visit http://drupalapi.org/api/directory/admin_menu/ and get summary information from the admin_menu .info file? I don't know, just throwing around ideas here.

Closing old issues. Please re-open if needed.

This should still happen. We have Views & Example modules in API now, trying out how the UI works. We should ramp up more projects.

Moving to appropriate queue.

We've actually been working on this for a while. There are a number of issues tagged 'api.d.o contrib', in various projects, that need to be resolved before this can be done.

So I'll make this a Meta issue... Here's a search for the issues needing to be resolved before we can actually put all of contrib on api.drupal.org:
https://www.drupal.org/project/issues/search?status[]=Open&issue_tags=ap...

This is per-project information, which people would search for from the project page, so I really agree with #15 in having it available from the project page.

When it comes to what to index and when, what about indexing releases when they are created? When you write some code against a module, you usually do it against a specific release you are using.

There is definitely some logic in #23 for well-maintained contrib modules. But on the other hand, my experience with many contrib modules is that I have to run the -dev version because maintainers do not make releases often enough and features and bug fixes that I need are not included in the latest release. Indexing the released code would not really cover that.

This proposal is also very different from what we do for Drupal Core, where we index each branch (not connected at all to releases), and we update the branches continuously, so that api.drupal.org stays up-to-date with the latest development code with very little lag.

I just used that issue search in #23 (for the "api.d.o contrib" tag, across all projects) to figure out what the list of issues is and add them to Related Issues. Including the closed ones. Also updating issue summary with what needs to be done to actually make this happen.

As we are moving to branch forks and merge requests. Is it possible to have one centralized repo where all the contrib modules are indexed. So that we can search the repo for API information ?

That is the intention of this issue, so we can have all of contrib on api.drupal.org. But it has not happened yet.

Also will be good to provide a code search tool through all modules, as requested in #3301944: Create a service to search in code of all Drupal Contrib Modules.

If we are talking about all contrib modules, GitLab already offers that functionality and is probably more powerful than anything we can build: https://git.drupalcode.org/search?group_id=2&scope=blobs&search=hook_menu

Wow, that's great! But when I use the default search (just typing "hook_menu" in top navbar - it shows me no results!
Here is the url: https://git.drupalcode.org/search?search=hook_menu&nav_source=navbar

And I've decided that it doesn't work at all, and don't continue to try to investigate and change search options. And most of the other users will be confused too!

Seems the "Group = Any" condition doesn't work, because if I select "Group = project" - I see the results!

So, can we change the default search behavior to give suitable results (search in "group_id=2") with default settings? Or fix the search with the "Group = Any" condition.

Yeah, that seems to be the default behaviour on GitLab. Even gitlab.com does the same: https://gitlab.com/search?search=hook_menu&nav_source=navbar

You need to select a group to look into, and that will give you not only code matches, but commits, comments, and many others. Once you select a group like this:

Then you get all the tabs: https://git.drupalcode.org/search?group_id=2&scope=blobs&search=hook_menu

If the feature to return code search results without selecting a group ever gets added to GitLab, we'll get it automatically too.

Thanks for the detailed explanation, yeah - this is a GitLab issue, and I've found a report about it: https://gitlab.com/gitlab-org/gitlab/-/issues/14232
By the way, GitHub has introduced a separate code search tool on https://cs.github.com/ and it works great!

We only have the “project” group indexed since the others are issue forks and sandboxes. If we indexed everything, the index would get bloated with duplicate results that are not useful most of the time.

Since GitLab is a product, there is no way to add in-context hints directing people to search within the project group.