Devise an extensible way to document extensible data structures

Part of the problem is that we don't currently have a way of doing a YML equivalent of an api.php file, which would be needed for this, and for an example to do with structures that are created in YML.

So what I might do instead is an example using FormAPI arrays, since those are all in PHP, and we have ways of documenting PHP.

@Crell, yup, that's pretty much the gist of it. Thank you for expanding on what I wrote and making the examples :)

There's a few things where I was thinking along very slightly different lines, so I'll expand on those first here before changing the summary straight away.

First, I was thinking that the tag that defines the base structure could also include the parts of the structure that are defined by the original inventor.

So with the Router example, the Router class recognizes some basic keys in the route structure. So we might have:

/**
 * // ...
 *
 * @datastructure route
 *   A route defines a mapping from an incoming request to the
 *   code that will handle it.
 * - *: The name of the route. An arbitrary string, which must be
 *   unique across the site. It's a good idea to prefix this with
 *   the name of your module.
 *   - defaults: Properties of the route that yada yada (I have no idea what
 *     the difference between defaults and options is, as it happens ;)
 *     - _controller: The callable that will be invoked for this route.
 *       This may
 *       be one of (and this is where we hit a problem, because I want 
 *       to make a
 *       bullet list here, but it's NOT part of the structure!): 
 *       a) a function
 *       name, b) a static method, in the form 
 *       '\Namespace\Class::method', c) a
 *       method on a service, in the form 'service.name:method'.
 */
class Router extends UrlMatcher implements RequestMatcherInterface, RouterInterface {

(Sorry for the awful wrapping -- I had it wrapped to 80 in my text editor, but d.org's formatting seems to be set to narrower, so having to hack at it in haste.)

> I don't quite understand the proposed structure for the @datastructurekey line, though. A lot of / and * and stuff. What does it mean?

With the core structure as a single tree as above this is hopefully a bit clearer:

/*
 * @datastructurekey route */defaults/ _title: The title yada yada.
 */

This says:
- An extra item is being defined, which should be spliced into the tree for the 'route' structure.
- This item should be inserted as a child of the */defaults item, where the / is a path separator in the tree structure.
- The key of the item is _title.
- The description is 'The title yada yada.'.

Basically, to the developer reading the documentation page rendered by API module, it would be as if the _title property's definition had been written alongside the definition of _controller in the documentation for the base structure, rather than added by another component or module elsewhere in the codebase.

The other thing that #8 doesn't cover is that I envisioned that the definition of the base structure could be in some sort of YAML equivalent of api.php files, analogous to the way we document hooks. But I'm happy to drop that for now, as it's an extra complicated thing to figure out. If anything, documenting the base structure in PHP keeps it closer to the code that invents it, which is a good thing.

And one thing I didn't cover in my summary is that sometimes we need to define possible values, such as service tags.

Now for Crell's concerns, which I agree are things that need to be figured out, and I'm not yet sure exactly how.

> 1) It's still using an unstructured format to define unstructured formats. I fear the inception problem, and documentation problem of the documentation.

I'd be happy for this to have a strictly-defined format, and indeed it probably should, since it's to be machine-parsed. Is that what you mean?

> 2) Some keys are non-global, or contextual. The requirements block of a route, for instance, can have a regex for any parameter.

It occurred to me as I was trying to refine my explanation of this (and thank you for beating me to it, I really appreciate it!) that this is maybe slightly related to our config schema system:

core.entity_form_mode.*.*:
  type: config_entity
  label: 'Entity form mode settings'
  mapping:

That's saying 'This is the definition of all configurations whose keys are of this form, with wildcards in these positions'.

One complication I'm thinking about is that there's probably a difference between saying '*/foo' where '*' represents the top-level keys which are arbitrary, such as route names or form API element names, and something where the '*' represents any key from the valid ones (I can't think of an example right now...)

> This could also become an issue with FAPI, where we have keys that are on 1/3 of all form elements

Indeed, here we sometimes want to say that #options is a valid key for elements with a certain value of #type... not sure how we handle that yet.

> I also don't like the idea of parsing bullet lists to figure out what the structure is.

That doesn't seem to me to require handling that's much different to what API module already does for something like hook_entity_info() on D7.

But let me say now that I will be happy to roll up my sleeves and help with the work needed on API module. (I keep meaning to get involved in API module in general...)

> Side note: Annotations are actually documented already as they have a corresponding class that defines them.

True, but I feel that documentation for annotations is very definitely been a step backwards from the documentation we had for info hook structures on D7 -- see #2092757: specialize output for Annotation classes for my reasoning (with pictures!).

Also, annotations don't handle structure extensibility at all. For instance, here in \Drupal\Core\Entity\EntityType, we find:

  /**
   * The route name used by field UI to attach its management pages.
   *
   * @var string
   */
  protected $field_ui_base_route;

Field UI, a lowly non-required core module, has got its mitts into a core component!

> Swallowing that dog to catch the cat to catch the mouse to catch the fly doesn't seem like it leads anywhere useful. At some point we need to just accept that undocumentable, unlintable, informal data formats are a problem, not a solution, and adjust accordingly.

Well it's inventing one new format to document several undocumented formats. So that's an improvement. And the new format would be much simpler that the formats it's documenting. And it wouldn't be extensible in the same way; it would have all of its own documentation in one place. So I'd say it's more like a single spider that's catching lots of flies ;)

Docblock seems a good idea to me because it allows the documentation to be directly alongside the code that is related to it, in the class or method docblock.

But I am not wedded to my suggestion of inventing new docblock tags. The real meat of what I'm suggesting here is the dovetailing idea, so that we document pieces of structures in the module or component that invents each piece.

Another possibility would be using YAML, since it works nicely for documenting schema structures, but it sounds like @Crell thinks that's not ideal either.

> Using for-reals PHP classes and types to define something is nice

I can see that it has advantages. I'm not sure what we can use to represent the structure pieces that are to be dovetailed together though. Have an interface represent the structure a piece is a part of? How do we specify the path into the structure? I'll have a ponder and see if I can think of anything, but I emphasise I'm going to be pluckign things out of thin air...

Finally, might there be existing systems for this sort of thing that we could use? I have no idea what to search for though.