Help Topics module roadmap: the path to beta and stable

As mentioned on #2920309: Add experimental module for Help Topics, we need to discuss here on this issue: should we get rid of (i.e., deprecate) hook_help()?

Adding this question to the issue summary, and editing slightly (rearranged and formatted stuff but didn't change it really).

HTML format error oops.

I prefer to deprecate hook_help() in terms of page elements it could be anything starting from event to block plugin extension

I mean thet "help's level-up" should be focused on placement, accessibility & context of "help storage"

Out of box help block could be enough to get contextual help, but then it could be "hints" to element level and issues like #2933984: 508 Compliance - Tooltips not displayed for keyboard navigation

I personally am not sure why we really need the Help block anyway for contextual help:
- It isn't very contextual. All it offers is a block that collects implemetations of hook_help() that applied to that page, so the only context is "page".
- The text is generally short. Forms could just add it directly to the $form for the page (or whatever else is generating the page) as a formatted text piece and be done with it, which would put the help near the form generating code, meaning it might be more likely to get updated if the page code gets updated.
- And other modules that want to add to that text could form alter it.

@alexpott help_url looks more like relation which were called "list_on" meanwhile it fits into "context" - some internal routes could not be accessible by current user

I would be in support of another meta tag appears_on, which was a set of route names, which would then give us the same per-page contextual help

RE #6 -- Translatability -- We have had this discussion before on the parent issue or its related Ideas issue. Yeah -- it is item (i) in the issue summary of #2592487: [meta] Help system overhaul, so we must have discussed it on that issue. Several times as I recall. And on previous issues where we developed the current hook_help() standard format, which is now many years ago.

Summary as I recall from many discussions of this:
- On localize.drupal.org, translators want no more than paragraph-size chunks to translate. Larger text just doesn't work well for that system.
- HTML tags and things like that are problematic on localize.drupal.org. It's easy to screw them up during translating.
- The translation database also has a size limit of some sort. I can't remember if it was for localize or in Drupal itself. But you don't want a really long help topic all translated together for that reason too. There was another issue about that... let's see, I'm pretty sure it was marked Related... Yes, it's Related on both the Idea issue and the Parent issue. Here we go:
#1885192: Field locales_source.source is not suitable for long texts and huge config objects

I would also note that finding translators for large chunks of text is difficult. The User Guide is a good example. We have the ability to translate it, but after several years and 10 language groups that formed to translate it, we have only 2 translations that are more or less complete (aside from updates). It's just a larger commitment to translate a whole topic, and then to edit the translation later, than it is to log on to localize and translate a few strings in your spare time.

Is there a specific issue to deprecate/remove/replace hook_help() - I really think making this stable should involve removing that.

Yes, if we leave it as a separate module, there's a serious UX regression, because you will now have help in two disparate places and you'll get different help depending on which modules you enable. So at a minimum stabilizing the module means moving the code into the Help module proper and displaying the help topics on the same page as other help.

Even if we do that, I think it might be a UX gate issue to have even more categories on the help page, some of which are modules and some of which are topics that may encompass several modules. So that implies converting existing core hook_help() to be consumed by topics, or at least adding an interoperability layer where hook_help() can specify which topic should contain the help (and adding a rough best-fit topic to each module's hook_help() implementation).

It's also not great to have two systems terms of code duplication and technical debt. From a release management perspective, I think we need to deprecate hook_help() if we add this module. (Which is cool, because hook_help() is very legacy.)

And then given that converting all the hook_help() to the new format is a big task, we shouldn't try to do that before we release Drupal 9, which means that hook_help() should be deprecated for removal in Drupal 10. (This is one of the things we would deprecate for removal in Drupal 10 even if it were deprecated in Drupal 8.) It might be possible to script a megapatch that converts all core help to Twig templates in the relevant locations. (Edit: Though I shudder to think about reviewing that.) Otherwise, this is the sort of deprecation that we might add to the list of skipped deprecations as we proceed through the conversions.

Finally, in terms of the release schedule, 8.8 is likely to be the last minor release of Drupal 8 to add significant new features, so if this module is not stable by October 2019 (when 8.8.0-alpha1 will be released), then it will need to wait until 9.1 at a minimum to become stable. Most experimental modules take 12-18 months to stabilize after they are added to core, so October (8 months from now) would be pretty ambitious. :)

Thanks everyone!

Oh, one more thing. Given that we're planning to move the code, we should either go straight from alpha to stable (not shipping a beta in a tagged release), or ensure that all service names and other machine name patterns have the proper core namespace, and plan to have Help Topics eventually wrap the Help module code (whereupon the beta module would be hidden from the core UI, but not removed before 10.0.x).

I have created
#3031642: Deprecate hook_help() and combine with Topics
and added it to the issue summary. Moved the discussion of hook_help() that was in the summary to the other issue, and added some of comment #13 to that issue too. Let's continue that hook_help() deprecation discussion there.

Also added notes from comments 13-14 about release issues to the summary -- added a new section about release plans.

Adding some more issues, and creating issues for things that didn't have them.

There has been some question about what in this issue is "must" and what is "maybe". Adding notes to issue summary to bring clarity to this -- everything is a Must Have except for one issue that is a question.

Minor clarification.

Looks like we still need a way to resolve the issue of conditionally linking to something if the module is enabled. This came up again in https://www.drupal.org/project/drupal/issues/3047703.

It's not clear to me reading above if this is something we can solve now and I just don't know how to do it. Or if this requires some additional code to be written first before we can solve the problem in topic .twig files. If it's the later maybe we should open an issue specifically to sort that out?

I think we need a new issue. It's related to
#2996305: Add support for the url() function to twig {%trans%}
but not identical.

I created an issue to continue to discuss solutions regarding conditional links in help topics: https://www.drupal.org/project/drupal/issues/3047830

@jhodgdon: Kudos that you are doing such a marathon on this!

I really love the way this brings the core help system. Now bear with me, as our use cases come from a completely different mindset:
As authors of complex Drupal distributions, we need a system for efficiently documenting our sitebuilding.
What we're doing:
* We leverage help_topics in feature modules.
* For efficiency, we use @markcarver's markdown-2.x to use markdown for help topic authoring (translating is a nonissue for this)
* The only thing that was missing then was backlinks from the config we document to the relevant help topics. For this we authored the contrib module Help Topics Backlinks and created #3048650: Add "Relevant help topics" block (or re-purpose help block).

Is this use too much an abuse? Whether or not, i want to let you know that
* even before this is finally committed to core, it is extremely useful in a slightly different context for us.
* this experimental module already has a (experimental) contrib module that relies on it.

Also proposing #3048641: Let module help fallback to help_topic:mymodule.about as child issue.

That is interesting... Right now we are not looking to add anything else to the roadmap for this module to get it to stable. I think your new issues should be decoupled, since the Product managers have signed off on this roadmap idea, and we're trying to get Release manager sign-off too, so we can get the initial patch committed. Adding more tasks will not help us get the patch in... once it is in and stable, then we can think about adding more features.

So, can you remove this issue as being the parent of the new issues you created, and just leave them as Feature Requests in help.module for later? Thanks!

OK, done! HTH!

I can't find issues about redesign or review starting help screen, looking at all discussions it clear that we to define purpose of it in context of hook_help removal and core gate requiring this hook for any module

Adding this suggestion from markcarver to the "maybe" section.

Adding a link to #3041924: [META] Convert hook_help() module overview text to topics to the issue summary, since some of the other issues listed here are blocked on it.

I just created one more issue that I've added to the "Stable" area: #3065632: Add more developer docs for Help Topics

Someone on groups.drupal.org made an interesting suggestion that I turned into an issue and marked as Related to this, but I'm not adding it to the Roadmap, since it's a drupal.org feature proposal and not something we would do in Core:
#3066332: Can we get Help Topics displayed on drupal.org?

Updating summary, as the POTX issue got finished.

We have spun off part of the "more tests" issue into a separate Coder issue. So, adding that to the Roadmap Beta section.

Adding another spin-off issue to the roadmap.

Fix typo in summary.

Updating summary for current state of parent/child issues that need to be done vs. are already done, because #3066512: Add checks for syntax and display of help topic Twig template files has been moved back from Coder to Drupal Core, and so the follow-up issue to make sure that those Coder checks are performed is no longer necessary (that is, #3066510: Add test coverage for help topic syntax by applying coder coding standards).

Adding a new issue for Beta, as I ran into difficulties making the Configurable Help module integrate with this.

Here's a fun question: where can we find documentation for the experimental help topics module :-)

I added a task to add some documentation here: https://www.drupal.org/docs/8/core/modules

Refreshing issue summary to show progress.

Updating one item.

Forgot to mark that new issue as Related.

@markcarver added a new blocker on that last issue, so adding it to the summary.

Found a bug... #3079810: core/help_topics directory does not work... probably post-beta is OK for that one? And this other bug: #3079595: The Help breadcrumb builder does not set cache contexts

Just adding a note here. Not sure where to put them into the roadmap...

Updating summary with previous comment and another update.

Ah. We had a line item in the Stable list for writing docs, but I just noticed we already had an issue in the list for that, so moving those notes to the issue instead.

Can you elaborate how many of the 'maybe to do' items are important for beta given the timeframe we have between now and 8.8.0

Thanks

This is getting closer to beta, but #3069109: Replace help_topic meta tags with front matter while there's been lots of activity on the blockers, is still PP-2. Is it OK if that doesn't get done prior to beta? If it doesn't, then it would probably still be possible to make the change from 9.1.x onwards.

The Maybe To Do item is really 1 item (with two dependencies that are outside of Help Topics, but need to be done before we can do our issue). It is really up to the Core Committer team (framework managers?) whether it can be done after Beta or needs to be done before.

The issue in question would mean changing the format of the meta-data section of every Twig help topic file, as well as some small corresponding changes in the HelpTopicDiscovery class to detect the meta-data using the new Core code coming from the other two issues our issue depends on.

So, the problem with doing it after Beta is that we are working on writing a bunch more topics, plus Contrib might also be working on topics, so the longer we wait, the more topics need to be updated. Also, it seems like doing it after Beta would violate the Experimental modules policy:

Beta experimental modules are considered API- and feature-complete for the minimum viable product.
...
Developers can begin using beta modules' APIs, but should be aware that some things may still change to address bugs.

Since this changes the format of all help topic files, it seems like it is the API, and it needs to be stable before we go to Beta. So I think if we do it after Beta, we would possibly need to put in a BC layer (ugh!). Or push the change off to D10?

So we need a decision from I think the Backend Framework managers about whether #3069109: Replace help_topic meta tags with front matter is required for Help Topics or not. Adding tag. I've updated the issue summary on that issue, so you can see what the change would look like, and there's a patch on that issue showing the changes to both help topic Twig files and the Discovery code.

What we need from the Framework Managers is a decision on whether we want this new API for Help Topics to be required (i.e., if we want to change to this new syntax for meta-data in the Twig files). If it is going to happen, it probably needs to happen prior to the Beta release of Help Topics. That will undoubtedly push the release of Help Topics as a beta Experimental module out at least 6 months, because the two dependencies of that issue are not really making quick progress. Either that or we could add it later with a BC layer that would allow help topics to have either syntax.

My opinion is that we should not make this a requirement for Beta. I don't think the meta-tags current syntax is that bad, and I think postponing Help Topics based on this syntax change is not a win for the project. I guess that maybe we need Product Manager review to weigh in on how important getting Help Topics into Core is vs. having a (maybe?) nicer syntax for the meta-data. So maybe you can discuss it at the next Committer meeting?

The only reason that issue is blocked is due to Symfony/Twig essentially blocking #3064854: Allow Twig templates to use front matter for metadata support.

I'm going to go over that issue again to try and see if I can decouple it in any way.

I don't think the meta-tags current syntax is that bad

As I've stated before, the introduction of these <meta> tags just adds to the confusion of "how to supply additional contextual data" that we're already currently doing in core:

1. Hooks
2. PHP Annotations
3. YAML Definitions
4. Now... <meta> tags in Twig templates

These <meta> tags are a one-off for just help topics and are not used anywhere else.

IMO, this should either implement proper YAML definitions or, as a compromise to inlining YAML definitions, wait for the proper implementation to support front matter inside Twig templates (which can kind of be a blending of 2 and 3).

My personal opinion is it shouldn’t be help topics module’s problem. If we get an API for metadata in twig, then we convert and have a BC layer and then its Help topics' problem. But until then, we want to be using the feature. And it doesn’t look like a quick fix unfortunately because of the issues with Twig extension points that may require upstream changes.

So what I'd be in favour of is using help topics now and dealing with a BC layer in D9 when we get frontmatter support.

For the record, I think the frontmatter proposal is excellent, and support unifying *.layouts.yml in the same fashion.

I've asked other committers to chime in too.

And it doesn’t look like a quick fix unfortunately because of the issues with Twig extension points that may require upstream changes.

Yeah, that isn't going to happen (at least not anytime soon, if ever). I've altered the original approach to the front matter issue to essentially avoid Twig as much as possible. I hope that instead of just talking about it, people actually help to get it in.

I really don't think it's wise to introduce yet another definition paradigm just for the sake of "stability" here. It is help_topics problem because it's what introduces it.

If you really want this to move forward, just help the front matter issues real quick. There's really not much left to be done.

I was asked by @jhodgdon for my opinion on the placement of #3064854: Allow Twig templates to use front matter for metadata support within this initiative's roadmap.

My read of that issue is:

a) There are /already/ multiple ways of adding template data to HTML (*.layout.yml for Layout Builder; <meta> within twig files for Help System)
b) The reason for the deviation with <meta> in Help Topics seems to be that we were trying to include the metadata with the template file ala Doctrine annotations for DX (vs. Layout Builder is more like its own concept, I guess?)
c) This issue proposes to use what seems to be a fairly emerging defacto standard “front matter” for embedding metadata within templates (basically:

---
yaml: foo
---

…at the top)
d) The upstream Twig people do not have buy-in to it becoming part of Twig (despite repeated, valiant attempts)
e) Lauri and WIm seem in support of the idea.
f) But, the patch from *checks watch* an hour ago is a totally fresh start, indicating the likelihood of this happening in time for alpha (which it would need to) feels pretty slim.
g) And even if it did, if we require Help Topics to then refactor on top of it prior to it becoming beta, that (further) endangers their timeline.

So, IMO, no. The front matter issue seems like a very nice feature to have. But it should be evaluated entirely independently of Help Topics (IOW, it would be a "should have" in this module's roadmap). If that issue makes it in, and we run out of time prior to core's beta deadline to the convert this module to use the new API, then we’d need to provide a BC layer. (Same as we will have to do for Layout Builder regardless, since that's already shipped as stable code.)

I think comparing this to Layout Builder is comparing apples to oranges given that Layout Builder was using a previously established pattern for specifying its configuration. Help Topic is proposing to introduce a completely new pattern for configuration.

I'm feeling very troubled to make a call in any direction. I feel like we shouldn't use the proposed meta tags approach given that it would end up adding a significant amount of technical debt, but on the other hand, I don't want to prevent this initiative from making progress.

Could we use the most recent approach proposed on #3064854: Allow Twig templates to use front matter for metadata support to add support for front matter as part of Help Topic module? This could allow us to make progress faster given that the solution would be first included only in an experimental module.

I like @lauriii's last idea in #54, that seems worth a go. If there turn out to be major blockers there though then I don't think we should block beta stability on it.

I don't think we can get #54 to RTBC/commit in a week, as a patch entirely within the Help Topics module... do you?

#3069109: Replace help_topic meta tags with front matter is in :D

Updated IS and moved it up a bit.

Obviously, disregard #56. Nice work! :) It has added some work on the two remaining issues, but that's nearly handled...

Adding #3086075: Use Twig to strip Twig syntax from help topics files in the syntax checker to the Stable roadmap (thanks @chx). [already is a child issue of this one]

I created a POTX issue that is a follow-up to #3069109: Replace help_topic meta tags with front matter.

Adding it to the Stable section of roadmap.

Adding two follow-ups from #2664830: Add search capability to help topics to stable section of roadmap.

Added #3087027: [policy, no patch] Mark the help topics module as beta stability 🎉

Related makes search in help available after module install

I asked in Slack and found out from @xjm that all of the code *and topics* related to the Help Topics module need to stay under core/modules/help_topics until we are Stable.

This means that all of the help topics we write need to go into core/modules/help_topics/help_topics for the time being.

I am rearranging the issues under Needed for Stable, and splitting/combining things, so that we can make that happen. Also will need to update all of the "Write topics for..." issues so that we put all topics into that directory instead of spreading them out.

Adding #63 to Stable section.

Adding 2 issues to roadmap. #3087496: Change prefixes of Help Topics machine names to help and make sure APIs are marked @internal for beta, and #3087499: Merge Help Topics classes into Help with BC layer for final merge.

Adding #3087667: Set up class aliases for Help Topics classes to Beta section.

#68 was wrong, we don't want to do that after all, my mistake. Removing that one from the roadmap.

Search results can use admin theme, unpostponed #3087879: Use admin theme annotation for help topics search

Added new issue to roadmap. Also reorganized stable section slightly.

Added new issue to roadmap.

Someone has posted #3113236: Help topics module - documentation page not found. It's not unusual for a module's handbook page to not be completed until it's approaching stable, but we should probably at least create a stub page or verify that the URL in the codebase is not wrong.

Will respond on that issue...

added #3144933: Add help_search plugin to the base admin theme test

Commited #3087879: Use admin theme annotation for help topics search

Removing an outdated POTX issue.

#3064854: Allow Twig templates to use front matter for metadata support got in. Do we have a follow-up issue for Help Topics to use the now-in-core /core/lib/Drupal/Component/FrontMatter/FrontMatter.php class instead of our own? I don't see it in the Roadmap here...

See previous. Adding 3 follow-ups to roadmap.

Because we have
#3090659: Make a way for help topics to generate links only if they work and are accessible
in the roadmap, there is no need for
#2996305: Add support for the url() function to twig {%trans%}
for purposes of help topics.

We're making progress on this!!! The help topics on #3041924: [META] Convert hook_help() module overview text to topics are down to 4 at RTBC, 1 at Needs Work, and 1 postponed on one of the RTBC issues. And then we have 1 coding issue here at Needs Review (should be close to done), and 2 that need some coding work. (Mostly adding this comment to refresh the issue list.)

Adding follow-up issue to Roadmap.

Also just a note: the topics issues are now all RTBC!

Adding follow-up issue to Roadmap. #3194120: Need a way to say it's time to run cron

commited - #3087218: Help searches fail if site is not fully indexed, and users do not know why

rtbc - #3086075: Use Twig to strip Twig syntax from help topics files in the syntax checker

needs works - #3192585: Fix up topics to use new help_topic_link function

3 child issue left in #3041924: [META] Convert hook_help() module overview text to topics

Adding follow-up #3213022: When generating link to non-existent help topic, put topic ID in fallback text to roadmap.

Issue to use the new link functions got split into two pieces, so adding the second piece to the Roadmap.

typo in summary

Editing issue summary to bring it up-to-date with its current status and timeline for Drupal 10.

Added a current status:

- Currently (as of Drupal 9.3.x), Help Topics is an experimental module and located in core/modules/help_topics.
- We have 4 issues remaining in our roadmap to stable.
- Drupal 10's first beta release is scheduled for the week of May 16, 2022. Help topics is not a high priority, so it would be best to target mid-April 2022 (at the latest) to RTBC all remaining path-to-stable issues, if Help Topics has a chance of being marked as stable in Drupal 10.

Edited the release plan to update it for Drupal 10.

Added a note about deprecated modules with help topics, and added links to the issues (that I know of).

Another IS update clarifying "needs discussion" issues on roadmap to stable.

I think it's time to make active #3037230: Finalize the merge of Help Topics into Help and #3025577: Move help topics to core or the correct modules

as only 2 issues left in #3215784: [META] Fix up topics to use new help_route_link function

Updated IS

- Closed as useless complexity #2960220: Do we need hierarchy/order for help topics?
- Now unrelated as topics using twig #3068483: Automatically map !translate YAML tag to TranslatableMarkup
- Frontend does not need #3075427: Create TemplateDiscovery for plugin managers to use all that time

The only actionable issue left is #3041924: [META] Convert hook_help() module overview text to topics

As last blocker fixed it needs new issue to consider and set module stable

Remaining minors
- points in #3121340: Fix up minor copy problems in help topics are only related to future renames in UI
- extra test addition #3219923: Add tests to enforce correct use of help_topic_link and help_route_link functions is follow-up to #3090659: Make a way for help topics to generate links only if they work and are accessible

I set active #3037230: Finalize the merge of Help Topics into Help because we can't deprecate hook_help before help_topics is stable and replacement for the help block is ready

So I filed #3355659: Mark Help topics as a stable module

Updated is

- not a part of roadmap but great follow-up #3031642: Deprecate hook_help() and combine with Topics
- ordered issues as

- first we need to merge #3087499: Merge Help Topics classes into Help with BC layer
- then move topics #3025577: Move help topics to core or the correct modules
- clean-up tests

New issue happened #3368825: Standardize how external links are wrapped with Twig trans/endtrans tags in help topics as consequence of #3121340: Fix up minor copy problems in help topics

Help topics are merged into help, now polishing

next target is #3025577: Move help topics to core or the correct modules

Added #3204810: Add new permission for accessing help pages to IS as it will have collision with #3144933: Add help_search plugin to the base admin theme test

I find #3074040: Move testing of help topic rendering into child of GenericModuleTestBase optional from 2 remaining issues from summary, no reason to slowdown tests for testing install of shipped topics within each module

So now it needs to announce deprecation of help_topics module and "long live help topics in help" at the same time

So left overs are

- #3121340: Fix up minor copy problems in help topics
- #3144933: Add help_search plugin to the base admin theme test

I bet we can mark the issue fixed now, 4 years!

The remaining issue from summary should be postponed on deprecating hook help #3074040: Move testing of help topic rendering into child of GenericModuleTestBase

Also there's #3087499: Merge Help Topics classes into Help with BC layer which is RTBC and waiting for release manager to set it fixed.

So I bet we can set this issue fixed as well and continue with remaining issues in help module's queue

Deprecation of hook help moved to plan to consider BC #3031642: Deprecate hook_help() and combine with Topics

Updated the IS. Per @andypost in #99 (and I, as co-maintainer, concur), moving some items in the IS that were in the section "Needs discussion" to "Not part of the roadmap".

1. Moved #2960220: Do we need hierarchy/order for help topics? to the section "Not part of the roadmap".
2. Moved "#3064854: Allow Twig templates to use front matter for metadata support, and if that is done, #3085972: Replace Drupal\help_topics\FrontMatter with Drupal\Component\Utility\FrontMatter" to the section "Not part of the roadmap"
3. Moved #3074040: Move testing of help topic rendering into child of GenericModuleTestBase to "Not part of the roadmap" as it's not a part of this roadmap, but a child of #3031642: Deprecate hook_help() and combine with Topics (per #111)

Hopefully the IS now clearly reflects that there are no action items remaining.

Edit: formatting

The last child been marked as fixed #3087499: Merge Help Topics classes into Help with BC layer

Consider maintenance of help module now going by the help module core's issue queue

Adding credit.