Decide on the future of the user guide

+1 to Use GitLab pages

If I may make a suggestion: let's try out what would happen if we convert the English original to MKdocs. I can only speak from my own experience. Aciidoc is not necessarily the most technically simple solution for motivating the average Joe to contribute to the continued existence of the Drupal user manual. I remember that it looked me at least 4 years to get a glimpse on how to work with that syntax correctly.

Markdown should be simple enough for many people to use. MKDocks automates the build process with one script and one general configuration file per language. Even converting existing HTML into Markdown is not that difficult. MKdocs material could even be hosted using github pages.

There is the great MKDocks material, that looks awesome

If I can make a wish, please let's work with it, it will definitely look great in the end result and is much easier to manage than aciidoc.

Whatever we do here I would like to see us reduce the burden on the DA for maintaining one-off infrastructure and workflows that the current system creates.

Personally I would love for this to remain in Git, and not become standard nodes. For all the same reasons outlined when we originally decided to go this path. I think it lends to better editorial control, consistency, and supports things like translations in a way that would be really hard without VCS. And selfishly, as the person who currently does the majority of the content and screenshot maintenance for the guide doing so via editing individual nodes would be a nightmare.

If we keep doing the importing to Drupal.org as nodes I think moving the build process to GitLab and creating the HTML artifacts there would be fairly straight forward, and a great idea.

The feeds import process and surrounding customization would need to be rebuild for Drupal 11. This is not ideal for ongoing maintenance.

We also import the user guide content on Drupalize.Me, and use a custom migrate source plugin and a couple of migration YAML files that's far less code, and easier to maintain, than the asciidoc_feeds code used by Drupal.org. I'm happy to share that work if it's ever useful.

Also, regarding the conversation about converting the files to Markdown. Drupalize.Me has code for importing markdown documents like this into Drupal via the Migrate API as well. And would be open to sharing that too.

The user guide is not part of www.drupal.org and does not appear in Drupal.org site search.

This for me is a big reason for not using GitLab pages (or Mkdocs) or anything else that is self contained. I'm worried that it potentially makes it less discoverable. I'm not opposed, something like learn.drupal.org could be cool. But I think burying it in GitHub pages could have a negative effect on people's already pretty negative view of Drupal's documentation. There are some benefits to going this route too, if for example we built a static site out of the user guide it could potentially be more shiny and have a different kind of impact on people's perception of the documentation. I think ideally if we do go this route that the site is themed to be consistent with the Drupal.org brand so that it continues to be as official as possible.

Converting to Markdown is an interesting idea (aside, there are static site generators that support asciidoc natively). Doing so would require figuring out a few technical things, here's a few I can think of off hand:

• Ebooks. Asciidoc has far superior support for ebook generation. Though I'm sure it's possible with Markdown. Probably requires something like converting to a single .md file, and then using pandoc to create the books.
• Markdown doesn't support reference style links. In asciidoc for example we can do [[extend-module]] and it'll be converted to a link to the document with that ID and the text of the link will be the title of the document. We use this a lot in the content of the guide. I'm sure we could come up with a custom solution in Markdown, but that's a thing we would need to figure out.
• Indexing. Asciidoc supports the creation of an index by using ((Term,term)) style metadata in documents. This is another thing we use a lot. And again, could probably be figured out with some kind of frontmatter in Markdown but isn't something Markdown supports natively.

I've tried using tools like pandoc in the past to automate conversion to Markdown as an experiment. But because we use features of asciidoc that have no Markdown equivalent it fails pretty miserably. So the conversion would either be manual. Or someone would need to script a custom conversion tool.

I think it would be a significant amount of work to convert things. And I'm not convinced that doing so would suddenly get more people contributing. Personally, it doesn't make much difference to me if I'm editing the files in asciidoc or markdown so I'm happy to do either.

And ... I think that if it is converted to markdown there are a lot more options for using something like Mkdocs and just general support in that many more people use Markdown on a regular basis.

I tell you what I did.

Used mkall.sh and built the guide just as usual
Then I used this little guy with this particular addon and all I have to do to get the 11.x version of the guide now, is to fix the tables.

Besides that, importing the HTML files from the guide's German version worked effortlessly.

So perhaps there is a chance we can host markdown files directly on GitHub pages in the future so we can iterate much faster then?

Converting markdown into other formats like epub and mobi as well as PDF shouldn't be that hard I like to test that as soon as I have fixed the tables in the markdown of the guide. If it helps I can take the English source and upload a fixed markdown version here.

mkdocks is, the right approach, at least iin my opinion. This way, we can continue to work directly in the GitHub editor and, if necessary, simply build a new output every x days or hours. Even I can manage that, and I think I know a lot less about DevOps than some other members of the Drupal community. Perhaps a version written in Markdown would just help to speed up the whole process, and thus relieve the DA relatively fast.

On drupal.org domain

I agree @eojthebrave:

 

The user guide is not part of www.drupal.org and does not appear in Drupal.org site search.

This for me is a big reason for not using GitLab pages (or Mkdocs) or anything else that is self contained. I'm worried that it potentially makes it less discoverable. I'm not opposed, something like learn.drupal.org could be cool.

Whatever the solution ends up being, keeping it on the drupal.org domain and not on a third-party domain, is crucial in my opinion.

Version control

Personally I would love for this to remain in Git, and not become standard nodes. [...] I think it lends to better editorial control, consistency, and supports things like translations in a way that would be really hard without VCS.

I agree -- this is the way to do it, to keep it structured and coherent, and importantly translatable. And you're doing a stellar job with that already, so thanks for that!

Use Markdown

@eojthebrave:

Also, regarding the conversation about converting the files to Markdown. Drupalize.Me has code for importing markdown documents like this into Drupal via the Migrate API as well. And would be open to sharing that too.

@joachim namyslo:

Used mkall.sh and built the guide just as usual. Then I used this little guy with this particular addon and all I have to do to get the 11.x version of the guide now, is to fix the tables.

Markdown is a joy to format text with, so I would definitely second that. It would be interesting if you each did a trial run of the conversion process, and shared the results, so that the process could be compared for efficiency, quality, ease of use, and so on?

All valuable points absolutely worth finding some freetime to read through all your arguments. However, until then just briefly dropping here a question: is this thread the good place to think about whether we want to consider updating “The Next Big Edition” of the user guide for Drupal CMS instead? Or maybe I have missed the point when it has been already decided that the User Guide must be about Core?

Having a Drupal CMS User Guide is a great idea @baluertl!

Since Drupal CMS builds on top of the content of the User Guide, it could be added as a new Chapter 14, before the last chapter, like this?

• Chapter 13. Security and Maintenance
• Chapter 14. Drupal CMS
• Chapter 15. Final Thoughts

... or it could get its own Drupal CMS User Guide?

Glad to hear others also like the idea. From what I gather about the Starshot project through its communications, its aim seems to focus on serving an audience that Drupal Core alone has struggled to engage: non-coder site builders who prefer working exclusively through a graphical user interface. While I can’t speak on behalf of the original authors of the User Guide (Jennifer, Amber, Joe, and many others whose contributions deserve to be honored, though I regretfully can’t recall all their names), my experience working on the Hungarian translation left me with the impression that it was written with a very similar audience in mind.

As Drupal 8 was era-shifting in its terms, I believe, Drupal CMS is also expected to be so too. Publishing the 1. edition of the User Guide at the time of such a groundbreaking change was an extremely valuable aid to assist many over the gap. I think Drupal CMS's initial adoption could be further supported if we dedicate the 2nd edition of the User Guide solely to them.

Oppositely, I don't really see the point in mixing Drupal Core × Drupal CMS in the same book. It just makes no sense at all to me, because those are two very distinctive products sold to different audiences.

Adding Drupal CMS documentation related issue: #3454527: [META] Track 3: Drupal Starshot documentation. There were also some Slack discussions about this in the #starshot channel.

I think the most important point from there (which was also mentioned in #4 and #7 here), is to have the docs on drupal.org domain and indexable by search engines, not on drupalcode.org, which is hidden a not indexed (currently).

It looks like a Drupal CMS guide is underway, see the comment #3454527-13: [META] Track 3: Drupal Starshot documentation posted just now.

In light of Lenny's recent updates, it seems there’s no need to shift the target audience of this User Guide project to focus on the Drupal CMS product. A dedicated documentation effort is already being handled by the excellent team at Drupalize.me. As such, I’m withdrawing the point I raised in #8, as it’s now clear this isn’t a concern.

That said, I’m left wondering whether Drupal Core (which is primarily aimed at developers, unlike Drupal CMS) truly needs a hands-on guide at all.

That's a perfect point.

Over the past 13 years, I've encountered Drupal developers who prefer reading English texts rather than localized ones.

Users just starting to explore software like Drupal are very grateful when you provide them with information about the software in their native language.

These are two completely different points of view.

I would like to know whether we host the documentation directly on drupal.org or use a static page generator like Vitepress.

I don't think we can easily activate the multilingual features of Drupal for drupal.org. But maybe someone responsible for d. o. can tell more about that.

Nevertheless, I would be happy if we could localize the documentation for Drupal CMS.

The question that concerns me here is whether this would really work if we did it through the documentation on Drupal.org.

If the answer to that question is yes, then nobody needs any new system to continue offering the user guide in multiple languages.

I bet Drupal will used by more people in the future when they can easily use it. Easy use means getting the whole UI + a great onboarding in their native language unconditionally.

I've started work on migrating this project to use Zensical (formerly Mkdocs). This is based on the DA suggesting Zensical as the tool they will likely support for importing content onto Drupal.org. See https://git.drupalcode.org/project/gitlab_templates/-/work_items/3568729. And it's what Drupal CMS is using now for the Drupal CMS guide.

It looks like Zensical will support translations. And most (if not all) of the Asciidoc features we use.

The end result is that the source content will be in Markdown instead of Asciidoc once the migration is complete. Assigning to me for now while I work on this.

Oof, artifacts are too large on pages:deploy job. That could be a blocker.

Also, that's amazing that you were able to do all this in a single day, @eojthebrave!

Converting this to markdown has actually been on my radar for a very long time and I've made steps towards it in bits and pieces in the past so had a pretty solid foundation to start from.

I was able to get GitLab to build a preview of the EN and DE versions of the guide. As is mentioned above, the artifacts created by the build are too large to deploy if all languages are built. This is probably because every language contains all the images (they often contain translated text) and it's a lot of images. And maybe we can get the DA to adjust the limit for this project. Here's an example where it failed before I configured GitLab to only build 2 languages for now https://git.drupalcode.org/project/user_guide/-/pipelines/790727

But this proof-of-concept is working a least. Still some bugs to iron out.

Preview: https://project.pages.drupalcode.org/user_guide/preview-74/en/ or https://project.pages.drupalcode.org/user_guide/preview-74/de/

The MR here includes:

• Migrated all the .asciidoc files to .md for all languages. This was done with some scripting, so it's possible there are glitches here and there since I haven't checked every single page in every language yet.
• Added support to build the markdown files into a static site using Zensical. This will create a static site that works on GitLab pages. And also aligns with what the DA is working on with regards to being able to import this content into pages on Drupal.org
• Markdown doesn't support asciidoc style reference links. Right now the Markdown files all have links like this [file-name](./file-name.md) replacing all the old autoref links. And a script that will create a copy of all the .md files with those links updated so that the [file-name] is replaced with [Title of the linked page].
• Removed all the old asciidoc scripts and what not
• Tried to make sure all the docs for contributing are updated as well, but I'm sure I missed something.

What this does NOT include:

• The old toolchain supported the creation of e-books. PDF and epub format. This current setup doesn't have that yet. Though it should be possible to achieve if we still want to support those.
• There's no index any longer. Asciidoc supports adding indexing tags to documents and then during the build process collecting all of those and creating and Index for the book. Neither Markdown nor Zensical have that. We could in theory do something with frontmatter in the .md files, but I'm honestly not sure it's worth it. This is the page that's missing in this update https://www.drupal.org/docs/user_guide/en/_index.html. Thoughts?

The page has broken link to node for "URL Aliases"

thank you! impressive works and mostly no changes needed as some modules removed from core

Sorry for the noise, I was playing around a bit with different approaches to getting GitLab pages to build this.

I chatted with Neil and Fran a little in Slack about GitLab CI hitting the artifact build limit when generating a preview that includes all language sites and here's what we figured out.

First, we can convert all the PNG images to WebP and save a good amount of space. I'm going to write a script for the repo that does that, and then run it and commit the results. That'll also give us an easy way to compress/convert any future images.

That by itself doesn't get us below the GitLab CI limit though.

Which for now at least might be okay. The guide content is ultimately going to be imported to Drupal.org via some custom PHP code that the DA maintains which reads the Zensical config directly from the repo and essentially does its own build. So the importer doesn't need the GitLab CI artifact at all.

The end result is NOT a GitLab pages hosted site. It's nodes on Drupal.org.

Where we do still want the the GitLab build is for previews. So I'm going to wire that up so that it always builds the English content and one other language. And then will only build content from any other language if that languages source content changes. That way an editor working on content will always be able to see a preview of the page(s) they are working on. Even if you can't view all other languages.

Thanks for working on this @oejthebrave!

About file format for compression and quality, AVIF seems superior, and in the example it converted a 1.4 MB PNG down to just 120 KB, where WebP ended at 180 KB. AVIF is very close to WebP in terms of support:

• AVIF: 95.36%
• WebP: 96.83%
• JPEG: 97.03%
• PNG: 97.27%

I would think the target audience for the User Guide mostly use modern browsers, and Drupal made AVIF the default file format in #3524206: Use avif with webp fallback for all core image styles. So perhaps AVIF can be considered? Or perhaps it would be trivial to switch from WebP to AVIF at a later date? I just fear it might not get done, which is why it might be worth considering using that format from the start.

Oh interesting. I hadn't really looked at AVIF at all before. The little I've read makes it sound like WebP might still be better for text heavy screenshots. Which is what these images all are.

Regardless, this makes me think that we should really just keep the lossless PNG files in the repo. And then let whatever tool is used to build the display version of the content do the image conversion. Basically, keep the originals.

I was doing this as an experiment to see if I could get the overall artifact size down enough to not trigger GitLab errors. But that didn't work. So maybe reverting to PNG images stored. And then having the build step do the conversion is a better approach.

Keeping the original PNGs for conversion into the final release sounds like a good plan. And great point about situations like this, where pixel-perfection is desired, and perhaps neither WebP nor AVIF should be used, but instead possibly compressing the PNGs, if still possible?

Actually, from the article I linked (and elsewhere) for screenshots, PNG is still recommended:

Lossless PNG compression
Don't use lossy formats for screenshots with text. The compression artifacts around letters look unprofessional. Lossless PNG compression (oxipng, pngquant) gives you 20-60% savings without touching a single pixel.

So the PNGs could (if they are not already) get compressed as a final step with one of these tools, whichever one is easiest/most efficient?

• https://github.com/oxipng/oxipng
• https://pngquant.org/
• https://tinypng.com/

Okay, back to PNG for the canonical screenshots. Though I haven't run them through any PNG compressor yet. I also removed the code in the Zensical build that was doing PNG -> WebP conversion for now too. I'm thinking keeping this simple so we can get the asciidoc to markdown conversion done. And then we can re-visit any changes we want to make to the Zensical build.

I also updated the GitLab CI configuration so that it will build the en and de languages by default. As well as any other language that has changes to it in the MR being built. With an exception for MR's that touch all languages.

Personally, I'm feeling like this is probably good enough that we could merge it and then open new issues for anything that needs to get resolved post asciidoc to markdown migration. Especially given that neither the asciidoc or the markdown importers for Drupal.org are working right now so we're not going to break them. And this gets us closer to being able to test the markdown importer.

Plan in #24 seems good to me since the change set is large.

It seems like the merge request needs a rebase.

Good call. Rebased to clean up all my WIP commits and CI testing. Then updated with `git push --force-with-lease=3470837-decide-on-the:e39e9e7d user_guide-3470837 3470837-decide-on-the`

Okay, I'm tearing the band-aid off and merging this.

First, I tagged the final pre-markdown conversion commit `git tag 11.x-asciidoc-final 7cfd3465` incase we ever need to find it again.

Then I merged this in to the 11.x branch. This makes 11.x incompatible with the drupal.org import, but that was broken anyway so ¯\_(ツ)_/¯. And now we can move forward with getting the Zensical based import working when the DA team has time to assist with that.

I'm sure there will be some issues to sort out with the Zensical conversion but we can open issues for those as we encounter them.

Thank you everyone for the ideas and assistance. Hopefully this leads to a future user guide that is more up-to-date and easier for everyone to contribute to.

Next up, I'll ping Neil at the DA again and figure out what steps we need to take to get this building on drupal.org. And open an issue for the known task of figuring out how to get all languages (not just EN and DE) to build on gitlab pages by raising the maximum deployment artifact size.

Fantastic @eojthebrave, thank you for keeping the momentum, and pushing this one over the hill!