Build a *.Drupal.org site for the User Guide and possible future books

I can answer question 2 in comment #1: Yes, the editorial workflow is via the User Guide project, which is using issues to manage changes to the source in the Git repo.

Regarding question 1, I believe there will not be many pages, but I would think there might be a page or two explaining what the user manual is, with links to the User Guide project for potential contributors? I don't really know, maybe someone in the Docs WG can comment.

Regarding question 3, multi-site search sounds like a good idea.

But I am not sure what you mean by "tagging" pages or "relating" to other pages. I believe that the only relationships between pages on this proposed sub-site and pages on Drupal.org would be links between them (in both directions)... I don't know what type of "tagging" you are referring to.

OK. I don't think that the DocsWG has a strong opinion on whether this should be on its own site or on drupal.org. The important points are:
- The guide will hopefully be massively multilingual. However, at the moment the Asciidoc Display module doesn't really integrate with Drupal languages. I was going to file an issue on that point today actually.
- The guide is not really directly related to drupal.org pages in any way.
- We want to make sure people can find the guide, and either view it online or download a PDF or other e-book.
- We want to have a Jenkins job that builds/updates the online content and the e-books.

I think there may be just a very small number of editable pages on the site, that a few designated administrators would need to be able to log on and edit, such as a home page explaining what the site is. We could possibly live without this by designating an AsciiDoc-Jenkins-generated page as the site home page.

The rest of the content would be generated from Jenkins scripts.

Should be doable. That's if it's a separate site at all and not on drupal.org directly.

Regarding "without any security issues", at least I thought the plan would be to display the AsciiDoc through a Drupal site, and someone would still need to be the site admin.

I am not sure about that, actually, the more I think about it.

Ideally, we'd want to:
- Display the HTML output of at least two books on the site. Right now we already have two: the Contributor guide and the User Guide to Drupal 8 itself. There may be more.
- Display the User Guide and possible follow-up volumes (like maybe "Advanced Guide to Views" or something else the docs team might decide to do in the future) in multiple languages, with a language switcher.
- Also provide links to download the PDF and other e-book formats.
- Allow people to edit and/or translate using the on-line editors, and download the resulting source text files.

Much of this functionality would require some kind of scripting rather than being static HTML. I would think Drupal would be the logical choice for a framework to put the scripting in?

Updating summary with some of the recent discussions.

That's already there:

[discover] We'll need to direct people to this Guide, and ideally if they search drupal.org these pages would come up in the search results

Perhaps not written the best but there. ;)

So... Any decision here? I still think that the best option is a Drupal-based site, because we want people to be able to edit on-line. We already have a Drupal module that allows for that, and I do not really want to (a) redo that code so it is independent of Drupal or (b) maintain this hypothetical independent code.

I also think that for navigating a set of potentially many languages and many books, the Drupal options (navigation blocks, etc.) are a plus, rather than probably having to maintain an HTML-based navigation separately, which seems like a pain. It's definitely possible to output *1* book as HTML, but not so easy to automate the navigation without some kind of scripting... and as I stated before, if we need some kind of scripting, it seems like Drupal would be the logical choice.

Given that, I still don't know about www.drupal.org vs. another site. But I'd like to not have to manually build and transfer the files over to the temporary site every day. It is getting to be a pain, and right now I am the only one who is doing it, so it's a bottleneck.

Maybe at least we could set up a Jenkins job that would do the build/transfer part on the temporary site (which we could then use as a proof of concept for the real site)?

Sounds good! That will be very very helpful. I am currently the only person who is updating the site and it's kind of tedious.

The workflow in #16 is incomplete. Actual workflow that I am doing:

a) Maintainer commits to the 8.0.x branch of project/user_guide

b) I do a local git pull and build the HTML output on my local machine. I am currently doing this using a script called "mkoutput.sh" in the scripts directory of the repository. That script builds both the HTML and e-book output, but for purposes of the dev site, it could stop after just doing the HTML. The script has copious comments in it so should be fairly easy to figure out. There is also a README file that explains what tools are needed (command-line tools to build the HTML output from the AsciiDoc source).

c) I ssh to devwww.drupalsystems.org and do a git pull on /var/www/dev/userguide-drupal.redesign.devdrupal.org/userguide_source

d) I use sftp to transfer my local output to /var/www/dev/userguide-drupal.redesign.devdrupal.org/userguide_output -- after running the script, this is located in my local project/user_guide/output/html directory and I transfer the en and guidelines directories into userguide_output

e) Clear the cache on the dev site using Drush

I am unfortunately not going to be available much or at all for the next 2.5 weeks to help in getting this done... or for that matter for updating the site, since I'm going on a bicycle trip. Sorry.

I'm back from my trip; sorry for the delay in responding here.

Thanks for setting up the git pull job -- that seems to be working (or at least, when I ssh'd into the site, the git pull was up to date except for the modification to scripts/mkoutput.sh, which I'll look at shortly on the other issue).

Regarding asciidoc versions, I am running 8.6.9 locally, vs. the 8.4.5 version you are running.

Also, I am running 0.0.25 version of xmlto, vs. 0.0.23 on the dev site.

Can we update the dev site to those versions?

Hm, that I don't know. I will have to check and see what the differences are between the output being created on my local machine (which we like) and the output on the d.o machine (which I haven't really looked at). I'll take a look at some of the intermediate output and see what is going on. Probably tomorrow.

Or maybe today. ;)

So, it looks like the asciidoc command is the problem. The version on the devdrupal.org machine is not producing the right output from the source, and the output cannot be parsed by the xmlto command. For instance, it is not finding the assigned section IDs to assign to each topic, so that cross-references to topics can be made. Then when the xmlto command is run, there are all sorts of validation errors about not finding the cross-referenced sections. I'm not familiar with the history of AsciiDoc, so I'm not sure when/why that macro for putting in cross-reference IDs changed, but that seems to be the main problem.

To look at the xmlto step, I copied in the guide.docbook (intermediate docbook file created by the asciidoc command) from my local machine and ran the xmlto command. This then at least ran... But the output is slightly different here and there (some different classes added to divs and the like) and it is not working quite right with the AsciiDoc Display module. It is *mostly* OK but not perfect.

xmlto is basically just an engine for applying XSL style files to XML output (docbook is XML). So probably the difference between your version and mine is that the XSL style files are slightly different. So... xmlto allows you to have an "overrides" file (and we already have one) to use different XSL style commands here and there. This means that I could probably coerce this to work. It's pretty close to the output the AsciiDoc display module is expecting, so it probably wouldn't take too much to fix it up.

So:

a) We definitely need to update the AsciiDoc command on the machine that runs the scripts. The version on there now is not working at all, and there is not an easy way to override it or fix it.

b) Ideally we would also update the xmlto package, but I could get around this with some work in the scripts and/or AsciiDoc display module if necessary. That might be a good idea anyway, since others theoretically using this module may also be using different versions of the xmlto package and/or its XSL style files.

For information on installing AsciiDoc, see
http://www.methods.co.nz/asciidoc/INSTALL.html

Thanks! I'll test this out later today.

The xmlto script has additional issues. It ran for a while after complaining about grep, and then it stopped at:

/usr/share/xmlto/format/docbook/xhtml: line 10: /usr/bin/cp: No such file or directory

so apparently both cp and grep are misplaced. I wonder why they would have those hard-wired in the script -- seems very stupid.

Anyway, those two errors should be easy enough to fix, as you said, with symlinks. There may be others, but it looks like the script was pretty much done by that point... probably just had to copy its output from a temp directory somewhere into its final destination? Not sure.. I didn't look at the script.

I also took the output from the asciidoc command and copied it to my local site. From there I was able to run xmlto and verified it works fine for display. So it does look like the new asciidoc script you installed works fine.

Anyway, if you want to test whether all the necessary symlinks have been made, you could do this:

cd /var/www/dev/userguide-drupal.redesign.devdrupal.org/userguide_source/scripts
./mkoutput.sh --no-ebooks

That will not overwrite anything important, and if it runs without any errors, it's probably fine. I don't think I probably have permission to make symlinks on that server in /usr/bin. Or that you'd want me to if I could. ;)

Thanks!

The script runs, and the output looks good now. Thanks!

So. What we would need the Jenkins job to do is -- and this is all relative to the userguide-drupal.redesign.devdrupal.org directory on the dev server:

a) git pull on the userguide_source directory [it is doing that now]

b) Remove the userguide_source/output directory completely (to get rid of old output).

c) cd to userguide_source/scripts

d) run ./mkoutput.sh --no-ebooks

e) If successful, replace what is currently in the userguide_output directory with the contents of userguide_source/output/html

So there should end up being userguide_output/en and userguide_output/guidelines directories.

Yes, seems to be working great! Thanks!

The dev site that the User Guide project is currently using seems unstable. I would really like to see this project get a permanent home as soon as possible.

The first step is that we still need to figure out whether this permanent home is on drupal.org or a different *.drupal.org. And then get it into the drupal.org infrastructure plan.

Any thoughts?

The dev site has been a big pain for us throughout the development of the User Guide, which we started at the end of June.

At least we now have Jenkins scripts that have removed the need for me to manually update the site whenever we make a commit -- THANK YOU @drumm.

But the site itself has been very problematic. For instance, so far we have had two different URLs -- dev sites are really not meant to be used for longer-term projects like this, and at some point between last June and now, the infra team changed how dev sites were being done and we had to move this one and the URL changed, which meant we had to go find all the links and change them, which was very painful. We had a note on the old site for a while telling people about the new URL, but with the latest crash that has been lost.

Then we have also had to deal with the .htaccess login, which even though we tell people as often as possible how to log in, they still don't get it all the time. And the dev sites have been unstable and/or slow multiple times, not just this week.

So it would be really very helpful if someone could just make a decision one way or another what we want to do with this, and do it. Having an active development project for something as (I think) important as the User Guide to the Drupal project, living on what is supposed to be used for one-off temporary development projects, is not really working all that well. We're making do but the sooner we can have a more permanent home, the better.

And just to say this one more time... really having a static HTML site is not going to work well, because of the online editing feature, which a static HTML site would not have. Even after the English guide is done for 8.0.x, we will need to make updates for 8.1.x, 8.2.x, ... 9.0.x, etc. (all of them may have UI changes), and these versions are coming out every 6 months I think. Also there will hopefully be translations.

I personally am comfortable editing in a plain text editor as I'm very familiar with the formatting, but most people aren't and having an online editor with buttons for the formatting, and preview capability, has been VERY helpful. I think it will continue to be. So I wish that the static HTML option would be taken off the table. There's no way to offer editing on-line in a static HTML site that is just displaying the User Guide output, and we really do need editing.

Our development site for the User Guide has been down with some MySQL error for almost a week, although I understand the infrastructure team is supposedly working on fixing the problem... no sign of a fix though.

The User Guide project is really suffering from the lack of a more permanent and supported home -- we've had quite a bit of down time throughout the lifetime of the project so far. Is there any hope of moving this forward?

Updating the issue summary with the current URL...

Thanks for making a new site again!

I have created the AsciiDoc config items, plus put back the needed blocks and set up permissions. The site is back up.

Hm. I also tried to turn off the page cache from admin/config/development/performance but it seems to be locked on. Why is that? On this site, we have run into problems with the page cache because the Jenkins job doesn't clear the cache for the AsciiDoc books, leading to stale content.

The drush cc all should do it. Thanks!

Adding some notes to the summary.

Outcome of meeting today (please correct if I missed something or got something wrong):

Having the User Guide in ordinary nodes on Drupal.org is very desirable because:
- It would be like other documentation (easier to tie things together, theme, etc.)
- Solr indexing

At the same time, we really need to keep the source for the User Guide in Git.

So, the plan is to build a way to import the HTML output of the AsciiDoc source into Drupal nodes in the appropriate language(s). This would be a modification (or sub-module) of the AsciiDoc Display module, which would provide a Drush command to import a "book" into Drupal nodes, and another one to update existing nodes (so it would need to have a database table to keep track of the correspondence). We would plan to do the import of a given language when the guide is deemed to be "stable" in that language, and then update it (by running a Jenkins job that would be available but not run automatically) when deemed to be a good idea.

Features to be added later:
- Ability to do on-line editing (using AsciiDoc Display module, connected to the nodes but reading the source as it does now, should be easy enough since we will have the table with the correspondences)
- Ability to download PDF and ePub ebooks (Jenkins can build and copy to a standard location, and links can be put into a block that is displayed on these pages)

I'll be working on the AsciiDoc Display module to make it do the node stuff.

Updating the summary...

I've also filed two related issues in the AsciiDoc display module:
#2757925: Add a way to link to ebook output
#2757921: Import AsciiDoc HTML output into nodes

@tvn and @drumm: I have some questions about importing into Nodes.... and I may have more. Can you follow #2757921: Import AsciiDoc HTML output into nodes please?

Just to clarify... Editing within nodes that save to VCS is not actually what is being proposed here.

What is being proposed here:
- Source would be in VCS in text files
- We would periodically build the HTML output from the AsciiDoc source, and re-import the output into Drupal nodes (a Jenkins process that we could kick off whenever we decide it's time to make an update, similar to the human decision of when to make a new released version for Drupal Core or a module)
- We will hopefully have available the existing on-line editor that lets you edit/preview on the Drupal site, but doesn't save anything anywhere. Instead you can download a revised .txt file and attach it to an issue (or save it to your local git repo and make a patch file)

The permanent site for the docs is up at
https://www.drupal.org/docs/user_guide/en/index.html
and the Guidelines book is up at
https://www.drupal.org/docs/user_guide_guidelines/index.html
so this is fixed.