Problem/Motivation

On #2522024: Build a *.Drupal.org site for the User Guide and possible future books, we decided we wanted the ability to import AsciiDoc HTML output into Drupal.org nodes.

The import needs to be language-aware, and updatable.

Proposed resolution

(done) Split the AsciiDoc Display module into a base module, direct display module, and a feeds module. The combination of base + direct display will work how this module used to work: allow you to directly take AsciiDoc output for a book, and display it at admin-defined URLs on a Drupal site. The combination of base + feeds will allow you to use the Feeds module to import AsciiDoc output for a book into nodes of one or more content types.

(in progress) Get this actually working on drupal.org, with a Jenkins script that will allow us to update the imported content when necessary.

Remaining tasks

This is mostly working on a dev site:
https://guide-drupal.dev.devdrupal.org/docs/user_guide_guidelines/index.... (the Contributor Guide book)
https://guide-drupal.dev.devdrupal.org/docs/user_guide/en/index.html (the User Guide itself).
Note that source editing has only been enabled for the User Guide, not the Contributor guide.

Outstanding issues and questions:

  1. [fixed] Make sure it works with translations. See #2774891: For Feeds import, handle translations.
  2. [Done!] The User Guide itself needs to have summaries of each chapter/topic added to the source. See #2775689: Create summaries for topics in the User Guide
  3. [done, removed from HTML -- but only works for the topic pages, not the chapter pages] The page title is duplicated in the body of the HTML. There are two ways we could fix that:
    a) Remove it from the HTML when we import. [kind of difficult, but the better solution]
    b) Hide it with CSS. This should work:
    .asciidoc-display-main-content h1.title,
    .asciidoc-display-main-content h2.title {
       display: none;
    }
    

  4. [done, removed from HTML] Remove the navigation area from the body (previous/next/up). Again, the easiest way would probably be to hide it with CSS, but removing it from the HTML source would be preferable. If doing in CSS, something like:
    .asciidoc-display-main-content div.navfooter {
       display: none;
    }
    

  5. [done] The body needs to have a text format that does not turn line breaks into P and BR tags. This text format does not currently exist on drupal.org. Note: as a bonus, this text format is set for use for only Administrators, which (coupled with the drupalorg node access permissions) locks the nodes to editing except for Administrators.
  6. [fixed] When importing, a link is being added to the Main Navigation menu for the main guide book, each time the import is run. That needs to not happen.
  7. [fixed - set author to "System Message" and edited view so this doesn't generate a maintainers block, so block is gone] We will need to set up the Guide Maintainer field when we import, or else hide the Guide Maintainers block on these pages. How do we do that? And what should we set it to?
    Note: it looks like this is governed by the Group field, and then each Group has Maintainers. We can use the "System Message" user to be the group maintainer, and edit the View so it doesn't display this user, but what Group should we use, and how do we set it on import?
  8. [done] Change the URLs of created pages to start with "docs" not "documentation".
  9. [done] Figure out if there is a better way we can deal with the copyright/attributions information rather than displaying it on every page. The issue is that for regular d.o pages, you can see the revision history on the page to figure out who contributed to it. But on these pages, the attributions/contributions information is not there in the revision history. It is instead in the Attributions section of the Guide. So, we need to link to that somehow (perhaps in the sidebar block?).
  10. [done] Permissions: We need to get rid of permissions for comments and editing nodes on these guides. Also decide who should have the "edit asciidoc source" permission (only logged-in users).

    For editing, we are OK due to the text format. Only administrators can use that text format, so the nodes cannot be edited by the general public.

    For commenting, we need to set the comment permissions when we import. Had to add a custom feeds field mapper to take care of this on import.

    For edit source, this is a normal Permission, so it can be set on the admin/people/permissions page.

  11. [done] Local tasks: need to hide the local tasks menu for these guides.
  12. [fixed] CSS/formatting issues in the Body area:
    #2711127: Vertically-centered tables are difficult to read.
    #2718975: Need a better font for documentation pages
  13. [fixed] Add some text to the Editing page that gives people a clue about what they are doing, how to create an issue and a patch, etc.
  14. [fixed] Apparently the Guide pages are importing as the wrong user, see comment #59.
  15. [fixed - vanished when we made this Edit page not a local task of node/NID] Formatting for the Edit AsciiDoc Source page is bonkers. Local task tabs are up in the header. Text heading for the page needs a clear:both in the CSS so it doesn't go to the right of the page title.
  16. [fixed] On non-imported docs pages, the Edit AsciiDoc Source link is still showing up in the Local Tasks. This should not be. To fix it, use a custom loader in hook_menu() and return false/null or whatever it is that indicates a 404/403 so it will vanish. Also make it a Callback not a Local Task anyway, so that the only link is from the block.
  17. [fixed]Before we go live, we need to change the settings on the Parser so that the "ignore hash" setting is turned off. This will mean that it will only update a given node if its source has changed in some way. (During the debugging process, it is handy to have it set to ignore this, so that whenever you import, all nodes are updated whether Feeds thinks they need it or not.)
  18. [done] To finalize, we need to export various things into patches/features. See Deployment Steps below.
  19. [done, removed] Remove the "Source file" lines, or move to bottom of the page. See #2788151: Move "source file" line to end of output, or remove it, for Feeds display.
  20. [done] Add missing space in "AsciiDocsource" in the Source information block (needs update to the Features patch).
  21. [figured out] Make the Drush commands for importing feeds work. See also #2788265: Add a --source option for drush feeds-import

Deployment Steps

Steps that will need to happen to get this onto drupal.org:

  1. Set up an area for the user guide source and output to be built. It should be at
    /usr/local/user_guide_source
    The output of AsciiDoc, for input into Feeds, gets built in the output/html_feed directory, and under that: /en (for the guide) or /guidelines (for the contributor guide)

    This should be a git clone of the User Guide project https://www.drupal.org/project/user_guide

    So basically, run:

    cd /usr/local
    git clone --branch 8.x-2.x https://git.drupal.org/project/user_guide.git user_guide_source
    
  2. Whenever the source is updated and you need to build the AsciiDoc output (for import into Drupal with Feeds),go to the "scripts" directory in the source area and run ./mkfeeds.sh
    ====> Some software may need to be installed in order for this script to run. It is already on the dev site. See parent issue #2522024: Build a *.Drupal.org site for the User Guide and possible future books

  3. Install needed modules. Using Drush and/or Git. Note that job_scheduler is required for feeds, and version 7.x-2.0-alpha3 is what was downloaded on the dev site. The dev version of feeds is required for Drush integration (latest beta release does not have the Drush integration).

    Commands....

    # Run this from sites/all/modules:
    drush dl feeds-7.x-2.x
    drush dl job_scheduler
    drush en feeds feeds_ui
    drush dl asciidoc_display
    drush en asciidoc_display asciidoc_display_feeds
    drush cc all
    drush cc drush
    

    Note that Feeds UI can be disabled on the Production site.

  4. Follow the instructions in the AsciiDoc Display project's README to install libraries for editing AsciiDoc source, under INSTALLATION - EDITING. This will create directories asciidoctor, asciidoctor_images, and markitup under sites/all/libraries.

    Or better yet, unzip the attached asciidoc_edit_libs.tgz file into sites/all/libraries -- much easier.

  5. Set up permissions for:
    - administer feeds [Administrator only]
    - administer asciidoc [Administrator only]
    - edit asciidoc source [Authenticated user]

    Add these to the Documentation feature.

  6. From admin/structure/types, in the Publishing Options vertical tab for the Documentation Page and Documentation Guide types, set them to multilingual, with translation.

    Add this to the Documentation feature.

  7. From admin/structure/types, add a text field called field_asciidoc_source_file to both content types (this will store the source file for the imported nodes). Use a hidden widget for it (so it is not on edit screens).

    Add this to the Documentation feature.

  8. Add a new "Asciidoc Import" text format, without the "turn URLs into links" and "automatic paragraphs" filters. Set it so only Administrators can use it.

    Add this text format and its permission to the Documentation feature.

  9. Create a feed importer for the User Guide on admin/structure/feeds, and one for the
    Guidelines book.

    Add these to the Documentation feature.

  10. Import these feeds by visiting /import (Feeds UI module) and using these source directories:

    /usr/local/user_guide_source/output/html_feed/
    /usr/local/user_guide_source/output/html_feed/guidelines/

    Thereafter, you can import via Drush:

    drush feeds-import asciidoc_user_guide_guidelines
    drush feeds-import asciidoc_user_guide
    

    Note that the first time you go to /import you will need to specify the directories to import from. Then when you use Drush, it will remember this and use the same directory. There is not a way to export this setting into Features. It is stored in a database table feeds_source, and this table is not exportable.

  11. Place the "AsciiDoc Display Feeds edit source" in the sidebar, in Panelizer (admin/structure/panelizer). (It will only display for pages that were imported and for Feeds importers that have the editing source set up.) for Documentation Page and Documentation Guide content.

    Add this to the Documentation feature.

  12. Edit the view for Group Maintainers to exclude the "System Message" user (user ID 180064), which is the user ID that the nodes are created by.

    Add this to the Documentation feature.

  13. Add code from comment #100 to the drupalorg module so that it doesn't display the local tasks. This goes into drupalorg_menu_local_tasks_alter().
  14. Make a patch for Drupalorg module with all of these feature/code changes.
  15. Make a Jenkins job, triggered on new releases for the User Guide, that will update the imported pages.

Comments

jhodgdon created an issue. See original summary.

jhodgdon’s picture

Assigned: Unassigned » jhodgdon

I'll be working on this...

jhodgdon’s picture

I have a question about this: What should we do about the hierarchy of Book -> Chapters -> Topics (topics are pages)? I'm assuming that the Book itself, each Chapter, and each Topic should be a page. But how do we make the relationship between them?

Options might be:

a) Node reference field for parent (parent of a Topic node is a Chapter node, and of a Chapter is a Book).

b) Use the core Book module.

tvn’s picture

Using our new content types a Book and a Chapter would use 'documentation guide' content type and Topic would use 'documentation page' content type.

'Documentation guide' is an organic group content type. 'Documentation page' is a 'group content' content type.

Both of them have og_group_ref_documentation entity reference field to specify parent guide (guides can be located inside guides and thus have parents).

You can see content types and fields e.g. here:
https://docs-migrate-drupal.dev.devdrupal.org/admin/structure/types/mana...
https://docs-migrate-drupal.dev.devdrupal.org/admin/structure/types/mana...

Example page https://docs-migrate-drupal.dev.devdrupal.org/docs/8/setting-up-cron/ove...
Example guide with one page inside of it https://docs-migrate-drupal.dev.devdrupal.org/docs/8/setting-up-cron

jhodgdon’s picture

Thanks for the detailed information, and for providing it so quickly! Probably most people wouldn't be able to see the manage fields pages you linked to, but of course I can ssh to the dev server and do drush uli to get access. Carefully. ;}

So, I think in this AsciiDoc Display / Import module, what I will do then is make it flexible, so when you do an import/update, you can specify:
- Content type to use for the book/guide as a whole
- Content type to use for each chapter, and field to specify the parent guide's node ID
- Content type to use for each topic/page, and field to specify the parent chapter's node ID

For the moment, it seems like I can just support that having the reference fields be Entity Reference field types. If someone else trying to use this module needs another field type that is incompatible, the module can be expanded then.

And I guess we'll need to have the ability to set some other values on each created node. The AsciiDoc stuff would only give us the title and body, but we'll need to specify author, some OG fields, etc., I would think. I'll take a look at the fields on these content types and see what would be needed, but it seems like the import procedure will need to have a flexible way to fix values for arbitrary node fields.

jhodgdon’s picture

Another question... What about the weight (to determine the order of chapters within the guide/book, and the order of topics within the chapter)?

I attempted to log in to your dev server to see the fields on the content types, but I cannot run drush uli -- see
#2760527: Permission denied running drush on devwww

If you are able to log in to the site, maybe you could post a screen shot of the Manage Fields pages of both of those content types for me, which might save me asking more questions for a while. :) Thanks!

tvn’s picture

I changed your password on that dev site to your username and made you an admin, so you should be able to login and access all the things.

Regarding the weight, it is done via menu system in the guides. So e.g. on that example Guide, if you click Admin group -> Menus -> Setting up cron you'll get to the menu of the guide: https://docs-migrate-drupal.dev.devdrupal.org/group/node/2735145/admin/m... where you can change the order of child pages and child guides (current just one child page there).

drumm’s picture

For the moment, it seems like I can just support that having the reference fields be Entity Reference field types.

Yes, that should cover it.

Regarding the weight, it is done via menu system in the guides. So e.g. on that example Guide, if you click Admin group -> Menus -> Setting up cron you'll get to the menu of the guide

The og_menu module makes the menus for guides. They are core menus, with og_menu handling creating them and (hopefully) adding links. Some migration code did end up needing to create the menu links: http://cgit.drupalcode.org/drupalorg/tree/features/drupalorg_docs_migrat...

jhodgdon’s picture

OK, I'll take a look at that code.... but adding code like that will make the node import code in the AsciiDoc module very drupal.org-specific. I do not think that is really very desirable.

Also, what will happen if we need to update a book from source (one that has been previously imported), and some of the pages have been moved, added, removed? Updating the menus seems problematic.

Do you think this is really possible to do/maintain?

drumm’s picture

I could see og_menu doing more of the heavy lifting, and/or drupalorg helping out. OG Menu relies on the form UI to get the data set up for saving. That could be a bit more programmatic friendly.

That leaves the piece of information that this module knows as the menu weight only. (Hierarchy is the entity references.) I think setting $node->menu['weight'] can fall within the scope here.

jhodgdon’s picture

OK. I was thinking that this module, if being generic, would need to have a way to say "Which field needs to be set to the topic/chapter weight?", and it would put an int there. That way it could apply to Book module, or another field, as long as the module that sets up the weights/navigation can take a node save and set up the required database stuff.

So... let's see. In sort of summary:
- Module will let you choose what content type to use for the overall book, as well as Chapter and Topic pages.
- Module will let you choose what field to use on each content type for the order/weight (integer, lower number comes first in order)
- Module will let you choose what field to use on each content type for the parent node ID
- Module will let you choose what field to use on each content type for the body. For main book page, the body is the cover image and/or table of contents. For chapters, the body is the table of contents. For topic pages, the body is the AsciiDoc formatted HTML output, plus copyright notice that is picked out from a special spot in the AsciiDoc source.
- The title is the topic/chapter/book title. No settings there.
- Module will let you set up other fields to have fixed values for each content type. We can use that for setting the group stuff for OG, taxonomy terms, or whatever is needed.

And we'll assume that if the module sets up all that stuff and does a node save, it will Just Work. (TM).

Is that about right?

tvn’s picture

Module will let you choose what field to use on each content type for the body. For main book page, the body is the cover image and/or table of contents. For chapters, the body is the table of contents.

Do you have any description for the chapters to be used as a body? We might not need 'table of contents as body' here because on guide pages we automatically display contents, so that would be duplication.

jhodgdon’s picture

No, the chapters do not have any descriptive text, at least currently. Using the current method of displaying them, the first topic of the chapter is displayed on the chapter page. For example:
https://userguide_new-drupal.dev.devdrupal.org/d8guide/en/content-struct...

drumm’s picture

Since we have one use case so far, I wouldn't spend too much time on flexibility/configurability. If there are compromises that make sense and save significant time/complexity, go ahead and take them. For example, configuration with drush variable-set, skipping building a UI, might make sense.

jhodgdon’s picture

I think it's going to be a *lot* easier for me to do the initial testing/development on my local machine, where I will not have all the fields on the fairly complex drupal.org site (probably not going to install all the OG stuff, for instance). Once I get the basics working, I would then try it out on a dev clone. So, I'll probably need at least some flexibility. Over time, there will possibly be some additional fields that get added to the content types that will need to be set to default values too, so I'd hate to hard-wire too much.

Too bad D7 doesn't have a YAML parser.... A YAML config file seems like it would be ideal for a way to submit the config... maybe the D7 .info format could be used though.

drumm’s picture

  • jhodgdon committed 0d85742 on 7.x-1.x
    Issue #2757921 by jhodgdon: Separate module into Base and Display sub-...
jhodgdon’s picture

I've just made a fairly big commit on this issue. This first commit separates out the module into a Base module and a Direct Display module -- the Direct Display module contains the current functionality, which displays the AsciiDoc output directly on the site by reading it from the output files, and allows on-line editing.

The next step will be to develop the node import sub-module.

jhodgdon’s picture

What do you (i.e., @drumm and/or @tvn) think of the idea of making the node import integrate with either Migrate or Feeds (possibly with Feeds Tamper)?
https://www.drupal.org/project/migrate
https://www.drupal.org/project/feeds
https://www.drupal.org/project/feeds_tamper

The motivation would be that both Migrate and Feeds already have mechanisms in place to (a) import nodes from a data source and (b) update nodes once they've been imported and (c) use Drush. So, the problem would be reduced to providing a data source for one of these modules, which would have some kind of a unique ID. That reduces the work to do this considerably.

Would either of those routes to getting this done be OK on drupal.org?

drumm’s picture

Either would need a bit of testing to make sure there are no side effects, etc, but they are possible to use.

I'd try out feeds first. Feeds is intended to be run long-term on a site, while migrate is usually used once and disabled. So I expect feeds to be a better fit.

jhodgdon’s picture

Yeah, Feeds would be my inclination too. I think it's quite a bit easier to work with than Migrate too.

jhodgdon’s picture

I looked into the Feeds API today. See
https://www.drupal.org/node/622700
https://www.drupal.org/node/622710
So... to set up a feed in Feeds, you need a Fetcher, a Parser, and a Processor.

The Feeds module comes with the FeedsFileFetcher fetcher, which can read from a file or directory.

And it has a FeedsNodeProcessor processor, which can write the output as nodes.

So, I think what we need is just a Parser, which can make sense of the files and present output that Feeds can understand. All the rest of what we want to do, I believe, can be set up in a feed already (such as setting values for various fields on the node). The Feeds module has several examples of parsers (CSV, RSS, Atom, OPML), so it shouldn't be too hard, hopefully.

jhodgdon’s picture

Actually, we may need a Fetcher too, so that we can specify where the copyright information can be found and parse that out first. And it must be a directory not a file. But that should be a simple extension of the existing FeedsFileFetcher class, not too difficult either.

jhodgdon’s picture

And I guess the output of the Parser will have two "fields" returned in it: the Body, and the embedded images.

drumm’s picture

Good to know Feeds will help reduce the new code needed.

I noticed http://cgit.drupalcode.org/user_guide/tree/source/en/attributions.txt could use 2015 replaced with 2016.

jhodgdon’s picture

Regarding copyright date, thanks! Just fixed that.

I'm working on the Feeds code. It does seem pretty straightforward. I have the parser written (about 100 lines of code, including the config form), and am just about to do the fetcher, then will test on my local site to see what I missed. :)

One thing I think we'll need to do is to modify the mkoutput.sh script so that it outputs the chapters on their own pages instead of putting the first topic of the chapter there. The current say (first topic there) is good for the direct output display method we are currently using, but not so good for importing. So, I'll have to figure out how to do that, but I'm pretty sure it was an option for the "chunked HTML" DocBook output, so it should be easy. I'll make it an option in the script, so that we can just switch Jenkins over to using the new version when we're ready, and it won't affect the current building/displaying.

I'll also need to modify the script so that it can handle multiple languages. We're about to start translating with one or two test languages... the Catalan team has already volunteered!

Sorry, that is a bit off topic... just on my mind at the moment...

jhodgdon’s picture

A bit of an update: I have the Feeds integration mostly working. A few notes and questions:

a) What do we want to do with the images? Many of the topics have embedded screenshot images. Two things we can do:
1. Feeds should be able to grab the images and put them into an Image field on the imported nodes.
2. Whatever import process we use (Jenkins) could put them in a standard location in the site's file directory.
Thoughts?

b) This process will require Feeds Tamper, for the URL aliases at least. The problem is that there are a lot of internal links in the AsciiDoc source, which go to bare topic file names like href="structure-taxonomy.html". I already have the functionality in place in my Parser to alter the href attributes by putting a prefix on them, but we'll need to put the same prefix on the URL aliases when we save the nodes, using Feeds Tamper. Hope this makes sense.

c) It doesn't look like the default Node processor in Feeds supports creating nodes of different node types based on feed fields. So I will probably have to add a Processor class to this too.

d) Sigh, just realized Feeds doesn't have a way to do menu items. See
#1633014: Add menu item map to Node Processor
There is a sandbox module that people have had success with:
https://www.drupal.org/sandbox/attisan/2144379
It looks *very* straightforward. If I have to do a processor class anyway, I can probably adopt this code, or something very similar.

e) Speaking of which, I don't have the "figure out what the parent is" stuff working yet either... I will need that to set up the menus, and probably what I'd need to do is make sure the Fetcher figures that out, and orders the files so that parents are processed before children. This should not actually be too hard, because the table of contents, which is in the Index file, has the entire hierarchy in it.

Anyway... making progress, and I think this is all doable.

jhodgdon’s picture

Update: I got the "figure out what the parent is" stuff working. Also, I think that instead of Feeds Tamper, in my custom Processor for nodes, I will add the functionality of prefixing the URL alias, so that it will go well with the Parser step.

@drumm or @tvn - I still am not sure what direction to go on the images. Any thoughts on the image question (item A in comment #27)? Thanks!

tvn’s picture

I personally think that image field on nodes makes more sense, but not sure if re-import and image updates will create a mess or not. I'll defer to drumm on this one.

drumm’s picture

a) Re images: documentation pages on Drupal.org have a file upload field, which can have images. This hasn’t changed from book pages. The only upcoming change we hope to do is better WYSIWYG integration, which won’t affect this issue.

The import should drop in an img tag where the image should appear.

Ideally, the import should be as much of a self-contained drush command as practical, to keep all the code in one place. Anything else needed in Jenkins can of course be done.

drumm’s picture

b) Re URL aliases: we have pathauto turned on for documentation pages, like https://www.drupal.org/drupalorg/docs/user-accounts/user-roles-and-permi....

If there’s an easy way to use those paths, that would be ideal. I expect it wouldn’t actually be easy at all - would need to look up the imported node by pre-import path and use url('node/' . $nid). So, Feeds Tamper makes sense.

drumm’s picture

So, Feeds Tamper makes sense.

I mean your approach in #28 makes sense.

drumm’s picture

d) Re menus: we can leave that for later. Either OG menu or drupalorg should handle that.

jhodgdon’s picture

The pages already have img tags. The problem is that they are pointing to just "images/file-name.png". Similar problem to within-book links: they are pointing to, for example, "structure-taxonomy.html". Pathauto will not be good for the within-book links, not at all. What I've been doing in my tests so far is:

a) Make the path aliases for each page (on import) equal to, for example, "test/structure-taxonomy.html".

b) Alter the link URLs for internal links by prepending a prefix, so they also point to "test/structure-taxonomy.html" (for example).

I can do something similar pretty easily for images. So if Feeds can import the images that are embedded in the page (which I have NOT, by the way, seen it do successfully yet), then it can put them in the Image field, which will put them in, for example, sites/default/files/doc_images/whatever.png. The image tag src attributes can then be prepended, on import, so that it ends up pointing to this same location.

But path auto I don't think would work at all... When I import page A, it may have references to several other pages B, C, and D. I will not have a way to figure out what the URLs will be for those pages from pathauto, given just the HTML for A. All I can really do is detect that it is "structure-taxonomy.html" and transform that somehow. I don't think this is even possible with Feeds Tamper, because pages B and C might not exist yet.

I guess maybe your comment in #31/32 is agreeing with that? I hope so. :)

Menus -- I think I have this working anyway. The problem with letting OG menu or drupalorg handle it is that they need to know what the parent item is, and what the order is, in order to make the menu items. This has to come from Feeds.

jhodgdon’s picture

OK... So, here's where I am on this...

a) I got the menus and images and path aliases stuff all working. Hooray! I can now import AsciiDoc output into Drupal nodes, make a nice menu of the table of contents hierarchy, and the internal links work fine.

b) However, there is one caveat. If you run the feeds import the first time, all is well. But the next time you run it, it tries to replace the images with new ones. This would be OK, except that when it does this, the uploaded file names in the Image field get mangled. For instance, I have an image file called issue-summary.png. The first time I import the feed, the image is created in the upload directory set aside for book images (configured on my Image field) with that name, and my "prefix the image src attributes" process makes it appear on the page where it should in the img tag. But the next time I run the import, that file name already exists in the files table in that directory, so the image that is in the book images field gets uploaded and saved with a new name, issue-summary_0.png. And the old uploaded image then gets deleted when the node is saved. So, the image tag is still pointing to issue-summary.png, which doesn't exist, and I get a broken image tag.

Oops.

Anyway, I am going to do a little bit of code cleanup and commit this as it is.

Then we will have our choice of two solutions, I think:

a) Somehow force the File module not to make new names for the images if they already exist. However, this is most likely not going to work out well, unless we have a dedicated place for the images for each AsciiDoc book, and for that matter, for each language. There's no guarantee someone won't have a duplicate image name somewhere.

b) Don't upload/import the images using Feeds. Instead, when we do an import, outside of Feeds (i.e., in Jenkins), we would put the images in a known location, and use that location to prefix the image tags' src attributes in the import. So the img tags would know where to find the images, but we wouldn't have Feeds import them.

c) Is there a third way? I can't think of it...

Anyway, I'll get that code committed and then try to talk to @drumm about next steps in IRC sometime in the next day or two.

  • jhodgdon committed 4d26c26 on 7.x-1.x
    Issue #2757921 by jhodgdon: Add ability to import AsciiDoc HTML output...
jhodgdon’s picture

Oh, I should mention that this image problem is compounded by the fact that you need to run the import twice to get the menu hierarchy right -- the first time, you have the problem that node A is the parent of node B in the menu, but they are not necessarily imported in that order. ... although I could probably fix that come to think of it...

Anyway, I've made the commit.

I'm also making a commit to the User Guide scripts -- separating the build scripts into 3 separate scripts: one for direct display (as we're using now on the dev site), one for feeds, and one for ebooks. The existing Jenkins job will still work fine (same script name as it has been using, and it will ignore the --no-ebooks argument).

jhodgdon’s picture

OK. I think I know what to do to fix all of these issues, and also I need to do more for the multilingual case (and test it):

a) Add an option for the link and image URL prefixing to add the language code as a "directory". So for instance if the prefix is "test/", make it "test/en", "test/es", etc. This is necessary because the file names are the same for each language, and the Feeds processor does all languages in one import.

b) Add a feeds mapper that will copy the image files to a specified location (also with a language code subdirectory option) within the public files directory. It will need options about what to do if there are already files there with the same filenames. This would be an alternative to uploading to an Image field where the file names get munged. So we wouldn't have Image fields for these files, and they wouldn't be part of the Files database, but the image tags would work at least.

c) For the menu hierarchy problem, I should be able to have the Fetcher grab the HTML files in hierarchy order, rather than in whatever order the PHP directory lister gets them. This will mean that the parents will exist before the child menu items are created, and the menu will be correct the first import.

None of this should be too difficult.

  • jhodgdon committed 1190a80 on 7.x-1.x
    Issue #2757921 by jhodgdon: Add more nuances to the Feeds import...
jhodgdon’s picture

I got all of the above done... I think it is all working. It is past business hours where @drumm is, I think, so hopefully we can connect in IRC in the next few days to see what the next steps are. Anyway, it's working!

jhodgdon’s picture

@drumm and @tvn - I am still a bit unsure about the different languages we will have for the User Guide, and how we should aim to handle them, with regards to:
- Should imported nodes for the same page in different languages be marked as translations of each other?
- What about menus for each language version of the User Guide?

drumm’s picture

I guess maybe your comment in #31/32 is agreeing with that? I hope so. :)

Yes. And we're on the latest version of pathauto which is better behaved for not overriding custom paths.

drumm’s picture

(Not sure if I'm fully caught up on all of these, I'm making my way through the comments.)

Add a feeds mapper that will copy the image files to a specified location (also with a language code subdirectory option) within the public files directory. It will need options about what to do if there are already files there with the same filenames. This would be an alternative to uploading to an Image field where the file names get munged. So we wouldn't have Image fields for these files, and they wouldn't be part of the Files database, but the image tags would work at least.

This sounds like a good solution.

Our CDN and browsers might aggressively cache images, and/or we might configure it more aggressive in the future. The most reliable way to make sure a new image gets served is to use a new URL. That can be part of the guide editing process.

drumm’s picture

Should imported nodes for the same page in different languages be marked as translations of each other?

Yes, we want to stick close to the core locale & translation modules.

What about menus for each language version of the User Guide?

We'll end up with a menu for each language. (We looked at the i18n_menu module, part of internationalization. It looked like more trouble than it was worth, although that was a rushed decision for the D8 launch page translations.)

jhodgdon’s picture

@drumm set up a dev site for me to try this out on for drupal.org -- so I'll work on that and report back.

We're also going to want to use Drush. To facilitate that... I am going to move all the configuration for the feed to being "importer" config rather than "source" config. the reason is that you cannot specify "source" config when you do an import with Drush, aside from the file path that you are importing.

Also the distinction in Feeds between "source" config and "importer" config is rather arbitrary and weird, because you cannot put any mapping config in the "source" config anyway, and for this setup, there's interaction between the Fetcher/Parser config and the Mapping config.

Therefore, it just doesn't make sense to me to have the config that I currently added to the "source" config for the Fetcher and Parser to be there. I'll put it on the "importer" config, which we can then export into a Feature. The only thing in the "source" config will then be the directory we are importing, which we can specify on the command line with Drush.

Anyway this is a bit technical but ... anyway, just recording what I'm doing so that if someone ever looks at the commits they can in theory come read this issue and find out why things were done. :)

  • jhodgdon committed 0e8ddd3 on 7.x-1.x
    Issue #2757921 by jhodgdon: Move nearly all the configuration for Feeds...
jhodgdon’s picture

Well, that was very straightforward to do. Works like a charm, and much more intuitive, I think, to have all the configuration on one page except the directory to import from.

Also wanted to note that I talked to drumm in IRC about languages. Basically, each Guide node on d.o has an OG menu associated with it. The top-level of the User Guide, as well as each Chapter, are guides (guides can contain both other guides and pages). So, each language will have its own top-level guide, hence its own menu. This should be fine. I just need to see if the feeds import process on the d.o dev site will set fields properly so OG can create the right menus. We shall see.

The other thing we will eventually want to do is when we import a foreign language item, mark it as a translation of the English page. I'll need to add a bit to the code to do that, but I think it can wait until later, so I'm putting it on a separate child issue: #2774891: For Feeds import, handle translations

jhodgdon’s picture

I got the import working on this dev site:
https://guide-drupal.dev.devdrupal.org/documentation/user_guide/en/index...

Yippee!!!

A few concerns:

a) No source editing yet. See #2775493: Make a way to edit from Feeds import

b) Each Guide and Page in a guide has a "Summary" field. I simply put the topic/chapter titles themselves in as the summaries, which is not ideal. We can probably do better. See #2775503: Make a way to have summaries

I also need to make a commit with a few more minor changes/updates that I did in order to get the menus and URL aliases working on drupal.org. Coming shortly.

  • jhodgdon committed 90741ce on 7.x-1.x
    Issue #2757921 by jhodgdon: Add code to support importing on drupal.org...
jhodgdon’s picture

Issue summary: View changes

Oh, and here is a summary of what I did on the dev site to get this working (noting this in the issue summary):

a) In the userguide_new asciidoc_source/scripts directory, ran ./mkfeeds.sh -- this made AsciiDoc output, saved in
/var/www/dev/userguide_new-drupal.dev.devdrupal.org/userguide_source/output/html_feed
--> under that: /en (for the guide) or /guidelines (for the contributor guide)

b) In the guide dev site -- https://guide-drupal.dev.devdrupal.org -- on the server, ran:
drush dl feeds # (7.x-2.0-beta2)
drush dl job_scheduler # (required for feeds, 7.x-2.0-alpha3)
drush en feeds feeds_ui
git clone --branch 7.x-1.x https://git.drupal.org/project/asciidoc_display.git
drush en asciidoc_display asciidoc_display_feeds
drush cc all

===> Note: Will need to set up appropriate permissions for:
- administer feeds
- administer asciidoc

c) Logged into the site with drush uli

d) On admin/structure/types, in the Publishing Options vertical tab for the
Documentation Page and Documentation Guide types, set them to multilingual, with
translation. Also added a field for asciidoc_source_file to both content types (to store the source file, so that we can eventually set up the editor).

e) Created a feed for the User Guide on admin/structure/feeds, and one for the
Guidelines book.

===> We will eventually need to export these to a Feature.

f) You can import these feeds on /import using source directories:

/var/www/dev/userguide_new-drupal.dev.devdrupal.org/userguide_source/output/html_feed
/var/www/dev/userguide_new-drupal.dev.devdrupal.org/userguide_source/output/html_feed/guidelines

jhodgdon’s picture

Update: I got the summary part working!

For an example, see https://guide-drupal.dev.devdrupal.org/documentation/user_guide_contribu...
This is the contributor guide for the User guide project, not the User Guide itself, but it has the same structure.

I have an open issue that will create summaries for the User Guide topics and chapters too:
#2775689: Create summaries for topics in the User Guide
Hopefully we'll get that done within the next week. With 100 topics to summarize, we should take a little care... :)

The Feeds Importers on the dev site have been updated to import the summary fields too, so comment #50 is still accurate as to what needs to be done to get this working on the live site, once the summary fields content is created for the User Guide.

jhodgdon’s picture

Status update:

a) I got the editing of AsciiDoc source files working on my test site... still need to test it out on the dev site. I will report back on that soon, and update the issue summary with additional steps needed to go live.

b) One of our regular/great/wonderful User Guide contributors is taking on the task of making summaries for all of the topics/chapters in the User Guide. I expect it will be done in a day or two, as he's pretty fast. :)

c) I guess one other missing piece is to set up a Jenkins job that can be triggered whenever we think it's a good idea, to re-import the source. I will have to test the drush commands on the dev site and report back on that too.

So... After lunch, I should be able to see about (a) and (c)... (b) should be done later this week... Anything else we should think about or do before this can go live?

jhodgdon’s picture

Issue summary: View changes
Status: Active » Needs review

Updating summary with current status, and steps to get this onto drupal.org.

Here is a sample page with editing enabled:
https://guide-drupal.dev.devdrupal.org/documentation/user_guide/en/secur...

You'll see a new Source information block on the right sidebar. The text in that block is configurable... I probably made it way too wordy. Anyway, if you click "Edit AsciiDoc source file" at the bottom of that block, you will see the editor page, and after editing, you can download the revised source file.

So... thoughts?

drumm’s picture

Here’s a draft of the message with the text cut down quite a lot:

This guide is maintained in the User Guide project. If you have a suggestion or want to report a problem, create an issue.

jhodgdon’s picture

Good idea. I put that up.

jhodgdon’s picture

Ah. One other thing I think we should maybe do is, when importing with Feeds, cut out the title within the HTML source of the page. As it is now, it appears twice.

Possibly we should also get rid of the navigation div at the bottom of each page. It seems redundant given the sidebar menus that are inherent in Guides.

Third item: I guess we should set the Guide Maintainers fields... For that, we would need Feeds Tamper I guess, because there is no field to set it to inherently. Is there some way we can instead set this entire Guide as being part of the User Guide project? I am not sure if Guides can be associated with projects, but that is what would be best if it can happen...

Thoughts?

tvn’s picture

Agree re: removing page title and navigation div.

There is no way to set a guide as a part of User Guide project. We do plan to make it possible to associate guides and projects later, but that'll be a simple entity reference, maintainers or permissions will not be merged/copied in any way. So yes, we should set Guide maintainers field.

I am looking at the dev site now and will comment with more feedback.

jhodgdon’s picture

Issue summary: View changes

Adding a list of several things that need to be addressed before we can finalize this to the issue summary...

@drumm and @tvn, can you take a look at the list? Several of them I don't know how to do or don't know which route to take.

jhodgdon’s picture

I do not see the Guide Maintainers field in the list of fields on Documentation Guide. Is it perhaps part of the Group it is in? I guess in that case we would need to set up a Group, and then ... somehow ... in Feeds, set that group to a value.

tvn’s picture

I think overall it looks really good and almost ready. A few things and questions:

1. The top url should be /docs instead of /documentation to match the rest of the new section (since all docs pages now get human friendly urls, we are trying to shorten them up hence why 'docs').

2. On the closer look, agree the navigation should be gone both on guides and pages.

3. Do we need a separate Copyright block? The standard copyright block at the bottom of the page seems to cover most of it, with the only exception of 'Section i.1 Copyright' link. If it is very important to highlight that link specifically, could we perhaps have it in the sidebar block under Source info copy?

4. Did the import script create all the guides including the very top level guide, or did you create any manually?

5. Permissions: I assume we don't want anyone to be able to Edit the nodes (via normal edit form) or comment on them. So we should lock those down for this particular guide.

6. Will there be any revision history at all for e.g. node updates when we re-run the script?

7. Related to above: I think the local tasks menu (the green button with a drop down) should probably be gone completely. Definitely for guides, for pages itself I see that there is one menu item there and it goes to the edit AsciiDoc source file now. There is a link for that in the sidebar as well. We should probably keep just one. Keeping the sidebar one seems fine to me, as it is different interaction from the regular Edit button on docs pages.

8. Because the content type uses Panelizer template for layout, we can't use the core Block UI to place blocks on it. e.g. https://guide-drupal.dev.devdrupal.org/documentation/user_guide/en/preve..., all content is currently moved to the left. We need to come up with a different way of adding that block.

tvn’s picture

Cross-posting :) I'll look at your comments now.

The Guide maintainers are managed via group interface. On any guide go to Admin -> People (e.g. https://www.drupal.org/group/node/2764139/admin/people). Thinking about it some more, for regular guides maintainers get permissions like edit and admin, etc. For the user guide none of that matters. So really the only reason we would add maintainers is to have the show up like https://www.drupal.org/drupalorg/docs. I am not sure if it is worth figuring out how to do it. What do you think?

tvn’s picture

Note: I initially tested while not-logged-in and was able to see and access 'Edit Ascii source'. I assume that's ok, but wanted to bring up nevertheless. Now as logged in I see that all normal for doc pages permissions are available - Edit, Discuss, etc. So we should definitely talk about which permissions make sense for you and lock the rest down.

jhodgdon’s picture

Issue summary: View changes

Thanks for your comments! Adding some To Dos to the summary.

Most of what you put in #60 looks easy to do. A couple of notes on #60 items that I did not add to the issue summary:
4. All imports were completely automatic, including the top-level guide page.

6. There is revision history, but not log messages. This comes from Feeds. If you are logged in on the dev site, see
https://guide-drupal.dev.devdrupal.org/node/2774132/revisions
for an example.

8. I did use Panelizer to place the block (see "how to deploy" section in the issue summary, last step currently).... Oh yeah, I also put it in regular Blocks first, before I figured that out. Oops -- removed that. I think it is fixed now. The pages now look similar to existing docs guide pages like
https://guide-drupal.dev.devdrupal.org/drupalorg/docs/content/front-page...

Regarding #61, I don't think we need to necessarily set up a group and permissions... but for some reason Dries is showing up as the "Guide Maintainer" for these guides, based on me being logged in as user 1 when I imported it I think. So that is not great.

jhodgdon’s picture

Issue summary: View changes

Yes, we should discuss the permissions. There is a permission called "edit asciidoc source" that I granted to even anonymous users on the dev site, but we can definitely set this to only logged-in users. Adding this as a to do to the summary.

  • jhodgdon committed ece792e on 7.x-1.x
    Issue #2757921 by jhodgdon: Tweak a few settings for AsciiDoc import
    

  • jhodgdon committed 5b9437b on 7.x-1.x
    Issue #2757921 by jhodgdon: Tweak a few settings for AsciiDoc import
    

  • jhodgdon committed b144967 on 7.x-1.x
    Issue #2757921 by jhodgdon: Tweak a few settings for AsciiDoc import
    

  • jhodgdon committed 277ceeb on 7.x-1.x
    Issue #2757921 by jhodgdon: Tweak a few settings for AsciiDoc import
    
jhodgdon’s picture

Issue summary: View changes

OK, I took care of several of the issues in the Issue Summary. Updating that...

Note: changed the URLs, so the Contributor Guideline is now at:
https://guide-drupal.dev.devdrupal.org/docs/user_guide_contributors/inde...
And the User Guide is at:
https://guide-drupal.dev.devdrupal.org/docs/user_guide/en/index.html

Note about copyright: I took out the copyright notice and added "written by many contributors" to the sidebar source block and linked to the attributions page. See
https://guide-drupal.dev.devdrupal.org/docs/user_guide/en/structure-cont...
for an example.

jhodgdon’s picture

So, here are some questions for @drumm and/or @tvn:

a) Are you OK with hiding the navigation and titles via CSS? If so, should it be added to Bluecheese or drupalorg module or where?

b) Should we hide the Guide Maintainers block, or make it relevant? I think maybe hide it?

c) If we are hiding the Guide Maintainers block... how? I guess maybe we can do this by detecting that the AsciiDoc Source field is set on the page, and hiding it in Panelizer with a condition -- is that the right approach?

d) I guess do something similar to (c) with the local tasks drop-down on these pages?

e) I have no idea how to set it up so that no one can either edit the nodes or comment on them for just these pages/books?

===> Would setting up a Group for these guides help or take care of (b) through (e)?

jhodgdon’s picture

I also looked over a few outstanding issues with formatting, and added a link to this issue to the summary:
#2711127: Vertically-centered tables are difficult to read.
This really should be fixed ASAP. it makes several pages hard to read.

drumm’s picture

b) Should we hide the Guide Maintainers block, or make it relevant? I think maybe hide it?

c) If we are hiding the Guide Maintainers block... how? I guess maybe we can do this by detecting that the AsciiDoc Source field is set on the page, and hiding it in Panelizer with a condition -- is that the right approach?

Making the block relevant would be ideal, but probably not a worthwhile use of time for now.

I think we can:

  • Import as the System Message user or another non-human instead of user 1.
  • Add a filter to the View for that block to exclude that user.
  • Make sure the title and everything with the block goes away when empty.
drumm’s picture

e) I have no idea how to set it up so that no one can either edit the nodes or comment on them for just these pages/books?

For edit, something along the lines of:

function asciidoc_display_feeds_node_access($node, $op, $account) {
  if ($op === 'update' && $node is imported) {
    return NODE_ACCESS_DENY;
  }
}

For comment: Import the nodes with $node->comment = COMMENT_NODE_CLOSED.

d) I guess do something similar to (c) with the local tasks drop-down on these pages?

That will get the local tasks drop down under control. We can either

  • Rename “Edit AsciiDocs” to “Edit” to make it fit.
  • Alter the menu or local task in drupalorg to do that.
  • Make the menu CSS more flexible to make the full text fit.
drumm’s picture

a) Are you OK with hiding the navigation and titles via CSS? If so, should it be added to Bluecheese or drupalorg module or where?

Can the import process keep those out of the imported body field, so we can rely on the navigation & titles that come with the page?

tvn’s picture

Thanks for fixing the sidebar block and moving copyright there, jhodgdon.

Can I suggest two things for the block:
- it would be better to move it below the navigation, so that navigation is consistently on the top of the sidebar and does not jump around when you move from the guide to the page
- add some whitespace or empty line above 'Edit AsciiDoc source file' link to make it a bit more prominent next to all the text.

Also, since the block is indeed added via panelizer, this means we modify default panelizer templates for both guides and pages. Those are storied as a part of Documentation feature. So the feature would need to be re-exported and committed. Perhaps add as a last step for deployment instructions.

tvn’s picture

#72 sounds like a good plan.

That will get the local tasks drop down under control. We can either

Rename “Edit AsciiDocs” to “Edit” to make it fit.
Alter the menu or local task in drupalorg to do that.
Make the menu CSS more flexible to make the full text fit.

If we are keeping 'Edit AsciiDocs' in the local tasks, I think the title should be as short as possible - 'Edit AsciiDocs' or 'Edit source'.

Also I think the AsciiDocs source file edit page should have some sort of message to give people more info about why this process is not a regular node edit, and how to actually make an edit and submit it as a patch.

Lastly, if the link is kept in local tasks, disregard my earlier comment about making it more prominent in the sidebar. There should be only one prominent link for this action on the page.

If we remove 'Edit AsciiDocs' from the local tasks completely, then I am afraid most of the pages will have 'View history' as the only local task available. Having that as a main CTA on the page is non ideal. We would need to hide it somehow.

jhodgdon’s picture

Issue summary: View changes

Lots of great feedback, thanks! Going through the comments from yesterday one by one... noting these in issue summary as well:

#72:
- I changed the imports to create the nodes with author System Message. Good idea.
- Edited the view to exclude the System Message user. That will need to be exported into its feature.

===> However, this didn't seem to work. Even when I changed the authoring information for the User Guide to be "System Message" and not Dries, it still showed the same contributor block on the sidebar (even after clearing the cache). I think there is more that has to be done, because that block appears to be connected to the owning Group, not the author, and I am not setting the group. I need more advice on how to set the Group on the guide when I import, and what I should set it to.

#73: Those changes do not belong in the asciidoc_display project. I will see about a patch to drupalorg to prohibit node editing, and Feeds Tamper to set the comment settings on import.

#74: I can make it so that the title and navigation are not in the HTML. Kind of a pain, but probably the right solution.

#75: taken care of. I was aware that things are in features; noted in issue summary so it is there too. :)

#76: ... will respond in a separate comment.

jhodgdon’s picture

Issue summary: View changes

Regarding comment #76, about the local tasks...

I think that the best solution is to hide the local tasks area entirely. I think that the Edit AsciiDoc Source link is better if it is near the explanation of what it means (in the block) and not in the local tasks.

So... I can definitely, in Panelizer, hide both the Contributors block and the Local Tasks tabs, using visibility rules. To me, this seems like the easiest solution -- we can just hide both of them if the asciidoc_source_file field is non-empty. Would that be OK?

The other part:

Also I think the AsciiDocs source file edit page should have some sort of message to give people more info about why this process is not a regular node edit, and how to actually make an edit and submit it as a patch.

I think the best way to handle that is to create a custom block and place it at the top of this page. The difficulty is that this page doesn't really know anything about what project on drupal.org this is related to, so it would need to be kind of generic. We may want to have other books in the future that are maintained like this one.

Or really... if they clicked the link, which is in this block giving information about the project... isn't that enough?

Anyway, if we do want to do this, is there a panelizer page that governs how this page is laid out? I don't have a clue... Something has, for some reason, moved the message that the module puts at the top of the editing form, and moved it into the sidebar. How/why is this happening?

Anyway I have added this bit as a To Do in the issue summary.

jhodgdon’s picture

Issue summary: View changes

Hm. There seems to be a bug in the import process in Feeds. Even though I told it to import content and set the creator to "System Message", it seems to have still left most of it as Dries -- I deleted the content and re-imported for the Guidelines document. Hm. The Guide pages are imported as Dries, and the Topic pages are imported as System Message. Anyway, I'll investigate.

  • jhodgdon committed caa0a5b on 7.x-1.x
    Issue #2757921 by jhodgdon: Updates to Feeds import
    
jhodgdon’s picture

Issue summary: View changes

I've taken care of a few items from the issue summary's Remaining tasks section... a few notes, hopefully coherent...

Item 4 - navigation is now gone from the HTML source.

Item 3 - titles in the HTML source... well this worked on my local site. However, the XSLT templates on the dev site are a different version (apparently), and on Chapter pages, my fix (which removed H1 class=title) did not work (because the titles generated on the dev site are H2 and not H2). But the topic pages are fine now. I am inclined to leave it at that. The chapter pages don't have any source on them anyway.

14 -- fixed bug that was resetting the user for imported nodes when I changed the content type. I re-imported everything. URL aliases are the same; node IDs are new. And the maintainers block went away! This is good.

13 -- Added some more explanatory text for the Edit AsciiDoc Source page. See what you think. Sample page:
https://guide-drupal.dev.devdrupal.org/node/2774377/asciidoc_edit

Note that the formatting for that page is bonkers. Again, where is this page generated? The CSS is wonky. The tabs drop-down is up in the header, and it's putting the explanatory text over to the right, until it reaches the end of the page title, and then it wraps over to the main column. Something is floating that shouldn't be. I'll add this to the issue summary as yet another To Do.

jhodgdon’s picture

My current questions for tvn or drumm:

a) Is my fix for item 13 in issue summary OK (added explanatory text to the edit asciidoc source page)

b) Item 15 - How to fix the formatting for the edit asciidoc source page. CSS is wonky.

c) Item 11 - What do you think about using panel visibility to hide the local tasks for these pages?

d) If we get rid of Local Tasks, and the nodes are owned by System Message, I don't think we need to worry about who has edit permissions, right? Well, I'm still not sure what Group they are in... who would have permission to edit the pages, and do we need to set the group?

e) I guess I still need to look into Feeds Tamper to set the Comments field to disallowed, and maybe to set the group field.

f) Can we fix #2711127: Vertically-centered tables are difficult to read. ? This is a problem for these guide pages, and IMO for any docs pages it would be better to have the tables top-justified and not vertically centered anyway. Any table with potentially large text, without borders, is subject to this same problem.

jhodgdon’s picture

sigh. I attempted to use Panelizer visibility rules to hide the local actions tabs on pages where asciidoc_source_file is not empty, but it didn't seem to work... There doesn't seem to be a way to set a rule for "is empty" or "is not empty", just "string equal to some value", and it did not work when I tried leaving it blank -- the local tabs were always visible.

So... I guess to get rid of local tasks on these pages... I have no idea. I am no Panels expert. Any ideas?

tvn’s picture

a) Re: #13 - I think the message is good and generic enough.

Regarding the styling, once we panelized documentation content types, styles explicitly apply to the modes/sub-pages we were gonna use: view, edit, discuss, view history. Anything new (like the ascii edit source page) essentially needs to be styled. I am not sure why there isn't a "base" layout/styles picked up by every mode/form for the same node. I'll let Neil to comment on how/where to style that.

d) If we get rid of Local Tasks, and the nodes are owned by System Message, I don't think we need to worry about who has edit permissions, right? Well, I'm still not sure what Group they are in... who would have permission to edit the pages, and do we need to set the group?

Doc Guide itself is a group. Doc pages are inside of one of the guides, which is their group. As long as the 'Guide' field is set, means the group is set.
E.g. here https://guide-drupal.dev.devdrupal.org/node/2774377/edit page is inside of 'Chapter 5. Basic Page Management' group.
Guides can be inside of other guides too, but they don't inherit any permissions of their parent.

Current permissions for documentation content types are as follows:
- any confirmed user can edit any documentation page.
- any confirmed user can edit any documentation guide
- only guide maintainers can administer their guides (e.g. add other maintainers, modify menus)

So even if we hide local tasks, any confirmed user can in theory go to node/2774377/edit and edit it. So we should still do something like what Neil said above to prevent that access.

tvn’s picture

Looks so much better without navigation in the HTML and titles. On Chapters titles are duplicated, but I think this is OK for the launch. Also thanks for moving the sidebar block down. Overall this is getting very close!

jhodgdon’s picture

Issue summary: View changes

OK. A few updates (thanks drumm/tvn for discussing and troubleshooting in IRC with me!):

Issue summary item 1 (translations) - I am close to a solution on the linked issue, will update soon.

Item 10 (permissions) - it turns out that because the import uses a new text format that is limited to Administrator use only, no one but Administrator can edit the content items, so this part is done. still need to take care of commenting.

New problem: noticed that "Edit AsciiDoc source" links are present in the local tasks on non-imported pages. Need to fix that by using a custom loader in the hook_menu. Adding this to the issue summary.

jhodgdon’s picture

Issue summary: View changes

I thought of something else, doh! I can make the "Edit asciidoc source" thing in hook_menu() just be a callback instead of a local task. Then it won't appear in Local Tasks at all -- just in that block. :) Also still need to fix the load/permissions thing so the URL doesn't work for a non-imported node. But anyway if I do that and we close down the comments, we'll be good.

hestenet’s picture

Issue tags: +Community Initiative

  • jhodgdon committed c67ca82 on 7.x-1.x
    Issue #2757921 by jhodgdon: More tweaks for importing AsciiDoc HTML...

  • jhodgdon committed b0f2605 on 7.x-1.x
    Issue #2757921 by jhodgdon: Fix entity load error message by changing...
jhodgdon’s picture

Issue summary: View changes

OK, I've made some more updates:

a) The node/###/asciidoc_edit path is now a callback URL not a local task.

b) This path returns 404 on any node that was not imported with Feeds from AsciiDoc source.

c) I added the ability in the feeds code to override the default comment settings for nodes that are imported.

d) On a separate issues, I made some changes so that translations will work: when we import the Hungarian or Catalan version of the guide (those should be the first two), individual guide pages will be marked as translations of the English version.

e) I tweaked some of the settings on the dev site in accordance with these changes.

f) About half of the summaries have been created!! See, for instance:
https://guide-drupal.dev.devdrupal.org/docs/user_guide/en/planning-chapt...
(or anything else through chapter 6).

Updated summary... the Outstanding Issues list in the Issue Summary is actually getting shorter for a change!

jhodgdon’s picture

Issue summary: View changes

So, we are down to just a few items -- tvn/drumm -- your input is requested!

a) The Local Tasks are still there. For normal users, the only thing they will see is "View history". Should we get rid of that or just leave it so it looks like other docs pages?

b) If we should get rid of it, does anyone have any ideas how to do it? I am not a Panelizer expert.

c) We still need to fix #2711127: Vertically-centered tables are difficult to read. but I guess that should not be a blocker. We do have a lot of tables in the User Guide though!

d) We'll need to export stuff to Features and make scripts etc. to finalize this.

e) That's really about it, from my point of view! Anything else you think we should update/fix?

Note: The summaries are about half written, and if past experience is anything to judge by, our star contributor will probably get them done this weekend. The problem with the bad formatting on the Edit AsciiDoc Source page went away when I made it not a Local Task of node/1245 -- so I will mark that as fixed in the issue summary. The page is much cleaner. Such as:
https://guide-drupal.dev.devdrupal.org/node/2774379/asciidoc_edit

tvn’s picture

a) Get rid of it. That is styled as the main CTA on the page, and View history is in no way that.
b) Do the opposite of what you've done for the 'Source information' block to show up only on these pages? Not an expert either. :)
c) I don't think that is a blocker. We should be able to fix it soon anyway.

I am still able to edit pages of the guide via node edit form. I suppose because my user is an admin?
And do you have another example of a node with revision history present?

Can't think of anything else now, dev site looks good to me overall. Perhaps let's remove [fixed/done] points from the list of outstanding issues, so we could clearly see what's left?

Last piece of feedback, unrelated to our implementation - I think having 'Concept' in page title for a lot of topic does not help the users much and makes it less readable. I realize you must've had a reason for doing it like that, but perhaps that could be moved to summary field.

jhodgdon’s picture

Issue summary: View changes

If you feel strongly about the Concept topic titles, you can file an issue in the User guide project (https://www.drupal.org/project/user_guide), and we can discuss it there... We did have a reason for making the page titles start with "Concept", because we wanted to clearly distinguish between topics that told you how to do something (task topics) and topics that provided background (concept topics), in the user guide, and not mix concepts and tasks together in the same topics. But we might be open to changing the title format, if you have a better suggestion that would still get across the point that these are not "how-to" topics. Personally, I think that is super helpful when scanning navigation or looking at search results, to know "this page is going to tell me how to do something" vs. "this page is going to give me background information"... so whatever convention we use, I think we want to make this distinction.

Regarding the local tasks... The Source Information block is not shown on non-AsciiDoc pages because the block code itself detects this. The Local Tasks are put in by Panelizer, and I could not figure out an easy way to get Panelizer to detect "this is an AsciiDoc source page" and hide the local tasks. Again, I'm not much of a Panels expert... maybe @drumm has an idea? There is a field that is set to a value for these pages (field_asciidoc_source) and should be empty for regular docs nodes, but Panels doesn't seem to have a way in a visibility rule to say "hide if this field has a value".

And yes, admins can still edit the nodes. I guess I could remove that by setting the text format to not even have permission for admins to use, come to think of it -- then only user 1 would be able to edit the nodes. I'm not sure if that is sensible or not... maybe in an emergency we would want an admin to be able to edit the node?

Edited the issue summary to make it easier to scan what is done and still needs doing. I didn't want to completely remove items from the summary because some of the comments referred to numbers... so used DEL tags. Hope that is OK.

tvn’s picture

And yes, admins can still edit the nodes. I guess I could remove that by setting the text format to not even have permission for admins to use, come to think of it -- then only user 1 would be able to edit the nodes. I'm not sure if that is sensible or not... maybe in an emergency we would want an admin to be able to edit the node?

If it is only users with the role 'administrator', then it is fine. There are only about 10 people and most are staff. However if it is anyone with 'administer content' permission, then it is much bigger number of people and I would suggest we limit it.

jhodgdon’s picture

The text format is only granted permissions for role "Administrator" specifically.

tvn’s picture

Perfect!

jhodgdon’s picture

OK then, it seems the only issue remaining is the Local Tasks hiding.

I did have one idea: since the revision history is the only remaining task, probably if we got rid of that (in drupalorg module) for the pages that were imported, the local tasks button/menu thing would vanish entirely. Does that sound like a good way to proceed? There is already apparently code in drupalorg somewhere that is hiding the View local task on docs pages, so we can probably do the same for docs pages that we detect are imported (I can write a few lines of code to detect this).

Other than that, I haven't come up with any other ideas on how to hide the local tasks for these pages, since Panels doesn't seem to have an obvious visibility criterion that would do it...

jhodgdon’s picture

Issue summary: View changes

Oh, forgot to update the issue summary...

jhodgdon’s picture

Issue summary: View changes

Update: The summaries are done! I have updated the Guide, so you can see the summaries at:
https://guide-drupal.dev.devdrupal.org/docs/user_guide/en/index.html

And... I realized that Panelizer had the ability to add PHP code for visibility. So I added a visibility rule for the Tabs (local tasks) for both Documentation Guides and Documentation Pages. These are the two lines of code:

$node = $contexts['panelizer']->data;
return empty($node->field_asciidoc_source_file);

Pretty simple... It seems to be working fine on the dev site: no local tasks drop-down for the imported pages (link above in this comment), and it is still there on the drupalorg docs book:
https://guide-drupal.dev.devdrupal.org/drupalorg/docs

So... I think everything is ready to go except actually finalizing this. Any thoughts? Should I spin off an issue in the drupalorg project and make a patch?

Actually, I'm out Friday-Sunday so... no patch until Monday. But anyway, are we ready?

If so, there is one more piece of information I need, which is where we plan to put the User guide source/output directory that we would use for imports and source editing. This is part of the Feeds configuration for this book (it has to know where the source files are in order to put up the source editor).

jhodgdon’s picture

Uh oh, https://guide-drupal.dev.devdrupal.org/ has an "500 Internal Server Error".

drumm’s picture

Panelizer had the ability to add PHP code for visibility

On production we have Paranoia module disabling PHP code fields. That's disabled on dev since I think it might over-lock-down things. Even though this would export cleanly, we shouldn't use Drupal's PHP executing anti-patterns.

If I'm understanding the issue comments properly, this is for hiding the “View history” local task. We can do this in drupalorg.module’s drupalorg_menu_local_tasks_alter(). At the end of that function is how we are hiding the View local task. The code for this will be similar, maybe breaking up the giant if condition.

jhodgdon’s picture

OK, when we get an operational dev site going again, I'll make that change in drupalorg module instead. And try all the deploy steps. And export them to features and make a patch so that hopefully if the site dies again, we won't have to do it all over again. ;)

I still will need to know what directory we would plan to use for the user guide git pull and build source directory.

drumm’s picture

what directory we would plan to use for the user guide git pull and build source directory

Let’s stick with ../userguide_source, next to htdocs. The important parts are that it isn't under htdocs, and has a memorable name.

jhodgdon’s picture

Unfortunately, I think that the Feeds config will need to store either an absolute path, or a path relative to the public files directory, which is the "normal" location for reading feeds (for whatever reason). So ../userguide_source will not help me very much...

drumm’s picture

We keep the files directory configured consistently for dev/staging/prod. Any chance ../../userguide_source works?

jhodgdon’s picture

I'll try it, when I have a dev site up again.

drumm’s picture

I just remembered we did this whole move for API.Drupal.org too. /usr/local/userguide-source will work better in the long run. Please use that

We'll have one copy for devwww in the one place, no extra steps needed when spinning up a fresh dev site. I'll get that set up once devwww is back online.

jhodgdon’s picture

Sounds good.

One note: I cannot currently share the userguide_source directory between the userguide dev site (the one that we have up there temporarily for viewing/editing the guide through direct method) and the guide dev site (the one from this issue, where we're testing importing into nodes using feeds). This is due to the Jenkins script that builds the userguide dev site output barfing when it cannot delete the output/html_feeds directory I have built for use in the guide dev site. (hope that makes sense)

But hopefully we will not need either of these dev sites for much longer. What a nightmare it has been...

drumm’s picture

Like the other restorations, I expect we'll keep the files associated with the existing dev sites. Or recreate the userguide_new one if necessary. I'll be setting the two up differently, keeping them completely separate is good.

jhodgdon’s picture

OK. I've recreated the dev site. I am certain it is not identical to what it was before -- some of the text may of course be different from the last go-around, since the exact text for various things wasn't recorded here and we lost the dev site database. So take a look and see if you want anything updated.

I also used a slightly different URL scheme for the Guidelines book imported nodes. See near top of issue summary for links to the pages. Again, I can change that.

Files attached:

a) For one of the steps, we need to download a bunch of stuff to the sites/all/libraries directory. This tgz file is a shortcut.

b) A patch for the drupalorg module with the updated Documentation feature and 2-line code change, that implements all of this stuff. Of course, as usual with Features, there are some extraneous differences in there, but it's a clean export... should be OK. Needs testing. ;)

I also updated the deployment instructions, as I followed them to create the site. They were all there... just not as cleanly written as I hope they are now. Enjoy!

jhodgdon’s picture

Issue summary: View changes

Minor update to deploy instructions

eojthebrave’s picture

Taking a look at the dev site. Woohoo! This is great. :)

So far the only thing that really stands out to me is the "Source file: views-create.txt" line at the top of every page. I'm wondering if maybe we can move this to the bottom of the copy? This information is important for maintainers, or anyone editing the copy. But not really relevant for someone reading it I don't think.

Thanks for all the awesome work you've put into this so far.

jhodgdon’s picture

Issue summary: View changes

Thanks for taking a look!

We could also just not add that "Source file" line at all -- we will have the "Edit ____" link in the sidebar... maybe we do not need it at all? Hm. I guess you only have the "Edit ___" line if you are logged in, come to think of it. But maybe it is just not necessary at all.

Anyway, I can easily either not add it or move it to the end -- it's done by a PHP script as part of the AsciiDoc output build process.

Thoughts on which would be better (remove or move to the end)?

Anyway, I filed:
#2788151: Move "source file" line to end of output, or remove it, for Feeds display
and added this to the summary.

jhodgdon’s picture

Issue summary: View changes

Oops. I just also noticed that the "Source information" block has no space in "AsciiDocsource". Need to update patch.

jhodgdon’s picture

Here's another thought on #113 - if the person has permission to edit files, we could have an Edit link. If they don't, we could just say what the source file is (this would be in the Source information block).

tvn’s picture

I like #116. Either that or remove the line completely.

jhodgdon’s picture

Yes, I think so too. I'll take care of that on the linked issue, as well as the typo in the Source Block.

jhodgdon’s picture

Issue summary: View changes

OK. I've got these two issues fixed. Here's a new patch for the drupalorg features with the typo fixed.

Unfortunately, I've run into a bug in the Feeds module and the drush commands to import do not actually work. I had only tested it in the UI previously... :( I will need to fix that.

jhodgdon’s picture

Issue summary: View changes

Ah. You can import without the --file option. It uses the location that had previously been entered into the UI at /import. Anyway, the issue I filed for this is #2788265: Add a --source option for drush feeds-import, adding that to the summary and updating instructions.

Unfortunately, this location is not exported into Features. Feeds allows you to export "importers", which have most of the configuration. But the location is part of a "source", not an "importer", and there is no way to export them to a feature.

So, updating the deployment instructions. We will need to visit /import once manually and then can use the drush command forever more.

jhodgdon’s picture

OK... I think everything is pretty much ready to go, pending final review of the new site by tvn/drumm. Any further changes to make?

If it is all fine, what is our next step -- staging? The deployment instructions were written from the point of view of "what I did on the dev site". It will be somewhat simpler on Staging if the patch is applied to Drupalorg (most of the steps say "put this into Features", and the patch will take care of that). But I'm not sure how to get the modulesdownloaded/enabled... or the addition to sites/all/libraries .

And I forgot to upload the new patch for drupalorg before with the typo fixed.

eojthebrave’s picture

I'm not sure if this is just a byproduct of the development process or something else. But I'm currently seeing duplicate chapter names in the navigation.

screenshot showing duplicate items in navigation on user guide development site

This doesn't happen in the main content area when viewing a top-level page that shows the grid of child pages.

jhodgdon’s picture

Huh, that's weird. It's in the Guidelines book too. I am pretty certain it was not like that when I looked at it last! Not sure what happened.

So... I logged in as admin on the site. The OG menus (which provide the navigation for the book as a whole, as well as each chapter) all have double entries, and if you look at a given duplicated entry, the two entries have the same node ID in them. Weird. I do not know how that could have happened -- you would think OG would clean out old entries when updating a node? It definitely should...

So, I wiped out the Guidelines book, and re-imported it. Then without wiping it out, I imported again (with the setting that says "ignore the fact that the data is the same and redo the nodes anyway", so that the nodes were re-created). Then I did the import again. No duplicate entries happened. That book was fixed.

So then I went to the Import page, and without first wiping it out, I imported the User Guide. That didn't help -- the duplicate entries were still there (but it didn't make 3rd entries). So I wiped it out and re-imported, and imported again. No duplicates.

So... I don't know what happened.... But it seems to be fixed, and it's an easy fix if it ever happens again (at least, easy for someone with Administrator priveleges on drupal.org).

Meanwhile... can we get this moving towards actual deployment on drupal.org?

jhodgdon’s picture

The Guide site, as well as https://userguide_new-drupal.dev.devdrupal.org are both WSOD right now (I filed an issue in Infrastructure about it).
#2790893: Dev sites WSOD

Sigh.

Is there any chance we could get this deployed on drupal.org soon, rather than having to build these two sites *again*, assuming that it's the same problem we've been having over and over?

hestenet’s picture

@jhodgon - fortunately this does not appear to be the same issue - it looks like a suhosin config issue on the new dev servers (that are supposed to replace the super fragile ones). Hoping we can fix it quickly, but we're in a bit of disarray having just finished our move out of the office -> full remote today. I'll try to reach out to @basic.

hestenet’s picture

@jhodgdon - back up now!

tvn’s picture

I looked through the dev site, apart from a couple of php notices all looks good to me. Neil will be reviewing in the next few days, and will start preparations for the deployment.

The notices were:

Notice: Undefined property: stdClass::$content in drupalorg_section_contents() (line 32 of /var/www/dev/guide-drupal.dev.devdrupal.org/htdocs/sites/all/modules/drupalorg/drupalorg/plugins/content_types/section_contents.inc).
Notice: Undefined index: und in _asciidoc_display_feeds_get_feeds_info() (line 746 of /var/www/dev/guide-drupal.dev.devdrupal.org/htdocs/sites/all/modules/asciidoc_display/asciidoc_display_feeds/asciidoc_display_feeds.module).

  • jhodgdon committed 2a9bf94 on 7.x-1.x
    Issue #2757921 by jhodgdon: Fix PHP notice
    
jhodgdon’s picture

Thanks for taking a look!

That first error in drupalorg_section_contents() is not from this change. It is because of code that basically looks like this:

$block = new stdClass();
if (something ... several nested if statements) {
  foreach (something) {
     $block->content .= (something);
  }
}

The first time that is hit, $block->content is not defined. I'll let @drumm or you file a separate issue if that is something you want to fix.

The second error, I have added code to the asciidoc_display module to fix it. Diff looks like this:

-    $source_file = $stuff['und'][0]['value'];
+    if (isset($stuff['und'][0]['value'])) {
+      $source_file = $stuff['und'][0]['value'];
+    }

I didn't yet deploy this on the dev site, but it should fix the problem and I've committed the change.

Note: I have friends staying at my house through tomorrow morning so am not online except at the moment, but will be around Wed-Fri this week to help get this deployed if possible. Thanks!

jhodgdon’s picture

Issue summary: View changes

See #2789433: Branch/Release strategy for User Guide, I have just made an 8.x-2.x branch for the User Guide. We should use/import that branch.

drumm’s picture

Progress so far on deployment:

jhodgdon’s picture

Looks good! Glad to see progress on this, thanks!

So... #3 (unzip the files and add to sites/all/libraries)... Can you commit the contents of that zip file or do we need to figure out wget-type commands to download the libraries? If so, let me know and I'll work on that.

#4 and a bunch of others are the patch to drupalorg project that I attached earlier. Actually, that patch takes care of #4 - #13, with the exception of #9, which I believe has to be done manually once from the UI, and thereafter can be done from a Drush command.

jhodgdon’s picture

I'm going to be out of town and off Internet for a few days next week, then back for a few days, then probably gone again for a few days the following week (weather permitting)... I will check back in and/or help out when I can... sorry for any delays that may occur!

jhodgdon’s picture

Issue summary: View changes

Adding one more point to the issue summary, for the Jenkins job. I'm thinking that we should set it up to trigger when there is a new release, not every commit. The reason is that we are embarking now on translations, and it is pretty easy for someone to commit a change that screws up the formatting and makes the build fail. We will check this when we make a Release (because we build e-books) so we can make sure everything builds. Also we can add new languages to the scripts at an appropriate point.

Either that or have it manually triggered.

drumm’s picture

All dependencies go into the drush make file. So far I've added https://bitbucket.org/drupalorg-infrastructure/drupal.org/src/ace16c68db...

I included only the dist directory from asciidoctor because example HTML files like https://github.com/asciidoctor/asciidoctor.js/tree/master/examples are notorious for bringing along security vulnerabilities. Similarly, I took out markitup’s index.html: https://bitbucket.org/drupalorg-infrastructure/drupal.org/src/ace16c68db...

That leaves

You'll also need to copy the js/asciidoc_display.edit.css file and the
js/images/monospace.png file from the AsciiDoc Display project into the
asciidoctor_images folder.

Can those be moved to a css directory so they can be operated on as a directory? That's more doable with drush make.

drumm’s picture

set it up to trigger when there is a new release, not every commit

How about using “dev” and “release” branches? They can be named anything, as long as the names change rarely, if at all. Jenkins is good at monitoring branches for commits, but I'm not sure how good it can do with tags. And we don’t want to be manually updating something like a tag more than a couple times a year.

I think we should keep staging monitoring all commits, and notifying of errors, so they can be caught before getting to a production.

drumm’s picture

We keep the permissions all exported in the drupalorg_permissions Feature, so it can be more-closely monitored. I have those committed & deployed.

I also went ahead and enabled translation for guides and pages so we can start spotting unintended side-effects. So far:
- The comment form got a language selection, now hidden.
- New guides and pages defaulted to language-neutral, but they didn't get good pathauto paths. Now overriding to English.

I committed the last chunk of the drupalorg patch to hide the system message user from the maintainers block.

jhodgdon’s picture

RE #135, yes that could be done... I will not have time to work on it until the 23rd or the following week, sorry!

RE #136, I am not very good with git and merging between one branch and another... Could we perhaps just leave the Jenkins job as manually triggered only? That might be the best thing.

jhodgdon’s picture

One more note about #138... It's not just me and eojthebrave that we have to worry about. With the translations, we will be adding a committer to this project for each new language, and the potential is there for any of them to make a mistake, such as:
- Committing to the wrong branch
- Committing to the wrong language
We won't give them permission to make release nodes, and Git gives us the ability to revert changes that were made in error, but I don't think we want the User Guide to get updated on drupal.org based on commits that have the potential (or even likelihood) to screw everything up, delete the existing guide, etc.

jhodgdon’s picture

I'm coming back after trips... not here for very many days until the next trip, but I should be able to get #135 done tomorrow. I could have done it today actually but it dropped off my radar. Sorry!

  • jhodgdon committed 45455b6 on 7.x-1.x
    Issue #2757921 by jhodgdon: Move Asciidoctor extra files for CSS and...
jhodgdon’s picture

OK, I've made a commit for #135. It moves the two files into an asciidoc_extra directory that need to go into sites/all/libraries/asciidoctor_images.

Readme now says:

You'll also need to copy the contents of the asciidoctor_extra directory into
sites/all/libraries/asciidoctor_images (two files: asciidoc_display.edit.css and
monospace.png).

I also made a small update to the JS for the Monospace button, because we've found that `text` works more consistently than +text+ to make Monospace in AsciiDoc (some processors do not support + apparently).

Hope this helps! Anything else I can do in the next few days to move this forward?

jhodgdon’s picture

jhodgdon’s picture

I will be around tomorrow, and then off for the next 3 weeks, with very limited Internet and no computer (I'll have a tablet for reading email and may be able to respond to questions occasionally). Is there anything else I can do to move this forward?

drumm’s picture

I think I have all the dependencies in place now. See #135 & #142.

Drush make replaced the whole directory with asciidoctor_extra instead of overlaying it. I didn’t look around to see if there was a way to solve this with just Drush make; instead I copied those to https://bitbucket.org/drupalorg-infrastructure/drupal.org/src/5d68032936... which our build script uses to move in arbitrary files.

jhodgdon’s picture

Sounds reasonable... Thanks!

drumm’s picture

FileSize
25.01 KB

Attached is the updated drupalorg patch for the remaining bits, and catching up with changes upstream.

jhodgdon’s picture

I gave this patch a quick look-through, and it all looks correct to me.

drumm’s picture

I committed & deployed the rest of the code. I made an additional commit to lock down the text format to more-closely match filtered HTML, limiting the allowed tags and restricting local images.

Cron is all hooked up on staging, https://www.staging.devdrupal.org/docs/user_guide/en/index.html exists for now, although it will go away with the staging rebuild in the next 24 hours.

Next up is getting the Jenkins job configured for production.

drumm’s picture

I got the script running on production, but the output isn't looking good: https://www.drupal.org/docs/user_guide/en/index.html

I suspect the problem is in the asciidoc parsing. The production server is a bit behind staging/dev:

drumm@wwwdev1:/var/www/dev/drumm-drupal.dev.devdrupal.org/htdocs/sites/all/modules$ asciidoc --version
asciidoc 8.6.9
drumm@wwwdev1:/var/www/dev/drumm-drupal.dev.devdrupal.org/htdocs/sites/all/modules$ xmlto --version
xmlto version 0.0.25

[bender@jenkins1 ~]$ asciidoc --version
asciidoc 8.4.5
[bender@jenkins1 ~]$ xmlto --version
xmlto version 0.0.23

I'll have to leave this as-is for now, I'll see about getting those upgraded.

drumm’s picture

mixologic got asciidoc upgraded and things are looking good. There are extra titles leftover, like on https://www.drupal.org/docs/user_guide/en/planning-chapter.html. I'll see about deleting those, or they might be part of the import we can improve.

drumm’s picture

Status: Needs review » Fixed

I cleaned up the extra guide titles, re-ran the import, and this looks good.

eojthebrave’s picture

Aww yeah! This is so awesome. Thanks Neil and Jennifer!

Just FYI, I'm seeing duplicate menu/navigation entries on some pages https://www.drupal.org/docs/user_guide/en/understanding-distributions.html. Same problem as #122, with a potential solution in #123.

drumm’s picture

Running the import again added another set of menu items. There must be something else unexpected left over from the import with the old asciidoc version. Rather than tracking that down, I'm going to delete all the nodes and do a clean import.

jhodgdon’s picture

WOOOHOOOOOO!!!!! Thanks @drumm!

drumm’s picture

We upgraded asciidoc, but that hasn't seemed to help. We'll try upgrading xmlto next.

On production, /usr/local/user_guide_source/output/html_feed/en/appendix.html has

<h2 class="title"><a id="appendix"></a>Appendix A. Appendix</h2>

instead of

<h1 class="title"><a id="appendix"></a>Appendix A. Appendix</h1>

I've been concentrating on that problem since it is easier to troubleshoot than the extra menu items. I'm hoping the extra menu items will fix themselves once the extra titles are fixed.

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.

eojthebrave’s picture

This appears to still be displaying duplicate links in the sidebar when viewing a specific page within the guide. https://www.drupal.org/docs/user_guide/en/install-run.html

Should we re-open this issue or create a new one to resolve the duplication?

drumm’s picture

Either for new/reopening this issue.

We upgraded xmlto, but #156 is still happening. I’m a bit stumped since I don’t know the AsciiDoc compile toolchain well. Is that something that's been seen before? Is there some system-level configuration/XSL to poke at?

jhodgdon’s picture

Status: Closed (fixed) » Active

I'll take a look today or tomorrow and get on IRC to discuss this -- I just got back from 3 weeks cycling trip so am wading out of a mound of email etc.

The HTML is generated by the xmlto part of the toolchain, which applies an XSL template to the XML output (docbook) that is produced by the previous part of the toolchain. We already have some overrides in xsl files that are checked into the User Guide project, so we can add more if version differences are causing problems.

drumm’s picture

FileSize
592.19 KB

We have xmlto version 0.0.25 everywhere now. We have upgrades making their way through the stack, so they were installed different ways. The best I can come up with for a version difference right now is:

[drumm@jenkins1 ~]$ xsltproc --version
Using libxml 20706, libxslt 10126 and libexslt 815
xsltproc was compiled against libxml 20706, libxslt 10126 and libexslt 815
libxslt 10126 was compiled against libxml 20706
libexslt 815 was compiled against libxml 20706

vs.

drumm@wwwdev1:~$ xsltproc --version
Using libxml 20901, libxslt 10128 and libexslt 817
xsltproc was compiled against libxml 20901, libxslt 10128 and libexslt 817
libxslt 10128 was compiled against libxml 20901
libexslt 817 was compiled against libxml 20901

I don’t want to wade deeper into that unless there’s a good chance it will help.

jenkins1 is currently on 8.x-2.0 of the guide. I’ve also tried 8.x-2.x, which doesn’t seem to make a difference.

The guide.docbook files seem to only have cosmetic differences on jenkins1, from the tag vs. branch. I’ll attach a copy from jenkins1 in case it is useful.

The -m feeds.xsl is doing something, if I remove it on jenkins1, the filenames come out as less human readable. It seems like the <xsl:template name="section.heading"> section might not be getting applied properly.

jhodgdon’s picture

OK...

So here is what I am seeing today, as far as problems:

a) On a Chapter page, such as

https://www.drupal.org/docs/user_guide/en/extend-chapter.html

The chapter title is displayed twice on the page. One is the Drupal page title, and the other is an H2 class="title".

However, on a Topic page, such as

https://www.drupal.org/docs/user_guide/en/extend-maintenance.html

the output in the main body area of the page seems to be OK (only one title, no summary, text all there, images OK).

Since it is just on the Chapter pages, my inclination is to ignore it... but we could try to fix it if you think it's important.

b) On a Topic page, such as

https://www.drupal.org/docs/user_guide/en/extend-maintenance.html

the sidebar navigation shows duplicate entries for the topics (each one appears twice). This is (I believe) due to the OG menus for the chapters all having two entries for each node ID, and I don't know why it is happening. I actually had this problem earlier on the dev site (see #122/#123), and I "fixed" it by wiping out the imported nodes and starting over from scratch, and it never recurred. So, we could try on drupal.org doing the same thing -- you go to /import, click on the guide that needs to be wiped, and go to the Delete Items page to delete the existing items. Then run the Jenkins script again to import it fresh. That might help.

I don't know why it occurs, because OG should be removing existing menu entries when a node is updated.

c) I also noticed that the Source Information block (which comes from a setting in the Feeds importers, which is saved in a Feature in the drupalorg project), there is a typo in the URL for the User Guide project. I filed a separate issue with a tiny patch to drupalorg to fix that:
#2822931: User Guide "Source information" block has a typo

d) There is also a problem with the "Edit source file" link, such as https://www.drupal.org/node/2814377/asciidoc_edit -- it says "Could not read extend-theme-find.txt". Perhaps the directory where the AsciiDoc source is has changed, or isn't readable on production? That is also a field in the Feeds importer config. I filed a separate issue for that too:
#2822936: Cannot edit AsciiDoc source in User Guide on drupal.org

e) The User Guide Contributor Guidelines book at https://www.drupal.org/docs/user_guide_guidelines/index.html has all the same problems except for not (b), so let's try the simple solution there and see if it fixes the User Guide and hope it is a temporary glitch... Although one difference between the Contributor Guidelines book and the User Guide itself is that the User Guide has translations... hm....

jhodgdon’s picture

I talked to drumm on IRC. Have filed a separate issue to get rid of the duplicate titles on the Chapter landing pages:
#2823353: Get rid of duplicate titles in User Guide chapters when imported by Feeds
This is assigned to me. It is not maybe the highest priority, but would be nice to do.

drumm also deleted the User guide, and reimported twice. The problem with the duplicate menu items recurred. Filed another issue for that:
#2823355: Re-import of User Guide is leading to duplicate menu items
Also assigned to me.

jhodgdon’s picture

Adding one more issue, because there is no language switcher, so you cannot see the other language versions of the Guide.
#2823757: Add language switcher to translated documentation guides & pages

eojthebrave’s picture

Thanks for looking into this Jennifer, and Neil. Let me know if there's anything I can do to help.

jhodgdon’s picture

Status: Active » Fixed

OK. Neil just fixed the last outstanding issue (duplicated menu links), so I think we can set this back to Fixed. I'll email @eojthebrave about some vaguely-related To Dos.

eojthebrave’s picture

Thanks for getting the duplicate menu items fixed.

jhodgdon’s picture

Someone today filed this issue about previous/next navigation, which we used to have on all drupal.org documentation pages and currently don't (including the user guide):
#2828230: Navigating through the guide is cumbersome

Also this one about the CSS for the docs pages:
#2828234: CSS padding in H1 page titles on docs pages causes strange wrapping

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.