Please update the metatag documentation to make it clear which tags are stored at node creation, which are generated at serving time, and how to force past-added nodes to pick up the current configuration when stored metatag configurations are changed.

Details follow:

On 9/30/2016, I changed the content type Home Slide to include noindex on the robots metatag. All home slides created after that date generate noindex on the robots metatag in the source code. However, home slides created before that date do not generate noindex on the robots metatag, even when viewed on 8/23/2017.

I expect configuration settings to affect all content of the content type, not just new content.

Configuration is as follows:

$config = new stdClass();
$config->disabled = FALSE; /* Edit this to true to make a default config disabled initially */
$config->api_version = 1;
$config->instance = 'node:home_slide';
$config->config = array(
  'robots' => array(
    'value' => array(
      'noindex' => 'noindex',
      'noarchive' => 'noarchive',
      'nosnippet' => 'nosnippet',
      'noodp' => 'noodp',
      'noydir' => 'noydir',
      'noimageindex' => 'noimageindex',
      'index' => 0,
      'follow' => 0,
      'nofollow' => 0,
      'notranslate' => 0,
    ),
  ),
);

Conversely, changes to Canonical URL take place immediately, even for old content.

The documentation/UI does not appear to indicate which settings are stored for future serving and which are generated every time the page is served. It would be better if these were clearly distinguished, and that instructions to truncate the metadata table were provided for the stored items should regeneration be required, along with any performance warnings if appropriate.

Comments

Charles Belov created an issue. See original summary.

DamienMcKenna’s picture

DamienMcKenna
Location NH, USA
CreditAttribution: DamienMcKenna at Mediacurrent commented
Version: 7.x-1.22 » 7.x-1.x-dev

Could you please check your existing nodes with the Devel module to see if they have any values saved for 'robots'?

Charles Belov’s picture

Charles Belov
he/him
Primary language English
Location San Francisco, CA, US
CreditAttribution: Charles Belov at San Francisco Municipal Transportation Agency commented

DamienMcKenna, thank you.

If I go to one of the affected pages and display the page array using devel, robots turns up as follows:

metatags (Array, 1 element)
  en (Array, 1 element)
    robots (Array, 1 element)
    $...['content']['system_main']['nodes'][179936]['field_expiration_date']['#object']->metatags['en']['robots']
      value (Array, 11 elements)
        noarchive (String, 9 characters ) noarchive
        0 (Integer) 0
        index (Integer) 0
        follow (Integer) 0
        noindex (Integer) 0
        nofollow (Integer) 0
        nosnippet (Integer) 0
        noodp (Integer) 0
        noydir (Integer) 0
        noimageindex (Integer) 0
        notranslate (Integer) 0

and

node:home_slide (Array, 21 elements)
  robots (Array, 1 element)
  $...['content']['metatags']['node:home_slide']['robots']
    #attached (Array, 1 element)
      drupal_add_html_head (Array, 1 element)
        0 (Array, 2 elements)
          0 (Array, 6 elements)
            #theme (String, 7 characters ) metatag
            #tag (String, 4 characters ) meta
            #id (String, 16 characters ) metatag_robots_0
            $...['content']['metatags']['node:home_slide']['robots']['#attached']['drupal_add_html_head'][0][0]['#id']
            #name (String, 6 characters ) robots
            #value (String, 9 characters ) noarchive
            #weight (Float) 5
          1 (String, 16 characters ) metatag_robots_0

I'm not sure how to tell whether any of that is node data as opposed to coming live from the module.

Charles Belov’s picture

Charles Belov
he/him
Primary language English
Location San Francisco, CA, US
CreditAttribution: Charles Belov at San Francisco Municipal Transportation Agency commented
Issue summary: View changes
Charles Belov’s picture

Charles Belov
he/him
Primary language English
Location San Francisco, CA, US
CreditAttribution: Charles Belov at San Francisco Municipal Transportation Agency commented
Issue summary: View changes
Charles Belov’s picture

Charles Belov
he/him
Primary language English
Location San Francisco, CA, US
CreditAttribution: Charles Belov at San Francisco Municipal Transportation Agency commented

In my database backup, I do see the following, which implies whatever has been set on a node is permanent:

INSERT INTO `metatag` (`entity_type`, `entity_id`, `data`, `language`, `revision_id`) VALUES ('node',157431,'a:1:{s:6:\"robots\";a:1:{s:5:\"value\";a:11:{s:9:\"noarchive\";s:9:\"noarchive\";i:0;i:0;s:5:\"index\";i:0;s:6:\"follow\";i:0;s:7:\"noindex\";i:0;s:8:\"nofollow\";i:0;s:9:\"nosnippet\";i:0;s:5:\"noodp\";i:0;s:6:\"noydir\";i:0;s:12:\"noimageindex\";i:0;s:11:\"notranslate\";i:0;}}}','en',277411),('node',157436,'a:1:{s:6:\"robots\";a:1:{s:5:\"value\";a:11:{s:9:\"noarchive\";s:9:\"noarchive\";i:0;i:0;s:5:\"index\";i:0;s:6:\"follow\";i:0;s:7:\"noindex\";i:0;s:8:\"nofollow\";i:0;s:9:\"nosnippet\";i:0;s:5:\"noodp\";i:0;s:6:\"noydir\";i:0;s:12:\"noimageindex\";i:0;s:11:\"notranslate\";i:0;}}}','und',277416),

But that's not how I expect the module to behave; I expect the server to generate the robots metatag from the rules in effect at the time the page is served. For instance, about a month or two ago I changed Canonical URL from
https://[site:url-brief]/node/[current-page:url:unaliased:args:value:1]
to
[current-page:url:absolute]
and it immediately took effect for all nodes, including those created in the past.

Charles Belov’s picture

Charles Belov
he/him
Primary language English
Location San Francisco, CA, US
CreditAttribution: Charles Belov at San Francisco Municipal Transportation Agency commented
Title: Robots tag changes not working retroactively » Improve documentation as to which tags are stored and which are generated on the fly
Component: Other tags » Documentation
Issue summary: View changes

I performed an experiment on my test system and discovered that if I performed the following command:

drush sql-query 'TRUNCATE TABLE metatag'

that the correct metatags were now being generated for the robots metatag on old content.

Charles Belov’s picture

Charles Belov
he/him
Primary language English
Location San Francisco, CA, US
CreditAttribution: Charles Belov at San Francisco Municipal Transportation Agency commented
Issue summary: View changes
DamienMcKenna’s picture

DamienMcKenna
Location NH, USA
CreditAttribution: DamienMcKenna at Mediacurrent commented
Title: Improve documentation as to which tags are stored and which are generated on the fly » Robots tag changes not working retroactively
Component: Documentation » Code

I think there's a bug which causes this, like maybe setting the defaults to a specific value which is then being saved in the database on each entity when it shouldn't be.

anbarasan.r’s picture

anbarasan.r CreditAttribution: anbarasan.r commented

I am experiencing the same issue, seems it is setting the default value to the DB which then get displayed on the page,
<meta name="robots" content="index follow 0 0 0 0 0 0 0 0" />

DamienMcKenna’s picture

DamienMcKenna
Location NH, USA
CreditAttribution: DamienMcKenna at Mediacurrent commented
Status: Active » Postponed
Related issues: +#2924181: Default Value for Robots on config form does not reflect stored configuration

FYI a duplicate of this was opened as #2924181: Default Value for Robots on config form does not reflect stored configuration and includes a patch which may help, could someone please give it a test and see?

entrepreneur27’s picture

entrepreneur27 CreditAttribution: entrepreneur27 commented

As best I can tell this is still an issue. I am using 7.1.28dev (downloaded today) on a site that has been live for years. I changed the default metatagsi from "nothing checked" in the advanced robots section to "no index, follow".

New nodes have this set as expected. However all my old nodes continue to show 0,0,0,0, for the metatags in a view, and when I look at the page source there is no meta tag for robots. Of course, if I edit the individual node, and check the robots no index box, then that shows up as expected in the page source code.

My conclusion: changing the metatag default robots setting does NOT change that setting for existing nodes, as described in the original bug report above. It does not seem to be fixed.

I have hundreds of such nodes and was hoping there would be a way to bulk update them. Any ideas?