Invalid translations of the ContentModerationState entity created before #2946402 exist in some installations of content_moderation

This sounds a lot like #2946402: Content moderation incorrectly always assumes a language is being added when the default language of an entity is being changed. Can you please re-test this against 8.6.x or 8.7.x HEAD? If you can reproduce it, can you please provide detailed instructions on how to reproduce it from scratch, including modules installed, translation settings configured, content created etc.

Okay, then it's possible the fix for the other issue broke this for you, but without actually being able to put a breakpoint in your project, I need to be able to reproduce it on my machine. It's possible the steps to reproduce include a combination of "content is created", "xyz moderation/translation is enabled", "content is edited".

I am also seeing the issue and can confirm that it does seem to be a regression caused by #2946402: Content moderation incorrectly always assumes a language is being added when the default language of an entity is being changed.

My client has already upgraded to 8.6.4 and needs a temporary solution. The attached patch reverts only the code changes.

I will try my best to figure out how to reproduce this issue and document it here.

Thanks for taking the time to write up some instructions to reproduce this, I still haven't been able to.

We are also experiencing this after the 8.6.4 update. Just updated to 8.6.5 but issue remains. Applied patch in #6 and it seems to be working for us. Thanks @jrockowitz for that.

I did run into this as well. Created a duplicate before I found this: #3024816: EntityOperations::updateOrCreateFromEntity(): A translation already exists for the specified language.

From the code, it would happen if the moderated entity default translation changes to one that already exists, possibly because it was deleted and re-created.

Hi, I'm on Drupal 8.6.5. I had the same issue and the patch fixed the save error above.

However, now, when editing a specific translation, the field content shows up with the default (english) language and global fields that are not translatable show up. Essentially, it appears the whole step of loading translation info doesn't happen.

@Berdir Hm, I can test that scenario, but we do clean up the composite entities as translations are deleted.

Disregard my previous comment about the issue persisting for me. It had to do with my environment and appears to be unrelated. Specifically, how I was handling language negotiation settings.

I also did hit this issue on a client website. I'm still not sure why this happened, but the only page where this occured is one of the earliest created. All recent pages does not have this issue (at least for now).

The "patch" in #6 also made the issue disappear.

Note that the only real line to change is to comment the `$content_moderation_state->langcode = $entity_langcode;`. Once this is commented the issue disappear.

As @Berdir commented, if I inspect the `$content_moderation_state` variable before the error happen the `langcode` is "de" even if I'm editing the "fr" version. I don't know what the editor made with this page but I guess it was first created in german and when the language was changed to french the $content_moderation_state langcode was not properly updated.

Is it possible this is being caused by dud translation data that was created before folks updated to include #2946402: Content moderation incorrectly always assumes a language is being added when the default language of an entity is being changed. If you created an 'fr' page, then changed the language of that page to 'en' and then attempted to add a 'fr' translation, that may be causing this?

If that's the case it would be good to prove it by examining the entity tables for both node and content_moderat_state. The following two queries would probably provide enough information:

select nid, vid, langcode from node_field_revision where nid = '99999';
select id, revision_id, langcode, content_entity_id, content_entity_revision_id from content_moderation_state_field_revision where content_entity_id = '99999';

Where 99999 is the ID of the node that is broken.

If this is NOT based on historical data from before that fix, I still can't reproduce it in HEAD, so if anyone can do that, working towards some clear instructions would be awesome.

With my system totally updated, I have tried that Querys and this was the result, in a test machine I have tried to remove those result and after doing it the error dissapear in that id.

select id, revision_id, langcode, content_entity_id, content_entity_revision_id from content_moderation_state_field_revision where content_entity_id = '377';
 id  | revision_id | langcode | content_entity_id | content_entity_revision_id 
-----+-------------+----------+-------------------+----------------------------
 145 |         581 | en       |               377 |                       6073
 145 |         584 | en       |               377 |                       6076
 145 |         584 | de       |               377 |                       6076
 145 |         588 | en       |               377 |                       6080
 145 |         588 | de       |               377 |                       6080
 145 |         589 | en       |               377 |                       6081
 145 |         589 | de       |               377 |                       6081
(7 rows)

select nid, vid, langcode from node_field_revision where nid = '377';
 nid | vid  | langcode 
-----+------+----------
 377 | 1897 | en
 377 | 1897 | de
 377 | 6073 | en
 377 | 6073 | de
 377 | 6076 | en
 377 | 6076 | de
 377 | 6080 | en
 377 | 6080 | de
 377 | 6081 | en
 377 | 6081 | de
(10 rows)

But It doesn't disappear all the errors related. I still getting this error

The entity cannot be translated since it is language neutral (und).
That appears in a content that was created as a Not Specified language, but later it was changed to a German when we try to edit the German version or to add translation I get this error.
The patch #6 didn't fix my error.

We ran into this error on an existing site that's been updated over the last two years. This is what I'm seeing with a specific node. Looking at the database table content_moderation_state_field_revision using the query in a previous comment we can see that the values from default_langcode are not correct.

We have English (en) and French (fr) languages and for some reason en is not set as the default with a value of 1 even though the site configuration is set to English as the default. The fr language is set as the default, but shouldn't be.

One solution seems to be to change the default_langcode to 1 for en and 0 for fr in the table content_moderation_state_field_revision. Then when trying to save the en content it now works where it errored before.

Don't know how these got messed up, but maybe during an upgrade or something.

Hi,

I can reproduce this following the steps outlined below. The project was using Drupal 8.6.2 with English as default language with French as second language. The content type was using a content moderation workflow. I'm surprised that when changing languages in step 3 below that the node table then has two records as seen in Step 3.4.Never mind, this is standard node revisioning.

1. Start with Drupal 8.6.2
2. Create content and save as published with French language
   1. Query:select * from content_moderation_state_field_revision where content_entity_id = '99999' order by revision_id desc;
   2. Result: 32978 39215 fr 1 editorial published node 99999 52440 1 1
   3. Query: select nid, vid, langcode from node_field_revision where nid = '99999';
   4. Result: 99999 52436 fr
3. Edit and change language to English, save and publish
   1. Query: select * from content_moderation_state_field_revision where content_entity_id = '99999' order by revision_id desc;
   2. Result:
      

      32978 39216 fr  1 editorial published node  99999  52441 1 NULL
      32978 39216 en  1 editorial published node  99999  52441 0 1
      32978 39215 fr  1 editorial published node  99999  52440 1 1
            

   3. Query: select nid, vid, langcode from node_field_revision where nid = '99999';
   4. Result:
      

      99999  52437 en
      99999  52436 fr
            

4. Update to 8.6.7 and process updates: node module's post update "Clear caches due to updated views data."
5. Edit same node and attempt to save
6. Observe the reported error, "A translation already exists for the specified language (en)."

Here's more info about the node table.
select * from node_field_revision where nid = '99999'\G"

*************************** 1. row ***************************
                          nid: 99999
                          vid: 52437
                     langcode: en
                        title: Test Article
                          uid: 1
                       status: 1
                      created: 1549121118
                      changed: 1549121286
                      promote: 1
                       sticky: 0
revision_translation_affected: 1
             default_langcode: 1
   content_translation_source: und
 content_translation_outdated: 0
*************************** 2. row ***************************
                          nid: 99999
                          vid: 52436
                     langcode: fr
                        title: Test Article
                          uid: 1
                       status: 1
                      created: 1549121118
                      changed: 1549121203
                      promote: 1
                       sticky: 0
revision_translation_affected: 1
             default_langcode: 1
   content_translation_source: und
 content_translation_outdated: 0

When I repeat the above steps with 8.6.7 and new a node, step 3 produces different records in the content_moderation_state_field_revision table. And, the resulting error does not repeat in steps 5 and 6 above.

select * from content_moderation_state_field_revision where content_entity_id = '99999' order by revision_id desc;

32980	39217	en	1	editorial	published	node	99999	52443	1	1
32980	39216	fr	1	editorial	published	node	99999	52442	1	1

I was able to reproduce the issue following the above steps using standard install profile with content moderation, workflow and content translation modules enabled along with configuring the basic page content type to be moderated and translated after adding French language. B/c #2946402: Content moderation incorrectly always assumes a language is being added when the default language of an entity is being changed was added into 8.6.4, I dropped back to 8.6.3, created content following the above steps and then updated to 8.6.7.

When changing the language, it appears that an additional content_moderation_state_field_revision record is created for the content's original language.

select * from content_moderation_state_field_revision where content_entity_id = '1' order by revision_id desc\G
*************************** 1. row ***************************
                           id: 1
                  revision_id: 2
                     langcode: fr
                          uid: 1
                     workflow: editorial
             moderation_state: published
       content_entity_type_id: node
            content_entity_id: 1
   content_entity_revision_id: 2
             default_langcode: 1
revision_translation_affected: NULL
*************************** 2. row ***************************
                           id: 1
                  revision_id: 2
                     langcode: en
                          uid: 1
                     workflow: editorial
             moderation_state: published
       content_entity_type_id: node
            content_entity_id: 1
   content_entity_revision_id: 2
             default_langcode: 0
revision_translation_affected: 1
*************************** 3. row ***************************
                           id: 1
                  revision_id: 1
                     langcode: fr
                          uid: 1
                     workflow: editorial
             moderation_state: published
       content_entity_type_id: node
            content_entity_id: 1
   content_entity_revision_id: 1
             default_langcode: 1
revision_translation_affected: 1

By removing the, what I think is erroneous, fr content_moderation_state_field_revision record that has revision_translation_affected NULL and update the latest en content_moderation_state_field_revision record's default_langcode value to 1 or TRUE resolves the issue as @kamkejj described in #18. I suspect removing the fr content_moderation_state_field_revision record that has revision_translation_affected NULL may not be necessary.

I think this is at least a major bug, so I'm escalating per priority page definition.

• Render one feature unusable with no workaround.
• Cause a significant admin- or developer-facing bug with no workaround.
• Cause user input to be lost, but do not delete or corrupt existing data.

I haven't looked into the code changes that is surfacing this issue. Hopefully, this can be resolved via a change to logic instead of cleaning up data.

Thanks @jasonawant. I think that more than confirms my suspicions in #15. I think ultimately it would be good to clean up the dud data instead of adjusting the logic, then at least all of the installations of content moderation would be in a known predictable state and we don't have to account for or test features against having dud languages floating around.

Great steps to reproduce in #19, next step should be creating a database fixture with the invalid data created before 8.6.2.

Also agree with the "Major" classification.

I'm not quite sure the best path to take to move this forward. I followed the directions found on Writing Automated Update Tests for Drupal 8 page to generate a fixture from a 8.6.2 site with a single node with the dud moderation data. Let me know if this isn't correct. It looks like this file could be simpler and only contain specific content. Instead, it appears to contain the entire db.

What are the next steps, write an update hook that updates the content_moderation_state_field_revision records?

We came across this today as well.

Not sure how the invalid translations were created.

If we need a quick fix would a couple of blanket db_queries to set the default default_langcode in the content_moderation_state_field_revision table to 0 where the langcode is a language that is not supposed to be default and one to the default_langcode to 1 where the langcode is the default language be enough to sort this out?

We tested this on a single piece of content that we needed fixed urgently and it seemed to do the trick.

Edit: A blanket query for all entries where the default langcode is not the one it's supposed to be is not a good solution. It causes some of the content to break with another error (empty state value).

This is my attempt, not sure if it is the right approach.

Well, updating the default translation is not supported by the API. How should we proceed?

Using patch from #26, getting the following when running the update (en is default language, zh-hans is second language):
Failed: LogicException: The default translation flag cannot be changed (zh-hans). in Drupal\Core\Entity\ContentEntityBase->onChange() [error]
(line 831 of docroot/core/lib/Drupal/Core/Entity/ContentEntityBase.php).

As @yongt9412 says, changing this is not supported by the API. Seems like running DB queries is the only available solution at the moment.

Following @Thibaud G comment in #27 to fix the database, these are the SQL needed to verify and update the default_langcode in the table content_moderation_state_field_revision to match the default_langcode in the table node_field_data.

Important: Always backup your database before running any SQL statements.
Following SQL query will list the nodes with mismatched default_langcode

# Select the default language code.
SELECT n.nid, n.vid, n.langcode, c.default_langcode FROM node_field_data n INNER JOIN content_moderation_state_field_revision c ON (n.nid = c.content_entity_id AND n.vid = c.content_entity_revision_id AND n.langcode = c.langcode) WHERE n.default_langcode = 1 AND c.default_langcode = 0;
# Select the languages.
SELECT n.nid, n.vid, n.langcode, c.default_langcode FROM node_field_data n INNER JOIN content_moderation_state_field_revision c ON (n.nid = c.content_entity_id AND n.vid = c.content_entity_revision_id AND n.langcode = c.langcode) WHERE n.default_langcode = 0 AND c.default_langcode = 1;

To fix the database if the previous statements returned results (and after taking your verified backups):

# Fix default language code.
UPDATE node_field_data n INNER JOIN content_moderation_state_field_revision c ON (n.nid = c.content_entity_id AND n.vid = c.content_entity_revision_id AND n.langcode = c.langcode) SET c.default_langcode = 1 WHERE n.default_langcode = 1 AND c.default_langcode = 0;
# Fix other language codes.
UPDATE node_field_data n INNER JOIN content_moderation_state_field_revision c ON (n.nid = c.content_entity_id AND n.vid = c.content_entity_revision_id AND n.langcode = c.langcode) SET c.default_langcode = 0 WHERE n.default_langcode = 0 AND c.default_langcode = 1;

+++ b/core/modules/content_moderation/content_moderation.post_update.php
@@ -94,3 +94,89 @@ function content_moderation_post_update_update_cms_default_revisions(&$sandbox)
+  if (!isset($entity_type_id)) {
+    $sandbox['bundles'] = [];
+    $sandbox['entity_type_ids'] = [];
+    /** @var \Drupal\workflows\WorkflowInterface $workflow */
+    foreach (Workflow::loadMultipleByType('content_moderation') as $workflow) {
+      /** @var \Drupal\content_moderation\Plugin\WorkflowType\ContentModeration $plugin */
+      $plugin = $workflow->getTypePlugin();
+      foreach ($plugin->getEntityTypes() as $entity_type_id) {
+        $sandbox['entity_type_ids'][$entity_type_id] = $entity_type_id;
+        foreach ($plugin->getBundlesForEntityType($entity_type_id) as $bundle) {
+          $sandbox['bundles'][$entity_type_id][$bundle] = $bundle;
+        }
+      }
+    }
+    $sandbox['offset'] = 0;
+    $sandbox['limit'] = Settings::get('entity_update_batch_size', 50);
+    $sandbox['total'] = count($sandbox['entity_type_ids']);
+    $entity_type_id = array_shift($sandbox['entity_type_ids']);
+  }

Maybe it would be simpler to batch over every ContentModerationState entity instead, and compare to being able to load an associated entity and language? Might make for a lot simpler batching logic.

Changing the default language can also be done by simply loading the default entity and updating the 'langcode' field.

This is something we should also make sure we have extensive testing for, since it's such a tricky data integrity problem.

For me, #28 doesn't help.
Because after this changes some nodes become unpublished. And it is not ok to have many unpublished nodes.

Here is what we ended using to fix this in our system in production. After extensive testing, we were not able to find any problem with the impacted nodes anymore. Based on #28.

to put in a post update hook:

  // Get the nids of the nodes to fix.
  $connection = \Drupal::database();
  $query = $connection->query('SELECT n.nid FROM node_field_data n INNER JOIN content_moderation_state_field_revision c ON (n.nid = c.content_entity_id AND n.vid = c.content_entity_revision_id AND n.langcode = c.langcode) WHERE n.default_langcode = 1 AND c.default_langcode = 0;');
  $nodes = array_keys($query->fetchAllKeyed(0,0));

  // Update english language rows (aka en rows).
  $connection->query('UPDATE content_moderation_state_field_revision c SET c.default_langcode = 1 WHERE c.langcode = :langcode AND c.default_langcode = 0 AND c.content_entity_id IN (:nids[])', [':langcode' => 'en', ':nids[]' => $nodes]);

  // Update chinese language rows (aka zh-hans).
  $connection->query('UPDATE content_moderation_state_field_revision c SET c.default_langcode = 0 WHERE c.langcode = :langcode AND c.default_langcode = 1 AND c.content_entity_id IN (:nids[])', [':langcode' => 'zh-hans', ':nids[]' => $nodes]);

Note that you ll need to adapt this to the languages installed on your website, as they are hardcoded in the code above. English is for us the default language, and Chinese the only other language we have, that is not the default.
Also note that the problem we had with #28 was that it was only updating the default revision, so any existing forward revision was still wrong and creating issues. Thus, our code above fixes the default language and all the existing revisions of the affected nodes.

What happens if the content is not created in the default language... I guess SQL suggestion above will not be valid for those nodes. I guess we need a stable solution to this problem. I guess d.o core team should look into this on urgent basis. Thanks

I manually applied the solution in #28 on a production website and it fixed the issue on all the pages where this error occured. After checking all the pages in all translations I didn't found any unpublished node like in #31, maybe because all affected nodes I had where published.

About #29 I tried by updating the code to load the default entity and update its langcode field, I get the same error as the reported here, that there is a translation already for that language. We need to do something for the revisions still I believe, maybe use queries already?

Test leftover. :/

#36 patch looks like doesn't need to be rerolled against 8.6.x

I was trying to work on top of #35 in order to find a way to remove dud data and update the default_langcode property in affected content moderation state entities. So far, I haven't found a way to avoid API restrictions and errors that are the reason for this issue as described in #27 and #26.

I would like to know if #32 is an acceptable solution and good starting point in order to commit this issue?

No needs reroll

I updated Drupal from 8.5.15 to 8.7.1 and now I'm facing the same problem.

InvalidArgumentException: A translation already exists for the specified language (en). in Drupal\Core\Entity\ContentEntityBase->onChange() (line 810 of /home/user/www.domain.com/core/lib/Drupal/Core/Entity/ContentEntityBase.php).

The site is multilingual (/en, /de).
used PHP-Version: 7.1.25

I tried to implement the patch from #36, but when I run the 'database update script' it fails and reports another error:
Error: Call to undefined method stdClass::setNewRevision() in content_moderation_post_update_update_default_langcode() (line 252 of /home/user/www.domain.com/core/modules/content_moderation/content_moderation.post_update.php) ...

Does anyone has the same problem, or what is the best solution to fix this issue yet?

Setting back to Needs Review & triggering bots.

This patch aims to prevent the exception error from the issue description by matching the content moderation state entity language from the current language of the moderated entity (ContentModerationStateEntity::loadFromModeratedEntity).

Let's see the tests...

No interdiff as it's not related to the idea from #26.

I've updated the issue description; hope this helps clarifies and makes it easier for others to jump in.

The last patch goes in a different direction that previously described in comment #22. The issue description outlines this path forward instead of changing logic.

The patch in #36 throws an error.

Error: Call to undefined method stdClass::setNewRevision() in /var/www/docroot/core/modules/content_moderation/content_moderation.post_update.php on line 176

Throws "Error: Call to undefined method stdClass::setNewRevision()"; Likely caused by a missing $default_entity that has sense been deleted.

+++ b/core/modules/content_moderation/content_moderation.post_update.php
@@ -94,3 +94,93 @@ function content_moderation_post_update_update_cms_default_revisions(&$sandbox)
+        $default_entity = $storage->load($revision->id());
+        $default_entity->langcode = $key;
+        $default_entity->setNewRevision(FALSE);
+        $default_entity->save();

Found some time to look more closely at this. This parts do not make sense to me from the patch in #36 comment.

1. +++ b/core/modules/content_moderation/content_moderation.post_update.php
   @@ -94,3 +94,93 @@ function content_moderation_post_update_update_cms_default_revisions(&$sandbox)
   +  /** @var \Drupal\Core\Entity\ContentEntityInterface[] $revisions */
   ...
   +
   

   This variable name is confusing. I think $content_moderation_state_storage->loadMultipleRevisions(array_keys($result)); returns \Drupal\content_moderation\Entity\ContentModerationState entities.

2. +++ b/core/modules/content_moderation/content_moderation.post_update.php
   @@ -94,3 +94,93 @@ function content_moderation_post_update_update_cms_default_revisions(&$sandbox)
   +  $storage = \Drupal::entityTypeManager()->getStorage($entity_type_id);
   ...
   +        $default_entity = $storage->load($revision->id());
   

   If I understand this correctly, for $entity_type_id of node, we have \Drupal\node\NodeStorage as $storage. Then, this is trying to use the id of the previously loaded \Drupal\content_moderation\Entity\ContentModerationState entity as if it's the Node's ID to load the node. Is that right? Seems that if we are trying to load the Node or other entity, we should use the content_entity_id value of the moderation state entity.

The queries on #28 solved the issue for me.

I have ran it in multiple sites, all of them with "en" as source language of the corrupted nodes, and different translations on each site.

The following code works for me (if content moderation is used for nodes only):

  $database = \Drupal\Core\Database\Database::getConnection();

  $query = $database->query("SELECT DISTINCT c.id, n.nid FROM {node_field_revision} n INNER JOIN {content_moderation_state_field_revision} c " .
    "ON (n.nid = c.content_entity_id AND n.vid = c.content_entity_revision_id AND n.langcode = c.langcode AND content_entity_type_id = 'node') " .
    "WHERE n.default_langcode <> c.default_langcode");

  $problem_items = $query->fetchAllKeyed();

  foreach ($problem_items as $cmid => $nid) {
    foreach ($database->query('SELECT vid, langcode, default_langcode, revision_translation_affected FROM {node_field_revision} WHERE nid = :nid', [':nid' => $nid])->fetchAll() as $record) {
      $database->update('content_moderation_state_field_revision')
        ->fields([
          'default_langcode' => $record->default_langcode,
          'revision_translation_affected' => $record->revision_translation_affected,
        ])
        ->condition('id', $cmid)
        ->condition('content_entity_revision_id', $record->vid)
        ->condition('langcode', $record->langcode)
        ->execute();
    }

    foreach ($database->query('SELECT langcode, default_langcode, revision_translation_affected FROM {node_field_data} WHERE nid = :nid', [':nid' => $nid])->fetchAll() as $record) {
      $database->update('content_moderation_state_field_data')
        ->fields([
          'default_langcode' => $record->default_langcode,
          'revision_translation_affected' => $record->revision_translation_affected,
        ])
        ->condition('id', $cmid)
        ->condition('langcode', $record->langcode)
        ->execute();
    }

    foreach ($database->query('SELECT revision_id, langcode FROM {content_moderation_state_field_revision} WHERE id = :id AND default_langcode = 1', [':id' => $cmid])->fetchAll() as $record) {
      $database->update('content_moderation_state_revision')
        ->fields([
          'langcode' => $record->langcode,
        ])
        ->condition('id', $cmid)
        ->condition('revision_id', $record->revision_id)
        ->execute();
    }

    foreach ($database->query('SELECT langcode FROM {content_moderation_state_field_data} WHERE id = :id AND default_langcode = 1', [':id' => $cmid])->fetchAll() as $record) {
      $database->update('content_moderation_state')
        ->fields([
          'langcode' => $record->langcode,
        ])
        ->condition('id', $cmid)
        ->execute();
    }
  }

Tentatively closing this as outdated. I know there was no committed fix for this, however a really large update process might be riskier for the majority of sites which likely weren't affected by this bug.

No we're a couple of releases down the line, hopefully most affected installations found solutions in this thread for dealing with the problem.

If anyone feels strongly this should remain open, please feel free to do so.