[UNSUPPORTED] Migrate translation for enable revision

Sorry, I already stated in #2396103: [DATA LOSS] Translations deleted I'm not willing to support this functionality. I'd suggest you to move this code to a sandbox project if you are feeling bold enough to maintain it :)

However, if you need any change in the ET codebase to support this from a separate migration module, you are welcome to post patches.

New patch attached which is a cleaned up version of Calystod's patch. It may still need work (I'm not convinced all of the data was updated successfully and that everything is working, and I think this update can be simpler).

I'm reopening this issue because I disagree with plach that this can't be done in a reliable way and I don't see the point in shutting down work on an update that lots of people need, even if you never commit it.

I came to this issue working on a project involving workbench moderation revisions and translations. We found that translations didn't have revision support but upon reading some issues we learned we could update to the latest dev to get revision support. After updating we got a schema that looked like it had revision support but we eventually discovered revision support wasn't working at all. Figuring we had to dive deep into debugging this module to fix revision support we allocated a bunch of time in our sprint to it. I started looking into how revision support worked and discovered function entity_translation_update_7007 had disabled revision support through an undocumented setting that I could find out about no other way than reading the entire .install file.

The problem is that once this is committed I will have to support any bug that it may introduce. And supporting bugs involving data migrations is definitely something I have no will to do. I don't see why this code cannot be part of a separate sandbox project, as I suggested above.

For everyone following this issue: I'm not going to play status pong here, but I won't commit any patch dealing with this. As I stated above I'm totally open to commit patches that will make dealing with this in a separate project easier.

Oh, btw the setting will be documented in the next release notes.

d.o.-- :D

Here are the release notes:

https://www.drupal.org/node/2412615

I linked this issue from there.

+++ b/entity_translation.install
@@ -446,3 +446,90 @@ function entity_translation_update_7007() {
+    $info_entity = entity_get_info($entity_type);
+    // For each row, we copy information into revision table
+    foreach ($result as $row) {
+      $entity = entity_load_single($row->entity_type, $row->entity_id);
+      $revision_id = $entity->{$info_entity['entity keys']['revision'] != "" ? $info_entity['entity keys']['revision'] : $info_entity['entity keys']['id']};

Just to clarify my position: hooks should not be called in update functions, and working with entities in a generic fashion without relying on hooks is almost impossible. This is why I stated I'm not sure this is actually possible to implement reliabily. It may work in most situations, but it will almost certainly break in some edge cases.

I tried migrating with the patch from #5. It failed, however, because I have a lot of orphaned rows in the entity_translation for deleted entities. Should the patch consider this or is something wrong with my setup because the oprhaned rows should not be there?

So you're saying the patch might 'need work'?

If translation revisioning is an option, then one could begin with normal node revisioning turned on, but not translation revisions. If then one decides to allow translation revisioning, the software should accommodate this by creating revision records for translations. It should either be optional or not. If one already has node revisioning turned on, then the option should no longer be available until translation revision records have been created manually.

Might be related

Steps to reproduce related bug at https://www.drupal.org/node/2030943#comment-9721545

I discovered a bug in the patch. When grabbing the 25 records in each batch, it was dependent on the entity_id column. If you have ids in that column that are farther apart than 25 then it enters an infinite loop. You need to use the range() method and key off the $sandbox['progress'] counter.

See this gist for my version. The change I made is in the db_select calls on lines 33-38.

https://gist.github.com/silent-e/32cd2b9c93737eed987c

I can't confirm this is 100% bug free because it failed on me when it got to the next entity type. I'm leaving it here for posterity and review. :)

(e)

I needed to face and solve this same use case (big update of existing site with entity translations and workbench moderation), so I had to update/upgrade entity translations, with revisions ...
Both #5 patch and ruckweb's amend didn't work out of the box for me, due to unexpected stops in unfinished records processing or the generation of infinite loops in presence of two or more entity_types (entity_translation_update_7008 never finishes ...).

I investigated and fixed the main bug, that was in the fixed range (error) of the db select on the entity_translation table.
The following is the code that perfectly works for me now for this entity_translation_update_7008 function:

/**
 * Create revision for existing translation.
 */
function entity_translation_update_7008(&$sandbox) {
  if (!isset($sandbox['progress'])) {
    $sandbox['progress'] = 0;
    $sandbox['current_type'] = 0;
    $sandbox["entity_type"] = array();
    $sandbox['max'] = db_query('SELECT COUNT(*) FROM {entity_translation}')->fetchField();
    $types = db_query('SELECT DISTINCT(entity_type) FROM {entity_translation}');
    $i = 0;
    // For each entity type, we initialize some information.
    foreach ($types as $type) {
      $sandbox["entity_type"][$i]["type"] = $type->entity_type;
      $sandbox["entity_type"][$i]['max'] = db_query('SELECT COUNT(*) FROM {entity_translation} WHERE entity_type=:entity_type', array(':entity_type' => $type->entity_type))->fetchField();
      $sandbox["entity_type"][$i]['progress'] = 0;
      $sandbox["entity_type"][$i]['current'] = 0;
      $sandbox["entity_type"][$i]['finish'] = 0;
      $i++;
    }
    $sandbox["entity_type_count"] = $i;
  }

  // If we have entities to update.
  if ($sandbox["entity_type_count"] > 0) {
    $i = $sandbox['current_type'];
    $entity_type = $sandbox["entity_type"][$i]["type"];

    // Define the new starting point for the database fetch range,
    // depending upon the entity_type query.
    $sandbox_progress = isset($sandbox["entity_type"][$i]['progress']) ? $sandbox["entity_type"][$i]['progress'] : $sandbox['progress'];

    // Get 25 records at a time to create revision table records.
    $result = db_select('entity_translation', 't')
      ->fields('t')
      ->orderBy('t.entity_id', 'ASC')
      ->condition('t.entity_type', $entity_type, "=")
      ->range($sandbox_progress, $sandbox_progress + 25)
      ->execute();

    $info_entity = entity_get_info($entity_type);
    // For each row, we copy information into revision table.
    foreach ($result as $row) {
      $entity = entity_load_single($row->entity_type, $row->entity_id);
      $revision_id = $entity->{$info_entity['entity keys']['revision'] != "" ? $info_entity['entity keys']['revision'] : $info_entity['entity keys']['id']};

      db_insert('entity_translation_revision')
        ->fields(array(
          "entity_type" => $row->entity_type,
          "entity_id" => $row->entity_id,
          "revision_id" => $revision_id,
          "language" => $row->language,
          "source" => $row->source,
          "uid" => $row->uid,
          "status" => $row->status,
          "translate" => $row->translate,
          "created" => $row->created,
          "changed" => $row->changed,
        ))
        ->execute();

      db_update("entity_translation")
        ->fields(array(
          "revision_id" => $revision_id,
        ))
        ->condition("entity_type", $row->entity_type, "=")
        ->condition("entity_id", $row->entity_id, "=")
        ->condition("language", $row->language, "=")
        ->execute();

      // End operation, we update the sandbox.
      $sandbox['progress']++;
      $sandbox["entity_type"][$i]['progress']++;
      $sandbox["entity_type"][$i]['current'] = $row->entity_id;
      $sandbox["entity_type"][$i]['finish'] = empty($sandbox["entity_type"][$i]['max']) ? 1 : ($sandbox["entity_type"][$i]['progress'] / $sandbox["entity_type"][$i]['max']);
    }
    if ($sandbox["entity_type"][$i]['finish'] >= 1) {
      $sandbox['current_type']++;
    }
  }

  // Batch is finished when all operation have been process.
  $sandbox['#finished'] = empty($sandbox['max']) ? 1 : $sandbox['progress'] / $sandbox['max'];

  if ($sandbox['#finished'] >= 1):
    variable_set('entity_translation_revision_enabled', TRUE);
  endif;

  return '';
}

I feel this now a very feasible solution for batch update entity_translation to entity_translations with revisions, also on existing sites (by hand would have been impossible due to more than 20.000 translated contents ...)
Hope this will help ...

Cheers.

I tried patch #5 and ran also in an infinite loop like itamair with my 5000 entity records. Then I tried the script of itamair. It managed more records but there's a problem with empty 'revision_id' (=NULL). I got the following error message.
----
The following updates returned messages
entity_translation module
Update #7008

Failed: PDOException: SQLSTATE[23000]: Integrity constraint violation: 1048 Column 'revision_id' cannot be null: INSERT INTO {entity_translation_revision} (entity_type, entity_id, revision_id, language, source, uid, status, translate, created, changed) VALUES (:db_insert_placeholder_0, :db_insert_placeholder_1, :db_insert_placeholder_2, :db_insert_placeholder_3, :db_insert_placeholder_4, :db_insert_placeholder_5, :db_insert_placeholder_6, :db_insert_placeholder_7, :db_insert_placeholder_8, :db_insert_placeholder_9); Array ( [:db_insert_placeholder_0] => node [:db_insert_placeholder_1] => 5293 [:db_insert_placeholder_2] => [:db_insert_placeholder_3] => de [:db_insert_placeholder_4] => en [:db_insert_placeholder_5] => 9 [:db_insert_placeholder_6] => 1 [:db_insert_placeholder_7] => 0 [:db_insert_placeholder_8] => 1417594936 [:db_insert_placeholder_9] => 1417603388 ) in entity_translation_update_7008() (Line 510 ...

----

Does anyone know how to manage this problem?

#15 worked for me. Thanks!!

Hi itamair, thanks for your contribution :) Do you want to upload your code in a patch file format? You could get credit for your contribution and it makes it easier for the community to review and commit the patch.

Hi all, if you're interested and available to create that patch but you're a newcomer to creating a patch you might be interested in that page https://www.drupal.org/node/707484

#15 did work for me, but:
- I got a time-out on max_execution-time (30 seconds), though its hard to see anything in that patch that could take so long. The 2nd time (after restoring the backup) it run the whole batch in under 30 seconds...
- If people are using this, they should be aware that before updating this module anywhere in the future, they should manually change the schema column of the record for this module in the system table back to 7007. So better do that directly after running this code, but then you'll have to (re)move and/or rename this function (or it will run again the next time you execute update.php).

I think it's worth looking at this from a different angle: there's really no need to do an entity_load for each entity to populate this table. You just need two steps, which can be done purely using SQL queries. Content must be frozen before starting and a backup taken of course.

1. Ensure every row of the entity_translation table has a non-null revision_id
2. Copy the content of the entity_translation table to entity_translation_revision

For step 1, where there's a NULL revision_id, this is because the entity / bundle is not subject to revision. The entity_id can be used as the revision_id, and I believe the table should have been set up this way in the first place to be consistent with other Drupal APIs.

UPDATE entity_translation SET revision_id = entity_id WHERE revision_id IS NULL

For step 2, it's just a straight copy.

INSERT INTO entity_translation_revision 
(entity_type, entity_id, revision_id,LANGUAGE, source, uid, STATUS, translate, created, CHANGED)
SELECT 
entity_type, entity_id, revision_id,LANGUAGE, source, uid, STATUS, translate, created, CHANGED
 FROM entity_translation

This can be done much more quickly than running the code in previous patches, and should achieve the same end.

Patch based on the suggestion in #20. The update process works like a charm with 10.000+ entities.

Getting a little too used to modern PHP.

One missed [].

Some time has passed and the install file has now two new update functions. I suppose that in order to use the patch from #25 I should:

1. Rename the functions to 7010 and 7011 and add them to the .install file
2. Run the update
3. Change the schema_version in the system table of the entity_translation module back to 7009, since the two functions in the patch are never going to be committed

Am I correct?

@plach : are not you going to reconsider your position after seeing the latests patch (#25) ?

It seems really simple and with no corner issues, I don't quite understand why you will not take it. Having a functionality that can not be activated is like not having the functionality at all. It's a pity.

I'm sorry but there's not proof that that code won't break badly in some unforeseen edge cases, people are welcome to use it if it works for them. I just don't want to deal with any (more) bug report dealing with enabling revision support :)

Re-roll.

Link referenced issue.

Since this is not going to get merged, perhaps there should be a Drush command that does what these update hooks do. That way, most users are not effected. Those users who are having a problem with migration can run the command to try to fix it.

This is a note to myself. I updated my attribution fields.