1. Dial back the content_moderation schema version using: drush ev "drupal_set_installed_schema_version('content_moderation', 8000);"
  2. Apply the patch.
  3. Run drush updb.
CommentFileSizeAuthor
#9 2890189-832-833-upgrade-9.patch665.09 KBsam152
#9 interdiff.txt509 bytessam152
#4 interdiff.txt3.71 KBsam152
#4 2890189-832-833-upgrade-5.patch664.6 KBsam152
#3 2890189-832-833-upgrade-3.patch663.48 KBsam152

Comments

Sam152 created an issue. See original summary.

sam152’s picture

sam152 commented
Parent issue: » #2890187: [META] Unofficial content_moderation upgrade path
sam152’s picture

sam152 commented
StatusFileSize
new 2890189-832-833-upgrade-3.patch663.48 KB

Initial start of this with @jhedstrom approach from #2846618: Unofficial content_moderation 8.2.x to 8.3.0 upgrade path.

I couldn't get this test to pass. Just running this approach I get:

After all updates ran, entity schema is up to date.

I tracked this down to the indexes the issue introduced being missing.

If I try and use the update manager to update the entity definition, it says the field needs to be updated, despite running updateFieldStorageDefinition.

sam152’s picture

sam152 commented
Status: Active » Needs review
StatusFileSize
new 2890189-832-833-upgrade-5.patch664.6 KB
new interdiff.txt3.71 KB

I've had another look at this. I managed to come up with quite a hideous patch (with code pulled from a few sources) which doesn't seem to leave any pending entity updates, ensures the new indexes exist and changes the column size.

My guess is the entity update manager will never change the column size of a table that already has data, so it has to be done manually. The ->applyUpdates is needed to create the indexes, which is has no problem doing (could possibly be made more specific, only update our entity type).

Schema before update:

mysql> describe content_moderation_state_field_data; describe content_moderation_state_field_revision; show indexes from content_moderation_state_field_data; show indexes from content_moderation_state_field_revision;
+----------------------------+------------------+------+-----+---------+-------+
| Field                      | Type             | Null | Key | Default | Extra |
+----------------------------+------------------+------+-----+---------+-------+
| id                         | int(10) unsigned | NO   | PRI | NULL    |       |
| revision_id                | int(10) unsigned | NO   | MUL | NULL    |       |
| langcode                   | varchar(12)      | NO   | PRI | NULL    |       |
| uid                        | int(10) unsigned | NO   | MUL | NULL    |       |
| workflow                   | varchar(255)     | YES  | MUL | NULL    |       |
| moderation_state           | varchar(255)     | YES  |     | NULL    |       |
| content_entity_type_id     | varchar(255)     | YES  | MUL | NULL    |       |
| content_entity_id          | int(11)          | YES  |     | NULL    |       |
| content_entity_revision_id | int(11)          | YES  |     | NULL    |       |
| default_langcode           | tinyint(4)       | NO   |     | NULL    |       |
+----------------------------+------------------+------+-----+---------+-------+
10 rows in set (0.00 sec)

+----------------------------+------------------+------+-----+---------+-------+
| Field                      | Type             | Null | Key | Default | Extra |
+----------------------------+------------------+------+-----+---------+-------+
| id                         | int(10) unsigned | NO   | MUL | NULL    |       |
| revision_id                | int(10) unsigned | NO   | PRI | NULL    |       |
| langcode                   | varchar(12)      | NO   | PRI | NULL    |       |
| uid                        | int(10) unsigned | NO   | MUL | NULL    |       |
| workflow                   | varchar(255)     | YES  | MUL | NULL    |       |
| moderation_state           | varchar(255)     | YES  |     | NULL    |       |
| content_entity_type_id     | varchar(255)     | YES  |     | NULL    |       |
| content_entity_id          | int(11)          | YES  |     | NULL    |       |
| content_entity_revision_id | int(11)          | YES  |     | NULL    |       |
| default_langcode           | tinyint(4)       | NO   |     | NULL    |       |
+----------------------------+------------------+------+-----+---------+-------+
10 rows in set (0.00 sec)

+-------------------------------------+------------+----------------------------------------------------------+--------------+----------------------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
| Table                               | Non_unique | Key_name                                                 | Seq_in_index | Column_name                | Collation | Cardinality | Sub_part | Packed | Null | Index_type | Comment | Index_comment |
+-------------------------------------+------------+----------------------------------------------------------+--------------+----------------------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
| content_moderation_state_field_data |          0 | PRIMARY                                                  |            1 | id                         | A         |           2 |     NULL | NULL   |      | BTREE      |         |               |
| content_moderation_state_field_data |          0 | PRIMARY                                                  |            2 | langcode                   | A         |           2 |     NULL | NULL   |      | BTREE      |         |               |
| content_moderation_state_field_data |          1 | content_moderation_state__id__default_langcode__langcode |            1 | id                         | A         |           2 |     NULL | NULL   |      | BTREE      |         |               |
| content_moderation_state_field_data |          1 | content_moderation_state__id__default_langcode__langcode |            2 | default_langcode           | A         |           2 |     NULL | NULL   |      | BTREE      |         |               |
| content_moderation_state_field_data |          1 | content_moderation_state__id__default_langcode__langcode |            3 | langcode                   | A         |           2 |     NULL | NULL   |      | BTREE      |         |               |
| content_moderation_state_field_data |          1 | content_moderation_state__revision_id                    |            1 | revision_id                | A         |           2 |     NULL | NULL   |      | BTREE      |         |               |
| content_moderation_state_field_data |          1 | content_moderation_state_field__uid__target_id           |            1 | uid                        | A         |           1 |     NULL | NULL   |      | BTREE      |         |               |
| content_moderation_state_field_data |          1 | content_moderation_state__09628d8dbc                     |            1 | workflow                   | A         |           1 |     NULL | NULL   | YES  | BTREE      |         |               |
| content_moderation_state_field_data |          1 | content_moderation_state__lookup                         |            1 | content_entity_type_id     | A         |           1 |      191 | NULL   | YES  | BTREE      |         |               |
| content_moderation_state_field_data |          1 | content_moderation_state__lookup                         |            2 | content_entity_id          | A         |           2 |     NULL | NULL   | YES  | BTREE      |         |               |
| content_moderation_state_field_data |          1 | content_moderation_state__lookup                         |            3 | content_entity_revision_id | A         |           2 |     NULL | NULL   | YES  | BTREE      |         |               |
+-------------------------------------+------------+----------------------------------------------------------+--------------+----------------------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
11 rows in set (0.00 sec)

+-----------------------------------------+------------+----------------------------------------------------------+--------------+------------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
| Table                                   | Non_unique | Key_name                                                 | Seq_in_index | Column_name      | Collation | Cardinality | Sub_part | Packed | Null | Index_type | Comment | Index_comment |
+-----------------------------------------+------------+----------------------------------------------------------+--------------+------------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
| content_moderation_state_field_revision |          0 | PRIMARY                                                  |            1 | revision_id      | A         |           4 |     NULL | NULL   |      | BTREE      |         |               |
| content_moderation_state_field_revision |          0 | PRIMARY                                                  |            2 | langcode         | A         |           4 |     NULL | NULL   |      | BTREE      |         |               |
| content_moderation_state_field_revision |          1 | content_moderation_state__id__default_langcode__langcode |            1 | id               | A         |           2 |     NULL | NULL   |      | BTREE      |         |               |
| content_moderation_state_field_revision |          1 | content_moderation_state__id__default_langcode__langcode |            2 | default_langcode | A         |           2 |     NULL | NULL   |      | BTREE      |         |               |
| content_moderation_state_field_revision |          1 | content_moderation_state__id__default_langcode__langcode |            3 | langcode         | A         |           2 |     NULL | NULL   |      | BTREE      |         |               |
| content_moderation_state_field_revision |          1 | content_moderation_state_field__uid__target_id           |            1 | uid              | A         |           1 |     NULL | NULL   |      | BTREE      |         |               |
| content_moderation_state_field_revision |          1 | content_moderation_state__09628d8dbc                     |            1 | workflow         | A         |           1 |     NULL | NULL   | YES  | BTREE      |         |               |
+-----------------------------------------+------------+----------------------------------------------------------+--------------+------------------+-----------+-------------+----------+--------+------+------------+---------+---------------+

Schema after:

mysql> describe content_moderation_state_field_data; describe content_moderation_state_field_revision; show indexes from content_moderation_state_field_data; show indexes from content_moderation_state_field_revision;
+----------------------------+------------------+------+-----+---------+-------+
| Field                      | Type             | Null | Key | Default | Extra |
+----------------------------+------------------+------+-----+---------+-------+
| id                         | int(10) unsigned | NO   | PRI | NULL    |       |
| revision_id                | int(10) unsigned | NO   | MUL | NULL    |       |
| langcode                   | varchar(12)      | NO   | PRI | NULL    |       |
| uid                        | int(10) unsigned | NO   | MUL | NULL    |       |
| workflow                   | varchar(255)     | YES  | MUL | NULL    |       |
| moderation_state           | varchar(255)     | YES  |     | NULL    |       |
| content_entity_type_id     | varchar(32)      | YES  | MUL | NULL    |       |
| content_entity_id          | int(11)          | YES  |     | NULL    |       |
| content_entity_revision_id | int(11)          | YES  |     | NULL    |       |
| default_langcode           | tinyint(4)       | NO   |     | NULL    |       |
+----------------------------+------------------+------+-----+---------+-------+
10 rows in set (0.00 sec)

+----------------------------+------------------+------+-----+---------+-------+
| Field                      | Type             | Null | Key | Default | Extra |
+----------------------------+------------------+------+-----+---------+-------+
| id                         | int(10) unsigned | NO   | MUL | NULL    |       |
| revision_id                | int(10) unsigned | NO   | PRI | NULL    |       |
| langcode                   | varchar(12)      | NO   | PRI | NULL    |       |
| uid                        | int(10) unsigned | NO   | MUL | NULL    |       |
| workflow                   | varchar(255)     | YES  | MUL | NULL    |       |
| moderation_state           | varchar(255)     | YES  |     | NULL    |       |
| content_entity_type_id     | varchar(32)      | YES  | MUL | NULL    |       |
| content_entity_id          | int(11)          | YES  |     | NULL    |       |
| content_entity_revision_id | int(11)          | YES  |     | NULL    |       |
| default_langcode           | tinyint(4)       | NO   |     | NULL    |       |
+----------------------------+------------------+------+-----+---------+-------+
10 rows in set (0.00 sec)

+-------------------------------------+------------+----------------------------------------------------------+--------------+----------------------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
| Table                               | Non_unique | Key_name                                                 | Seq_in_index | Column_name                | Collation | Cardinality | Sub_part | Packed | Null | Index_type | Comment | Index_comment |
+-------------------------------------+------------+----------------------------------------------------------+--------------+----------------------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
| content_moderation_state_field_data |          0 | PRIMARY                                                  |            1 | id                         | A         |           2 |     NULL | NULL   |      | BTREE      |         |               |
| content_moderation_state_field_data |          0 | PRIMARY                                                  |            2 | langcode                   | A         |           2 |     NULL | NULL   |      | BTREE      |         |               |
| content_moderation_state_field_data |          0 | content_moderation_state__lookup                         |            1 | content_entity_type_id     | A         |           1 |     NULL | NULL   | YES  | BTREE      |         |               |
| content_moderation_state_field_data |          0 | content_moderation_state__lookup                         |            2 | content_entity_id          | A         |           2 |     NULL | NULL   | YES  | BTREE      |         |               |
| content_moderation_state_field_data |          0 | content_moderation_state__lookup                         |            3 | content_entity_revision_id | A         |           2 |     NULL | NULL   | YES  | BTREE      |         |               |
| content_moderation_state_field_data |          0 | content_moderation_state__lookup                         |            4 | workflow                   | A         |           2 |     NULL | NULL   | YES  | BTREE      |         |               |
| content_moderation_state_field_data |          0 | content_moderation_state__lookup                         |            5 | langcode                   | A         |           2 |     NULL | NULL   |      | BTREE      |         |               |
| content_moderation_state_field_data |          1 | content_moderation_state_field__uid__target_id           |            1 | uid                        | A         |           1 |     NULL | NULL   |      | BTREE      |         |               |
| content_moderation_state_field_data |          1 | content_moderation_state__09628d8dbc                     |            1 | workflow                   | A         |           1 |     NULL | NULL   | YES  | BTREE      |         |               |
| content_moderation_state_field_data |          1 | content_moderation_state__id__default_langcode__langcode |            1 | id                         | A         |           2 |     NULL | NULL   |      | BTREE      |         |               |
| content_moderation_state_field_data |          1 | content_moderation_state__id__default_langcode__langcode |            2 | default_langcode           | A         |           2 |     NULL | NULL   |      | BTREE      |         |               |
| content_moderation_state_field_data |          1 | content_moderation_state__id__default_langcode__langcode |            3 | langcode                   | A         |           2 |     NULL | NULL   |      | BTREE      |         |               |
| content_moderation_state_field_data |          1 | content_moderation_state__revision_id                    |            1 | revision_id                | A         |           2 |     NULL | NULL   |      | BTREE      |         |               |
+-------------------------------------+------------+----------------------------------------------------------+--------------+----------------------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
13 rows in set (0.00 sec)

+-----------------------------------------+------------+----------------------------------------------------------+--------------+----------------------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
| Table                                   | Non_unique | Key_name                                                 | Seq_in_index | Column_name                | Collation | Cardinality | Sub_part | Packed | Null | Index_type | Comment | Index_comment |
+-----------------------------------------+------------+----------------------------------------------------------+--------------+----------------------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
| content_moderation_state_field_revision |          0 | PRIMARY                                                  |            1 | revision_id                | A         |           4 |     NULL | NULL   |      | BTREE      |         |               |
| content_moderation_state_field_revision |          0 | PRIMARY                                                  |            2 | langcode                   | A         |           4 |     NULL | NULL   |      | BTREE      |         |               |
| content_moderation_state_field_revision |          0 | content_moderation_state__lookup                         |            1 | content_entity_type_id     | A         |           1 |     NULL | NULL   | YES  | BTREE      |         |               |
| content_moderation_state_field_revision |          0 | content_moderation_state__lookup                         |            2 | content_entity_id          | A         |           2 |     NULL | NULL   | YES  | BTREE      |         |               |
| content_moderation_state_field_revision |          0 | content_moderation_state__lookup                         |            3 | content_entity_revision_id | A         |           4 |     NULL | NULL   | YES  | BTREE      |         |               |
| content_moderation_state_field_revision |          0 | content_moderation_state__lookup                         |            4 | workflow                   | A         |           4 |     NULL | NULL   | YES  | BTREE      |         |               |
| content_moderation_state_field_revision |          0 | content_moderation_state__lookup                         |            5 | langcode                   | A         |           4 |     NULL | NULL   |      | BTREE      |         |               |
| content_moderation_state_field_revision |          1 | content_moderation_state_field__uid__target_id           |            1 | uid                        | A         |           1 |     NULL | NULL   |      | BTREE      |         |               |
| content_moderation_state_field_revision |          1 | content_moderation_state__09628d8dbc                     |            1 | workflow                   | A         |           1 |     NULL | NULL   | YES  | BTREE      |         |               |
| content_moderation_state_field_revision |          1 | content_moderation_state__id__default_langcode__langcode |            1 | id                         | A         |           2 |     NULL | NULL   |      | BTREE      |         |               |
| content_moderation_state_field_revision |          1 | content_moderation_state__id__default_langcode__langcode |            2 | default_langcode           | A         |           2 |     NULL | NULL   |      | BTREE      |         |               |
| content_moderation_state_field_revision |          1 | content_moderation_state__id__default_langcode__langcode |            3 | langcode                   | A         |           2 |     NULL | NULL   |      | BTREE      |         |               |
+-----------------------------------------+------------+----------------------------------------------------------+--------------+----------------------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
12 rows in set (0.00 sec)

The last submitted patch, 3: 2890189-832-833-upgrade-3.patch, failed testing. View results
- codesniffer_fixes.patch Interdiff of automated coding standards fixes only.

mstef’s picture

mstef commented

Testing with 8.3.4. Patch applies cleanly and everything seems to be working. I'll continue testing.

Thank you very much for your work.

fgm’s picture

fgm
Primary language French
Location Paris, France
commented

With this patch applied:

* drush updb --entity-updates applies cleanly
* but then drush entup throws: 

The following updates are pending:

content_moderation_state entity type :
  Le champ Identifiant (ID) du type d'entité de contenu nécessite d'être mis à jour.
Do you wish to run all pending updates? (y/n): y
Drupal\Core\Entity\Exception\FieldStorageDefinitionUpdateForbiddenException: The SQL storage cannot change the schema for an existing field (content_entity_type_id in                    [error]
content_moderation_state entity) with data. in Drupal\Core\Entity\Sql\SqlContentEntityStorageSchema->updateSharedTableSchema() (line 1403 of
/somepath/web/core/lib/Drupal/Core/Entity/Sql/SqlContentEntityStorageSchema.php).
Failed: Drupal\Core\Entity\Exception\FieldStorageDefinitionUpdateForbiddenException: !message in Drupal\Core\Entity\Sql\SqlContentEntityStorageSchema->updateSharedTableSchema() (line [error]
1403 of /somepath/web/core/lib/Drupal/Core/Entity/Sql/SqlContentEntityStorageSchema.php).
Cache rebuild complete.                                                                                                                                                                   [ok]
Finished performing updates.
rop’s picture

rop commented

After applying the patch
drush updb --entity-updates
still throws:

Drupal\Core\Entity\EntityStorageException: Exception thrown while performing[error]
a schema update. SQLSTATE[42000]: Syntax error or access violation: 1071
Specified key was too long; max key length is 767 bytes: ALTER TABLE
{content_moderation_state_field_data} ADD UNIQUE KEY
`content_moderation_state__lookup` (`content_entity_type_id`,
`content_entity_id`, `content_entity_revision_id`, `workflow`, `langcode`);
Array
(
)
 in Drupal\Core\Entity\Sql\SqlContentEntityStorage->wrapSchemaException()
(line 1485 of
/var/www/*/public_html/core/lib/Drupal/Core/Entity/Sql/SqlContentEntityStorage.php).
Failed: Drupal\Core\Entity\EntityStorageException: !message in              [error]
Drupal\Core\Entity\Sql\SqlContentEntityStorage->wrapSchemaException()
(line 1485 of
/var/www/*/public_html/core/lib/Drupal/Core/Entity/Sql/SqlContentEntityStorage.php).
sam152’s picture

sam152 commented
Status: Needs review » Needs work
StatusFileSize
new interdiff.txt509 bytes
new 2890189-832-833-upgrade-9.patch665.09 KB

For some reason composer patches wont apply a patch correctly if it only contains new files. Adding an empty newline to the module file seems to fix things. I was getting the patch files in ./core/core/modules instead of the correct ./core/modules path.

sam152’s picture

sam152 commented
Status: Needs work » Needs review
fgm’s picture

fgm
Primary language French
Location Paris, France
commented

The upgrade path passes like a charm now, and the data look as expected. Thanks for the impressive work.

The most worrying part for me is the fact that this introduces content_moderation_update_800[1-3]() implementations, which might cause conflict when the module leave experimental status and official versions of these same functions start appearing. But so for, none exist in 8.3.3/8.3.4/8.3.5.

sam152’s picture

sam152 commented

You're welcome!

The instructions in #2846618: Unofficial content_moderation 8.2.x to 8.3.0 upgrade path say to dial back your schema version by using drush ev "drupal_set_installed_schema_version('content_moderation', 8000);".

It'd be great if someone had time to organise the instructions into the meta and make sure they were referenced from each issue summary.

damienmckenna’s picture

damienmckenna
Location TN, USA
commented

Just to voice my concern - not having an upgrade path between minor versions is one thing, but making architectural changes that result in broken configuration and/or data without providing the necessary update scripts is a pretty user-hostile decision. Huge thanks to everyone working on this update script.

rop’s picture

rop commented

I've tried the patch again, but it does not seem to fix the problem, as mentioned before.
I could be doing something wrong.
My situation is: I've started using content_moderation in 8.2.6 , I've then updated core successfully to core version 8.3.2 using the relevant patch from this (parent)issue. That worked fined. After this I've update core to 8.3.4 but despite apllying patch#9 I keep getting the errors about too long keys. (I've also tried 8.3.3)
The patch applies fine, it asks to overwrite the existing files, yet drush entity-updates fails.

Anyone an idea why it fails while others are succesfull?

fgm’s picture

fgm
Primary language French
Location Paris, France
commented

@roproprop : You need to run drush updb first. After that there remains nothing to update for drush entup.

gábor hojtsy’s picture

gábor hojtsy
he/him
Primary language Hungarian
Location Hungary
commented

Re @DamienMcKenna:

Just to voice my concern - not having an upgrade path between minor versions is one thing, but making architectural changes that result in broken configuration and/or data without providing the necessary update scripts is a pretty user-hostile decision. Huge thanks to everyone working on this update script.

Well, the goal with experimental modules is to move faster, which by definition means breaking things. Its like an 8.x-1.x-dev of a contrib module, anything may change. If developers would write an upgrade path then they would need to provide an upgrade path test (data dumps, etc), so moving fast is not really feasible anymore. What distinguishes experimental modules from non-experimental modules then? The only difference between a 8.x-1.x-dev contrib module and an experimental module in core now is where it is.

(Whether this is a good idea is a different question, but if the documentation or the warnings when enabling or having experimental modules enabled lead you to believe otherwise, then we need to fix those for sure).

xjm’s picture

xjm
she/her
Primary language English
commented
Related issues: +#2649768: [meta] No definition of "Experimental" & not nearly enough warning
xjm’s picture

xjm
she/her
Primary language English
commented
Related issues: +#2699539: Remove experimental modules from core (or treat them like first class modules)
rop’s picture

rop commented

@fgm I did run drush updb first. After that status still says entities need to be updated.

I've now tried again doing only drush updb and I again get these errors:

The following updates are pending:

content_moderation module : 
  8002 -   Add new storage indexes, by applying all updates. 
  8003 -   Manually change the storage schema to have the new field length. 

Do you wish to run all pending updates? (y/n): y
Exception thrown while performing a schema update. SQLSTATE[42000]: Syntax error or access violation: 1071 Specified key was too long; max key length is 767 bytes: ALTER TABLE     [error]
{content_moderation_state_field_data} ADD UNIQUE KEY `content_moderation_state__lookup` (`content_entity_type_id`, `content_entity_id`, `content_entity_revision_id`, `workflow`,
`langcode`); Array
(
)

Performing content_moderation_update_8002                                                                                                                                           [ok]
Failed: Exception thrown while performing a schema update. SQLSTATE[42000]: Syntax error or access violation: 1071 Specified key was too long; max key length is 767 bytes: ALTER   [error]
TABLE {content_moderation_state_field_data} ADD UNIQUE KEY `content_moderation_state__lookup` (`content_entity_type_id`, `content_entity_id`, `content_entity_revision_id`,
`workflow`, `langcode`); Array
(
)
sam152’s picture

sam152 commented
Issue summary: View changes
sam152’s picture

sam152 commented

Your schema version is at 8001 instead of 8000, you need to make sure 8001 runs by rolling it back. See the updated issue summary.

rop’s picture

rop commented

@Sam152 Thank you! that did the trick..all good now.
Sorry for the confusion.
Thanks for your work on this.

Version: 8.3.x-dev » 8.4.x-dev

Drupal 8.3.6 was released on August 2, 2017 and is the final full bugfix release for the Drupal 8.3.x series. Drupal 8.3.x will not receive any further development aside from critical and security fixes. Sites should prepare to update to 8.4.0 on October 4, 2017. (Drupal 8.4.0-alpha1 is available for testing.)

Bug reports should be targeted against the 8.4.x-dev branch from now on, and new development or disruptive changes should be targeted against the 8.5.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

sam152’s picture

sam152 commented
Status: Needs review » Closed (works as designed)