It seems that translatable content entities need to implement two tables: base_table and data_table. Group and GroupContent currently only implement the former. In order to have translatability function properly, it turns out we need to add this (rather poorly documented) table to the annotations.

However, as https://www.drupal.org/node/2554097 points out, this cannot be done automatically. We'll need to write proper update hooks to fix this.

Major because any new module install will basically get a faulty base table.

Support from Acquia helps fund testing for Drupal Acquia logo

Comments

kristiaanvandeneynde created an issue. See original summary.

kristiaanvandeneynde’s picture

Issue summary: View changes

We should also have a battle plan for people with existing views.

See https://www.drupal.org/node/2433153

joachim’s picture

> this (rather poorly documented) table to the annotations.

Is there an issue to improve the documentation of this? (I did notice that this module's entities didn't have a 2nd table, but as I don't fully understand it myself, I didn't say anything. Sorry!)

kristiaanvandeneynde’s picture

Just created one: #2715089: Document the type of tables an entity type can define

And it's not your fault :) I copied a lot of logic from Node and other content entity types, building it up from scratch. When I got to a point where it would install correctly, I thought I had all the necessary bits in place. Seeing as the module worked fine without a data_table I figured it was optional.

Guess it's not..

kristiaanvandeneynde’s picture

Here's an almost complete patch so I can ask core devs to see if this is the right approach. All it needs is a migration of the old data into the new tables which is why I'm trying to retrieve all of the column names. So I can build queries where I select the old data into the new columns.

Few notes:

  • The whole issue was caused because I did not set the 'translatable' annotation key to TRUE
  • I had to specify the data_table annotation keys even though Drupal can calculate default names, see: #2328565: Omitting the "base_table" or "data_table" for entity types is broken
  • I have spent hours going through the entity update process and think this is the safest way to move from a non-translatable entity type to a translatable one. It would be really cool if there were better examples of this somewhere.
ronaldtebrake’s picture

Nicely spotted and thanks for this great effort!

It seems to work well for me. But I will wait for one of the core developers to set it to RTBC I guess :).

A couple of questions:

1. Am I correct in thinking on a new install, with the 'translatable' annotation and the data table key in place the update_hook / migration is 'redundant' since there is nothing to migrate and the tables should be installed correctly?

If so maybe we can alter the documentation of the update hook a bit to make that more clear.

2. Regarding the examples, I found one over here https://www.drupal.org/node/2641828#comment-10711058 especially the interdiff could be useful: https://www.drupal.org/files/issues/choice-entity-type-2641828-5-interdi...

dawehner’s picture

  1. +++ b/group.install
    @@ -47,3 +49,77 @@ function group_update_8001(&$sandbox) {
    +      // At this point the database structure should match what is defined in
    +      // code. However, Drupal still thinks we are running the old schema
    +      // version because it cached the old definitions in the key/value storage.
    +      //
    +      // Inform Drupal of the new schema version.
    +      $schema_repository->setLastInstalledDefinition($entity_type);
    

    One problem with that is that this potentially ignores other update functions which want to deal with the same schema. The alternative would be to just set the new entity tables + translatable flag

  2. +++ b/group.install
    @@ -47,3 +49,77 @@ function group_update_8001(&$sandbox) {
    +        // @todo Looks like table mapping can't read $temp_table.
    +        $temp_cols = $storage->getTableMapping()->getAllColumns($temp_table);
    +        $base_cols = $storage->getTableMapping()->getAllColumns($base_table);
    +        $data_cols = $storage->getTableMapping()->getAllColumns($data_table);
    +
    

    You need to pass the new $entity_type to the getTableMapping method, so you get the latest code one, not the last installed version.

kristiaanvandeneynde’s picture

Just talked a bit more to dawehner on IRC and he suggested I load the old schema, adjust the necessary keys and then save that one. That way we don't undo any schema alterations someone else might have done during their update hook.

Furthermore, we could then have a variable containing the old schema and one with the new, pass those into $storage->getTableMapping() and be able to read the old column names like that.

All that is left then is to map the old columns' data to the new ones' and remove the temporary table.

kristiaanvandeneynde’s picture

Right, so this patch takes care of the data migration properly, but it made me rethink both points made by dawehner. I'll see if I can ask him to weigh in on them once more.

First of all, we are rebuilding the entity type's tables from scratch and from code, so I don't see why we need to load the previous schema definition and adjust it instead of simply overwriting it. After all, any module that wanted to adjust the schema also had to adjust the entity type's definition and we are using the latest copy of that in our update hook.

Secondly, I couldn't for the life of me find a way to retrieve the old storage definition to pass into getTableMapping(). Instead I tried to manually load the storage handler for the old entity type definition and the new entity type definition, but it turns out getTableMapping() then tries to load the storage definition from the entity type in code. So that didn't work and I had to lift the retrieval for the old columns to the top of the update hook. It feels like I'm doing something wrong here, so I'd love to find a way to retrieve the $storage_definition to pass onto getTableMapping().

Do not use this patch yet please, it works great but I want to have someone review it first.

dawehner’s picture

First of all, we are rebuilding the entity type's tables from scratch and from code, so I don't see why we need to load the previous schema definition and adjust it instead of simply overwriting it. After all, any module that wanted to adjust the schema also had to adjust the entity type's definition and we are using the latest copy of that in our update hook.

Well, let's assume a user updates from moduleA_8000 to moduleA_8002 and moduleB_8001 to moduleB__8009, while hitting update.php. Update.php will not guarantee you the order of those update functions, so other update functions might come before or after. On the other hand the code version doesn't change during that process, so the definition in code, is up to date already. update.php is tricy.

Secondly, I couldn't for the life of me find a way to retrieve the old storage definition to pass into getTableMapping().

I think it is something like:

$repo = \Drupal::service('entity.last_installed_schema.repository');
$fieldstorage_definitions = $repo->getLastInstalledFieldStorageDefinitions($entity_type_id);
$table_mapping = ... ->getTableMapping($fieldstorage_definitions);
kristiaanvandeneynde’s picture

Right, so this works.

All it needs now is exceptions for edge cases where the storage doesn't implement a certain interface.

Also note we are not deleting the migration tables so people can still access their old data should anything go wrong during the migration. They should have back-ups, but this is a nice gesture towards those who can't easily access or restore those.

alan.upstone’s picture

FileSize
149.18 KB

Hello Kristiaan

Applied the patch. I got the attached error when running update.php, and my views are all broken.

I get this when clicking on your flurb then selecting a group:

http://dev-mobilized.pantheonsite.io/group/20

The website encountered an unexpected error. Please try again later.

Drupal\Core\Database\DatabaseExceptionWrapper: Exception in Group actions[group_actions]: SQLSTATE[42S22]: Column not found: 1054 Unknown column 'unknown' in 'order clause': SELECT 'group_actions:block_2' AS view_name, MIN(group_content.id) AS group_content_id FROM {group_content} group_content GROUP BY view_name ORDER BY unknown ASC; Array ( ) in Drupal\views\Plugin\views\query\Sql->execute() (line 1452 of core/modules/views/src/Plugin/views/query/Sql.php).
Drupal\views\ViewExecutable->execute(NULL) (Line: 1412)
Drupal\views\ViewExecutable->render() (Line: 117)
Drupal\views\Plugin\views\display\Block->execute() (Line: 1587)
Drupal\views\ViewExecutable->executeDisplay('block_2', Array) (Line: 78)
Drupal\views\Element\View::preRenderViewElement(Array) (Line: 42)
Drupal\views\Plugin\Block\ViewsBlock->build() (Line: 202)
Drupal\block\BlockViewBuilder::preRender(Array)
call_user_func('Drupal\block\BlockViewBuilder::preRender', Array) (Line: 381)
Drupal\Core\Render\Renderer->doRender(Array) (Line: 448)
Drupal\Core\Render\Renderer->doRender(Array, ) (Line: 195)
Drupal\Core\Render\Renderer->render(Array) (Line: 462)
Drupal\Core\Template\TwigExtension->escapeFilter(Object, Array, 'html', NULL, 1) (Line: 104)
__TwigTemplate_2617f4b4428068cdd24e54b3cdcc7ae84d6ae2f3e1366ac6b01a2537a9c32694->doDisplay(Array, Array) (Line: 381)
Twig_Template->displayWithErrorHandling(Array, Array) (Line: 355)
Twig_Template->display(Array) (Line: 366)
Twig_Template->render(Array) (Line: 64)
twig_render_template('core/themes/bartik/templates/page.html.twig', Array) (Line: 384)
Drupal\Core\Theme\ThemeManager->render('page', Array) (Line: 435)
Drupal\Core\Render\Renderer->doRender(Array, ) (Line: 195)
Drupal\Core\Render\Renderer->render(Array) (Line: 462)
Drupal\Core\Template\TwigExtension->escapeFilter(Object, Array, 'html', NULL, 1) (Line: 88)
__TwigTemplate_039606928b737abc794de14b5f41a97b59176f66c9dcc0bb3b6df3a09ecc43a2->doDisplay(Array, Array) (Line: 381)
Twig_Template->displayWithErrorHandling(Array, Array) (Line: 355)
Twig_Template->display(Array) (Line: 366)
Twig_Template->render(Array) (Line: 64)
twig_render_template('core/themes/classy/templates/layout/html.html.twig', Array) (Line: 384)
Drupal\Core\Theme\ThemeManager->render('html', Array) (Line: 435)
Drupal\Core\Render\Renderer->doRender(Array, ) (Line: 195)
Drupal\Core\Render\Renderer->render(Array) (Line: 147)
Drupal\Core\Render\MainContent\HtmlRenderer->Drupal\Core\Render\MainContent\{closure}() (Line: 574)
Drupal\Core\Render\Renderer->executeInRenderContext(Object, Object) (Line: 148)
Drupal\Core\Render\MainContent\HtmlRenderer->renderResponse(Array, Object, Object) (Line: 90)
Drupal\Core\EventSubscriber\MainContentViewSubscriber->onViewRenderArray(Object, 'kernel.view', Object) (Line: 111)
Drupal\Component\EventDispatcher\ContainerAwareEventDispatcher->dispatch('kernel.view', Object) (Line: 144)
Symfony\Component\HttpKernel\HttpKernel->handleRaw(Object, 1) (Line: 62)
Symfony\Component\HttpKernel\HttpKernel->handle(Object, 1, 1) (Line: 57)
Drupal\Core\StackMiddleware\Session->handle(Object, 1, 1) (Line: 48)
Drupal\Core\StackMiddleware\KernelPreHandle->handle(Object, 1, 1) (Line: 98)
Drupal\page_cache\StackMiddleware\PageCache->pass(Object, 1, 1) (Line: 77)
Drupal\page_cache\StackMiddleware\PageCache->handle(Object, 1, 1) (Line: 47)
Drupal\Core\StackMiddleware\ReverseProxyMiddleware->handle(Object, 1, 1) (Line: 50)
Drupal\Core\StackMiddleware\NegotiationMiddleware->handle(Object, 1, 1) (Line: 23)
Stack\StackedHttpKernel->handle(Object, 1, 1) (Line: 628)
Drupal\Core\DrupalKernel->handle(Object) (Line: 19)

kristiaanvandeneynde’s picture

Hey Alan,

I actually expected Views to break because the underlying table structure has changed. You should be able to recover them by exporting their config (single export), changing 'groups' as the base table to 'groups_field_data' and then re-importing that.

Could you try that and let me know?

alan.upstone’s picture

Even your modules's own view - group/x - is expected to crash?

kristiaanvandeneynde’s picture

It's not crashing for me, you probably have a block with a view on that page?

kristiaanvandeneynde’s picture

Right, so I just broke my site on purpose to reproduce this error and fixed it like this:

  1. Apply the patch above
  2. Run update.php (or drush updb)
  3. Go to /admin/config/development/configuration/single/export and copy over all of the affected views (one txt file per view for instance)
  4. Go to /admin/config/development/configuration/single/import and paste in one view at a time
  5. Change all occurrences of groups with groups_field_data where the key states base_table: or simply table:
  6. Do the same for group_content and group_content_field_data
  7. Under the "Advanced" fieldset, enter the view's machine name (ID)
  8. Click Import, confirm the import and your view should be fixed
  9. Repeat for all other views using Group or GroupContent

Again: pay special attention to the keys base_table: and table:. Changing entity_type: group_content will break your view.

I'll commit the patch now so other issues are no longer blocked and will link to this issue in the next release's notes.

kristiaanvandeneynde’s picture

Status: Needs review » Fixed
alan.upstone’s picture

Kristiaan was right re #14. There was a block in the sidebar using a view that was crashing; not the main Group view. Changing the appearance setting to another theme temporarily turns off such blocks and allows you to do the stuff in #16. Then you can reinstate your blocks by reverting the theme.

Thanks for taking the time to do #16.

alan.upstone’s picture

I have had all sorts of problems since doing this.

My status report says

Entity/field definitions

Mismatched entity and/or field definitions
The following changes were detected in the entity type and field definitions.
Group entity
The Group entity entity type needs to be updated.
The Default translation field needs to be uninstalled.
Group content entity
The Group content entity entity type needs to be updated.
The Default translation field needs to be uninstalled.

Running drush entity-updates gets...

The following updates are pending:

group entity type :
The Group entity entity type needs to be updated.
The Default translation field needs to be uninstalled.
group_content entity type :
The Group content entity entity type needs to be updated.
The Default translation field needs to be uninstalled.
Do you wish to run all pending updates? (y/n): y

...then the error:

Drupal\Core\Entity\EntityStorageException: The SQL storage cannot change the schema for an existing entity type (group) with data. in
Drupal\Core\Entity\Sql\SqlContentEntityStorageSchema->onEntityTypeUpdate()
(line 303 of /srv/bindings/96df09bfe929475f9e54ce28f76f5177/code/core/lib/Drupal/Core/Entity/Sql/SqlContentEntityStorageSchema.php).
Failed: Drupal\Core\Entity\EntityStorageException: !message in Drupal\Core\Entity\Sql\SqlContentEntityStorageSchema->onEntityTypeUpdate()
(line 303 of /srv/bindings/96df09bfe929475f9e54ce28f76f5177/code/core/lib/Drupal/Core/Entity/Sql/SqlContentEntityStorageSchema.php).

kristiaanvandeneynde’s picture

You should not have to run drush entity-updates if you just ran update.php. The update hook should have cleanly migrated your existing data structure into the new data structure, after which the status report should no longer list mismatched entity definitions.

Do you still have a DB backup from before the update?

alan.upstone’s picture

Hello Kristiaan
drush entity-updates was the last report after trying the usual things like update.php with no luck.

I restored from a backup. I ran update.php again and deleted every view named in the error message (see image in comment #12) as causing a taxonomy_index_tid error - until all errors were gone and I could get update.php to complete. Then I recreated the deleted views.

kristiaanvandeneynde’s picture

Hi Alan,

When it comes to updates like these, you should steer clear from drush entity-updates. Data migrating entity updates should always happen via an update hook. I'm glad you got it to work in the end, but it left me wondering a bit about why it went wrong.

Anyway, I've updated the alpha4 release notes to mention the specific steps you need to take to upgrade and linked people to this issue if they had any views that needed updating. On the bright side: with this out of the way there should be no more massive data changes like this one.

Sorry to all who were inconvenienced by this patch, even though it was absolutely necessary to get it in as soon as possible. One roadblock less towards beta!

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.

The last submitted patch, 9: group-2715011-9-WIP.patch, failed testing.

Status: Closed (fixed) » Needs work

The last submitted patch, 11: group-2715011-11-WIP.patch, failed testing.

kristiaanvandeneynde’s picture

Status: Needs work » Closed (fixed)

Bad testbot! :)