Problem/Motivation

SqlContentEntityStorage::countFieldData() is called when a field storage definition gets updated to detect whether data exists for this field already. It does this by either querying the base table or the data table, depending on whether the entity type is translatable or not.

The revision metadata fields (revision_timestamp, revision_uid, revision_log), however, are only stored in the revision table. Thus, query fails.

The UUID field is only stored in the base table so for translatable entities the query fails.

Steps to reproduce

  1. Install Drupal in German. Because of #2363099: Using translated field definition descriptions in entity schema results in different schema definitions, resulting in update.php changes the status report will inform you that your database schema is out of date.
  2. Try to update the database schema using /update.php

Expected result:
The update worked.

Actual result:

Drupal\Core\Database\DatabaseExceptionWrapper: SQLSTATE[42S22]: Column not found: 1054 Unknown column 'revision_log' in 'where clause': SELECT COUNT(*) AS expression FROM (SELECT DISTINCT t.nid AS nid, 1 AS expression FROM {node_field_data} t WHERE (revision_log IS NOT NULL ) LIMIT 1 OFFSET 0) subquery; Array ( ) in Drupal\Core\Entity\Sql\SqlContentEntityStorage->countFieldData() (line 1757 of core/lib/Drupal/Core/Entity/Sql/SqlContentEntityStorage.php).

Proposed resolution

  • Introduce a TableMappingInterface::getFieldTableName() method to determine in which table a field is stored. In case the field is stored in multiple tables, inspect them in relevance order (data table, base table, revision table) and return the most relevant, where the field is stored.
  • Use the new method in SqlContentEntityStorage::countFieldData() to determine the table to be queried.

Remaining tasks

None

User interface changes

None

API changes

  • Added the TableMappingInterface::getFieldTableName() method.
  • Added an $entity_type parameter to DefaultTableMapping::__costruct().
Support from Acquia helps fund testing for Drupal Acquia logo

Comments

plach’s picture

Assigned: Unassigned » plach

On this

plach’s picture

xjm’s picture

Issue tags: +Triaged D8 critical
xjm’s picture

Issue tags: +D8 upgrade path
plach’s picture

Issue tags: +entity storage
larowlan’s picture

Status: Active » Needs review
Issue tags: +CriticalADay
FileSize
1000 bytes

Here's a test

Status: Needs review » Needs work

The last submitted patch, 6: content-entity-revision-table-2371605.1.patch, failed testing.

larowlan’s picture

Status: Needs work » Needs review
FileSize
3.38 KB

And a fix

Still needs a test for the revision metadata stuff.

larowlan’s picture

And a test for revision meta fields

tstoeckler’s picture

Wow, awesome!

This looks great.

One thing which I have not grokked yet is what happens when a field shows up in multiple tables. It currently doesn't matter because all we use countFieldData() for is checking whether any data exists at all or not.
But per the documentation the function should return a count, and e.g. for revisionable fields it totally makes a difference if I COUNT (DISTINCT ) on the data table or on the revision data table. However, we have no way of specifying what you want.
The current implementation returns different things depending on the order in which the table mapping got populated, which could become a pretty big WTF at some point.

Not really sure what to do. I think the easiest solution would be to simply document that for revisionable fields we always return the number of "non-revisioned field values" (or however you call that) and then fix the table order to make sure that is actually true.

Thoughts?

larowlan’s picture

well I could reverse the table names, by nature of the way they are constructed, you end up with this order:

  • base
  • data
  • revision_base
  • revision_data

Thoughts?

andypost’s picture

Status: Needs review » Reviewed & tested by the community

Looks ready

alexpott’s picture

Status: Reviewed & tested by the community » Needs work

I think we need to address #10

dawehner’s picture

+++ b/core/lib/Drupal/Core/Entity/Sql/DefaultTableMapping.php
@@ -334,4 +334,17 @@ protected function generateFieldTableName(FieldStorageDefinitionInterface $stora
+   * {@inheritdoc}
+   */
+  public function getFieldTableName($field_name) {
+    foreach ($this->getTableNames() as $table_name) {
+      $columns = $this->getAllColumns($table_name);
+      if (in_array($field_name, $columns, TRUE)) {
+        return $table_name;
+      }
+    }
+    return FALSE;
+  }
+
 }

mh, what about fields like 'nid' or 'type' which are stored in both?

Let's at least wait until plach can bring up his proposal he was working on already.

yched’s picture

[edit: crosspost]

  1. +++ b/core/lib/Drupal/Core/Entity/Sql/DefaultTableMapping.php
    @@ -334,4 +334,17 @@ protected function generateFieldTableName(FieldStorageDefinitionInterface $stora
    +  public function getFieldTableName($field_name) {
    +    foreach ($this->getTableNames() as $table_name) {
    +      $columns = $this->getAllColumns($table_name);
    +      if (in_array($field_name, $columns, TRUE)) {
    +        return $table_name;
    +      }
    +    }
    +    return FALSE;
    +  }
    

    Is that really correct ? A table storing a field is in no way guaranteed to contain a column named exactly after that field.

  2. +++ b/core/lib/Drupal/Core/Entity/Sql/TableMappingInterface.php
    @@ -102,4 +102,16 @@ public function getReservedColumns();
    +   * Returns the table name for a given column.
    +   *
    +   * @param string $field_name
    +   *   The field name for which the table name is sought.
    ...
    +  public function getFieldTableName($field_name);
    

    So is the param a column name or a field name ?

plach’s picture

Discussed this with Daniel and Alex:

We can return a single table and inspect them in order of "importance": data table, base table, revision table. Views would use ::getFieldTableName() to determine in which table the field should be listed so it would appear just once.

Working on ::getFieldTableName().
(just a warning since this is already assigned to me ;)

plach’s picture

This implements #16 and should address #15. It also provides additional test coverage for the new method.

I added a @todo to revisit some code in ::countFieldData() because, after various attempts, I was not able to make every test pass without falling back to the data or base table in case no field information is available. We should be able to fix this in #2274017: Make SqlContentEntityStorage table mapping agnostic .

plach’s picture

larowlan’s picture

+++ b/core/lib/Drupal/Core/Entity/Sql/DefaultTableMapping.php
@@ -129,6 +140,53 @@ public function getFieldNames($table_name) {
+      $storage = \Drupal::entityManager()->getStorage($this->entityType->id());

+++ b/core/lib/Drupal/Core/Entity/Sql/SqlContentEntityStorage.php
@@ -294,7 +294,7 @@ public function getTableMapping(array $storage_definitions = NULL) {
+      $table_mapping = new DefaultTableMapping($this->entityType, $definitions);

Can we pass the storage controller in as an argument? It should be in scope as $this.

Other than that, I think this is good to go

plach’s picture

Sorry, I should have clarified that: I did not inject the storage itentionally because in #2274017: Make SqlContentEntityStorage table mapping agnostic we are likely to be doing the opposite, that is making the storage rely on the table mapping, which will allow us to remove that line. Hence the current approach is likely to avoid API changes in the future.

larowlan’s picture

Status: Needs review » Reviewed & tested by the community

works for me, I think I'm ok to rtbc this, I added the tests but the rest is largely re-done from scratch

plach’s picture

Status: Reviewed & tested by the community » Needs review
FileSize
21.91 KB
1.49 KB

A couple of small clean-ups.

larowlan’s picture

Status: Needs review » Reviewed & tested by the community

unless bot says no

alexpott’s picture

Status: Reviewed & tested by the community » Needs work
Issue tags: +Needs issue summary update

I don't think we need a CR for the changes to DefaultTableMapping's constructor since this is internal to SQL entity storage.

The issue summary does not outline the proposed resolution. Can be re-rtbc'd once it does.

  • alexpott committed 8005955 on 8.0.x
    Issue #2371605 by plach, larowlan: SqlContentEntityStorage::...
alexpott’s picture

Status: Needs work » Fixed

Well I committed it by mistake. I'm not going to revert as this only needs an issue summary update and @plach promised me one so all good here.

plach’s picture

Issue summary: View changes
Issue tags: -Needs issue summary update

And here it is :)

plach’s picture

Issue summary: View changes

Moar

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.

amateescu’s picture

There's one thing which was forgotten here: we also need to provide a way to get the revision table name for a field if the SQL storage chooses to store default / all revisions in separate tables.

Opened a new issue to fix this: #2955442: Add a way to get all the tables in which a field is stored from TableMappingInterface