Problem/Motivation

Given an entity that has been translated (=== has more than 1 translation) that has an untranslatable field. When you change only that untranslatable field, ContentEntityBase::hasTranslationChanges() will return FALSE for all translations of that entity, even though in effect all translations have changed because untranslatable fields affect every language.

As ContentEntityBase::hasTranslationChanges() is used in two places in core, this has two different consequences:

Changed time tracking
Because forms explicitly update the changed timestamp whenever they are saved, this is only a problem when calling $entity->save() explicitly. In that case, though, because the changed timestamp is not updated, you can have sequential $entity->save() calls overwriting the other's data, which EntityChangedConstraintValidator would otherwise prevent
Tracking affected translations of a revision
This would lead to a new revision being marked as affecting no languages, so that it would not be visible at all in the user interface and, thus, not revertable. Because (as stated above), the changed time of the translation that is being edited is being explicitly changed (and because the changed field is translatable) this problem is somewhat masked, because this means that the translation that is being edited is always being marked as affected. Arguably, however, all translations should be marked as affected.

Proposed resolution

Remove the FieldDefinitionInterface::isTranslatable() check from ContentEntityBase::hasTranslationChanges().

Remaining tasks

  1. Write tests for the revision translation affected problem
  2. Review patch
  3. Write change notice

User interface changes

None.

API changes

This is a change in behavior, but as argued above the current behavior is a bug. The implications of this will have to be discussed in more detail.

Data model changes

None.

Comments

tstoeckler created an issue. See original summary.

tstoeckler’s picture

Status: Active » Needs review
StatusFileSize
new2.13 KB
new5.04 KB

Here we go.

The last submitted patch, 2: 2808335-2--tests-only.patch, failed testing.

Status: Needs review » Needs work

The last submitted patch, 2: 2808335-2.patch, failed testing.

tstoeckler’s picture

Issue summary: View changes
Priority: Major » Critical
Status: Needs work » Needs review

OK, the test fails in 2808335-2.patch are other test classes, so we probably just need to update those tests. It does prove that this is a valid bug. Would like to get some reviews / comments first, though, before pursuing this.

Also marking critical now, because - even though the test doesn't explicitly test the data loss scenario, that's pretty easy to reproduce as long as the changed timestamp does not get updated. See the first two paragraphs of the issue summary for more info.

berdir’s picture

+++ b/core/lib/Drupal/Core/Entity/ContentEntityBase.php
@@ -1184,7 +1184,7 @@ public static function bundleFieldDefinitions(EntityTypeInterface $entity_type,
    */
-  public function hasTranslationChanges() {
+  public function hasTranslationChanges($check_untranslatable_fields = FALSE) {
     if ($this->isNew()) {
       return TRUE;

This is unfortunately an API change if subclasses override this. It doesn't matter if it's optional, if interface requires it, you must define it (you can provide a default value if the interface doesn't, but this doesn't work)

tstoeckler’s picture

Issue tags: +Dublin2016
catch’s picture

Status: Needs review » Needs work

Yes it would be better for bc to add a new method on the interface and implement that on the base class.

Also this has more generic use than checking whether the translation changed, so doesn't hurt to have a new method for that reason either.

tstoeckler’s picture

Issue summary: View changes
Priority: Critical » Normal
Status: Needs work » Needs review
StatusFileSize
new3.45 KB

We figured out that, even though the test and bug report is valid, it is not actually possible to get data loss with this, because ContentEntityForm::submitForm() always updates the changed timestamp, no matter what. Downgrading to normal therefore.

We also realized that the other usage of ContentEntityBase::hasTranslationChanges() is incorrect as well. I will try to write some test coverage for that, both for the API and for the UI because (arguably) ther is a user-facing bug there.(However not critical.) See the issue summary, for more information.

Also uploading a new patch that matches the updated issue summary.

Status: Needs review » Needs work

The last submitted patch, 9: 2808335-9--changed-untranslatable.patch, failed testing.

tstoeckler’s picture

Title: Changed time not updated when only non-translatable fields are changed on a translated entity » [PP-2] Changed time not updated when only non-translatable fields are changed on a translated entity
Issue tags: +Needs tests

So there are multiple problems that are exposed by the tests, but at the very least the test fail of Drupal\content_moderation\Tests\ModerationLocaleTest is postponed at least on #2615016: ContentEntityBase::hasTranslationChanges should exclude the "changed" fields from the comparison. There are other problems, as well, but I don't think it's worth pursuing them as of yet as the different dependencies of blockers make it very hard to keep track of what's going on.

hchonov’s picture

The failing test ContentTranslationUITestBase::doTestTranslationChanged is that now the field of the node entity "vid" gets also compared, which is untranslatable and now because of that changing a value of a translatable field and setting a new revision will lead to all translations having the same changed timestamp. This is also a problem that is solved in #2615016: ContentEntityBase::hasTranslationChanges should exclude the "changed" fields from the comparison.

wim leers’s picture

-      if (!$definition->isComputed() && (!$translated || $definition->isTranslatable())) {
+      if (!$definition->isComputed()) {

YES please. Running into this problem while working on #2737719: EntityResource: Provide comprehensive test coverage: for every entity type, every format, every method.

I'd go even further, and say that ContentEntityBase::hasTranslationChanges() should have something like this too:

if (!$translated) {
  return FALSE;
}

Because \Drupal\Core\Entity\ContentEntityInterface::hasTranslationChanges() has this documentation:

   * If the entity is translatable only translatable fields will be checked for
   * changes.

But perhaps that's wrong for a reason I can't see. In which case we should have test coverage for that. I see no unit test coverage for hasTranslationChanges() at all.

hchonov’s picture

I'd go even further, and say that ContentEntityBase::hasTranslationChanges() should have something like this too:

if (!$translated) {
  return FALSE;
}

hasTranslationChanges could be used also for non translated entities to check for changes on saving and then e.g. set a new revision. So it still should check for changes against the current translation no matter if the entity is translated or not.

wim leers’s picture

#14: if so, both the name and documentation are misleading. Let's also improve that here then.

The last submitted patch, 9: 2808335-9--changed-untranslatable.patch, failed testing.

hchonov’s picture

Version: 8.2.x-dev » 8.3.x-dev
Status: Needs work » Needs review
StatusFileSize
new5.09 KB
new2.12 KB

@Wim Leers: in the latest patch here from @tstoeckler this sentence has been removed from the documentation :).

In order to make this issue not dependent on #2615016: ContentEntityBase::hasTranslationChanges should exclude the "changed" fields from the comparison I've fixed the failing tests by hard coding the revision metadata fields which will now be checked for changes but should not.

I've marked the term field "parent" as computed as it has its own storage and this is currently the only way of it being skipped by ContentEntityBase::hasTranslationChanges.

Also I think this should be targeting 8.3.x.

hchonov’s picture

Issue tags: -Needs tests
StatusFileSize
new5.27 KB
new674 bytes

Also assuring in the new test that the changed time across all translations is equal after a non-translatable field is being updated.

hchonov’s picture

StatusFileSize
new6.05 KB
new687 bytes

We also need an update function to update the field storage definition of the taxonomy term field "parent".

hchonov’s picture

StatusFileSize
new6.05 KB
new682 bytes

Copy paste mistake :).

The last submitted patch, 19: 2808335-19.patch, failed testing.

hchonov’s picture

The last submitted patch, 17: 2808335-17.patch, failed testing.

hchonov’s picture

StatusFileSize
new6.11 KB
new0 bytes

I've just learned that we have to retrieve the field storage definition to update from the entity definition update manager instead from the entity field manager.

The last submitted patch, 18: 2808335-18.patch, failed testing.

hchonov’s picture

StatusFileSize
new6.26 KB
new1.09 KB

I've just learned from @berdir in IRC that we have to uninstall the field storage definition if we make it computed as it is not tracked by the entity definition update manager.

The last submitted patch, 20: 2808335-20.patch, failed testing.

The last submitted patch, 24: 2808335-23.patch, failed testing.

Status: Needs review » Needs work

The last submitted patch, 26: 2808335-26.patch, failed testing.

hchonov’s picture

StatusFileSize
new8.81 KB
new2.43 KB

I am happy to see the test \Drupal\node\Tests\NodeRevisionsUiTest::testNodeRevisionsTabWithDefaultRevision as it shows that the current behavior was wrong and not how actually the revisions overview page was intended to behave - it should show only revisions with translation changes and the test is saving multiple revisions without any translation changes and asserting that they are all shown, but actually they have to be filtered out and that was not happening until now because the revision metadata fields were not skipped by the comparison in ContentEntityBase::hasTranslationChanges.

I've adjusted the test for the correct behavior.

hchonov’s picture

Status: Needs work » Needs review
tstoeckler’s picture

  1. +++ b/core/lib/Drupal/Core/Entity/ContentEntityBase.php
    @@ -1277,13 +1277,24 @@ public function hasTranslationChanges() {
    +      'revision_uid', 'revision_user',
    +      'revision_timestamp', 'revision_created',
    +      'revision_log', 'revision_log_message'
    

    I see that you are trying to semantically group the field names in one row, but this is a very unusual syntax in Drupal core. Can we just put each field name on its own line?

  2. +++ b/core/lib/Drupal/Core/Entity/ContentEntityBase.php
    @@ -1277,13 +1277,24 @@ public function hasTranslationChanges() {
    +      if (in_array($field_name, $revision_metadata_fields)) {
    

    I guess it's not exploitable here, but it's almost always best to pass TRUE as the third parameter to in_array(), because PHP is weird...

  3. +++ b/core/modules/taxonomy/src/Entity/Term.php
    @@ -157,7 +157,11 @@ public static function baseFieldDefinitions(EntityTypeInterface $entity_type) {
    +      // Marking as computed as the parent field has its own storage and in
    +      // order for ContentEntityInterface::hasTranslationChanges to skip it
    +      // from the comparision.
    +      ->setComputed(TRUE);
    

    This is not correct. Yes, computed fields are weird, but this is not actually a computed field, it's just not stored in the normal way.

    I'm not exactly sure off-hand why this was needed, but we need to solve that in a different way.

tstoeckler’s picture

Title: [PP-2] Changed time not updated when only non-translatable fields are changed on a translated entity » Changed time not updated when only non-translatable fields are changed on a translated entity

BTW, I think it's a great idea to hardcode the field names (for now), so removing the postponed title prefix.

hchonov’s picture

StatusFileSize
new7.54 KB
new3.04 KB

I've made the field computable in order for it to not get checked as somehow on translation edit there were changes in the "parent" field even if no sorting has occurred.

I've found out the problem in TermForm::form where the parent is set to 0 and this gets into the field values, but later is not saved, so this was causing problems in hasTranslationChanges and I removed it.

I've addressed all your remarks.

hchonov’s picture

StatusFileSize
new7.84 KB
new1.03 KB

@tstoeckler pointed me to the reason why in TermForm::form we set parent=0 and this is in order to preselect 'root', but as 0 is a non-valid target_id it does not get saved, but only makes hasTranslationChanges return a false result, so we have to filter out the parents before assigning them.

Status: Needs review » Needs work

The last submitted patch, 35: 2808335-35.patch, failed testing.

hchonov’s picture

Status: Needs work » Needs review
StatusFileSize
new8.33 KB
new3.38 KB

I've talked with @tstoeckler and we both agreed that the term parent field is currently broken and therefore it is currently breaking the checks of hasTranslationChanges for non-translated entities and by the change in the current issue it would not be skipped anymore for translated entities and will break the hasTranslationChanges check for them as well. We both agreed that fixing this would be out of scope of this issue so therefore we skip checking the parent field for now and I've opened #2843060: Term field "parent" could not be handled properly by ContentEntityBase::hasTranslationChanges to find a solution without having to block the current issue on it.

wim leers’s picture

Status: Needs review » Needs work
Related issues: +#2392845: Add a trait to standardize handling of computed item lists

#32.3: I hate to say it, but your interpretation of "computed" may not be correct. See #2392845: Add a trait to standardize handling of computed item lists :)

  1. +++ b/core/lib/Drupal/Core/Entity/ContentEntityBase.php
    @@ -1245,6 +1245,29 @@ public static function bundleFieldDefinitions(EntityTypeInterface $entity_type,
    +  protected function getFieldsToSkipFromTranslationChangesCheck() {
    

    So this is a global list of hardcoded fields. #33 suggests this is a temporary work-around.

    Let's make that more explicit by adding a @todo that points to an issue where that will be solved.

  2. +++ b/core/lib/Drupal/Core/Entity/ContentEntityBase.php
    @@ -1245,6 +1245,29 @@ public static function bundleFieldDefinitions(EntityTypeInterface $entity_type,
    @@ -1277,13 +1300,17 @@ public function hasTranslationChanges() {
    

    For me personally (see #13), the test that proves whether this solves it or not, is whether you can remove the @todo from \Drupal\Tests\rest\Functional\EntityResource\EntityResourceTestBase.

    i.e. if you can change

          // Set a default value on the field.
          $this->entity->set('field_rest_test', ['value' => 'All the faith he had had had had no effect on the outcome of his life.']);
          // @todo Remove in this if-test in https://www.drupal.org/node/2808335.
          if ($this->entity instanceof EntityChangedInterface) {
            $changed = $this->entity->getChangedTime();
            $this->entity->setChangedTime(42);
            $this->entity->save();
            $this->entity->setChangedTime($changed);
          }
          $this->entity->save();
    

    to:

          // Set a default value on the field.
          $this->entity->set('field_rest_test', ['value' => 'All the faith he had had had had no effect on the outcome of his life.']);
          $this->entity->save();
    

    Plus, that's a @todo pointing to this issue, so it must be solved by this issue anyway :)

hchonov’s picture

Status: Needs work » Needs review
StatusFileSize
new9.42 KB
new1.42 KB

@WimLeers thank you for the review. I've addressed all your remarks.

hchonov’s picture

So according to #2392845: Add a trait to standardize handling of computed item lists the parent field of the term entity is a "computed field" as the definition of a computed field almost matches the current behavior of the parent field :

Sample case: a "back reference" field (a field that lists all the entities that have an entity_ref field pointing to that entity)
Behavior : don't save or load *anything*, we don't know whether there are items and how many, that will be determined on read at runtime.

So do we want to make it computed here and so skip it automatically from the comparison in hasTranslationChanges or we prefer the current approach where we explicitly skip the parent field?

Status: Needs review » Needs work

The last submitted patch, 39: 2808335-39.patch, failed testing.

hchonov’s picture

Status: Needs work » Needs review
StatusFileSize
new12.01 KB
new1.85 KB

The changed timestamp has been hard-coded but saving the entity with changes on a non-translatable field causes the changed timestamp to get updated, so added dynamically estimation of the changed timestamp for the rest resource tests.

wim leers’s picture

The changed timestamp has been hard-coded but saving the entity with changes on a non-translatable field causes the changed timestamp to get updated, so added dynamically estimation of the changed timestamp for the rest resource tests.

Doesn't that mean this logic is still broken? Because:

  /**
   * Determines if the current translation of the entity has unsaved changes.
   *
   * If the entity is translatable only translatable fields will be checked for
   * changes.
   *
   * @return bool
   *   TRUE if the current translation of the entity has changes.
   */
  public function hasTranslationChanges();

Ah, but no, I already raised that in #13 and you answered that in #14.

Then I'm fine with this.

tstoeckler’s picture

#32.3: I hate to say it, but your interpretation of "computed" may not be correct. See #2392845: Add a trait to standardize handling of computed item lists :)

No worries, I'm well aware of that issue. I'll let the master himself speak for me on this one ;-), see #2207193: Don't allow validation constraints for computed properties, where in #3 I said:

As far as I know 'computed' means that the data is not stored in the entity tables along with the other fields.

To which @fago replied in #4:

Nop, computed means it's not stored as entity field/data at all. It's computed out of other values, or eventually lazy-loaded (this has yet to become clear).

The "(this has yet to become clear)" part is what #2392845: Add a trait to standardize handling of computed item lists is about.

To reply more specifically to #40 on the same topic:

So according to #2392845: Clarify the notion of "computed field" the parent field of the term entity is a "computed field" as the definition of a computed field almost matches the current behavior of the parent field :

Sample case: a "back reference" field (a field that lists all the entities that have an entity_ref field pointing to that entity)
Behavior : don't save or load *anything*, we don't know whether there are items and how many, that will be determined on read at runtime.

The important word in that in that example is "back" in "back reference". I.e. the actual references are already stored in the DB, and we can compute the value of the back reference with an entity query and the only purpose of having this back reference as a field is for performance/DX/UX (i.e. using formatters for it, etc.)

The "*anything*" in "don't save or load *anything*" of that quote is meant quite literally. And for term parents that's not the case, we do load things for that field, specifically we load the parents from the taxonomy_term_hierarchy table. It just happens to be the case that during a Term::load() that never happens. Granted that is super weird and causes problems - but it still doesn't make the field computed.

Edit: Mixed up the issue links

wim leers’s picture

Haha oops, I didn't realize you were on that issue too :D

hchonov’s picture

According to #44 the solution we've chosen to explicitly skip the "parent" field from the comparison of hasTranslationChanges has been the right decision instead of making the field computed as it looks like it is part-computed part-not-computed :P.

tstoeckler’s picture

Yes, definitely! (At least in my opinion definitely! ;-))

gábor hojtsy’s picture

Status: Needs review » Reviewed & tested by the community
Issue tags: +D8MI, +language-content, +sprint

The goals set out in the issue summary make sense, the patch looks to be right implementing the goals and skipping the backend fields properly. Also thanks for the expanded test coverage. Let's get this in :)

Status: Reviewed & tested by the community » Needs work

The last submitted patch, 42: 2808335-42.patch, failed testing.

hchonov’s picture

Status: Needs work » Reviewed & tested by the community

It has been a random test fail -> back to RTBC.

Version: 8.3.x-dev » 8.4.x-dev

Drupal 8.3.0-alpha1 will be released the week of January 30, 2017, which means new developments and disruptive changes should now be targeted against the 8.4.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

xjm’s picture

Version: 8.4.x-dev » 8.3.x-dev
Priority: Normal » Major

@catch suggested that this issue is probably patch-release-safe. It is also at least major for data integrity risks per @alexpott.

Status: Reviewed & tested by the community » Needs work

The last submitted patch, 42: 2808335-42.patch, failed testing.

wim leers’s picture

Issue tags: +Needs reroll

Apparently this conflicts with the recent REST commits.

anish.a’s picture

Version: 8.3.x-dev » 8.4.x-dev
Status: Needs work » Needs review
Issue tags: -Needs reroll
StatusFileSize
new12 KB

Rerolled with 8.4.x branch.

hchonov’s picture

Version: 8.4.x-dev » 8.3.x-dev
Status: Needs review » Reviewed & tested by the community
StatusFileSize
new12.01 KB

@xjm agreed in #52 on it being major and flagging it for the 8.3.x branch.

So I am uploading a re-roll of the RTBC patch from #42 for the 8.3.x branch and since it is just only a re-roll I think I am allowed of putting back the RTBC status of the patch.

Status: Reviewed & tested by the community » Needs work

The last submitted patch, 56: 2808335-56.patch, failed testing.

hchonov’s picture

Status: Needs work » Needs review

It has been a CI error, therefore putting back to RTBC.

gábor hojtsy’s picture

Status: Needs review » Reviewed & tested by the community

Status: Reviewed & tested by the community » Needs work

The last submitted patch, 56: 2808335-56.patch, failed testing.

gábor hojtsy’s picture

Patch failed to apply.

hchonov’s picture

Status: Needs work » Reviewed & tested by the community
StatusFileSize
new12.06 KB

#2770921: Feb 21st: Convert chunk of WTB to BTB by just moving classes, changing use statements adding traits has converted the test NodeRevisionsUiTest from WTB to BTB and therefore moved the test class to another location, but the test itself is still the same.

So this is simply a re-roll and therefore I think it is fine to put the RTBC status back as there are no real changes.

catch’s picture

Status: Reviewed & tested by the community » Needs work
Issue tags: +Needs re-roll

Hmm this doesn't apply to 8.4.x

error: patch failed: core/modules/taxonomy/src/Entity/Term.php:232
error: core/modules/taxonomy/src/Entity/Term.php: patch does not apply

hchonov’s picture

Issue tags: -Needs re-roll
StatusFileSize
new12.06 KB
new12.04 KB

The patch failed to apply because of the commit of #2850691: Deprecate TermInterface::getVocabularyId() only in 8.4.x. So I've prepared two patches one for 8.3.x and one for 8.4.x.

hchonov’s picture

Status: Needs work » Needs review
catch’s picture

Status: Needs review » Reviewed & tested by the community

  • catch committed d837f01 on 8.4.x
    Issue #2808335 by hchonov, tstoeckler, anish.a, Wim Leers, catch, Berdir...

  • catch committed d50593a on 8.3.x
    Issue #2808335 by hchonov, tstoeckler, anish.a, Wim Leers, catch, Berdir...
catch’s picture

Status: Reviewed & tested by the community » Fixed

Committed/pushed to 8.4.x and 8.3.x, thanks!

gábor hojtsy’s picture

Issue tags: -sprint

Yay, thanks!

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.