Problem/Motivation

The new method FieldItemList::equals() introduced by #2297817: Do not attempt field storage write when field content did not change is not reliable.

Due to the nature of PHP value types get converted into various types when dealing with form values or database result sets. So the final check return $value1 === $value2; fails on comparing on these values:

timestamps:

array(1) {
  [0] =>
  array(1) {
    'value' =>
    int(1429624933)
  }
}

array(1) {
  [0] =>
  array(1) {
    'value' =>
    string(10) "1429624933"
  }
}

booleans:

array(1) {
  [0] =>
  array(1) {
    'value' =>
    bool(false)
  }
}
array(1) {
  [0] =>
  array(1) {
    'value' =>
    int(0)
  }
}

Proposed resolution

Replace the type safe comparison in FieldItemList::equals() by a non type safe one.

Remaining tasks

Think carefully if the propsed solution can cause any problem ;-)

User interface changes

none

API changes

none

Support from Acquia helps fund testing for Drupal Acquia logo

Comments

Berdir’s picture

Depends on your understanding of reliable ;)

What if the type matters and those two should be considered something different?

For storing values, we definitely want to rather save too much than lose data. It's a best effort implementation.

That might be very different for your use case, but I'm not sure how to unify those two different use cases.

mkalkbrenner’s picture

For storing values, we definitely want to rather save too much than lose data. It's a best effort implementation.

I agree!

But FieldItemList::equals() became part of the field API and can therefor used elsewhere and not just to decide if we run an update statement on the storage. In this case it should do what it's documentation states - in a reliable way.
Personally I really appreciate this new API functionality and therefor it would be bad if we turn it into a somehow "private" function.

Let me think about it ...

hchonov’s picture

What about if the form_state values are type casted to the type set by their fields before they land in values?
In this case the entity values and the ones from form state should always be of the same type and so there should be no case where this function is going to check the same values but of different types.

cspitzlay’s picture

#3 Sounds like a good idea to me.

One might consider it a bug if one of the values to compare has the wrong type and is only equal if it is implicitely typecast. The data with the wrong type should never get this far.

Just one thought though: Will this force authors of custom code or contrib modules that manipulate form data to always get the type right, too? This may lead to lots of "My field is not saved. WTF?" bugs. Or would this typecast be done after any other alteration of the data?

cspitzlay’s picture

@mkalkbrenner: Did you get these type differences when just loading and manipulating the entity or is a form submission involved?

mkalkbrenner’s picture

Status: Active » Needs review
FileSize
2.76 KB

@hchonov: I thought about a kind of normalization of field values, too. But this will cause a bigger change.

What about a simple change that allows to "select" the desired behavior:

interface FieldItemListInterface {
  ...

  /**
   * Determines equality to another object implementing FieldItemListInterface.
   *
   * @param \Drupal\Core\Field\FieldItemListInterface $list_to_compare
   *   The field item list to compare to.
   *
   * @param bool $strict
   *   TRUE means that the values will be compared strictly.
   *
   * @return bool
   *   TRUE if the field item lists are equal, FALSE if not.
   */
  public function equals(FieldItemListInterface $list_to_compare, $strict = FALSE);

  ...
}
alexpott’s picture

Title: Method FieldItemList::equals() is not reliable » Method FieldItemList::equals() uses type safe comparison limiting usefulness

Re-titling to describe the issue.

alexpott’s picture

Where does the type casting occur? Does it occur on the way back out of the database? If it does then shouldn't == be good enough because if we're relying anywhere on the variable type this shouldn't be down to the storage.

Channeling my inner @yched - I'm not a huge fan of #6

mkalkbrenner’s picture

FileSize
474 bytes

Where does the type casting occur? Does it occur on the way back out of the database?

@alexpott: Have a look at Drupal\system\Tests\Entity\ContentEntityChangedTest::testChanged() in the patch posted at https://www.drupal.org/node/2428795#comment-9852123.
This test failed and the examples in this issue's description are var_dump() during this test.

The timestamp is form the "created" field, the boolean from "language_default" and the same happens for "user_id" with an integer type:

array(1) {
  [0] => 
  array(1) {
    'target_id' =>
    int(1)
  }
}

array(1) {
  [0] => 
  array(1) {
    'target_id' =>
    string '1' (length=1)
  }
}
Did you get these type differences when just loading and manipulating the entity or is a form submission involved?

In this test, no form handling is included, just the entity API and the entity SQL storage.

shouldn't == be good enough because if we're relying anywhere on the variable type this shouldn't be down to the storage

The arguments in #4 seems to be in the same direction. As well as https://www.drupal.org/node/2297817#comment-9852673
Let's see what the testbot says to such a simple change ...

mkalkbrenner’s picture

Issue summary: View changes
joelpittet’s picture

Whoops, sorry re-posting here:

shouldn't == be good enough because if we're relying anywhere on the variable type this shouldn't be down to the storage

PHP is weird sometimes:

php -r "var_dump(0 == 'all');"
bool(true)
mkalkbrenner’s picture

@joelpittet: you're right with that example. The string 'all' gets converted to integer in this context. That's PHP.

But I think that in our case we're just comparing "compatible" types like 0 == '0' or 0 == FALSE that could be converted into each other. And the array keys matter as well.

mkalkbrenner’s picture

The patch in #9 passed. How good is the test coverage? ;-)

joelpittet’s picture

I'm not sure of this part of the system that well, it really does depend on where that data is coming from for the comparison. As long as you've thought of those strange bits.

mkalkbrenner’s picture

@alexpott asked me to check where these type conversions are coming from. Basically they're caused by PDOStatement which returns every value as a string. That's an example of a result for a simple article:

object(stdClass)[965]
  public 'vid' => string '1' (length=1)
  public 'langcode' => string 'en' (length=2)
  public 'revision_timestamp' => string '1430315259' (length=10)
  public 'revision_uid' => string '1' (length=1)
  public 'revision_log' => string '' (length=0)
  public 'nid' => string '1' (length=1)
  public 'type' => string 'article' (length=7)
  public 'uuid' => string 'b41b9844-993a-4401-9100-a79e979f1fb2' (length=36)
  public 'isDefaultRevision' => string '1' (length=1)

These values are directly passed to entities' __construct() and cached internally in $this->values as strings without any type conversion.

From my point of view a type conversion within PHP makes no sense. Even if you assign a "PHP type" to each field the amount of places where you have to ask the field definition for the type and you have to convert the value accordingly seems to be endless.

Therefor the simple patch from #9 seems to be the right solution.

alexpott’s picture

Issue tags: +Needs tests

I agree - if the entity loaded from the db doesn't have typed properties then this makes sense. Let's improve the tests to cover this.

Aside: The temptation to say "doesn't have typed data" above was almost overwhelming.

mkalkbrenner’s picture

Here's a test that covers the SQL storage load issue as described in #15. This one should fail.

Status: Needs review » Needs work

The last submitted patch, 17: 2475237_test_only.patch, failed testing.

mkalkbrenner’s picture

Status: Needs work » Needs review
FileSize
1.45 KB

And now the patches from #9 and #17 combined.

mkalkbrenner’s picture

FileSize
1.84 KB

I extended the FieldItemListTest unit test In addition to the FieldSqlStorageTest from #17.
So we now have two different test that should fail without the patch from #9.

Status: Needs review » Needs work

The last submitted patch, 20: 2475237_test_only_20.patch, failed testing.

mkalkbrenner’s picture

Status: Needs work » Needs review
Related issues: +#2428795: Translatable entity 'changed' timestamps are not working at all
FileSize
2.3 KB

And now again the patch from #9 in combination with both tests.

mkalkbrenner’s picture

FileSize
1.31 KB
823 bytes

I removed the kernel test as requested by @alexpott in IRC, because that will be covered by related issues.

hchonov’s picture

Just to give a real example from contrib, where this patch will solve problems in the Relation entity.

From the preSave() of the Relation entity:

$this->arity = count($this->endpoints);

As we do not know if the number of the endpoints have changed, we always count them again and set the arity. In this case the arity is set as an integer, but the one loaded from the storage is a string.

The arity is an integer base field on the relation entity.

So just running

entity_load('relation', 1)->save();

causes equals to return false for the arity field.

Fabianx’s picture

Status: Needs review » Reviewed & tested by the community

RTBC, looks great to me.

alexpott’s picture

Status: Reviewed & tested by the community » Fixed

This issue addresses a major bug with test coverage and is allowed per https://www.drupal.org/core/beta-changes. Committed d560b48 and pushed to 8.0.x. Thanks!

  • alexpott committed d560b48 on 8.0.x
    Issue #2475237 by mkalkbrenner: Method FieldItemList::equals() uses type...

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.