FieldCrudTestCase::testUpdateField() assumes that a column type of 'numeric' will be automatically rounded by the database server. We cannot make such an assumption because it's generally wrong. On SQLite, fixed-precision numeric(digit, precision) types do not exist, and everything is stored as either integers or floats.

What we can assume is that the database engine will store this numeric type with at least the requested precision.

FieldCrudTestCase fails on SQLite because of this, this is consequently a critical.

Support from Acquia helps fund testing for Drupal Acquia logo

Comments

Damien Tournoud’s picture

Status: Active » Needs review
FileSize
1.04 KB
Stevel’s picture

Status: Needs review » Reviewed & tested by the community

This is a simple patch, and fixes the error on sqlite.

bjaspan’s picture

Status: Reviewed & tested by the community » Needs work

The purpose of testUpdateField() is not to make sure that numeric columns work but that the field_update_field() function works. The current code assumes that creating a numeric(5,2) field works as expected, then changes it to numeric(5,3) to verify that the change takes. The proposed patch actually just disables the test on SQLite, because the assumption about numeric(5,2) fields is wrong, and the modified test actually does not verify that field_update_field() does anything (since the numbers are stored as floats, the test would pass if field_update_field() was not even called).

Really, the to be correct on dbs that have numeric columns, the test should first verify the field behaves as a numeric(5,2) should, and then verify that the updated field behaves as a numeric(5,3) column. But this issue is not about belt-and-suspendering this test case.

A correct patch would change the test to change a field in some way that verifies field_update_field() is actually working. Incidentally, if necessary I think it is completely acceptable to have branches in the test code based on database type. Tests are not normal code, they just need to do whatever is necessary to test what needs to be tested. So if a correct test on all db types requires if (driver() == 'sqlite'), so be it.

Alternatively, the critical issue can be fixed by disabling the test on sqlite altogether (again with an if statement), since really that is all the present patch does anyway, just in an obscured manner.

Stevel’s picture

On SQLite, fixed-precision numeric(digit, precision) types do not exist, and everything is stored as either integers or floats.

From the Schema API:

'precision', 'scale': For type 'numeric' fields, indicates the precision (total number of significant digits) and scale (decimal digits right of the decimal point). Both values are mandatory. Ignored for other field types.

But sqlite completely ignores these parameters, so we need to change the comment there.
We no longer "support" fixed-precision fields (i.e. one can't rely on the fact that the values are rounded to a specific precision) , but perhaps we can support a kind of "minimal guaranteed precision" here?

Damien Tournoud’s picture

Just to mess-up the debate a little bit more, the precise behavior of the precision-loss is perfectly undefined.

From the MySQL manual:

When such a column is assigned a value with more digits following the decimal point than are permitted by the specified scale, the value is converted to that scale. (The precise behavior is operating system-specific, but generally the effect is truncation to the permissible number of digits.)

From the PostgreSQL manual:

If the scale of a value to be stored is greater than the declared scale of the column, the system will round the value to the specified number of fractional digits.

We should definitely test something more predictable here.

Damien Tournoud’s picture

And to even mess-up this a little bit more, there are some database engines, include Drizzle, that throw an error if you try an insert or update operation that will result in a loss of data (string truncation, number truncation).

We definitely need to fix the test.

sfyn’s picture

Status: Needs work » Needs review
FileSize
3.82 KB

Here's a simple patch that simply skips the test for sqlite.

Status: Needs review » Needs work

The last submitted patch, sqliteprecision-851602-7.patch, failed testing.

sfyn’s picture

Status: Needs work » Needs review
FileSize
3.69 KB

reroll

Status: Needs review » Needs work

The last submitted patch, sqliteprecision-851602-7.patch, failed testing.

chx’s picture

Status: Needs work » Needs review
FileSize
2.98 KB

I rewrote the test from ground up. It now changes cardinality from 4 to 5 and makes sure that amount of stuff can be inserted but not more.

chx’s picture

dereine asked to check the values loaded. And I do not like the way the patch is structured. (ie. 4 is used twice)

dawehner’s picture

Status: Needs review » Reviewed & tested by the community

This do while looks much better than before.

Looks fine.

dhthwy’s picture

looks good. +1 to RTBC.

webchick’s picture

Status: Reviewed & tested by the community » Fixed

This looks good, but that test is missing a big ass "why" you're using that method to test field updates, since it's kinda obscure. I wouldn't know from reading the issue, and only know because I happened to be in the room with chx when he explained it to me. :P

So I added this to the top of the first hunk:

    // Create a field with a defined cardinality, so that we can ensure it's
    // respected. Since cardinality enforcement is consistent across database
    // systems, it makes a good test case.

If there's a more betterer comment to describe this, feel free to post a follow-up patch to amend it.

Committed to HEAD, with that adjustment.

bjaspan’s picture

Title: FieldCrudTestCase::testUpdateField() assumes a fixed-precision storage » FieldCrudTestCase::testUpdateField() does not test changing the schema
Priority: Critical » Normal
Status: Fixed » Needs work

The original test, though flawed as described above, tested Field API altering the field storage schema. The new test does not; in fact the new test could pass without calling the field storage engine's update hook at all.

IMHO, we should have a field_update_field() test that changes the schema. Not critical, I suppose.

Damien Tournoud’s picture

Do we support adding columns to the schema? If so, we could easily test that.

chx’s picture

Barry, good luck figuring out a test that will test more than this and generic enough. I know that the mongodb field storage engine enforces from PHP this and will need an upgrade because currently it throws exceptions at invalid deltas. Edit: I mean, even this test needs PHP-level enforcement for mongodb and I have my doubts about enforcing anything else... Even in SQL, in SQLite you can store anything into any SQL field unless we begin to add CHECKs to the SQLite-schema code so I have no idea what else you can find...

Still edit:

  if ($has_data && $field['columns'] != $prior_field['columns']) {
    throw new FieldUpdateForbiddenException("field_sql_storage cannot change the schema for an existing field with data.");
  }

Didn't we agree that we put stuff into fields that are hard to upgrade, anyways?

bjaspan’s picture

In #3 I wrote: "Incidentally, if necessary I think it is completely acceptable to have branches in the test code based on database type. Tests are not normal code, they just need to do whatever is necessary to test what needs to be tested. So if a correct test on all db types requires if (driver() == 'sqlite'), so be it."

A more reasonable way to do that is to put the tests into the storage module's tests directly. Ironically, it turns out in this case we've ALMOST already done that. field_sql_storage.test has testUpdateFieldSchemaWithData() (the update should fail) and testFieldUpdateIndexesWithData() (the update should succeed). So, all we need to do is move the original 'numeric' test from field.test to field_sql_storage.test. Other storage engines which do not perform a similar test of field_update_field() will simply be less well tested than field_sql_storage.module.

bjaspan’s picture

Status: Needs work » Needs review
FileSize
3.15 KB

Test moved to field_sql_storage.test and improved slightly.

Damien Tournoud’s picture

Status: Needs review » Needs work

I'm sorry, but I thought it was clear for everyone that the behavior of numeric fields across database engines is *not* consistent, and that we need to test on something else.

bjaspan’s picture

Yeah, so, I forgot that the SQL storage engine runs on things other than mysql. :-) What can I say, I was up late last night.

I think the solution is to wrap the test in "if (driver() == 'mysql')" as described in #19.

TcRacer’s picture

Title: FieldCrudTestCase::testUpdateField() does not test changing the schema » Field.crud issues with commerce

Hey guys. I'm really new to all of this. I've worked a little on d6. I'm trying to make a website for my little cousins charity. I'm only 14, so this is all pretty overwhelming to me. I used to have a subscription on Lynda.com that helped me out a lot, but it expired. Anyway, I installed a suite of modules on my d7 installation to work for drupal commerce. I had to download the address field, entity API, rules, and views modules for it. When I enable all this modules, I get this error -

FieldException: Attempt to create field name billing_address which already exists, although it is inactive. in field_create_field() (line 277 of C:\Users\Daniel\Desktop\drupal-7.0-beta1\drupal-7.0-beta1\modules\field\field.crud.inc).

What can I do to fix this? download one of those files above? replace a file with one above? Thanks!
Dan

tstoeckler’s picture

Title: Field.crud issues with commerce » FieldCrudTestCase::testUpdateField() does not test changing the schema

Resetting the title.

I would recommend you just reinstall your whole site from scratch. If any problem persists, please open a new issue in the Drupal core or Commerce issue queue. That way people will be more likely to help you than in an issue, which is focused on something entirely unrelated. Thanks.

Garrett Albright’s picture

Status: Needs work » Needs review
FileSize
3.28 KB

Reroll with the db_driver() check. Only tested against SQLite currently; could someone with access to Postgres give it a run?

  • webchick committed 99d14f6 on 8.3.x
    #851602 by chx, bjaspan, Damien Tournoud: Fixed FieldCrudTestCase::...

  • webchick committed 99d14f6 on 8.3.x
    #851602 by chx, bjaspan, Damien Tournoud: Fixed FieldCrudTestCase::...

  • webchick committed 99d14f6 on 8.4.x
    #851602 by chx, bjaspan, Damien Tournoud: Fixed FieldCrudTestCase::...

  • webchick committed 99d14f6 on 8.4.x
    #851602 by chx, bjaspan, Damien Tournoud: Fixed FieldCrudTestCase::...

  • webchick committed 99d14f6 on 9.1.x
    #851602 by chx, bjaspan, Damien Tournoud: Fixed FieldCrudTestCase::...