I can't set an score as 0.5 with PostgreSQL

The errror.

Drupal\search_api\SearchApiException: SQLSTATE[22P02]: Invalid text representation: 7 ERROR:  invalid input syntax for integer: "0.5"         [error]
LINE 1: ...ATE search_api_db_default_index_text SET score=score * '0.5'
                                                                  ^: UPDATE search_api_db_default_index_text SET score=score * :mult
WHERE  (field_name = :db_condition_placeholder_0) ; Array
(
)
 in Drupal\search_api_db\Plugin\search_api\backend\Database->fieldsUpdated() (line 1034 of
/mnt/persist/www/visitlinkoping_prod/web/modules/contrib/search_api/modules/search_api_db/src/Plugin/search_api/backend/Database.php).

Way to replicate it.

Install a Drupal Standard profile with PostgreSQL the stable one from Ubuntu 16.04. Install Search API Database.

Create the defaults server, and index, add a field like body, put it with Score 0.5, try to index, you'll get the error.

CommentFileSizeAuthor
#49 2846932-49--db_backend_floating_point_boosts.patch4.22 KBdrunken monkey
#47 2846932-47--db_backend_floating_point_boosts.patch4.3 KBdrunken monkey
#47 2846932-47--db_backend_floating_point_boosts--interdiff.txt1.53 KBdrunken monkey
#45 2846932-46--db_backend_floating_point_boosts.patch3.76 KBdrunken monkey
#45 2846932-46--db_backend_floating_point_boosts--interdiff.txt972 bytesdrunken monkey
#44 2846932-44--debug.patch4.28 KBdrunken monkey
#44 2846932-44--debug--interdiff.txt930 bytesdrunken monkey
#42 2846932-42--db_backend_floating_point_boosts.patch3.69 KBdrunken monkey
#42 2846932-42--db_backend_floating_point_boosts--interdiff.txt908 bytesdrunken monkey
#41 2846932-41--debug.patch4.3 KBdrunken monkey
#41 2846932-41--debug--interdiff.txt930 bytesdrunken monkey
#39 2846932-39--db_backend_floating_point_boosts.patch3.7 KBdrunken monkey
#39 2846932-39--db_backend_floating_point_boosts--interdiff.txt522 bytesdrunken monkey
#37 2846932-37--debug.patch4.17 KBdrunken monkey
#37 2846932-37--debug--interdiff.txt924 bytesdrunken monkey
#35 2846932-35--db_backend_floating_point_boosts.patch3.7 KBdrunken monkey
#35 2846932-35--db_backend_floating_point_boosts--interdiff.txt2.77 KBdrunken monkey
#30 interdiff--14-30.patch4.06 KBkillua99
#30 2846932-30--db_backend_floating_point_boosts.patch5.64 KBkillua99
#21 2846932-19--db_backend_floating_point_boosts.patch3.37 KBkillua99
#18 2846932-18--db_backend_floating_point_boosts.patch3.38 KBkillua99
#17 2846932-17--db_backend_floating_point_boosts.patch3.27 KBkillua99
#14 2846932-14--db_backend_floating_point_boosts.patch3.24 KBdrunken monkey
#13 2846932-13--db_backend_floating_point_boosts.patch3.23 KBdrunken monkey
#13 2846932-13--db_backend_floating_point_boosts--interdiff.txt611 bytesdrunken monkey
#11 2846932-11--db_backend_floating_point_boosts.patch3.2 KBdrunken monkey
#11 2846932-11--db_backend_floating_point_boosts--interdiff.txt507 bytesdrunken monkey
#9 2846932-8--db_backend_floating_point_boosts.patch3.21 KBdrunken monkey
#9 2846932-8--db_backend_floating_point_boosts--interdiff.txt1.72 KBdrunken monkey
#7 2846932-7--db_backend_floating_point_boosts.patch2.35 KBdrunken monkey
#7 2846932-7--db_backend_floating_point_boosts--interdiff.txt571 bytesdrunken monkey
#4 2846932-3--db_backend_floating_point_boosts--tests_only.patch972 bytesdrunken monkey
#3 2846932-3--db_backend_floating_point_boosts.patch1.79 KBdrunken monkey
#3 2846932-3--db_backend_floating_point_boosts--tests_only.patch0 bytesdrunken monkey

Comments

killua99 created an issue. See original summary.

killua99’s picture

Title: PostgreSQL » PostgreSQL score 0.5 invalid input syntax for integer
drunken monkey’s picture

Component: General code » Database backend
Status: Active » Needs review
StatusFileSize
new0 bytes
new1.79 KB

Thanks for reporting this issue! Once again, MySQL is just way laxer than Postgres regarding these things.
Well, nothing for it. Let's see if the attached SQL syntax works for all three DMBSs we support.

drunken monkey’s picture

StatusFileSize
new972 bytes

Oops.

The last submitted patch, 3: 2846932-3--db_backend_floating_point_boosts--tests_only.patch, failed testing.

The last submitted patch, 3: 2846932-3--db_backend_floating_point_boosts.patch, failed testing.

drunken monkey’s picture

No idea what's happening there in Postgres. This should help debugging.

Status: Needs review » Needs work

The last submitted patch, 7: 2846932-7--db_backend_floating_point_boosts.patch, failed testing.

drunken monkey’s picture

Status: Needs work » Needs review
StatusFileSize
new1.72 KB
new3.21 KB

And this has a chance of fixing, or at least helping, with the MySQL fails.

Status: Needs review » Needs work

The last submitted patch, 9: 2846932-8--db_backend_floating_point_boosts.patch, failed testing.

drunken monkey’s picture

Status: Needs work » Needs review
StatusFileSize
new507 bytes
new3.2 KB

Status: Needs review » Needs work

The last submitted patch, 11: 2846932-11--db_backend_floating_point_boosts.patch, failed testing.

drunken monkey’s picture

Status: Needs work » Needs review
StatusFileSize
new611 bytes
new3.23 KB

This is getting exhausting …

drunken monkey’s picture

StatusFileSize
new3.24 KB
killua99’s picture

Let me help you.

The test is failing with this patch or is a test fail from the current branch ?

killua99’s picture

Also All the test are against Drupal 8.4.x, that .... is way into the future, any reason for this? what about 8.2.x ? or stable one? 8.2.5? I guess using Drupal 8.4.x does not affect the test failure.

Also I mange to control the issue with the unserialized I guess, I'm not 100% sure, but PostgreSQL do not use UNSIGNED, in that case is SERIAL, but I do not believe we're gonna have negative numbers.

The next issue is when I set a field with score 3. If I leave AS INTEGER, the postgresql has an error about "Error numeric out of range" ... Still if I set to BIGINT still there are numbers, or Scores that are way to big, I guess this is because the operation is at the end, (score * 15), that cause a really big number. I was trying to delete de index and reindex everything but I can, should I TRUNCATE the table manually?

killua99’s picture

StatusFileSize
new3.27 KB

This patch might work for all databases. I guess

killua99’s picture

StatusFileSize
new3.38 KB

Meh, it fails to apply. Reroll.

The last submitted patch, 17: 2846932-17--db_backend_floating_point_boosts.patch, failed testing.

Status: Needs review » Needs work

The last submitted patch, 18: 2846932-18--db_backend_floating_point_boosts.patch, failed testing.

killua99’s picture

Status: Needs work » Needs review
StatusFileSize
new3.37 KB

Well, when you try to edit too fast on notepad.

Status: Needs review » Needs work

The last submitted patch, 21: 2846932-19--db_backend_floating_point_boosts.patch, failed testing.

killua99’s picture

I find out the root of the unserialized.

The Task.php Content Entity create the table.

    $fields['data'] = BaseFieldDefinition::create('string_long')
      ->setLabel(t('Task data'))
      ->setReadOnly(TRUE);

On PostgreSQL that is equal to = text. That mean really short character. This need to be

        'type' => 'blob',
        'not null' => TRUE,
        'size' => 'big',

It require an update.

drunken monkey’s picture

Thanks for trying to help!
However, as you can see from the test results, #14 is already pretty good. There's just some pretty inexplicable fail on Postgres, with an illegal serialized value being stored, which I don't really know how to debug. Can you reproduce that problem/test fail locally? Debugging that would really help – I can neither reproduce nor explain it.

Using type "BIGINT" shouldn't be necessary as the database column itself uses normal "INT" as well. And I really don't know what the purpose of all these additional function calls would be.

drunken monkey’s picture

Oh, sorry, wrote that comment before seeing your last post. Awesome work, thanks for finding that out!
Not sure how this is caused/triggered by this patch, though. But if your conclusion is true, it's a big problem in any case, I guess.
Do you know where that SQL is created, and how we can fix this on our side?
However if Postgres really creates a field with a max length of (apparently) 47 characters for a type called string_long, this sounds to me like a Core bug.

killua99’s picture

Since patch #4 only test you could see the serialize is broken.

With PostgreSQL the way the task is safe cause in broken data. That is the process we should look first, then, the integer issue.

killua99’s picture

Is not a core problem tho, I'm concluding that type field might work too, the thing is, the task when is saved is broken.

drunken monkey’s picture

Is not a core problem tho, I'm concluding that type field might work too, the thing is, the task when is saved is broken.

Sorry, I don't understand this. Why do you say that this is not a Core bug?

killua99’s picture

The object that try to serialize is half, or less. Others objects serialized are complete. Just the one trying to change the score fails. The field string_long might work too.

killua99’s picture

Status: Needs work » Needs review
StatusFileSize
new5.64 KB
new4.06 KB
borisson_’s picture

Status: Needs review » Needs work

Not sure if we should do this, but I did see a couple of things that can be improved in the patch.

  1. +++ b/modules/search_api_db/src/Plugin/search_api/backend/Database.php
    @@ -930,9 +930,13 @@ class Database extends BackendPluginBase implements PluginFormInterface {
    +              $multiplier = min((int) $multiplier, 4294967295);
    ...
    +                  ->expression('score', 'CAST((score * CAST(COALESCE(:mult,\'0\') AS INTEGER)) AS INTEGER)', array(':mult' => $multiplier))
    

    We already cast to an integer, so why do it again here?

  2. +++ b/src/Entity/Task.php
    @@ -136,12 +138,29 @@ class Task extends ContentEntityBase implements TaskInterface {
    +      switch ($this->get('type')->value) {
    +
    +        case 'updateIndex':
    +
    +          $class = Index::class;
    +
    +          break;
    +
    +      }
    

    Coding standards: There's too many blank lines here, it should be like this, I think.

    switch {
    case:
    $class
    break;
    
    }
    
  3. +++ b/src/Entity/Task.php
    @@ -136,12 +138,29 @@ class Task extends ContentEntityBase implements TaskInterface {
    +        throw new $e->getMessage();
    

    Does this work? It doesn't look like it should. Don't we need to, at least, do: throw new \Exception($e->getMessage());?

  4. +++ b/src/Task/TaskManager.php
    @@ -114,11 +114,16 @@ class TaskManager implements TaskManagerInterface {
    +    $serializer = \Drupal::service('serializer');
    

    The task manager uses constructor DI to inject needed services, let's do that for this service as well.

killua99’s picture

Actually I see now that, my entire comment was strip away from this patch. No idea why.

I thing is, I'm agree with these changes, the really issue is, the update process of score have to be rewritten. The update score multiples for what do you have on the database, and I was trying to catch the bug, I see at the indexItem the score was rounded and checked to don't be so big. If you have a score of 200, and the multipler will be 10 the next score will be 2000, if we change the score again trying to figure why the score does not work, it will do another multiplier against the current score instead look at the token and then do the math.

Mainly this issue has to mayor for my point of view, the Class serialization, that's currently handle by the php function serialize that for Class and mostly for Entity get a broken serialize result, that's the main error on the Test. Later the score, when you update any field.

My thought is, the way the Task Entity store data should be changed to \Serialize. That will be more solid the way we rebuild an Entity from the serialize string.

This patch might not be taken for commit, this require work. and maybe a META issue to attend all this issues.

drunken monkey’s picture

I'm very sorry, but I really don't understand what you're trying to say, and your patch mostly looks like temporary hacks and nothing that could be committed to the module. And, again, most of the changes look like they're completely unnecessary seeing as #14 already almost passes.
Please try to explain again what the problem is that causes the test fails for #14. If it's a truncated field value – how would using a different serializer help?
If you have identified the problem and know how to fix it (and it's really not a problem in Core), then please use my patch in #14 as a base and only add the changes that are necessary to make the tests pass on Postgres.

killua99’s picture

The patch #14 does not work neither.

The error is clear, Drupal\Tests\search_api_db\Kernel\BackendTest::testBackend
Exception: could not unserialize: O:30:"Drupal\search_api\Entity\Index":27:{s:5:"

the taskmanager.php when serialize the Class Index get broken and with weird characters and save only the half of it. on MySQL it saved with those weird characters, but it should use the Serialize class instead the function.

On PHP documentation they mention about this with Class, that might get broken. The patch #14 to able to work with PostgreSQL that is this issue, the Serialize process has to be change.

I did an interfiff against you patch #14 to see the differences. The patch require more work, but is the only way to guarantee a good Serialize way. Also the UPDATE process of the Index is completely wrong. It do the score against the Score getting bigger and bigger the INTEGER.

drunken monkey’s picture

Status: Needs work » Needs review
StatusFileSize
new2.77 KB
new3.7 KB

Sorry, but it's really almost impossible for me to tell what you're trying to say. Do you maybe have a colleague with better English who can translate? I think this is working for neither of us.

Anyways, I think you're now saying that Postgres can't handle the special characters introduced by serializing an entity? In that case, we can just serialize it's values, can't we? Using the serializer service would a) introduce a dependency on the "Serialization" module and b) apparently require us to know beforehand what class the serialized data should have (which we don't).
Patch attached (interdiff compared to #3, which is apparently the only serious patch in this issue, without any debugging code).

Also the UPDATE process of the Index is completely wrong. It do the score against the Score getting bigger and bigger the INTEGER.

If I interpret this correctly, I don't think you're reading the code right there. Or how do you think it should look?

borisson_’s picture

Status: Needs review » Needs work

So, if we want to improve the postgres handling, we should make sure that this passes on postgres?

drunken monkey’s picture

Status: Needs work » Needs review
StatusFileSize
new924 bytes
new4.17 KB

Yes, of course! (All patches should pass on Postgres.)
So, seems like it's time for another debugging patch!

Status: Needs review » Needs work

The last submitted patch, 37: 2846932-37--debug.patch, failed testing.

drunken monkey’s picture

Status: Needs work » Needs review
StatusFileSize
new522 bytes
new3.7 KB

Oh. Right.

The last submitted patch, 37: 2846932-37--debug.patch, failed testing.

drunken monkey’s picture

StatusFileSize
new930 bytes
new4.3 KB

More debugging!

Tried sqlfiddle.com as an alternative, but that doesn't really seem to work.
Maybe I should try again to set up Postgres locally? Can't really be more tedious than this (he said, unconvinced).

drunken monkey’s picture

Forgot the Postgres test above, but might not matter, actually: I managed to install Postgres this time and found a cast syntax that seems to work for all three DBMSs! (Although I couldn't actually reproduce the initial bug reported here with my Postgres installation, so not completely sure.)
Let's give it a try! (Interdiff compared to #39, of course.)

Status: Needs review » Needs work

The last submitted patch, 42: 2846932-42--db_backend_floating_point_boosts.patch, failed testing.

drunken monkey’s picture

Status: Needs work » Needs review
StatusFileSize
new930 bytes
new4.28 KB

What?

drunken monkey’s picture

StatusFileSize
new972 bytes
new3.76 KB

Hm, OK, seems the cast doesn't really get us anything – Postgres refuses to multiply an integer with a float literal in any case, it doesn't even get to the surrounding cast.
This workaround should work in any case, though – no idea why I didn't think of that before.

The last submitted patch, 44: 2846932-44--debug.patch, failed testing.

drunken monkey’s picture

Looks pretty good in my local tests, but to avoid problems with very large numbers (as we already saw in a different issue) I guess we should only use that if necessary, and with as small a scaling factor as possible (maximum 1000, though – who really needs more than three digits of precision there?).

borisson_’s picture

Tests are green, I like the workaroud. It's not perfect but does sound like a good idea. Should we protect against too large numbers by using max(PHP_MAX_INT, pow(10, min(3, $after_point_digits)));?

If you don't think that's needed, we can commit this I think.

drunken monkey’s picture

StatusFileSize
new4.22 KB

Needs a re-roll. Some lunatic committed a patch changing all arrays to short syntax …

No, such a check is definitely not necessary, especially for the line you quote – that's at most 1000. And the total multiplier can't be much larger. The only integer overflow that might happen is in the database table, and I don't think we can really guard against that. The problem of the DBMS, I'd say. (At least until enough people complain.)

So, if this passes, I'll commit it for RC 2.

borisson_’s picture

Status: Needs review » Reviewed & tested by the community

Thanks for explaining! Looks good!

  • drunken monkey committed d9a68c4 on 8.x-1.x
    Issue #2846932 by drunken monkey, killua99, borisson_: Fixed error when...
drunken monkey’s picture

Status: Reviewed & tested by the community » Fixed

OK, good to hear.
Committed.
Thanks again for everyone's work here!

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.