Dear Drupal guys,
We have an e-commerce powered by drupal. In admin users can import new products (which are node-based content-types) from Excel.
And after the new nodes were added, they should be indexed. While indexing the first imported node, we get the following error:
PDOException: SQLSTATE[23000]: Integrity constraint violation: 1062 Duplicate entry '25313-4' for key 'PRIMARY': INSERT INTO {search_api_item} (item_id, index_id, changed) VALUES (:db_insert_placeholder_0, :db_insert_placeholder_1, :db_insert_placeholder_2); Array ( [:db_insert_placeholder_0] => 25313 [:db_insert_placeholder_1] => 4 [:db_insert_placeholder_2] => 1 ) in SearchApiAbstractDataSourceController->trackItemInsert() (line 626 of /path/to/drupal/sites/all/modules/search_api/includes/datasource.inc)
To fix the issue, I updated trackItemInsert() in sites/all/modules/search_api/includes/datasource.inc like so:
/**
* {@inheritdoc}
*/
public function trackItemInsert(array $item_ids, array $indexes) {
if (!$this->table || $item_ids === array()) {
return;
}
foreach ($indexes as $index) {
$this->checkIndex($index);
}
// Since large amounts of items can overstrain the database, only add items
// in chunks.
foreach (array_chunk($item_ids, 1000) as $chunk) {
$insert = db_insert($this->table)
->fields(array($this->itemIdColumn, $this->indexIdColumn, $this->changedColumn));
foreach ($chunk as $item_id) {
foreach ($indexes as $index) {
$duplicate_check_query = "SELECT COUNT(*) column_exists
FROM {search_api_item}
WHERE item_id = {$item_id}
AND index_id = {$index->id}";
$skip_insert = db_query($duplicate_check_query)->fetchField();
if (!$skip_insert) {
$insert->values(array(
$this->itemIdColumn => $item_id,
$this->indexIdColumn => $index->id,
$this->changedColumn => 1,
));
}
}
}
$insert->execute();
}
}
I know the query is not 100% drupalized, but you get the idea. So I'm wondering if it can be a candidate for an update in search_api module.
To more easily see the update, I've attached my commit in gitlab.
Thank you,
Mohammad
Comment | File | Size | Author |
---|---|---|---|
#2 | 3052798-2--ignore_duplicate_items_inserted.patch | 1.2 KB | drunken monkey |
Comments
Comment #2
drunken monkeyThanks a lot for reporting this problem!
Your changes would be easiest to see, though, by creating a patch file. Would be great if you could do it that way next time.
Anyways, we already made the same change in Drupal 8, so seems only reasonable to do the same here. I just copied and slightly adapted the code from Drupal 8 – see the attached patch, and please test/review!
PS: It seems you (like many others – it's really easy to misinterpret) are confused by the "Issue tags" field. As the guidelines state, they aren't meant for free text tags related to the issue, but only for specific categorization purposes, usually by module maintainers.
So, if you aren't sure your current usage is correct, please just leave the field empty.
Also, please read and follow the Status field documentation – “Active” would have been the correct status here, not “Patch (to be ported)”.
Comment #3
drunken monkeyWould you please test the patch, if possible? That would help me commit this to the project and fix the issue for everyone.
Comment #4
smhnaji CreditAttribution: smhnaji commentedThank you for the reply and fix.
I read about how to contribute, and how to gracefully propose a patch in Drupal community.
Also I compared the code I suggested and the query that you put in the patch file.
Anyway after the long time, while I have tried for some whole hours on that, I couldn't get to
trackItemInsert()
function to do the actual test.I put a
watchdog()
with a unique log type right aftertrackItemInsert()
function definition, with the hope that I can see the log type, but no luck.What I did to reproduce the problem (in fact, to see the log):
Now please guide me through how can I get to the function @drunken-monkey.
Thank you
Comment #5
drunken monkeyThe
trackItemInsert()
is triggered when you create a new node.Also, you were the one who reported the original problem, so why not just do what you did then and see if it works with my patch applied (instead of your changes)? I myself don’t really know how to trigger the error you saw.
Comment #6
smhnaji CreditAttribution: smhnaji commentedI put more than 10 hours dedicated to catch the problem and reproduce the bug, tested more than 5 other scenarios (some of them, took much time) to get to the
code block, but it was not successful. I think it may probably be because of reindexing that we cannot get to the buggy situation.
I always did successfully reach just before and after
if (isset($existing[$item_id]))
block. The patch worked seamlessly every time.I'm sure that the problem does exist, I resolved it, but I cannot get to it again.
Also am pretty sure that your patch will work well because we I got to the upper and down lines.
Anyway, I have no Idea now.
Comment #8
drunken monkeyAlright, thanks a lot for putting so much effort into this!
I’m also quite confident the new code works, so let’s just leave it at that.
Committed. Thanks a lot again!
Comment #9
smhnaji CreditAttribution: smhnaji commentedYou're welcome.
It feels nice to see commit of my first bug report has been applied in such a popular module :)
Thank you,
Have a nice time.
Comment #10
smhnaji CreditAttribution: smhnaji commented