Dear Drupal guys,

We have an e-commerce powered by drupal. In admin users can import new products (which are node-based content-types) from Excel.

And after the new nodes were added, they should be indexed. While indexing the first imported node, we get the following error:

PDOException: SQLSTATE[23000]: Integrity constraint violation: 1062 Duplicate entry '25313-4' for key 'PRIMARY': INSERT INTO {search_api_item} (item_id, index_id, changed) VALUES (:db_insert_placeholder_0, :db_insert_placeholder_1, :db_insert_placeholder_2); Array ( [:db_insert_placeholder_0] => 25313 [:db_insert_placeholder_1] => 4 [:db_insert_placeholder_2] => 1 ) in SearchApiAbstractDataSourceController->trackItemInsert() (line 626 of /path/to/drupal/sites/all/modules/search_api/includes/datasource.inc)

To fix the issue, I updated trackItemInsert() in sites/all/modules/search_api/includes/datasource.inc like so:

  /**
   * {@inheritdoc}
   */
  public function trackItemInsert(array $item_ids, array $indexes) {
    if (!$this->table || $item_ids === array()) {
      return;
    }

    foreach ($indexes as $index) {
      $this->checkIndex($index);
    }

    // Since large amounts of items can overstrain the database, only add items
    // in chunks.
    foreach (array_chunk($item_ids, 1000) as $chunk) {
      $insert = db_insert($this->table)
        ->fields(array($this->itemIdColumn, $this->indexIdColumn, $this->changedColumn));
      foreach ($chunk as $item_id) {
        foreach ($indexes as $index) {
          
          $duplicate_check_query = "SELECT COUNT(*) column_exists
            FROM {search_api_item}
            WHERE item_id = {$item_id}
            AND index_id = {$index->id}";
          $skip_insert = db_query($duplicate_check_query)->fetchField();

          if (!$skip_insert) {
            $insert->values(array(
            $this->itemIdColumn => $item_id,
            $this->indexIdColumn => $index->id,
            $this->changedColumn => 1,
            ));
          }
        }
      }
      $insert->execute();
    }
  }

I know the query is not 100% drupalized, but you get the idea. So I'm wondering if it can be a candidate for an update in search_api module.

To more easily see the update, I've attached my commit in gitlab.

Thank you,
Mohammad

Support from Acquia helps fund testing for Drupal Acquia logo

Comments

smhnaji created an issue. See original summary.

drunken monkey’s picture

Version: 7.x-1.26 » 7.x-1.x-dev
Component: Database backend » Framework
Priority: Major » Normal
Status: Patch (to be ported) » Needs review
Issue tags: -duplicate entry INSERT, -Integrity constraint violation, -PDOException: SQLSTATE[23000]: Integrity constraint violation: 1062 Duplicate entry
FileSize
1.2 KB

Thanks a lot for reporting this problem!
Your changes would be easiest to see, though, by creating a patch file. Would be great if you could do it that way next time.

Anyways, we already made the same change in Drupal 8, so seems only reasonable to do the same here. I just copied and slightly adapted the code from Drupal 8 – see the attached patch, and please test/review!

PS: It seems you (like many others – it's really easy to misinterpret) are confused by the "Issue tags" field. As the guidelines state, they aren't meant for free text tags related to the issue, but only for specific categorization purposes, usually by module maintainers.
So, if you aren't sure your current usage is correct, please just leave the field empty.

Also, please read and follow the Status field documentation – “Active” would have been the correct status here, not “Patch (to be ported)”.

drunken monkey’s picture

Would you please test the patch, if possible? That would help me commit this to the project and fix the issue for everyone.

smhnaji’s picture

Thank you for the reply and fix.

I read about how to contribute, and how to gracefully propose a patch in Drupal community.

Also I compared the code I suggested and the query that you put in the patch file.

Anyway after the long time, while I have tried for some whole hours on that, I couldn't get to trackItemInsert() function to do the actual test.

I put a watchdog() with a unique log type right after trackItemInsert() function definition, with the hope that I can see the log type, but no luck.

What I did to reproduce the problem (in fact, to see the log):

  • Changed Search API Index providers from Database to Solr
  • Reindexed both Database and Solr based indexes (separately)
  • Used "Clear all indexed data" button and then "Index now" button
  • Used "Queue all items for indexing" without hitting "Clear all indexed data"
  • With the option "Index items immediately" checked, I updated a node, and then I searched, and the new title appeared correctly in the search results

Now please guide me through how can I get to the function @drunken-monkey.

Thank you

drunken monkey’s picture

The trackItemInsert() is triggered when you create a new node.
Also, you were the one who reported the original problem, so why not just do what you did then and see if it works with my patch applied (instead of your changes)? I myself don’t really know how to trigger the error you saw.

smhnaji’s picture

I put more than 10 hours dedicated to catch the problem and reproduce the bug, tested more than 5 other scenarios (some of them, took much time) to get to the

          if (isset($existing[$item_id])) {
            continue;
          }

code block, but it was not successful. I think it may probably be because of reindexing that we cannot get to the buggy situation.

I always did successfully reach just before and after if (isset($existing[$item_id])) block. The patch worked seamlessly every time.

I'm sure that the problem does exist, I resolved it, but I cannot get to it again.
Also am pretty sure that your patch will work well because we I got to the upper and down lines.

Anyway, I have no Idea now.

  • drunken monkey committed 9f95bb7 on 7.x-1.x
    Issue #3052798 by drunken monkey, smhnaji: Fixed errors on duplicate...
drunken monkey’s picture

Status: Needs review » Fixed

Alright, thanks a lot for putting so much effort into this!
I’m also quite confident the new code works, so let’s just leave it at that.
Committed. Thanks a lot again!

smhnaji’s picture

Status: Fixed » Needs review

You're welcome.

It feels nice to see commit of my first bug report has been applied in such a popular module :)

Thank you,
Have a nice time.

smhnaji’s picture

Status: Needs review » Fixed

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.