Add import setting to prevent duplicate references from being imported

I too have this use case for a client, and plus 1 for this feature request. However, @lorisbel having issues with the above patch, have yet to dig in too deep, but the following seems to be erroring for me in ImportForm.php:

'bibcite_import_batch_callback', [$chunk, $format, $options],

Specifically, passing the "$options". When it is passed, even if the "skip_duplicates" is empty, I receive "Entities have not been found. Please check file and format." error, even though entities are in fact created.

Should be able to look deeper next week and have more information.

@lorisbel, Thanks for the update, was just about to dig in deeper when I saw your update!

Retested your new patch locally with my use case (BibTeX format, and DOI duplicate policy), and seems to be working as expected. I did have one minor issue I needed to change, around the "!=" operator in the checkDuplicates function. I found that I needed the operator to be "!==" vs. "!=", in order for checks to come out correctly.

I believe this is because it is reading "$options['skip_duplicates']['skip_by_doi']" as a string, not an integer. Per https://www.php.net/manual/en/language.operators.comparison.php I think the "!==" takes into account the different types (int vs. string), hence why it then works for me.

I went ahead and created a patch with the above, as well as started adding some comments and slight reformatting to get closer to coding standards, such as renaming "checkDuplicates" to "bibcite_import_check_duplicates", and a couple of other minor things. Please let me know if any issues with that work.

Thanks for all the work on this, would be a great feature to get committed in my opinion.

Thank you for addressing this need! I approached the problem by adding a 'UniqueField' constraint on the bibcite_doi field. I use it as a module to add after bibcite is installed. The nice thing is we get validation and error messaging very easily. I put the main files in a gist:

https://gist.github.com/slattery/fe49761a85ae2c39e322f84e9733b5b3

This is really a rough start. You put a lot more thought into the user experience. If I were to advance this I would need to do a lot more.

I reviewed changes. I think that users don't need to check options each time on import. And it should be global module settings.
Also what do you think about options to update existed entities not just skip it? https://www.drupal.org/project/bibcite/issues/2890092
This issue won't be included in next release.
We have some ideas about implementation this function.

Obviously you can use patch as a temporary solution until it will be finished.

This is a great patch, but I'm having one small issue with it that I haven't been able to figure out on my own.

I have a user that wants to delete a reference and then re-import it and even if the reference was deleted, this still says there is a duplicate entry found and won't add a new one.

I thought maybe that it was just a cached entry causing issues, but issue persists even if I do "Configuration -> Performance -> Clear All Caches" and then try to import it after deleting.

Open to any thoughts or solutions.

I could also use this feature on a project, +1. I agree that checking for duplicates should be a global module setting, and it would be nice to also have the option to update existing entities instead of just skipping them. That way the single source of truth is not confused.

+      $message = [
+        t('Entry has been skipped due to duplicate policy.'),
+        $entry['title'],
+      ];
+      \Drupal::logger('bibcite_import')
+        ->error(implode("\n", $message));

This part should be simplified as follows:

\Drupal::logger('bibcite_import')
  ->error(t('Entry has been skipped due to duplicate policy: @title', [
    '@title' => $entry['title'],
  ]);

Taking a crack at this.

The idea of using a simple unique field constraint was appealing. Based on some research, it looked like relying on DOI only could be unreliable, and so I looked into using PMID and PMCID as backups. Adding constraints to those fields resulted in error messages one might find confusing; three errors could result from a single import.

This is how I ended up with the approach taken in the merge request.

GPT-5.3-Codex and GPT-5.4 were used in this.

Hiding the patches as I found they could not be applied to 3.0.x, but if you want to use a patch for testing you can look in the merge request under the “code” button.

Heyo! Maintainer weighing in here. I'm looking for some more reasons to be convinced that expecting people to **add** a unique "bibcite_doi" value to prevent duplication is really the best way to go.

I'm hesitant to think this is the best way to go because:
(A) Sure, it'll work if someone *knows* they need to add a unique "bibcite_doi" value, but it won't for people who don't know this
(B) And even if they do this, it requires additional work

I'm wondering if it would be preferable to use a checksum methodology, whereby the actual contents of the bibcite entity generate a unique hash that can then be compared to existing hashes. This would mean that we would want an update hook to populate that checksum for existing entities on sites.

Thoughts?

After spending a bunch of time on this (it took a few iterations to land here), my knee-jerk reaction it “no, this is absolutely the right approach!”

But am trying not to have my judgement clouded.

I believe that DOI, PMID, and PMCID are common values to be included; at least, for the faculty publications at our university, they are. So I thought this would be a simple method of catching duplicate items.

I’m also biased against the notion of an update hook to populate the checksums of all existing publications, as our platform hosts thousands of sites, and while they don’t all have publications, there are plenty. We have gotten in situations where update hooks have been run incompletely, and had to do quite a bit of detective work to rectify the situation.

I will confer with our platform architect about this; he’s also one of your co-maintainers…

Mark, coming at this from a different view, is there a reason you can think that someone would want to have multiple records with the same DOI, PMID, or PMCID value? Those are supposed to be unique identifiers.

It's also being added as an optional opt-in, and the unique constraint still allows references to have an empty value for any of those fields (e.g., this won't prevent saving multiple references with empty DOI values).

I'm not to keen on the checksum approach. I think that for most users, the idea is to prevent them from accidentally adding a record that already exists for the same DOI. Think of this scenario:

1. Someone imports a reference a month ago and modifies a field value
2. Someone else imports the same reference today
3. The checksum uniqueness comparison would allow it because they aren't identical records, due to the value change in step 1.

But in this scenario, I think we DO want to stop the import.

I reviewed this and made two fixes. One which should fix the one failing test, and another to resolve issue where HTML in the violation message was being escaped in the error message on import.

Given Mark's concerns, I won't commit this until we get his buy-in, but I have no concerns with this approach.

Thanks for the perspectives, @byronveale and @bkosborne! As I originally framed it, I was looking for "reasons to be convinced" that using the DOI and variants as a proxy for uniqueness was sufficient!

I'm fine going with proxy value uniqueness as opposed to the checksums -- I agree that DOI, PMID, and PMCID should be commonplace, even if it won't cover every entry like the checksum would, and I'd clarify that my original vision of the checksum is that it would be calculated from the data alone, so the import date wouldn't factor in and Brian's scenario in #20 above wouldn't be a problem.

So, assuming we go with proxy value uniqueness as currently staged, let's make sure we consider an implication that I don't yet see explicitly discussed in the writeup above:

I think if someone is doing a bulk re-import, it is quite likely (i.e., not just a theoretical edge case) that they may have edited some of the entries and expect those to be **updated** on import, not **skipped** (as the current business logic would do). In this scenario, using the 'bibcite_doi', 'bibcite_pmid', 'bibcite_pmcid' to identify "uniqueness" would prevent the record from being updated.

Of course, if we went the checksum route, it would also result in a probably equally unexpected/undesirable result of creating multiple records for the "same" reference, rather than updating the existing entry.

So perhaps we turn the new setting into radios for "Import all references, including potential duplicates"/"Do not update existing duplicate references"/"Replace existing duplicate references"? (This is similar to how Feeds Import module is configurable.)

Relatedly, it does occur to me that another way of solving the original "problem" would be to provide a way to de-duplicate imported records. Bibcite already includes this for authors, from #2860034: Ability to Merge Duplicate Contributor Entries After Import. Adding the ability to do reference deduplication is not mutually exclusive of trying to prevent duplicates on import, of course, so maybe that's just a separate enhancement issue?

Hi Mark, thanks for the feedback!

I think if someone is doing a bulk re-import, it is quite likely (i.e., not just a theoretical edge case) that they may have edited some of the entries and expect those to be **updated** on import, not **skipped**

Agreed that's a valid use case. To keep the scope of this issue narrow, I think it would be best to build on the solution we have here in #2890092: Option to update existing references instead of creating new ones and address the ability to overwrite/update records there. Especially given we aren't going to breaking anyones workflow with this change as it's a config option that's disabled by default.

I think it would be best to build on the solution we have here in #2890092 and address the ability to overwrite/update records there.

That sounds like a good way to keep concerns separate. Consider me "bought-in" to this MR! :)

Thank you, Mark!