Decoding of special characters seems to fail during Bibtex import

I ran into the exact same issue as goetz. We are migrating one site from OpenScholar (with Biblio) to D8 with Bibcite, as well as another D7 site with the normal version of Biblio.

Looking into this issue, I never realized it was so complicated.

BibTeX has all of these character substitutions with the nested curly brace and backslash notation. It does this so that it can display the accented characters, but also allow for sorting by the non-accented character.
http://www.bibtex.org/Format/

{\"a} {\^e} {\`i} {\.I} {\o} {\'u} {\aa} {\c c} {\u g} {\l} {\~n} {\H o} {\v r} {\ss} {\r u}
is the substitution for:
ä ê ì İ Ø ú å ç ğ ł ñ ő ř ß ů

I ran across one reference, which also has LaTeX notation for mathematical symbols.
https://journals.aps.org/prb/abstract/10.1103/PhysRevB.79.195208
title = {$p$-type ${\text{Bi}}_{2}{\text{Se}}_{3}$ for topological insulator and low-temperature thermoelectric applications},

To further complicate things, the OpenScholar Drupal distro has a WYSIWYG field for their reference title. In that example above, they render the chemical formula numbers with HTML sub tags, but OpenScholar exports the BibTeX with no special formatting.
title = {p-Type Bi2Se3 for Topological Insulator and Low-Temperature Thermoelectric Applications},

Yet another reference had special punctuation (¡Viva la mitochondria!) in addition to the special characters.
https://molbio.princeton.edu/publications/viva-la-mitochondria-harnessin...
title = {{\textexclamdown}Viva la mitochondria!: harnessing yeast mitochondria for chemical production.},
author = {Duran, Lisset and L{\'o}pez, Jos{\'e} Monta{\~n}o and Avalos, Jos{\'e} L}

However, a different source for the same reference used the Unicode characters in the BibTeX export, which I am assuming is fine, but it might not have the sorting advantage.
https://academic.oup.com/femsyr/article-abstract/20/6/foaa037/5863938
title = "{¡Viva la mitochondria!: harnessing yeast mitochondria for chemical production}",
author = {Duran, Lisset and López, José Montaño and Avalos, José L},

It does look like the Biblio module had to solve the same problem years ago.
https://www.drupal.org/project/biblio/issues/183517

In that Drupal issue thread, soxofan made a good point:

The proper way should be IMHO:

• on import/insert: recode everything to unicode ("é")
• on export: encode in the proper format: "\'e" for bibtex export, "é" for html rendering and xml export

I could not find anything better than to copy the solution from Biblio module.

We have hundreds of references with LaTeX notation for mathematical symbols. So a working import would be nice :)

I tried the patch but it doesn't work.

The patch in #4 works in my limited testing. It fixed the import of the following entry, which previously showed the literal \textquoteright

@article {sullivan24ToH,
	title = {Comparing the Perceived Intensity of Vibrotactile Cues Scaled Based on Inherent Dynamic Range},
	journal = {IEEE Transactions on Haptics},
	volume = {17},
	number = {1},
	year = {2024},
	pages = {45-51},
	keywords = {Actuators, Frequency modulation, Frequency response, Haptic interfaces, psychometric testing, Resonant frequency, Vibrations, wearable devices, Wrist},
	doi = {10.1109/TOH.2024.3355203},
	author = {Sullivan, Daziyah H. and Chase, Elyse D. Z. and O{\textquoteright}Malley, Marcia K.}
}

Am I confused or has this issue been fixed on the 8.x-1.x branch but not the 3.0.x branch?

Sorry: I was confused.

There is an issue fork and a branch for this issue, but no one has committed the patch to the branch. So the HEAD of the branch is the same as 8.x-1.x, which led to my confusion.

I created merge requests for the 8.x-1.x and 3.0.x branches. In both cases, the patch from Comment #4 applied cleanly.

At work, we use that patch with version 3.0.1, and no one has complained. That is less conscientious testing than I normally do.

I am changing this issue to target the 3.0.x branch, since I assume the maintainers will follow the usual practice of fixing an issue first in the branch for current development, then in the legacy branch.

On each branch,

• The initial commit applies the patch from Comment #4.
• The second commit removes check_plain(), which is left over from Drupal 7. Sanitization is not needed in exception messages, so I did not replace it with Html::escape(). PHPStan caught the error.
• The third commit fixes a test.

Of course, you should be suspicious of a commit that "fixes" a test. In this case, it is the right thing to do. Since # is a special character (parameter placeholder) in TeX, it has to be escaped when encoding (i.e., exporting to a BibTeX file). Furthermore, BibtexCaseDecodeTest tests the other direction: starting with the BibTeX file, create a PHP array (and then json_encode() it).

Of course, you should be suspicious of a commit that "fixes" a test. In this case, it is the right thing to do. Since # is a special character (parameter placeholder) in TeX, it has to be escaped when encoding (i.e., exporting to a BibTeX file).

Thanks for the clarification.

Setting this to "Needs work," as I would like to add minimal test coverage for a subset of special characters. If I have time, I'll add the tests myself, but leaving this un-assigned for now.

Test coverage added!