Problem/Motivation

The Tika extractor sets the locale to en_US.UTF-8 to avoid stripping UTF-8 multibyte characters. This probably works for most systems, however if that locale is not present the indexing fails - the extract function throws an exception Tika Exctractor is not available (which has a typo btw). AFAIK the C.UTF-8 locale is more likely to be present on a system, so would be a more reliable choice.

I've set the priority to Minor since I presume this is an unusual scenario: I'm using Lando on WSL (Windows Subsystem for Linux on Win10) with the Pantheon recipe. The only locales present on the container Lando builds are:

$ locale -a
C
C.UTF-8
POSIX

Steps to reproduce

  1. Use a bare minimum Linux install, or remove the en_US.UTF-8 locale.
  2. Create a file with UTF-8 characters in the filename, eg 'Drupal’s contributors.docx' (character 0x2019 apostrophe)
  3. Index using the Tika extractor

Proposed resolution

Change line 39 and 56 of /src/Plugin/search_api_attachments/TikaExtractor.php to use the C.UTF-8 locale:

    // setlocale(LC_CTYPE, 'en_US.UTF-8');
    setlocale(LC_CTYPE, 'C.UTF-8');

    // shell_exec("LANG=en_US.utf-8");
    shell_exec("LANG=c.utf-8");
Command icon Show commands

Start within a Git clone of the project using the version control instructions.

Or, if you do not have SSH keys set up on git.drupalcode.org:

Comments

gaddman created an issue. See original summary.

izus’s picture

hi,
thanks for this catch
we maybe can add a check for the locale before using it
please feel free to suggest a patch/MR for checking en_US.UTF-8, else C.UTF-8 before setting locale
the typo can also be corrected here
Thanks for any contribution

gaddman’s picture

Issue summary: View changes

gaddman’s picture

Status: Active » Needs review

Thanks Izus. I've created a merge request. A few notes:

  • The en_US locale is the preferred one, since it appears C.UTF-8 is not especially widespread.
  • The original code was setting the LANG variable inside a separate call to shell_exec, which AFAIK wouldn't achieve the desired result.
  • This bit of code referenced a now deleted comment at php.net (see archived copy). Reviewing that comment, and a related issue for Apache Solr, I've just noted in the code that it's to support encoded filenames (rather than commands).
  • All tested OK with filenames and file contents containing UTF-8 characters (specifically "’" (0x2019) and "ö" (0x00F6).
izus’s picture

ok
thanks for this
will keep the issue open if someone wants to review and RTBTC
will come back to merge it in few days.

  • izus committed a7dcab9 on 8.x-1.x authored by gaddman
    Issue #3255205: Indexing fails due to missing locale
    

  • gaddman authored 627c3b2 on 9.0.x
    by gaddman, izus,  Indexing fails due to missing locale. See #3255205
    
izus’s picture

Status: Needs review » Fixed

This is now merged in 8 and 9 branch
Thanks

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.