Since Solr 3.x there are a bunch of stemmers available (apart from SnowballFactory) that can be used as well. I added them to the schema generation configurable by the "Multingual settings" (admin/settings/apachesolr/confgen/multilingual)

Further I extended the stopwords lists for various languages.

Comments

  • mkalkbrenner committed d64acba on 6.x-3.x authored by gnindl
    Issue #2361393 by gnindl: updated default stopwords for different...

  • mkalkbrenner committed 8f5bca5 on 7.x-1.x authored by gnindl
    Issue #2361393 by gnindl: updated default stopwords for different...
mkalkbrenner’s picture

Status: Active » Needs review
StatusFileSize
new14.47 KB

I already committed the stopwords and created a new patch that only contains the logic.
I applied these modifications to the code:

  • Display the minimum required solr version for a stemmer in the selection form.
  • Removed fallback to Porter Stemmer because it will create Illegal configurations. Display an error instead.

Beside some more testing, these additional TODOs need to be implemented within function apachesolr_multilingual_get_stemming_filters():

  • Distinguish between solr versions. Not all filters of a chain are available in each solr version. See https://wiki.apache.org/solr/MultitermQueryAnalysis
  • Do we need to disable LowerCaseFilterFactory if the filters contain a language specific lower case filter factory?
mkalkbrenner’s picture

StatusFileSize
new14.85 KB

The logic removed the protected words from the default snowball stemmer. Here's a updated patch.

  • mkalkbrenner committed 15a3696 on 6.x-3.x
    Issue #2361393 by gnindl, mkalkbrenner: Stemmers supported in Solr 3.x...

  • mkalkbrenner committed 021a8e3 on 6.x-3.x
    Issue #2361393 by mkalkbrenner: use language fallback for stemmer...
mkalkbrenner’s picture

Status: Needs review » Needs work

I already committed the patches to git. But the TODOs of #3 are still open and require additional work.

  • mkalkbrenner committed a1dd3af on 6.x-3.x
    Issue #2361393 by mkalkbrenner: removed filters from solr 1.x chain that...

  • mkalkbrenner committed 8f5bca5 on 8.x-1.x authored by gnindl
    Issue #2361393 by gnindl: updated default stopwords for different...
mkalkbrenner’s picture

Status: Needs work » Closed (fixed)