Add localized stopword lists

I would definitely like to add this capability, so if you could provide a german list, I'd be happy to add this functionality.

Thanks, I'm quite busy with another module I'm in the process of finishing soon, so I'll try to get this idea as soon as possible.

i'm interested in making this work in a Drupal-ish way. my suspicion is that we should follow these instructions to make the module "translatable", and then work out a way to include grab the list of stopwords from the relevant .po file, somehow. at very least, we should take care to integrate with the i18n mechanisms which are in core for D6, for providing language selection options, etc.

perhaps the porterstemmer module has tackled this problem already?

Agreed that the porterstemmer module be useful for making the related block more relevant at least in regards to the English language. I'm not sure that this process could be applied for multiple different languages.

Also, it seems from the list that nath provided that the usage of common words are somewhat similar, but not necessarily the same at least in terms of direct translation. This somewhat contradicts the system of the .pot files, as well as leaves the user unable to be able to specify more words that should be considered common.

To solve this, I suggest we implement somewhat of a locale specific list of common words, using the variable_get for common words.
This could be set up as an array with the key as the locale, and the list of common words similar to the current one.
Since we're doing only single words, this could be a string separated by spaces, which combined with an explode(' ',$variable), would provide the necessary array type to strip out the relevant words from whatever locale is currently selected. This would allow it to be integrated with modules such as the localizer module.

This would also allow a admin configurable method to edit the appropriate strings. This method could then use the .pot file to provide the default common string.

So I'm going to do it somewhat in a drupalish way, which is to have seperate files which are included on the basis of locale. Will have a commit sometime soon.

It's totally untested, including even for syntax errors. You'll need the files which should be stored within the updated module. these are located in the "ignore" directory, which will contain the word lists for the various languages.

If you have problems, please let me know immediately, so I can do something about it.

I've re-rolled this patch and committed it here: http://drupalcode.org/project/related_block.git/commit/8e987cb
This will effectively pull in a language-specific stopwords list to use when filtering out words to search for related content.

Also added the german list from http://drupal.org/node/191811#comment-629787 above, in this commit here: http://drupalcode.org/project/related_block.git/commit/8fc7dc1.

The process of adding new languages' stopwords is to create a file in the module's stopwords/ directory with the 2-letter prefix for the language in question, with a .inc suffix. The file should create a simple $words array containing all words which should be ignored by the related search algorithm. Thus related_block/stopwords/de.inc looks like this:

$words = array( '', 'aber', 'alle', 'als', 'auch', 'auf', 'aus', 'bei', 'beim', 'bis', 'brauchen', 'da', 'damit', 'dann', 'das', 'dass', 'dem', 'den', 'denn', 'der', 'des', 'die', 'dies', 'diese', 'doch', 'drei', 'durch', 'eigene', 'ein', 'eine', 'einem', 'einen', 'einer', 'es', 'gut', 'für', 'haben', 'hat', 'ich', 'ihnen', 'ihr', 'ihre', 'ihren', 'ihrer', 'im', 'ins', 'ist', 'kann', 'können', 'kommt', 'man', 'mit', 'müssen', 'muss', 'nach', 'neue', 'nicht', 'nur', 'oder', 'per', 'schon', 'sehr', 'seine', 'sich', 'sie', 'sind', 'so', 'sollten', 'sowie', 'über', 'und', 'unter', 'von', 'was', 'welche', 'wenn', 'werden', 'wird', 'wie', 'zum', 'zur');

Updating title to reflect what this feature actually adds. It is now possible to create a .inc file in the stopwords/ subfolder of this module which simply returns a list of "stopwords" which are ignored when comparing the text of the content to determine Relevance.

erf.