Probably the list of available locales is not the same as the list of languages that can be stemmed. I understood the question was about tokenization and full-text indexing in particular and not locales in general.
Maybe I got it wrong, but I would still appreciate hints to technical docs about supported languages with stemming. What components are used for this?
Cheers Kristian K
25.05.2016 20:21 Christian Grün kirjutas:
Is it possible to add the list of supported values in the doc for LANGUAGE at: http://docs.basex.org/wiki/Options#Indexing.
The list depends on your local Java environment. You can get a list via:
declare namespace locale = "java:java.util.Locale"; (locale:getAvailableLocales() ! locale:getLanguage(.)) => distinct-values() => sort()
I have added this example to the documentation.
LANGUAGE
SignatureLANGUAGE [lang] Defaulten SummaryThe specified language will influence the way how an input text will be tokenized. This option is mainly important if tokens are to be stemmed, or if the tokenization of a language differs from Western languages. See Full-Text Index for more details.
Thanks!
-- France Baril Architecte documentaire / Documentation architect france.baril@architextus.com