Hi,

I’m trying to use BaseX for linguistic queries on a TEI document containing annotated tokens (i.e. tei:w-elements with attributes). I’m specifically interested in distance queries that allow to search for combinations of token features within a given window (e.g. all nouns that have an adjective ending with ‘lein’ within a distance of 3.) Theoretically, this is rather easy to formulate with an XPath or XQuery expression, but performance is poor when the dataset gets a bit larger (in my case, I have a total of 190.000 tokens in my test document, attribute and text indexes created).

This is what I essentially try to do as a simple XPath:

declare default element namespace  "http://www.tei-c.org/ns/1.0";

//w[@type = "NN"][(subsequence(preceding::w, 1, 3), subsequence(following::w, 1, 3))/@type = "ADJA"]

Since tokens may be interwoven with markup, I have to use preceding::* or following::*

A simple XQuery returning all matches including their context would look like this:

declare default element namespace  "http://www.tei-c.org/ns/1.0";

let $window := 3

let $matches := //w[@type = "NN"]

return

for $m in $matches

let $pre := subsequence($m/preceding-sibling::w, 1, $window)

let $next := subsequence($m/following-sibling::w, 1, $window)

return

  if (($pre,$next)[@type = "ADJA"])

  then

    <conc>

      <pre>{$pre}</pre>

      <match>{$m}</match>

      <next>{$next}</next>

    </conc>

  else ()

With $matches being a sequence of ca. 50.000 elements, a FLOWR is a bit too costly, I fear; limiting $matches to ~ 1.000 items performs within 5600ms (returning 154 items), but performance decreases rapidly after that (not to speak about setting a larger distance) .

So, my question is: Is there a way to improve performance on operations like these (without resorting to changing the input document)?

I’d be glad to provide my dataset off list, if this helps.

Thanks,

Daniel