I've been looking forward to the new release because I really want to use an XQuery Fulltext system for a search engine for a reasonably large (200,000 printed page equivalent) XML structured text-oriented database. However, my initial test, on a large fraction of the total database (certainly over 120,000 pages printed equivalent) yields disappointing results.
The XML schema is roughly based on Docbook, so it is not unusual. Most of the actual text is within PARA elements.
My test query was:
//para[. ftcontains "sumter"]
On a dual quad Xeon system the query took about 28 seconds to run. This is completely unusable in my context.
I am pretty sure I know the reason for the poor performance. The derived query plan resolves to:
<QueryPlan> <IterPath> <Root/> <IterStep axis="descendant" test="*:para"> <FTContains> <Context/> <FTWords>sumter</FTWords> </FTContains> </IterStep> </IterPath> </QueryPlan>
Now, there are *millions* of PARA elements in the database - but not so many (hundreds) of references to the word "sumter". And this is by no means an uncommon XML structure for a text-oriented system.
The problem is obvious. There are an absolutely enormous number of PARA elements and very few actual text hits. The IterStep is evidently hitting every single PARA element. The query optimization should be checking the fulltext search index and using those results as an initial filter. Then the IterStep over PARAs from that filtered set would be highly productive.
Is there a way to get that kind of (properly) optimized query from the system, or is this hopelessly built into the current architecture?
I do hope that I get some kind of response this time, because at least two previous queries on earlier releases of BaseX went unanswered.