Hi Andy,
Am 06.04.2015 um 13:13 schrieb Andy Bunce:
I have compared the timings for databases created from sources with and without the -s option. In principle these should provide very similar results.
xmlgen /f 0.2 /o folder/auction
xmlgen /f 0.2 /o folder/auction /s 400
In the first case a single file auction is created. In the second 35 files (auction00000... auction00034) are created. In both cases the file or files are loaded to a single database to query. In most cases the query performance is very similar. The exception is q09.xq. This appears to be 2 or 3 orders of magnitude slower against the database created from the split sources.
The actual query result seems to be the same. I have seen the same performance effect for factors f= 0.2, 0.5, 1 but not the somewhat trival f=0.
Any idea what is happening here?
this is a case of heuristics not quite working out every time. The query working on a single file is so fast because it is rewritten to two nested index lookups:
for $p in db:open-pre("auction_full",0)/site/people/person let $a := for $t in db:attribute("auction_full", $p/@id) /self::person/parent::buyer/parent::closed_auction return element item { db:attribute("auction_full", $t/itemref/@item)/self::id/parent::item[ parent::europe/parent::regions/parent::site/parent::document-node() ]/name/text() } return element person { attribute name { $p/name/text() }, $a }
This rewriting only works if `$ca` as well as `$ei` are inlined into their respective `for` loops (which are then rewritten to XPath expressions and finally index lookups).
Since the expressions bound to those variables are not constants at compile time, inlining them into a loop (in this case `for $p in $auction/...`) will initially duplicate work, so it is not generally safe to do. Because of exactly those index rewritings we are talking about here, we still want to inline *cheap* axis paths. The cheapness is determined in the `Path#cheap()` method [1]. Currently it only allows for a single document node as root node, which is the cause of the differences you are seeing.
While every heuristic can be tweaked (and I have no opinion on this specific one), there will always be cases like this where small changes lead to unexpectedly big differences in running time. The alternative is to just be consistently slow ;-).
Hope that helps, Leo
[1] https://github.com/BaseXdb/basex/blob/20dbe4c/basex-core/src/main/java/org/b...