Hi Marc,
I can only agree that benchmarking is a complex issue. While it's usually easy to "prove" somehow that the own implementation is much faster than all others, it's much more difficult to do performance comparisons that do justice to all compared systems. Usually, a good question to start with is to first look for the components of a software framework could be potential bottlenecks. Next, it helps to have a real-life scenario in mind that can be simulated to some extent by the performance tests.
To get more specific and talk about BaseX: the GUI timings produced are pretty close to the timings of the web application architecture. By calling the basex standalone interpreter, most of the time will be spent for initializing the JVM, so the best solution here is to start a basexserver instance and run basexclient (it's reasonable to ignore the first results, because those will be affected by the JVM warmup time). The output produced by query can either be redirected to /dev/null or written to a file via the -o option; as long as it's not huge, it will make no big difference. I think it makes sense to only look at the total time, because the time for parsing, compiling etc. may change over different versions of BaseX.
All the best, Christian
On Sat, Dec 27, 2014 at 4:25 PM, Marc van Grootel marc.van.grootel@gmail.com wrote:
Hi,
For a few new blog posts I'm working on some examples where I compare traditional XQuery code with an Origami example.
I am not looking for scientific answers but I do want to be able to do a more or less fair comparison.
I noticed that when running code from the GUI timings can be quite different from running basex.
Currently I do something like basex -V example/foo.xq or basex -r10 -V example/foo.xq
What would be a honest way of including timings in an article. It's mainly for a qualitative assessment and comparing a traditional solution with an Origami solution. What to look for in the breakdown between parsing/compiling/evaluating and printing.
Tips?
Thanks, and I hope you are all having a good Xmas break.
--Marc