It would be great if you can explain me why the query and loop took so much time earlier and now it gets completed quickly.
In your original query...
transaction/* except (/transaction/traInfo)
...the second path expression was evaluated for each result of the first expression. While it would theoretically be possible for the query processor to cache the results of the second expression, it is difficult in practice to decide when this is reasonable. Beside that, you have a potentially large number of sets that need to be compared every time, resulting in max. n*n comparisons (or O(n²)). The following query (which is probably the one you chose?) will always be linear:
transaction/*[name() ne 'traInfo']
Apart from this I have one concern that in my application XQueries will be provided by end users. So, every time I wouldn't be able to change or optimize the query.
It is hardly possible to limit queries of end users to those that are fast enough to be processed in a given time. As an example, the following query will take hours or even days to compute, even if it looks that simple:
(1 to 10000000000)[. = 0]
However, you can limit evaluation time and memory consumption, as described here:
http://docs.basex.org/wiki/XQuery_Module#xquery:eval
does BaseX use any query optimizer or can you suggest me any external tool / lib for the same ?
BaseX would be pretty much worthless without query optimizer, so I don't quite get what you mean by that?
Christian