Hello all, putting the PARALLEL flag to 1 solved the issue of inconsistency between the query + scheduling time and the actual operation completion time. Now we have it stabilized. Thanks. Peaks are now rarely possible when an update from the insertion client serializes itself with one or more queries or updates from another. I assume that this could be reduced as soon as we complete the implementation of the splitting of the data into two different DBs. A cold one which is growing (with scheduled batch ops) but indexed and UPDINDEX=true (for rare update operations that still might occur) and a hot one for realtime data which has no index but whose size will be reduced and kept constant. Lowering in this way the general access times should also reduce the impact of peaks. Do the last statements make sense to the experts and implementers?
BR, Marco.
On 24/02/2015 15:04, Christian Grün wrote:
Hi Marco,
There are two clients which read and write through RESTXQ to/from 2 different logical DBs on the same server. Is it possible that behaving concurrently we cause some sort of race condition that serializes operations for a long time period?
Yes, this is absolutely possible. I guess that your hard disk, or even SSD, needs to do many write operations at the same time, leading to a somewhat random access pattern.
You could try to set the PARALLEL flag to 1 and see how this effects the total performance!
Please help us out [...]
Hope this helps ;) Christian
because it's getting to deeply nested into BaseX's
internals for us to understand.
Thanks a lot and BR, Marco.
[1] https://github.com/BaseXdb/basex/blob/master/basex-core/src/main/java/org/ba... [2] public Iter iter() throws QueryException { compile();
long time = System.currentTimeMillis(); Util.errln("QUERY [MARCO]: Start query iter " + time); try { [...] final Updates updates = resources.updates; if(updates != null) { // if parent context exists, updates will be performed by maincontext Util.errln("QUERY [MARCO]: Starting updates " + (System.currentTimeMillis() - time)); if(qcParent == null) { [...] Util.errln("QUERY [MARCO]: Completed updates " + (System.currentTimeMillis() - time)); } } Util.errln("QUERY [MARCO]: Completed query" + (System.currentTimeMillis() - time)); return cache;
} catch(final StackOverflowError ex) { [...] }}
On 22/02/2015 22:08, Marco Lettere wrote:
I'm planning to add some logging here [1] and here [2]. Do you think it will unleash some details? M.
[1] https://github.com/BaseXdb/basex/blob/master/basex-api/src/main/java/org/bas... [2] https://github.com/BaseXdb/basex/blob/master/basex-core/src/main/java/org/ba...
On 22/02/2015 20:05, Christian Grün wrote:
I'd like just to understand to what relates the huge difference (d) between "querying the db and scheduling an update to the PUL" which is what the internal function does and the time logged by the basexhttp server as the completion time of the whole RestXQ call which is T + d.
Hm, difficult to tell. In all cases, the PUL will be executed before the result of a query will be sent to the client.