Hi, I've e-mailed before about this issue, but that was a while ago so I wanted to start a new thread.
I'm currently experiencing a situation where all calls (updating or otherwise) are being blocked by any updating call within the same database, even when GLOBALLOCK is set to false.
The setup is that we have a restxq endpoint which is checks if the requested content is loaded and if it isn't, we use a java library to fetch the content and then add it to the database. We return a restxq:forward, to the requested endpoint which thens processes the request, assuming the content exists.
For some reason however, even with GLOBALLOCK set to false,
Hi Joe,
concurrency in BaseX still has some [limitations]. Possibly one of these is triggering a global lock in your case. You can find out what locks a query is triggering by running it in the GUI and looking at the "Query Information“ pane or on the command line by using `SET QUERYINFO ON`.
If you’re unsure what’s triggering the global lock in your case, feel free to send the query to either the mailing list or directly to me. Could take some time for an answer though as I’m currently abroad.
Regards, Jens
[limitations]: http://docs.basex.org/wiki/Transaction_Management#Limitations
So the database name which is being updated is based on variables passed in to the updating query, so I believe that would cause it to lock everything.
Is there any way I can specify exactly what should be locked when going into an expression? Or better yet have basex lock nothing? From the documentation I thought that is what GLOBALLOCK would do, but it seems like even with that set to false it locks everything.
I'm worried having everything lock for every updating query is going to be a real performance bottleneck
On Sat, Nov 2, 2013 at 10:55 AM, Jens Erat jens.erat@uni-konstanz.dewrote:
Hi Joe,
concurrency in BaseX still has some [limitations]. Possibly one of these is triggering a global lock in your case. You can find out what locks a query is triggering by running it in the GUI and looking at the "Query Information“ pane or on the command line by using `SET QUERYINFO ON`.
If you’re unsure what’s triggering the global lock in your case, feel free to send the query to either the mailing list or directly to me. Could take some time for an answer though as I’m currently abroad.
Regards, Jens
-- Jens Erat
PGP: 350E D9B6 9ADC 2DED F5F2 8549 CBC2 613C D745 722BAm 01.11.2013 um 11:49 schrieb Joe Templeman joe@inkling.com:
Hi, I've e-mailed before about this issue, but that was a while ago so I
wanted to start a new thread.
I'm currently experiencing a situation where all calls (updating or
otherwise) are being blocked by any updating call within the same database, even when GLOBALLOCK is set to false.
The setup is that we have a restxq endpoint which is checks if the
requested content is loaded and if it isn't, we use a java library to fetch the content and then add it to the database. We return a restxq:forward, to the requested endpoint which thens processes the request, assuming the content exists.
For some reason however, even with GLOBALLOCK set to false, _______________________________________________ BaseX-Talk mailing list BaseX-Talk@mailman.uni-konstanz.de https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk
Hi Joe,
Is there any way I can specify exactly what should be locked when going into an expression? Or better yet have basex lock nothing?
Unfortunately, the locking algorithms cannot be completely disabled, but we could think about providing such a mode. XQuery Locking Options [1] could then be used to manually control all lock operations. I’ll have some discussions about that with Jens.
From the documentation I thought that is what GLOBALLOCK would do, but it seems like even with that set to false it locks everything.
The documentation was a bit updated; I’ve updated the corresponding paragraph.
I'm worried having everything lock for every updating query is going to be a real performance bottleneck
Have you already tested if this is a real bottleneck?
Best, Christian
[1] http://docs.basex.org/wiki/Transaction_Management#XQuery_Locking_Options
Thank you for your reply. Unfortunately yes, this will be a serious performance issue, we will be serving 50+ requests per second from basex and to have all requests blocked for 1-2 seconds when a request requires content to be loaded into to the database is not acceptable. I would estimate around 10% of requests will require content to be added.
For the custom locking you linked to, do those lock names have to be declared in advance? Again, since we don't know the name of the database until the request comes in, we need a locking system which can lock on variable names. Currently we're using our own java library for this function.
I'm going to try and build my own version of basex with the option to turn off all locking. If I get something acceptable I will send a pull request for it to be merged in.
On Wed, Nov 6, 2013 at 11:13 AM, Christian Grün christian.gruen@gmail.comwrote:
Hi Joe,
Is there any way I can specify exactly what should be locked when going
into
an expression? Or better yet have basex lock nothing?
Unfortunately, the locking algorithms cannot be completely disabled, but we could think about providing such a mode. XQuery Locking Options [1] could then be used to manually control all lock operations. I’ll have some discussions about that with Jens.
From the documentation I thought that is what GLOBALLOCK would do, but it seems like even with
that
set to false it locks everything.
The documentation was a bit updated; I’ve updated the corresponding paragraph.
I'm worried having everything lock for every updating query is going to
be a
real performance bottleneck
Have you already tested if this is a real bottleneck?
Best, Christian
[1] http://docs.basex.org/wiki/Transaction_Management#XQuery_Locking_Options
For the custom locking you linked to, do those lock names have to be declared in advance?
Yes, the lock names must be declared statically in the query, as they must be known at parse time (before the query is evaluated). Otherwise, we could as well lock a database once it is about to be updated, but this leads to race conditions and deadlocks.
I'm going to try and build my own version of basex with the option to turn off all locking. If I get something acceptable I will send a pull request for it to be merged in.
We are looking forward to your feedback and experiences. Christian
On Wed, Nov 6, 2013 at 11:13 AM, Christian Grün christian.gruen@gmail.com wrote:
Hi Joe,
Is there any way I can specify exactly what should be locked when going into an expression? Or better yet have basex lock nothing?
Unfortunately, the locking algorithms cannot be completely disabled, but we could think about providing such a mode. XQuery Locking Options [1] could then be used to manually control all lock operations. I’ll have some discussions about that with Jens.
From the documentation I thought that is what GLOBALLOCK would do, but it seems like even with that set to false it locks everything.
The documentation was a bit updated; I’ve updated the corresponding paragraph.
I'm worried having everything lock for every updating query is going to be a real performance bottleneck
Have you already tested if this is a real bottleneck?
Best, Christian
[1] http://docs.basex.org/wiki/Transaction_Management#XQuery_Locking_Options
Hi Joe,
I'm going to try and build my own version of basex with the option to turn off all locking. If I get something acceptable I will send a pull request for it to be merged in.
did you already have to look at the code?
My impression is that we could quite easily add a QUERYLOCKING that, when disabled, only considers manual locking flag, which can e.g. be seat via the query:read-lock and query:write-lock options. But I dont’ know if that would help, as you mentioned you’d have to choose the database dynamically.
You mentioned that you are expecting the database to be locked for 1-2 seconds. Maybe there is some chance to speed up your write operations?
Best, Christian
What I actually did was add a DISABLELOCKING option which when true will stop all locking in the DBLocking class, then I use a map of locks in my java library to grab a write lock for a given database (using the db name as the key) when doing any update. However there is one problem with this, when we release the lock at the end of the query, as I understand it the pending update list may not have been flushed. To fix this I'm going to have to release the lock after a server side forward, but is there any better way of doing this?
I'm planning on submitting a pull request for the DISABLELOCKING feature soon.
Thanks, Joe
On Tue, Nov 19, 2013 at 5:26 AM, Christian Grün christian.gruen@gmail.comwrote:
Hi Joe,
I'm going to try and build my own version of basex with the option to
turn
off all locking. If I get something acceptable I will send a pull
request
for it to be merged in.
did you already have to look at the code?
My impression is that we could quite easily add a QUERYLOCKING that, when disabled, only considers manual locking flag, which can e.g. be seat via the query:read-lock and query:write-lock options. But I dont’ know if that would help, as you mentioned you’d have to choose the database dynamically.
You mentioned that you are expecting the database to be locked for 1-2 seconds. Maybe there is some chance to speed up your write operations?
Best, Christian
What I actually did was add a DISABLELOCKING option which when true will stop all locking in the DBLocking class, then I use a map of locks in my java library to grab a write lock for a given database (using the db name as the key) when doing any update. However there is one problem with this, when we release the lock at the end of the query, as I understand it the pending update list may not have been flushed. To fix this I'm going to have to release the lock after a server side forward, but is there any better way of doing this?
…to say more, we’ll probably have to look into your implementation first.
Thanks for your feedback, Christian
I'm planning on submitting a pull request for the DISABLELOCKING feature soon.
Thanks, Joe
On Tue, Nov 19, 2013 at 5:26 AM, Christian Grün christian.gruen@gmail.com wrote:
Hi Joe,
I'm going to try and build my own version of basex with the option to turn off all locking. If I get something acceptable I will send a pull request for it to be merged in.
did you already have to look at the code?
My impression is that we could quite easily add a QUERYLOCKING that, when disabled, only considers manual locking flag, which can e.g. be seat via the query:read-lock and query:write-lock options. But I dont’ know if that would help, as you mentioned you’d have to choose the database dynamically.
You mentioned that you are expecting the database to be locked for 1-2 seconds. Maybe there is some chance to speed up your write operations?
Best, Christian
Hi all,
## Notes on Implementing Non-Locking
If implementing such non-locking I’d propose following:
- If documented at all, provide a large red warning message in the docs. Something about really do not use this feature if you don’t know the internals very well and voiding all support or something like that. - We already have some locking options. I’d recommend to use them, but disable the recognition of which databases to lock (thus do not call the visitor, I think it should be in `AQuery#databases`). Do not break locking for commands, they’re able to recognize what they need to lock very well. - Add some function like `boolean lockTest(String db)`. See if `db` is actually locked before opening it. There is one function always called in the `Data` class, this is a great point for such a hook, I used it when writing a test implementation of optimistic concurrency control for my [thesis]. If the lock is not set, fail early (throw some QueryException). Your XQuery developers (and database consistency) will be very thankful.
The current `DBLocking` implementation expects conservative locking, thus fetching all locks in the beginning of the transaction and will fail if you try to fetch more locks afterwards. If you fetch locks using the existing XQuery options you will not have any problems with that. We used that to prevent deadlocks to prevent failure in case of external side effects which occur rather often, but this resulted in rather bad parallelism for complex queries (which means —- sadly —- most relevant ones).
## Efforts on Further Improved Concurrency Control
We have some discussion on extending our locking for use cases which can deal with external side effects (thus, restarting the transaction in case of deadlocks), but this is a rather large project as it delves deep in the core _and_ large parts of the query processor, but this will definitely take some time and effort. The concurrency control part is not too complicated itself (and there is at least a somewhat working, yet rather hacky, concept implementation), but making BaseX capable of gracefully terminating and restarting transactions will take some effort.
Kind regards from Lake Constance, Jens
[thesis]: http://kops.ub.uni-konstanz.de/handle/urn:nbn:de:bsz:352-235049
basex-talk@mailman.uni-konstanz.de
-
Christian Grün -
Jens Erat -
Joe Templeman