Hey Christian,
Your response makes me think that perhaps the locking is not the bottle neck, and I am simply catching the threads at the wrong times, but the fact that the majority of them are blocked makes me think this may be a bigger issue.
No files in the RESTXQ folder are changing at runtime, and from looking at the code it appears that the result of the parsing should be cached unless the files change anyways, but that does not appear to be the case, at least from my analysis of the thread dumps. Could this possibly be a bug?
Should I be getting better throughput than 0.5 - 10 requests per second? Generally it is 0.5 - 3 requests per second for standard size projects (6000+ XML files), where we are performing queries for nodes by attribute index, and faster 10 - 15 requests per second for smaller projects, same type of queries. Can you think of a way to speed this up, perhaps moving the Table files directly into memory, to minimize disk accesses?
Thanks,
Jeremy
Hi Jeremy,
> The most common place for threads to be blocking is at
> org.basex.io.random.TableDiskAccess.read*. The threads are waiting for other
> threads to finish, since these method calls are synchronized. Is it possible
> to concurrently read from disk?
Concurrent reads are indeed supported, but the most lower-level
operations need to be synchronized in order to avoid two operations
changing the file offset pointer at the same time. It’s interesting to
hear that this could be a potential bottleneck for your use case, as
it hasn’t been of importance in our benchmarks and profilings so far.
> Another place threads appear to be blocking is while waiting for the lock to
> call org.basex.http.RestXqModules.cache (another synchronized method). It
> seems like this should only be called once to parse the modules, and then
> all requests afterwards should be retrieved from the cache, but from the
> thread dumps it appears threads are still re-parsing the xquery modules. Is
> this expected behavior?
With each RESTXQ call, the restxq directory will be parsed for
changes. If XQuery Modules are modified or added, they will be parsed
before the actual request is answered. – Usually, this check shouldn’t
represent any bottleneck, but we could think about an option that
disallows changes to the RESTXQ directory while the server is running.
– Do you have many files in your RESTXQ directory?
Hope this helps,
Christian