Hi James, as to my experience the approach works. I've tried to use N Slaves BaseX (DB only) instances coordinated by one Master BaseX including HTTP server. On the latter I implemented a RestXQ [1] receiving a HTTP request which was then turned into a XQuery and executed on the different backend instances through the client module [2]. I have been able to achieve a scalability of nearly 80% on some particular queries. The results were encouraging indeed but a few details related to parallel computing in general have to be considered. 1) If the query is memory only then you can exploit multi-cores to actually achieve parallelism. If the query has to heavily access DBs on disk (most of the cases actually are io-bound) then you should ensure that your distributed Slaves access different disks or reside on different PCs otherwise all the gain of parallel computation is lost by overhead on disk access. 2) Uploading of documents from a single source usually doesn't exploit data-parallelism because all the data has to be sequentialized through a distributor node first. So if you are able to generate the data from the beginning in a distributed way, this could improve the distribution step too. 3) Another thing to keep in mind is that you should try to perform as much of the work on the Slaves (for instance query and transformation) and limit the work on the coordinator to a bare minimum of computing and memory usage in order to avoid the coordinator to be flooded with the results produced by all your slaves. I achieved this by injecting functions as external variables in the XQ to be executed on the Slaves.
Hope this is useful for your work and I'm really looking forward to know how it will proceed! Regards, Marco.
[1] http://docs.basex.org/wiki/RESTXQ [2] http://docs.basex.org/wiki/Client_Module
On 03/01/2018 19:50, James Sears wrote:
In the past I hit a scalability limit with BaseX - a billion+ nodes kind of a made querying it a bit slower than I liked.
I thought I'd try and address this, so I’ve written some code and placed it in GitHub: https://github.com/jameshnsears/xqa-documentation
What I've done is proof of concept, that's all - no way "finished". I'm emailing the list in the hope that what I've done so far might generate some constructive criticism. Maybe my approach has potential, maybe it doesn't?
There are only four components so far, the first three are Docker containers:
an ActiveMQ instance
a load balancer
a shard
a command line client exists to load the XML, from file, into an
ActiveMQ queue.
The software requires close to zero configuration. For example, each shard you start will automatically receive XML from the load balancer. And the load balancer distributes XML so that each shard holds the same # of documents.
There's a Travis project associated with the above link - it shows how easy it is to run the software end to end.
So far my effort is all about ingesting the XML, before I move further I thought I'd canvass some feedback - so if anyone has any then please give it :-)
Thanks.