Hi All, I create a collection of 4.5 gb xml document. and process some Xquery in sequencial manner then it is taking adequate time to process all queries. But when I converting process to multi threading then it is taking more than 10 times of time. Please someone suggest what is the best way to process Xquey in multithreading model.
MY code is following : Thread thread = new Thread(new XqueryProcessor( new Context(),"db1")); Thread thread1 = new Thread(new XqueryProcessor( new Context(),"db2")); Thread thread2 = new Thread(new XqueryProcessor( new Context(),"db3"));
thread.start(); thread1.start(); thread2.start();
Xquery Processor has all queries which need to process. and db1, db2 ,db3 are basically same xml files three collection.
Hi Pushpendra Singh,
more information on [transaction management] can be found in the documentation. Especially have a look at the chapter Limitations. Are you running any updates? If you have problems with your specific use case, please give us more information on the queries used.
Kind regards from Lake Constance, Germany, Jens Erat
[transaction management]: http://docs.basex.org/wiki/Transaction_Management
Hi Pushpendra,
first of all, it’s highly advisable to only create one instance of the Context class and pass it on as reference. Regarding the slow down, I assume that your queries read a lot of data sequentially, and the concurrent query execution leads to competing disk operations, leading to random access patterns. This problem can to some extent be alleviated by switching to SSDs, but a cheaper and more straightforward alternative could be to optimize your queries first.
Best, Christian ___________________________
On Wed, Nov 13, 2013 at 7:38 AM, Pushpendra Singh Sengar pushpendra1412@gmail.com wrote:
Hi All, I create a collection of 4.5 gb xml document. and process some Xquery in sequencial manner then it is taking adequate time to process all queries. But when I converting process to multi threading then it is taking more than 10 times of time. Please someone suggest what is the best way to process Xquey in multithreading model.
MY code is following : Thread thread = new Thread(new XqueryProcessor( new Context(),"db1")); Thread thread1 = new Thread(new XqueryProcessor( new Context(),"db2")); Thread thread2 = new Thread(new XqueryProcessor( new Context(),"db3"));
thread.start(); thread1.start(); thread2.start();Xquery Processor has all queries which need to process. and db1, db2 ,db3 are basically same xml files three collection.
-- Thanks & Regards , Pushpendra Singh
BaseX-Talk mailing list BaseX-Talk@mailman.uni-konstanz.de https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk
Hi Christian, I tried it with single Context also still it is taking same time as with different context. Queries written in optimized manner.Disk operation is also high taking 10 to 20 mb reading per second. I will try with SSD. Could you please guide me to sharing data between queries.
On Wed, Nov 13, 2013 at 4:11 PM, Christian Grün christian.gruen@gmail.comwrote:
Hi Pushpendra,
first of all, it’s highly advisable to only create one instance of the Context class and pass it on as reference. Regarding the slow down, I assume that your queries read a lot of data sequentially, and the concurrent query execution leads to competing disk operations, leading to random access patterns. This problem can to some extent be alleviated by switching to SSDs, but a cheaper and more straightforward alternative could be to optimize your queries first.
Best, Christian ___________________________
On Wed, Nov 13, 2013 at 7:38 AM, Pushpendra Singh Sengar pushpendra1412@gmail.com wrote:
Hi All, I create a collection of 4.5 gb xml document. and process some Xquery in sequencial manner then it is taking adequate
time
to process all queries. But when I converting process to multi threading then it is taking more
than
10 times of time. Please someone suggest what is the best way to process Xquey in multithreading model.
MY code is following : Thread thread = new Thread(new XqueryProcessor( new Context(),"db1")); Thread thread1 = new Thread(new XqueryProcessor( new
Context(),"db2"));
Thread thread2 = new Thread(new XqueryProcessor( newContext(),"db3"));
thread.start(); thread1.start(); thread2.start();Xquery Processor has all queries which need to process. and db1, db2 ,db3 are basically same xml files three collection.
-- Thanks & Regards , Pushpendra Singh
BaseX-Talk mailing list BaseX-Talk@mailman.uni-konstanz.de https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk
I tried it with single Context also still it is taking same time as with different context.
Well, this is not about performance. Several context instances may lead to the corruption of your data if you e.g. decide to perform write operations. The JavaDoc may give you some hints [1]. Beside that, the Wiki article on transactions tell you how concurrent operations are handled by BaseX [2].
Queries written in optimized manner.Disk operation is also high taking 10 to 20 mb reading per second.
So this means that all the data you are scanning really needs to be looked at? And that there is no way to benefit from index structures?
Could you please guide me to sharing data between queries.
I’m not sure why you want to do that, or if it would really help. As XQuery is a rather complex language, you’d first need to define what parts of the data are relevant for all queries, etc. Maybe you could give us a minimized query example instead, which shows us what your queries do?
Christian
[1] http://docs.basex.org/javadoc/org/basex/core/Context.html [2] http://docs.basex.org/wiki/Transaction_Management
basex-talk@mailman.uni-konstanz.de
-
Christian Grün -
Jens Erat -
Pushpendra Singh Sengar