Any query requiring hours to complete seems like you must be doing something very expensive, which usually means you need to build some intermediate indexes to support your query requirements or you have an inefficient algorithm that can be improved.
But the fact that you aren’t seeing the processor pegged suggests something else is going on. I have some long-running queries but the processor serving the BaseX instance is definitely at 100% while that query runs.
Can you post one your long-running queries or at least describe the kind of processing you’re doing?
Have you used the BaseX profiling module to see where time is being taken? That should allow you to isolate the time-takers and start to work out how to optimize them.
Cheers,
Eliot _____________________________________________ Eliot Kimber Sr Staff Content Engineer O: 512 554 9368 M: 512 554 9368 servicenow.comhttps://www.servicenow.com LinkedInhttps://www.linkedin.com/company/servicenow | Twitterhttps://twitter.com/servicenow | YouTubehttps://www.youtube.com/user/servicenowinc | Facebookhttps://www.facebook.com/servicenow
From: BaseX-Talk basex-talk-bounces@mailman.uni-konstanz.de on behalf of Darryl Pretto darryl.pretto@workday.com Date: Tuesday, April 12, 2022 at 2:11 PM To: basex-talk@mailman.uni-konstanz.de basex-talk@mailman.uni-konstanz.de Subject: [basex-talk] BaseX Query Performance [External Email]
Hi,
I have a BaseX 9.0 instance running on Jetty 9.4.8 on RHEL 7.9. I run many different types of queries on several thousand XML files, about 10gb total source files. These queries take several hours to complete. While I have been able to rewrite queries here and there to optimize them, I am wondering what is the best way to increase BaseX query performance overall. When I check the server CPU and memory neither of them have high utilization. The CPU use varies but mostly is at 12%. The memory use is similar. I have allocated 8gb of memory to Jetty via the Java options settings: -Xms8g -Xmx8g
Are there any other resources I can check or change to increase overall query speed?
Thanks,
Darryl