Hi Eliot,
The setting is probably too complex to give a short and satisfying answer. If you manage to get this reproduced with some test data (or a database that you can share with us), and if you could tell us step by step what do do, that would just be fine.
To avoid overlooking obvious flaws in the version 11 branch, I’ve just created a 1gb XMark database instance [1]. It took around 2 minutes to fully optimize this instance with db:optimize('1gb', true()) and all index structures enabled, and less than 1 minute without the full-text index.
Cheers, Christian
[1] https://projects.cwi.nl/xmark/downloads.html
On Sun, Mar 3, 2024 at 4:00 PM Eliot Kimber eliot.kimber@servicenow.com wrote:
I’m continuing to test my process for loading data which depends on optimizing databases, some of which are pretty large (100+MB with 100s of 1000s of elements). I’m testing on both macOS and linux using the 27 Feb build on macOS and the 29 build on linux (just a matter of when I downloaded them).
What I’m seeing is that when I test with a relatively small content set the optimization completes reliably and everything works as it should.
When I test with a realistically large data set, the optimization either takes a very long time (as much as an hour to complete) or never completes with the server at 100% CPU utilization. It seems to be worse on macOS but it’s difficult to verify, partly because a test takes several hours to run.
I have the BaseX source code available locally, although I’m unable to compile with maven due to internal maven issues (we have a pretty locked down maven proxy and I don’t know maven well enough to know how I might configure around that).
Is there anything I can do to diagnose this issue to at least confirm or deny that there are still deadlock issues with the optimization?
I assume that it should not take 10s of minutes to optimize even a large database.
Here's the details for a typical database that has failed to optimize on macOS:
Thanks,
Eliot
*Eliot Kimber*
Sr Staff Content Engineer
O: 512 554 9368
M: 512 554 9368
servicenow.com https://www.servicenow.com
LinkedIn https://www.linkedin.com/company/servicenow | Twitter https://twitter.com/servicenow | YouTube https://www.youtube.com/user/servicenowinc | Facebook https://www.facebook.com/servicenow
