Hello,
I’m seeing regular occurrences where I end up with extra folders in my database location with numbers after, for example:
ODF-PUSH-T-JU ODF-PUSH-T-JU_1079560404
These have some but not all of the files from the original database with the same name, crucially don’t have the inf.basex file and so prevent the /dba from listing databases.
What creates these folders?
* I have a job that runs each night to optimise all the databases that have changed that day, I think these are coming about due to a failure to optimise but I’m not sure, I’m trying to understand what might creates them to be created so I can look deeper into the cause.
Running 8.4 beta 5687c30 on Linux.
Many thanks for your help, James
Hi James,
As you already guessed, these folders are relics from OPTIMIZE ALL calls – or db:optimize($db, true) –, and they are removed after the operation if everything went alright.
Hope this helps, Christian
On Fri, Jan 8, 2016 at 12:25 PM, James Ball basex-talk@jamesball.co.uk wrote:
Hello,
I’m seeing regular occurrences where I end up with extra folders in my database location with numbers after, for example:
ODF-PUSH-T-JU ODF-PUSH-T-JU_1079560404
These have some but not all of the files from the original database with the same name, crucially don’t have the inf.basex file and so prevent the /dba from listing databases.
What creates these folders?
- I have a job that runs each night to optimise all the databases that have changed that day, I think these are coming about due to a failure to optimise but I’m not sure, I’m trying to understand what might creates them to be created so I can look deeper into the cause.
Running 8.4 beta 5687c30 on Linux.
Many thanks for your help, James
Hi Christian,
Thank you for that - so I’m looking in the right area.
As you already guessed, these folders are relics from OPTIMIZE ALL calls – or db:optimize($db, true) –, and they are removed after the operation if everything went alright.
So another question - these seem to come about from running OPTIMIZE ALL on databases with UPDINDEX TRUE. Is there any reason that OPTIMIZE ALL can’t be run on such databases?
(I’m running it because these databases become huge after lots of files are added)
Many thanks, James
So another question - these seem to come about from running OPTIMIZE ALL on databases with UPDINDEX TRUE. Is there any reason that OPTIMIZE ALL can’t be run on such databases?
It shouldn’t cause any problems. Here is a little command scripts that demonstrates that it should work alright:
<commands> <set option='updindex'>true</set> <create-db name='db'/> <add path='a.xml'><a/></add> <add path='b.xml'><b/></add> <optimize-all/> <info-db/> <close/> </commands>
Can you also reproduce the behavior with smaller data sets?
Thanks, Christian
Christian,
I can’t reliably reproduce the behaviour at all at the moment but I’m losing a fair number of databases to the problem. I first thought it was related to running out of disk space mid-operation but I don’t think that’s the cause. I just wanted to check that I wasn’t doing anything I shouldn’t before I try to find a reproducible error.
UPDINDEX is really useful for my use case but seems to leave me with database in GB range until I OPTIMIZE them. I shall continue monitoring.
Many thanks, James
On 8 Jan 2016, at 19:00, Christian Grün christian.gruen@gmail.com wrote:
So another question - these seem to come about from running OPTIMIZE ALL on databases with UPDINDEX TRUE. Is there any reason that OPTIMIZE ALL can’t be run on such databases?
It shouldn’t cause any problems. Here is a little command scripts that demonstrates that it should work alright:
<commands> <set option='updindex'>true</set> <create-db name='db'/> <add path='a.xml'><a/></add> <add path='b.xml'><b/></add> <optimize-all/> <info-db/> <close/> </commands>
Can you also reproduce the behavior with smaller data sets?
Thanks, Christian
So this happened again overnight but I have more information now.
1) I had switched off UPDINDEX on some of the databases and one of these was affected so this is not an UPDINDEX issue 2) I found that there was a hard crash of the JVM - which would account for why many databases were left locked and one had the folder with the numbers after 3) My Upstart script restarts the server quite happily which disguises 2) if you’re not looking closely!
The hard crash was due to the JVM not being able to allocate memory. What I’m not sure is why my script to OPTIMIZE all the databases seems to run quite happily manually but is failing when run by cron overnight.
I will continue monitoring.
Regards, James
On 8 Jan 2016, at 19:11, James Ball basex-talk@jamesball.co.uk wrote:
Christian,
I can’t reliably reproduce the behaviour at all at the moment but I’m losing a fair number of databases to the problem. I first thought it was related to running out of disk space mid-operation but I don’t think that’s the cause. I just wanted to check that I wasn’t doing anything I shouldn’t before I try to find a reproducible error.
UPDINDEX is really useful for my use case but seems to leave me with database in GB range until I OPTIMIZE them. I shall continue monitoring.
Many thanks, James
On 8 Jan 2016, at 19:00, Christian Grün christian.gruen@gmail.com wrote:
So another question - these seem to come about from running OPTIMIZE ALL on databases with UPDINDEX TRUE. Is there any reason that OPTIMIZE ALL can’t be run on such databases?
It shouldn’t cause any problems. Here is a little command scripts that demonstrates that it should work alright:
<commands> <set option='updindex'>true</set> <create-db name='db'/> <add path='a.xml'><a/></add> <add path='b.xml'><b/></add> <optimize-all/> <info-db/> <close/> </commands>
Can you also reproduce the behavior with smaller data sets?
Thanks, Christian
basex-talk@mailman.uni-konstanz.de
-
Christian Grün -
James Ball