I have a database that shows having the expected 2 documents in it in the DBA databases list but when I go to the database in the DBA it shows 0 items.
I think this is because the database was being written to and something happened that left it in a bad state.
Is there any possibility of recovering this database so it is usable?
The practical problem is that it takes several hours to populate this database, so recovering it rather than attempting another multi-hour load would be nice.
Here’s what I see for files in the database’s directory on the server:
-rw-r--r-- 1 eliot.kimber.adm domain users 73039559 Apr 18 14:50 atv.basex -rw-r--r-- 1 eliot.kimber.adm domain users 7544382 Apr 18 14:52 atvl.basex -rw-r--r-- 1 eliot.kimber.adm domain users 2730980 Apr 18 14:52 atvr.basex -rw-r--r-- 1 eliot.kimber.adm domain users 236 Apr 18 04:17 ftxx.basex -rw-r--r-- 1 eliot.kimber.adm domain users 218072 Apr 18 04:17 ftxy.basex -rw-r--r-- 1 eliot.kimber.adm domain users 1603678 Apr 18 04:17 ftxz.basex -rw-r--r-- 1 eliot.kimber.adm domain users 15 Apr 18 04:17 idp.basex -rw-r--r-- 1 eliot.kimber.adm domain users 15193 Apr 18 04:17 inf.basex -rw-r--r-- 1 eliot.kimber.adm domain users 67 Apr 18 04:17 pth.basex -rw-r--r-- 1 eliot.kimber.adm domain users 28 Apr 18 04:17 swl.basex -rw-r--r-- 1 eliot.kimber.adm domain users 83701760 Apr 18 14:52 tbl.basex -rw-r--r-- 1 eliot.kimber.adm domain users 9 Apr 18 04:17 tbli.basex -rw-r--r-- 1 eliot.kimber.adm domain users 112380335 Apr 18 14:52 tokl.basex -rw-r--r-- 1 eliot.kimber.adm domain users 2829520 Apr 18 14:52 tokr.basex -rw-r--r-- 1 eliot.kimber.adm domain users 1718692 Apr 18 04:17 txt.basex -rw-r--r-- 1 eliot.kimber.adm domain users 351326 Apr 18 04:17 txtl.basex -rw-r--r-- 1 eliot.kimber.adm domain users 285610 Apr 18 02:38 txtr.basex -rw-r--r-- 1 eliot.kimber.adm domain users 0 Apr 18 04:18 upd.basex
In the Databases list I see:
_temp_lrk_yokohama_link_recordshttp://mirabel.corp.service-now.com:8984/dba/database?name=_temp_lrk_yokohama_link_records 2 274 MB 2025-04-18, 14:52:42
But then when I go to the database itself it just say “0 entries”.
Thanks,
Eliot
_____________________________________________ Eliot Kimber Sr. Staff Content Engineer O: 512 554 9368
servicenow
servicenow.comhttps://www.servicenow.com LinkedInhttps://www.linkedin.com/company/servicenow | Xhttps://twitter.com/servicenow | YouTubehttps://www.youtube.com/user/servicenowinc | Instagramhttps://www.instagram.com/servicenow
Hi Eliot,
That's not fun.
I see your directory has a ` upd.basex`you could try the suggestion at: https://docs.basex.org/main/Transaction_Management#update_operations
Maybe you get lucky. /Andy
On Fri, 18 Apr 2025 at 15:59, Eliot Kimber via BaseX-Talk < basex-talk@mailman.uni-konstanz.de> wrote:
I have a database that shows having the expected 2 documents in it in the DBA databases list but when I go to the database in the DBA it shows 0 items.
I think this is because the database was being written to and something happened that left it in a bad state.
Is there any possibility of recovering this database so it is usable?
The practical problem is that it takes several hours to populate this database, so recovering it rather than attempting another multi-hour load would be nice.
Here’s what I see for files in the database’s directory on the server:
-rw-r--r-- 1 eliot.kimber.adm domain users 73039559 Apr 18 14:50 atv.basex
-rw-r--r-- 1 eliot.kimber.adm domain users 7544382 Apr 18 14:52 atvl.basex
-rw-r--r-- 1 eliot.kimber.adm domain users 2730980 Apr 18 14:52 atvr.basex
-rw-r--r-- 1 eliot.kimber.adm domain users 236 Apr 18 04:17 ftxx.basex
-rw-r--r-- 1 eliot.kimber.adm domain users 218072 Apr 18 04:17 ftxy.basex
-rw-r--r-- 1 eliot.kimber.adm domain users 1603678 Apr 18 04:17 ftxz.basex
-rw-r--r-- 1 eliot.kimber.adm domain users 15 Apr 18 04:17 idp.basex
-rw-r--r-- 1 eliot.kimber.adm domain users 15193 Apr 18 04:17 inf.basex
-rw-r--r-- 1 eliot.kimber.adm domain users 67 Apr 18 04:17 pth.basex
-rw-r--r-- 1 eliot.kimber.adm domain users 28 Apr 18 04:17 swl.basex
-rw-r--r-- 1 eliot.kimber.adm domain users 83701760 Apr 18 14:52 tbl.basex
-rw-r--r-- 1 eliot.kimber.adm domain users 9 Apr 18 04:17 tbli.basex
-rw-r--r-- 1 eliot.kimber.adm domain users 112380335 Apr 18 14:52 tokl.basex
-rw-r--r-- 1 eliot.kimber.adm domain users 2829520 Apr 18 14:52 tokr.basex
-rw-r--r-- 1 eliot.kimber.adm domain users 1718692 Apr 18 04:17 txt.basex
-rw-r--r-- 1 eliot.kimber.adm domain users 351326 Apr 18 04:17 txtl.basex
-rw-r--r-- 1 eliot.kimber.adm domain users 285610 Apr 18 02:38 txtr.basex
-rw-r--r-- 1 eliot.kimber.adm domain users 0 Apr 18 04:18 upd.basex
In the Databases list I see:
_temp_lrk_yokohama_link_records http://mirabel.corp.service-now.com:8984/dba/database?name=_temp_lrk_yokohama_link_records
2
274 MB
2025-04-18, 14:52:42
But then when I go to the database itself it just say “0 entries”.
Thanks,
Eliot
*Eliot Kimber*
Sr. Staff Content Engineer
O: 512 554 9368
*servicenow*
servicenow.com https://www.servicenow.com
LinkedIn https://www.linkedin.com/company/servicenow | X https://twitter.com/servicenow | YouTube https://www.youtube.com/user/servicenowinc | Instagram https://www.instagram.com/servicenow
I tried deleting the udp.basex file but it appeared to have no effect, unfortunately.
Cheers,
E.
_____________________________________________ Eliot Kimber Sr. Staff Content Engineer O: 512 554 9368
servicenow
servicenow.comhttps://www.servicenow.com LinkedInhttps://www.linkedin.com/company/servicenow | Xhttps://twitter.com/servicenow | YouTubehttps://www.youtube.com/user/servicenowinc | Instagramhttps://www.instagram.com/servicenow
From: Andy Bunce bunce.andy@gmail.com Date: Friday, April 18, 2025 at 11:48 AM To: Eliot Kimber eliot.kimber@servicenow.com Cc: basex-talk@mailman.uni-konstanz.de basex-talk@mailman.uni-konstanz.de Subject: Re: [basex-talk] Recovering database in bad state [External Email]
________________________________ Hi Eliot,
That's not fun.
I see your directory has a ` upd.basex`you could try the suggestion at: https://docs.basex.org/main/Transaction_Management#update_operationshttps://docs.basex.org/main/Transaction_Management#update_operations
Maybe you get lucky. /Andy
On Fri, 18 Apr 2025 at 15:59, Eliot Kimber via BaseX-Talk <basex-talk@mailman.uni-konstanz.demailto:basex-talk@mailman.uni-konstanz.de> wrote: I have a database that shows having the expected 2 documents in it in the DBA databases list but when I go to the database in the DBA it shows 0 items.
I think this is because the database was being written to and something happened that left it in a bad state.
Is there any possibility of recovering this database so it is usable?
The practical problem is that it takes several hours to populate this database, so recovering it rather than attempting another multi-hour load would be nice.
Here’s what I see for files in the database’s directory on the server:
-rw-r--r-- 1 eliot.kimber.adm domain users 73039559 Apr 18 14:50 atv.basex -rw-r--r-- 1 eliot.kimber.adm domain users 7544382 Apr 18 14:52 atvl.basex -rw-r--r-- 1 eliot.kimber.adm domain users 2730980 Apr 18 14:52 atvr.basex -rw-r--r-- 1 eliot.kimber.adm domain users 236 Apr 18 04:17 ftxx.basex -rw-r--r-- 1 eliot.kimber.adm domain users 218072 Apr 18 04:17 ftxy.basex -rw-r--r-- 1 eliot.kimber.adm domain users 1603678 Apr 18 04:17 ftxz.basex -rw-r--r-- 1 eliot.kimber.adm domain users 15 Apr 18 04:17 idp.basex -rw-r--r-- 1 eliot.kimber.adm domain users 15193 Apr 18 04:17 inf.basex -rw-r--r-- 1 eliot.kimber.adm domain users 67 Apr 18 04:17 pth.basex -rw-r--r-- 1 eliot.kimber.adm domain users 28 Apr 18 04:17 swl.basex -rw-r--r-- 1 eliot.kimber.adm domain users 83701760 Apr 18 14:52 tbl.basex -rw-r--r-- 1 eliot.kimber.adm domain users 9 Apr 18 04:17 tbli.basex -rw-r--r-- 1 eliot.kimber.adm domain users 112380335 Apr 18 14:52 tokl.basex -rw-r--r-- 1 eliot.kimber.adm domain users 2829520 Apr 18 14:52 tokr.basex -rw-r--r-- 1 eliot.kimber.adm domain users 1718692 Apr 18 04:17 txt.basex -rw-r--r-- 1 eliot.kimber.adm domain users 351326 Apr 18 04:17 txtl.basex -rw-r--r-- 1 eliot.kimber.adm domain users 285610 Apr 18 02:38 txtr.basex -rw-r--r-- 1 eliot.kimber.adm domain users 0 Apr 18 04:18 upd.basex
In the Databases list I see:
_temp_lrk_yokohama_link_recordshttp://mirabel.corp.service-now.com:8984/dba/database?name=_temp_lrk_yokohama_link_records 2 274 MB 2025-04-18, 14:52:42
But then when I go to the database itself it just say “0 entries”.
Thanks,
Eliot
_____________________________________________ Eliot Kimber Sr. Staff Content Engineer O: 512 554 9368
servicenow
servicenow.comhttps://www.servicenow.com LinkedInhttps://www.linkedin.com/company/servicenow | Xhttps://twitter.com/servicenow | YouTubehttps://www.youtube.com/user/servicenowinc | Instagramhttps://www.instagram.com/servicenow
This is with BaseX 11.6 on RHEL.
Cheers,
E.
_____________________________________________ Eliot Kimber Sr. Staff Content Engineer O: 512 554 9368
servicenow
servicenow.comhttps://www.servicenow.com LinkedInhttps://www.linkedin.com/company/servicenow | Xhttps://twitter.com/servicenow | YouTubehttps://www.youtube.com/user/servicenowinc | Instagramhttps://www.instagram.com/servicenow
From: Eliot Kimber eliot.kimber@servicenow.com Date: Friday, April 18, 2025 at 3:30 PM To: Andy Bunce bunce.andy@gmail.com Cc: basex-talk@mailman.uni-konstanz.de basex-talk@mailman.uni-konstanz.de Subject: Re: [basex-talk] Recovering database in bad state I tried deleting the udp.basex file but it appeared to have no effect, unfortunately.
Cheers,
E.
_____________________________________________ Eliot Kimber Sr. Staff Content Engineer O: 512 554 9368
servicenow
servicenow.comhttps://www.servicenow.com LinkedInhttps://www.linkedin.com/company/servicenow | Xhttps://twitter.com/servicenow | YouTubehttps://www.youtube.com/user/servicenowinc | Instagramhttps://www.instagram.com/servicenow
From: Andy Bunce bunce.andy@gmail.com Date: Friday, April 18, 2025 at 11:48 AM To: Eliot Kimber eliot.kimber@servicenow.com Cc: basex-talk@mailman.uni-konstanz.de basex-talk@mailman.uni-konstanz.de Subject: Re: [basex-talk] Recovering database in bad state [External Email]
________________________________ Hi Eliot,
That's not fun.
I see your directory has a ` upd.basex`you could try the suggestion at: https://docs.basex.org/main/Transaction_Management#update_operationshttps://docs.basex.org/main/Transaction_Management#update_operations
Maybe you get lucky. /Andy
On Fri, 18 Apr 2025 at 15:59, Eliot Kimber via BaseX-Talk <basex-talk@mailman.uni-konstanz.demailto:basex-talk@mailman.uni-konstanz.de> wrote: I have a database that shows having the expected 2 documents in it in the DBA databases list but when I go to the database in the DBA it shows 0 items.
I think this is because the database was being written to and something happened that left it in a bad state.
Is there any possibility of recovering this database so it is usable?
The practical problem is that it takes several hours to populate this database, so recovering it rather than attempting another multi-hour load would be nice.
Here’s what I see for files in the database’s directory on the server:
-rw-r--r-- 1 eliot.kimber.adm domain users 73039559 Apr 18 14:50 atv.basex -rw-r--r-- 1 eliot.kimber.adm domain users 7544382 Apr 18 14:52 atvl.basex -rw-r--r-- 1 eliot.kimber.adm domain users 2730980 Apr 18 14:52 atvr.basex -rw-r--r-- 1 eliot.kimber.adm domain users 236 Apr 18 04:17 ftxx.basex -rw-r--r-- 1 eliot.kimber.adm domain users 218072 Apr 18 04:17 ftxy.basex -rw-r--r-- 1 eliot.kimber.adm domain users 1603678 Apr 18 04:17 ftxz.basex -rw-r--r-- 1 eliot.kimber.adm domain users 15 Apr 18 04:17 idp.basex -rw-r--r-- 1 eliot.kimber.adm domain users 15193 Apr 18 04:17 inf.basex -rw-r--r-- 1 eliot.kimber.adm domain users 67 Apr 18 04:17 pth.basex -rw-r--r-- 1 eliot.kimber.adm domain users 28 Apr 18 04:17 swl.basex -rw-r--r-- 1 eliot.kimber.adm domain users 83701760 Apr 18 14:52 tbl.basex -rw-r--r-- 1 eliot.kimber.adm domain users 9 Apr 18 04:17 tbli.basex -rw-r--r-- 1 eliot.kimber.adm domain users 112380335 Apr 18 14:52 tokl.basex -rw-r--r-- 1 eliot.kimber.adm domain users 2829520 Apr 18 14:52 tokr.basex -rw-r--r-- 1 eliot.kimber.adm domain users 1718692 Apr 18 04:17 txt.basex -rw-r--r-- 1 eliot.kimber.adm domain users 351326 Apr 18 04:17 txtl.basex -rw-r--r-- 1 eliot.kimber.adm domain users 285610 Apr 18 02:38 txtr.basex -rw-r--r-- 1 eliot.kimber.adm domain users 0 Apr 18 04:18 upd.basex
In the Databases list I see: _temp_lrk_yokohama_link_recordshttp://mirabel.corp.service-now.com:8984/dba/database?name=_temp_lrk_yokohama_link_records 2 274 MB 2025-04-18, 14:52:42
But then when I go to the database itself it just say “0 entries”.
Thanks,
Eliot
_____________________________________________ Eliot Kimber Sr. Staff Content Engineer O: 512 554 9368
servicenow
servicenow.comhttps://www.servicenow.com LinkedInhttps://www.linkedin.com/company/servicenow | Xhttps://twitter.com/servicenow | YouTubehttps://www.youtube.com/user/servicenowinc | Instagramhttps://www.instagram.com/servicenow
Hi Eliot,
I’m sorry I cannot give a simple answer that leads to a positive result. Sometimes, the INSPECT command gives you some hints what may have gone wrong (but it doesn’t help you fix it).
I think this is because the database was being written to and something
happened that left it in a bad state.
Can you recollect what happened?
Best, Christian
On Fri, Apr 18, 2025 at 10:31 PM Eliot Kimber via BaseX-Talk < basex-talk@mailman.uni-konstanz.de> wrote:
This is with BaseX 11.6 on RHEL.
Cheers,
E.
*Eliot Kimber*
Sr. Staff Content Engineer
O: 512 554 9368
*servicenow*
servicenow.com https://www.servicenow.com
LinkedIn https://www.linkedin.com/company/servicenow | X https://twitter.com/servicenow | YouTube https://www.youtube.com/user/servicenowinc | Instagram https://www.instagram.com/servicenow
*From: *Eliot Kimber eliot.kimber@servicenow.com *Date: *Friday, April 18, 2025 at 3:30 PM *To: *Andy Bunce bunce.andy@gmail.com *Cc: *basex-talk@mailman.uni-konstanz.de < basex-talk@mailman.uni-konstanz.de> *Subject: *Re: [basex-talk] Recovering database in bad state
I tried deleting the udp.basex file but it appeared to have no effect, unfortunately.
Cheers,
E.
*Eliot Kimber*
Sr. Staff Content Engineer
O: 512 554 9368
*servicenow*
servicenow.com https://www.servicenow.com
LinkedIn https://www.linkedin.com/company/servicenow | X https://twitter.com/servicenow | YouTube https://www.youtube.com/user/servicenowinc | Instagram https://www.instagram.com/servicenow
*From: *Andy Bunce bunce.andy@gmail.com *Date: *Friday, April 18, 2025 at 11:48 AM *To: *Eliot Kimber eliot.kimber@servicenow.com *Cc: *basex-talk@mailman.uni-konstanz.de < basex-talk@mailman.uni-konstanz.de> *Subject: *Re: [basex-talk] Recovering database in bad state *[External Email]*
Hi Eliot,
That's not fun.
I see your directory has a ` upd.basex`you could try the suggestion at:
https://docs.basex.org/main/Transaction_Management#update_operations
Maybe you get lucky.
/Andy
On Fri, 18 Apr 2025 at 15:59, Eliot Kimber via BaseX-Talk < basex-talk@mailman.uni-konstanz.de> wrote:
I have a database that shows having the expected 2 documents in it in the DBA databases list but when I go to the database in the DBA it shows 0 items.
I think this is because the database was being written to and something happened that left it in a bad state.
Is there any possibility of recovering this database so it is usable?
The practical problem is that it takes several hours to populate this database, so recovering it rather than attempting another multi-hour load would be nice.
Here’s what I see for files in the database’s directory on the server:
-rw-r--r-- 1 eliot.kimber.adm domain users 73039559 Apr 18 14:50 atv.basex
-rw-r--r-- 1 eliot.kimber.adm domain users 7544382 Apr 18 14:52 atvl.basex
-rw-r--r-- 1 eliot.kimber.adm domain users 2730980 Apr 18 14:52 atvr.basex
-rw-r--r-- 1 eliot.kimber.adm domain users 236 Apr 18 04:17 ftxx.basex
-rw-r--r-- 1 eliot.kimber.adm domain users 218072 Apr 18 04:17 ftxy.basex
-rw-r--r-- 1 eliot.kimber.adm domain users 1603678 Apr 18 04:17 ftxz.basex
-rw-r--r-- 1 eliot.kimber.adm domain users 15 Apr 18 04:17 idp.basex
-rw-r--r-- 1 eliot.kimber.adm domain users 15193 Apr 18 04:17 inf.basex
-rw-r--r-- 1 eliot.kimber.adm domain users 67 Apr 18 04:17 pth.basex
-rw-r--r-- 1 eliot.kimber.adm domain users 28 Apr 18 04:17 swl.basex
-rw-r--r-- 1 eliot.kimber.adm domain users 83701760 Apr 18 14:52 tbl.basex
-rw-r--r-- 1 eliot.kimber.adm domain users 9 Apr 18 04:17 tbli.basex
-rw-r--r-- 1 eliot.kimber.adm domain users 112380335 Apr 18 14:52 tokl.basex
-rw-r--r-- 1 eliot.kimber.adm domain users 2829520 Apr 18 14:52 tokr.basex
-rw-r--r-- 1 eliot.kimber.adm domain users 1718692 Apr 18 04:17 txt.basex
-rw-r--r-- 1 eliot.kimber.adm domain users 351326 Apr 18 04:17 txtl.basex
-rw-r--r-- 1 eliot.kimber.adm domain users 285610 Apr 18 02:38 txtr.basex
-rw-r--r-- 1 eliot.kimber.adm domain users 0 Apr 18 04:18 upd.basex
In the Databases list I see:
_temp_lrk_yokohama_link_records http://mirabel.corp.service-now.com:8984/dba/database?name=_temp_lrk_yokohama_link_records
2
274 MB
2025-04-18, 14:52:42
But then when I go to the database itself it just say “0 entries”.
Thanks,
Eliot
*Eliot Kimber*
Sr. Staff Content Engineer
O: 512 554 9368
*servicenow*
servicenow.com https://www.servicenow.com
LinkedIn https://www.linkedin.com/company/servicenow | X https://twitter.com/servicenow | YouTube https://www.youtube.com/user/servicenowinc | Instagram https://www.instagram.com/servicenow
As discussed in a parallel thread, I’ve determined that the problem is caused by trying to transform a massive doc in the database using an XSLT transform that either never completes or fails in a way that is not reflected in the log.
I performed a full-scale test on my development server, meaning that this was the only process running on it. I could see through my server status dashboard that the process doing the XSLT was running and it took a *long* time to finish (at least 24 hours—I stopped watching it closely but checking it now it has completed successfully).
So the issue is with my attempt to transform a 300MB document, not an issue with BaseX itself.
The solution is clearly to not try to transform a 300MB document, which I’m pursing now.
I’ve yet to find a way, short of connecting BaseX to an interactive debugger, to get visibility into what it’s doing on the server—for example, I orchestrate my processes by creating a sequence of jobs but there doesn’t seem to be a way to see what job is running or get insight into what it’s actively doing.
I’m not sure it’s even possible to provide that kind of visibility but it does make knowing what’s happening a challenge. Of course I can use messages in my code to get some feedback in the log, and I do do that, but for just monitoring a running server it would be nice to have more visibility.
Cheers,
E.
_____________________________________________ Eliot Kimber Sr. Staff Content Engineer O: 512 554 9368
servicenow
servicenow.comhttps://www.servicenow.com LinkedInhttps://www.linkedin.com/company/servicenow | Xhttps://twitter.com/servicenow | YouTubehttps://www.youtube.com/user/servicenowinc | Instagramhttps://www.instagram.com/servicenow
From: Christian Grün christian.gruen@gmail.com Date: Thursday, April 24, 2025 at 8:11 AM To: Eliot Kimber eliot.kimber@servicenow.com Cc: Andy Bunce bunce.andy@gmail.com, basex-talk@mailman.uni-konstanz.de basex-talk@mailman.uni-konstanz.de Subject: Re: [basex-talk] Re: Recovering database in bad state [External Email]
________________________________ Hi Eliot, I’m sorry I cannot give a simple answer that leads to a positive result. Sometimes, the INSPECT command gives you some hints what may have gone wrong (but it doesn’t help you fix it). > I think this is because the database was being written to and something happened i This message needs your attention
* Someone new is on this email.
Provided by ServiceNow DT (Employee Portal KB0077950) - This banner is visible only to ServiceNow employees. CGBANNERINDICATOR Hi Eliot,
I’m sorry I cannot give a simple answer that leads to a positive result. Sometimes, the INSPECT command gives you some hints what may have gone wrong (but it doesn’t help you fix it).
I think this is because the database was being written to and something happened that left it in a bad state.
Can you recollect what happened?
Best, Christian
On Fri, Apr 18, 2025 at 10:31 PM Eliot Kimber via BaseX-Talk <basex-talk@mailman.uni-konstanz.demailto:basex-talk@mailman.uni-konstanz.de> wrote: This is with BaseX 11.6 on RHEL.
Cheers,
E.
_____________________________________________ Eliot Kimber Sr. Staff Content Engineer O: 512 554 9368
servicenow
servicenow.comhttps://www.servicenow.com LinkedInhttps://www.linkedin.com/company/servicenow | Xhttps://twitter.com/servicenow | YouTubehttps://www.youtube.com/user/servicenowinc | Instagramhttps://www.instagram.com/servicenow
From: Eliot Kimber <eliot.kimber@servicenow.commailto:eliot.kimber@servicenow.com> Date: Friday, April 18, 2025 at 3:30 PM To: Andy Bunce <bunce.andy@gmail.commailto:bunce.andy@gmail.com> Cc: basex-talk@mailman.uni-konstanz.demailto:basex-talk@mailman.uni-konstanz.de <basex-talk@mailman.uni-konstanz.demailto:basex-talk@mailman.uni-konstanz.de> Subject: Re: [basex-talk] Recovering database in bad state I tried deleting the udp.basex file but it appeared to have no effect, unfortunately.
Cheers,
E.
_____________________________________________ Eliot Kimber Sr. Staff Content Engineer O: 512 554 9368
servicenow
servicenow.comhttps://www.servicenow.com LinkedInhttps://www.linkedin.com/company/servicenow | Xhttps://twitter.com/servicenow | YouTubehttps://www.youtube.com/user/servicenowinc | Instagramhttps://www.instagram.com/servicenow
From: Andy Bunce <bunce.andy@gmail.commailto:bunce.andy@gmail.com> Date: Friday, April 18, 2025 at 11:48 AM To: Eliot Kimber <eliot.kimber@servicenow.commailto:eliot.kimber@servicenow.com> Cc: basex-talk@mailman.uni-konstanz.demailto:basex-talk@mailman.uni-konstanz.de <basex-talk@mailman.uni-konstanz.demailto:basex-talk@mailman.uni-konstanz.de> Subject: Re: [basex-talk] Recovering database in bad state [External Email]
________________________________ Hi Eliot,
That's not fun.
I see your directory has a ` upd.basex`you could try the suggestion at: https://docs.basex.org/main/Transaction_Management#update_operationshttps://docs.basex.org/main/Transaction_Management#update_operations
Maybe you get lucky. /Andy
On Fri, 18 Apr 2025 at 15:59, Eliot Kimber via BaseX-Talk <basex-talk@mailman.uni-konstanz.demailto:basex-talk@mailman.uni-konstanz.de> wrote: I have a database that shows having the expected 2 documents in it in the DBA databases list but when I go to the database in the DBA it shows 0 items.
I think this is because the database was being written to and something happened that left it in a bad state.
Is there any possibility of recovering this database so it is usable?
The practical problem is that it takes several hours to populate this database, so recovering it rather than attempting another multi-hour load would be nice.
Here’s what I see for files in the database’s directory on the server:
-rw-r--r-- 1 eliot.kimber.adm domain users 73039559 Apr 18 14:50 atv.basex -rw-r--r-- 1 eliot.kimber.adm domain users 7544382 Apr 18 14:52 atvl.basex -rw-r--r-- 1 eliot.kimber.adm domain users 2730980 Apr 18 14:52 atvr.basex -rw-r--r-- 1 eliot.kimber.adm domain users 236 Apr 18 04:17 ftxx.basex -rw-r--r-- 1 eliot.kimber.adm domain users 218072 Apr 18 04:17 ftxy.basex -rw-r--r-- 1 eliot.kimber.adm domain users 1603678 Apr 18 04:17 ftxz.basex -rw-r--r-- 1 eliot.kimber.adm domain users 15 Apr 18 04:17 idp.basex -rw-r--r-- 1 eliot.kimber.adm domain users 15193 Apr 18 04:17 inf.basex -rw-r--r-- 1 eliot.kimber.adm domain users 67 Apr 18 04:17 pth.basex -rw-r--r-- 1 eliot.kimber.adm domain users 28 Apr 18 04:17 swl.basex -rw-r--r-- 1 eliot.kimber.adm domain users 83701760 Apr 18 14:52 tbl.basex -rw-r--r-- 1 eliot.kimber.adm domain users 9 Apr 18 04:17 tbli.basex -rw-r--r-- 1 eliot.kimber.adm domain users 112380335 Apr 18 14:52 tokl.basex -rw-r--r-- 1 eliot.kimber.adm domain users 2829520 Apr 18 14:52 tokr.basex -rw-r--r-- 1 eliot.kimber.adm domain users 1718692 Apr 18 04:17 txt.basex -rw-r--r-- 1 eliot.kimber.adm domain users 351326 Apr 18 04:17 txtl.basex -rw-r--r-- 1 eliot.kimber.adm domain users 285610 Apr 18 02:38 txtr.basex -rw-r--r-- 1 eliot.kimber.adm domain users 0 Apr 18 04:18 upd.basex
In the Databases list I see: _temp_lrk_yokohama_link_recordshttp://mirabel.corp.service-now.com:8984/dba/database?name=_temp_lrk_yokohama_link_records 2 274 MB 2025-04-18, 14:52:42
But then when I go to the database itself it just say “0 entries”.
Thanks,
Eliot
_____________________________________________ Eliot Kimber Sr. Staff Content Engineer O: 512 554 9368
servicenow
servicenow.comhttps://www.servicenow.com LinkedInhttps://www.linkedin.com/company/servicenow | Xhttps://twitter.com/servicenow | YouTubehttps://www.youtube.com/user/servicenowinc | Instagramhttps://www.instagram.com/servicenow
Good to know.
So the issue is with my attempt to transform a 300MB document, not an issue
with BaseX itself.
You may be able to save lots of time if you manage to rewrite the XSLT script to XQuery (Update). Here’s a script that creates an element with 1 million nodes, which are immediately deleted again. It takes less than 1 second:
<a>{ (1 to 1000000) ! <b/> }</a> update { delete node ./b }
Best, Christian
Interesting—I will try using Update—the actual change is a simple modification of attribute values.
Cheers,
E.
_____________________________________________ Eliot Kimber Sr. Staff Content Engineer O: 512 554 9368
servicenow
servicenow.comhttps://www.servicenow.com LinkedInhttps://www.linkedin.com/company/servicenow | Xhttps://twitter.com/servicenow | YouTubehttps://www.youtube.com/user/servicenowinc | Instagramhttps://www.instagram.com/servicenow
From: Christian Grün christian.gruen@gmail.com Date: Thursday, April 24, 2025 at 8:52 AM To: Eliot Kimber eliot.kimber@servicenow.com Cc: Andy Bunce bunce.andy@gmail.com, basex-talk@mailman.uni-konstanz.de basex-talk@mailman.uni-konstanz.de Subject: Re: [basex-talk] Re: Recovering database in bad state [External Email]
________________________________ Good to know. So the issue is with my attempt to transform a 300MB document, not an issue with BaseX itself. You may be able to save lots of time if you manage to rewrite the XSLT script to XQuery (Update). Here’s a script that creates an element with 1 million nodes, which are i i This message needs your attention
* Someone new is on this email.
Provided by ServiceNow DT (Employee Portal KB0077950) - This banner is visible only to ServiceNow employees. CGBANNERINDICATOR Good to know.
So the issue is with my attempt to transform a 300MB document, not an issue with BaseX itself.
You may be able to save lots of time if you manage to rewrite the XSLT script to XQuery (Update). Here’s a script that creates an element with 1 million nodes, which are immediately deleted again. It takes less than 1 second:
<a>{ (1 to 1000000) ! <b/> }</a> update { delete node ./b }
Best, Christian
Interesting—I will try using Update—the actual change is a simple modification of attribute values.
The syntax is a bit verbose, but probably still better than XSLT ;·) Examples:
<a>{ (1 to 1000000) ! <b c=''/> }</a> update { replace value of node ./b/@c with 'new' }
<a>{ (1 to 1000000) ! <b c=''/> }</a> update { for $b at $p in b/@c return replace value of node $b with $p }
…or, if a database is directly updated:
replace value of node .//b with 'xyz'
I’ve updated my code to use XQuery update to update the attributes. When I run it as a job, the job returns immediately and the next job starts. I’m assuming the update must be on the update queue (this is in the context of a larger job sequence that constructs the link data).
I can tell the update is running because the database in question will show 0 files until the update is finished, at which point it will show the expected 2 files.
For my small-scale test, I have a doc with 79,632 attributes to be updated using this code:
declare updating function linkrk:updateWhereUsedIndexToProduction( $database as xs:string ) { let $whereUsedMap as element()? := db:get($database)/doc-where-used-index let $msg := message(``[[DEBUG] linkrk:updateWhereUsedIndexToProduction(): Updating `{$database}` to production using XQuery update]``) return for $att in $whereUsedMap//noderef/(@database|@baseuri)[starts-with(., '_temp_')] let $newValue := substring-after(string($att), '_temp_') return replace value of node $att with $newValue };
I see the debug message in my log and then the message for the start of the next job in the sequence.
The update seems to take about 1.5 minutes based on the log message time stamps, which is roughly 1.1ms/update, which seems about as fast as it could go without parallelizing the updates (this is on an M3 macBook Pro).
I tried using prof:time() to profile the update but prof:time() does not allow updating expressions.
Is there a technique for profiling this type of update better than the link timestamp analysis I’m doing?
In my full-scale content, I have about 1.5M attributes to be updated, so at 1.1ms/update, that comes to about 24 minutes, which isn’t too bad. Testing now.
Cheers,
E.
_____________________________________________ Eliot Kimber Sr. Staff Content Engineer O: 512 554 9368
servicenow
servicenow.comhttps://www.servicenow.com LinkedInhttps://www.linkedin.com/company/servicenow | Xhttps://twitter.com/servicenow | YouTubehttps://www.youtube.com/user/servicenowinc | Instagramhttps://www.instagram.com/servicenow
From: Christian Grün christian.gruen@gmail.com Date: Thursday, April 24, 2025 at 8:52 AM To: Eliot Kimber eliot.kimber@servicenow.com Cc: Andy Bunce bunce.andy@gmail.com, basex-talk@mailman.uni-konstanz.de basex-talk@mailman.uni-konstanz.de Subject: Re: [basex-talk] Re: Recovering database in bad state [External Email]
________________________________ Good to know. So the issue is with my attempt to transform a 300MB document, not an issue with BaseX itself. You may be able to save lots of time if you manage to rewrite the XSLT script to XQuery (Update). Here’s a script that creates an element with 1 million nodes, which are i i This message needs your attention
* Someone new is on this email.
Provided by ServiceNow DT (Employee Portal KB0077950) - This banner is visible only to ServiceNow employees. CGBANNERINDICATOR Good to know.
So the issue is with my attempt to transform a 300MB document, not an issue with BaseX itself.
You may be able to save lots of time if you manage to rewrite the XSLT script to XQuery (Update). Here’s a script that creates an element with 1 million nodes, which are immediately deleted again. It takes less than 1 second:
<a>{ (1 to 1000000) ! <b/> }</a> update { delete node ./b }
Best, Christian
Hi Eliot,
You can have a look into our documentation to see how XQuery Update works [1]. In particular, have a look at “Pending Update List”.
Best, Christian
[1] https://docs.basex.org/main/Updates
On Thu, Apr 24, 2025 at 10:26 PM Eliot Kimber eliot.kimber@servicenow.com wrote:
I’ve updated my code to use XQuery update to update the attributes. When I run it as a job, the job returns immediately and the next job starts. I’m assuming the update must be on the update queue (this is in the context of a larger job sequence that constructs the link data).
I can tell the update is running because the database in question will show 0 files until the update is finished, at which point it will show the expected 2 files.
For my small-scale test, I have a doc with 79,632 attributes to be updated using this code:
declare updating function linkrk:updateWhereUsedIndexToProduction(
$database as xs:string
) {
let $whereUsedMap as element()? := db:get($database)/doc-where-used-index
let $msg := message(``[[DEBUG] linkrk:updateWhereUsedIndexToProduction(): Updating `{$database}` to production using XQuery update]``)
return
for $att in $whereUsedMap//noderef/(@database|@baseuri)[starts-with(., '_temp_')]
let $newValue := substring-after(string($att), '_temp_')
return replace value of node $att with $newValue
};
I see the debug message in my log and then the message for the start of the next job in the sequence.
The update seems to take about 1.5 minutes based on the log message time stamps, which is roughly 1.1ms/update, which seems about as fast as it could go without parallelizing the updates (this is on an M3 macBook Pro).
I tried using prof:time() to profile the update but prof:time() does not allow updating expressions.
Is there a technique for profiling this type of update better than the link timestamp analysis I’m doing?
In my full-scale content, I have about 1.5M attributes to be updated, so at 1.1ms/update, that comes to about 24 minutes, which isn’t too bad. Testing now.
Cheers,
E.
*Eliot Kimber*
Sr. Staff Content Engineer
O: 512 554 9368
*servicenow*
servicenow.com https://www.servicenow.com
LinkedIn https://www.linkedin.com/company/servicenow | X https://twitter.com/servicenow | YouTube https://www.youtube.com/user/servicenowinc | Instagram https://www.instagram.com/servicenow
*From: *Christian Grün christian.gruen@gmail.com *Date: *Thursday, April 24, 2025 at 8:52 AM *To: *Eliot Kimber eliot.kimber@servicenow.com *Cc: *Andy Bunce bunce.andy@gmail.com, basex-talk@mailman.uni-konstanz.de basex-talk@mailman.uni-konstanz.de *Subject: *Re: [basex-talk] Re: Recovering database in bad state *[External Email]*
Good to know.
So the issue is with my attempt to transform a 300MB document, not an issue with BaseX itself.
You may be able to save lots of time if you manage to rewrite the XSLT script to XQuery (Update). Here’s a script that creates an element with 1 million nodes, which are immediately deleted again. It takes less than 1 second:
<a>{ (1 to 1000000) ! <b/> }</a> update { delete node ./b }
Best,
Christian
Following up: The XQuery update approach seems to be working reliably.
Cheers,
E. _____________________________________________ Eliot Kimber Sr. Staff Content Engineer O: 512 554 9368
servicenow
servicenow.comhttps://www.servicenow.com LinkedInhttps://www.linkedin.com/company/servicenow | Xhttps://twitter.com/servicenow | YouTubehttps://www.youtube.com/user/servicenowinc | Instagramhttps://www.instagram.com/servicenow
From: Eliot Kimber via BaseX-Talk basex-talk@mailman.uni-konstanz.de Date: Thursday, April 24, 2025 at 9:41 PM To: Christian Grün christian.gruen@gmail.com Cc: basex-talk@mailman.uni-konstanz.de basex-talk@mailman.uni-konstanz.de Subject: [basex-talk] Re: Recovering database in bad state [External Email]
________________________________ I’ve updated my code to use XQuery update to update the attributes. When I run it as a job, the job returns immediately and the next job starts. I’m assuming the update must be on the update queue (this is in the context of a larger job sequence that constructs the link data). i This message needs your attention
* Someone new is on this email.
Provided by ServiceNow DT (Employee Portal KB0077950) - This banner is visible only to ServiceNow employees. CGBANNERINDICATOR I’ve updated my code to use XQuery update to update the attributes. When I run it as a job, the job returns immediately and the next job starts. I’m assuming the update must be on the update queue (this is in the context of a larger job sequence that constructs the link data).
I can tell the update is running because the database in question will show 0 files until the update is finished, at which point it will show the expected 2 files.
For my small-scale test, I have a doc with 79,632 attributes to be updated using this code:
declare updating function linkrk:updateWhereUsedIndexToProduction( $database as xs:string ) { let $whereUsedMap as element()? := db:get($database)/doc-where-used-index let $msg := message(``[[DEBUG] linkrk:updateWhereUsedIndexToProduction(): Updating `{$database}` to production using XQuery update]``) return for $att in $whereUsedMap//noderef/(@database|@baseuri)[starts-with(., '_temp_')] let $newValue := substring-after(string($att), '_temp_') return replace value of node $att with $newValue };
I see the debug message in my log and then the message for the start of the next job in the sequence.
The update seems to take about 1.5 minutes based on the log message time stamps, which is roughly 1.1ms/update, which seems about as fast as it could go without parallelizing the updates (this is on an M3 macBook Pro).
I tried using prof:time() to profile the update but prof:time() does not allow updating expressions.
Is there a technique for profiling this type of update better than the link timestamp analysis I’m doing?
In my full-scale content, I have about 1.5M attributes to be updated, so at 1.1ms/update, that comes to about 24 minutes, which isn’t too bad. Testing now.
Cheers,
E.
_____________________________________________ Eliot Kimber Sr. Staff Content Engineer O: 512 554 9368
servicenow
servicenow.comhttps://www.servicenow.com LinkedInhttps://www.linkedin.com/company/servicenow | Xhttps://twitter.com/servicenow | YouTubehttps://www.youtube.com/user/servicenowinc | Instagramhttps://www.instagram.com/servicenow
From: Christian Grün christian.gruen@gmail.com Date: Thursday, April 24, 2025 at 8:52 AM To: Eliot Kimber eliot.kimber@servicenow.com Cc: Andy Bunce bunce.andy@gmail.com, basex-talk@mailman.uni-konstanz.de basex-talk@mailman.uni-konstanz.de Subject: Re: [basex-talk] Re: Recovering database in bad state [External Email]
________________________________ Good to know.
So the issue is with my attempt to transform a 300MB document, not an issue with BaseX itself.
You may be able to save lots of time if you manage to rewrite the XSLT script to XQuery (Update). Here’s a script that creates an element with 1 million nodes, which are immediately deleted again. It takes less than 1 second:
<a>{ (1 to 1000000) ! <b/> }</a> update { delete node ./b }
Best, Christian
basex-talk@mailman.uni-konstanz.de
-
Andy Bunce -
Christian Grün -
Eliot Kimber