I’m making good progress on our BaseX-based validation dashboard application.
The basic process here is we use Oxygen’s scripting support to do DITA map validation and then ingest the result into a database (along with the content documents that were validated) and provide reports on the validation result.
The practical challenge here seems to be running the Oxygen process successfully from BaseX—because our content is so huge it can take 10s of minutes for the Oxygen process to run.
I set the command time-out to be much longer than the process should run but running it from the HTTP app’s query panel it eventually failed with an error that wasn’t a time out (my code had earlier reported legit errors so I know errors will be properly reported).
As soon as the Oxygen process ends I want to ingest the resulting XML file, which is why I started with doing it from within BaseX.
But I’m wondering if this is a bad idea and I should really be doing it with i.e., a shell script run via cron or some such?
I was trying to keep everything in BaseX as much as possible just to keep it simple.
Any general tips or cautions for this time of integration of BaseX with the outside world?
Thanks,
E. _____________________________________________ Eliot Kimber Sr Staff Content Engineer O: 512 554 9368 M: 512 554 9368 servicenow.comhttps://www.servicenow.com LinkedInhttps://www.linkedin.com/company/servicenow | Twitterhttps://twitter.com/servicenow | YouTubehttps://www.youtube.com/user/servicenowinc | Facebookhttps://www.facebook.com/servicenow
Hi Eliot,
Can you be a bit more specific about how you set the timeout?
If you only set it as an option on one of the proc:*() functions but the the action gets triggered by a RESTXQ request, then still the default timeout of 30 seconds may kick in (https://docs.basex.org/wiki/Options#TIMEOUT). See the possibilities to alter this value at the top of said page. For example, you can set it in webapp/WEB-INF/web.xml or with -Dorg.basex.TIMEOUT=7200 in your BaseX startup script.
Gerrit
On 26.01.2022 22:33, Eliot Kimber wrote:
I’m making good progress on our BaseX-based validation dashboard application.
The basic process here is we use Oxygen’s scripting support to do DITA map validation and then ingest the result into a database (along with the content documents that were validated) and provide reports on the validation result.
The practical challenge here seems to be running the Oxygen process successfully from BaseX—because our content is so huge it can take 10s of minutes for the Oxygen process to run.
I set the command time-out to be much longer than the process should run but running it from the HTTP app’s query panel it eventually failed with an error that wasn’t a time out (my code had earlier reported legit errors so I know errors will be properly reported).
As soon as the Oxygen process ends I want to ingest the resulting XML file, which is why I started with doing it from within BaseX.
But I’m wondering if this is a bad idea and I should really be doing it with i.e., a shell script run via cron or some such?
I was trying to keep everything in BaseX as much as possible just to keep it simple.
Any general tips or cautions for this time of integration of BaseX with the outside world?
Thanks,
E.
*Eliot Kimber*
Sr Staff Content Engineer
O: 512 554 9368
M: 512 554 9368
servicenow.com https://www.servicenow.com
LinkedIn https://www.linkedin.com/company/servicenow | Twitter https://twitter.com/servicenow | YouTube https://www.youtube.com/user/servicenowinc | Facebook https://www.facebook.com/servicenow
For my initial attempt I’m just using proc:execute() from a manually-run XQuery script run from the BaseX admin query panel: let $timeout as xs:integer := 60 * 30 let $result := proc:execute($oxygenCmd, ('-i', $inputPath, '-r', $reportPath, '-s', $settingsPath ), map{ 'timeout' : $timeout } )
Subsequent tests seemed to work OK, in that the process ran until Oxygen ran out of memory. I haven’t been able to figure out why my attempts to set the JVM memory for Oxygen are failing and have contacted Oxygen support for guidance.
But it’s good to know about these other ways to set the timeout—I know I’ll need to attend to more considered BaseX settings once I get everything hooked up.
Cheers,
E.
_____________________________________________ Eliot Kimber Sr Staff Content Engineer O: 512 554 9368 M: 512 554 9368 servicenow.comhttps://www.servicenow.com LinkedInhttps://www.linkedin.com/company/servicenow | Twitterhttps://twitter.com/servicenow | YouTubehttps://www.youtube.com/user/servicenowinc | Facebookhttps://www.facebook.com/servicenow
From: BaseX-Talk basex-talk-bounces@mailman.uni-konstanz.de on behalf of Imsieke, Gerrit, le-tex gerrit.imsieke@le-tex.de Date: Wednesday, January 26, 2022 at 3:54 PM To: basex-talk@mailman.uni-konstanz.de basex-talk@mailman.uni-konstanz.de Subject: Re: [basex-talk] Managing Interactions with Long-Running Processes [External Email]
Hi Eliot,
Can you be a bit more specific about how you set the timeout?
If you only set it as an option on one of the proc:*() functions but the the action gets triggered by a RESTXQ request, then still the default timeout of 30 seconds may kick in (https://urldefense.com/v3/__https://docs.basex.org/wiki/Options*TIMEOUT__;Iw... ). See the possibilities to alter this value at the top of said page. For example, you can set it in webapp/WEB-INF/web.xml or with -Dorg.basex.TIMEOUT=7200 in your BaseX startup script.
Gerrit
On 26.01.2022 22:33, Eliot Kimber wrote:
I’m making good progress on our BaseX-based validation dashboard application.
The basic process here is we use Oxygen’s scripting support to do DITA map validation and then ingest the result into a database (along with the content documents that were validated) and provide reports on the validation result.
The practical challenge here seems to be running the Oxygen process successfully from BaseX—because our content is so huge it can take 10s of minutes for the Oxygen process to run.
I set the command time-out to be much longer than the process should run but running it from the HTTP app’s query panel it eventually failed with an error that wasn’t a time out (my code had earlier reported legit errors so I know errors will be properly reported).
As soon as the Oxygen process ends I want to ingest the resulting XML file, which is why I started with doing it from within BaseX.
But I’m wondering if this is a bad idea and I should really be doing it with i.e., a shell script run via cron or some such?
I was trying to keep everything in BaseX as much as possible just to keep it simple.
Any general tips or cautions for this time of integration of BaseX with the outside world?
Thanks,
E.
*Eliot Kimber*
Sr Staff Content Engineer
O: 512 554 9368
M: 512 554 9368
servicenow.com https://www.servicenow.com
LinkedIn <https://urldefense.com/v3/__https://www.linkedin.com/company/servicenow__;!!... > | Twitter <https://urldefense.com/v3/__https://twitter.com/servicenow__;!!N4vogdjhuJM!X... > | YouTube <https://urldefense.com/v3/__https://www.youtube.com/user/servicenowinc__;!!N... > | Facebook <https://urldefense.com/v3/__https://www.facebook.com/servicenow__;!!N4vogdjh... >
-- Gerrit Imsieke Geschäftsführer / Managing Director le-tex publishing services GmbH Weissenfelser Str. 84, 04229 Leipzig, Germany Phone +49 341 355356 110, Fax +49 341 355356 510 gerrit.imsieke@le-tex.de, https://urldefense.com/v3/__http://www.le-tex.de__;!!N4vogdjhuJM!XON4qC6R8fN...https://urldefense.com/v3/__http:/www.le-tex.de__;!!N4vogdjhuJM!XON4qC6R8fNj3sdIfTp9rLyz4U2wPMQRUkUD_R98BtbZGDCeu0h8HtKZ9q0l6tM2UK0Wqw$
Registergericht / Commercial Register: Amtsgericht Leipzig Registernummer / Registration Number: HRB 24930
Geschäftsführer / Managing Directors: Gerrit Imsieke, Svea Jelonek, Thomas Schmidt
basex-talk@mailman.uni-konstanz.de
-
Eliot Kimber -
Imsieke, Gerrit, le-tex