I’m getting my validation dashboard server almost ready for use and will be making available to real users.
I want to make sure that I’ve done everything I can to ensure maximum performance and stability of the server and I’m realizing I’m not entirely sure how to do that.
This is running on a Centos virtual machine with 16GB of ram and 1TB of available disk space and 4 cores—not the beefiest but it’s what IT gave us (and we’re not sure how to ask for more on short notice).
Here’s what I’ve done so far:
* Using the JRE provided with Oxygen, allocated with 4GB (we are also using this server to run Oxygen via scripting and it needs 8GB to handle our insanely huge DITA maps) * Set parallel to 4 (to match the number of cores, but just guessing that this is a useful setting based on the docs)
I have not modified any index settings but I’m also not doing anything other than querying on element types and attribute values, so from the docs it looks like the default indexes should be what I need.
I’m seeing some apparent occasional slowness on pages that should not be slow (don’t reflect long-running queries or huge data volumes) but I’m not really sure how to diagnose it or even verify that I’ve succeeded in giving BaseX all the resources it needs.
The same webapp performs well on my development mac, which makes me think the issue is one of resource allocation, so probably my user error setting up the server.
Any guidance on how to ensure best server performance would be most helpful.
Thanks,
Eliot
_____________________________________________ Eliot Kimber Sr Staff Content Engineer O: 512 554 9368 M: 512 554 9368 servicenow.comhttps://www.servicenow.com LinkedInhttps://www.linkedin.com/company/servicenow | Twitterhttps://twitter.com/servicenow | YouTubehttps://www.youtube.com/user/servicenowinc | Facebookhttps://www.facebook.com/servicenow
On Sun, 2022-02-06 at 03:34 +0000, Eliot Kimber wrote:
* Using the JRE provided with Oxygen, allocated with 4GB (we are also using this server to run Oxygen via scripting and it needs 8GB to handle our insanely huge DITA maps)
Make sure you have e.g. 64 gigabytes or more of swap configures; free -h will tell you this
* Set parallel to 4 (to match the number of cores, but just guessing that this is a useful setting based on the docs)
check /proc/cpuinfo (e.g, less /proc/cpuinfo) and you'll prolly find it can run 8 threads
I’m seeing some apparent occasional slowness on pages that should not be slow (don’t reflect long-running queries or huge data volumes)
make sure there are no xml catalogs or DTDs to be fetched externally - or, if there are catalogs, e.g. used with fn:transform(), that those catalog files do NOT start with a doctype that causes a network fetch of a dtd...
but I’m not really sure how to diagnose it or even verify that I’ve succeeded in giving BaseX all the resources it needs.
maybe in an ssh/terminal window, keep "top" running while you fetch a page, and see if the system gets really busy. Note also the centos system is probably using a hard drive, not an SSD, so file access may be slower - make sure you have indexes!
hope this helps at least a little,
Liam,
Here’s what free -h tells me:
/data/basex/git/dita-build-tools/src/main$ free -h total used free shared buff/cache available Mem: 15G 1.4G 8.3G 529M 6.0G 13G Swap: 0B 0B 0B
I read that as there being no swap. But note that this is a virtual machine (from AWS as far as I know)—I don’t know if that makes a difference.
If I understand what cpuinfo is telling me, I have 4 CPUs, each with 4 cores, so should be able to service 16 threads, yes?
I have DTD parsing turned off (which is the default)—if I didn’t do that it would take hours to load our content into the database, so that *shouldn’t* be a problem (I hope).
I did find prof:runtime() in the docs and verified that it has 4GB of RAM available.
_____________________________________________ Eliot Kimber Sr Staff Content Engineer O: 512 554 9368 M: 512 554 9368 servicenow.comhttps://www.servicenow.com LinkedInhttps://www.linkedin.com/company/servicenow | Twitterhttps://twitter.com/servicenow | YouTubehttps://www.youtube.com/user/servicenowinc | Facebookhttps://www.facebook.com/servicenow
From: Liam R. E. Quin liam@fromoldbooks.org Date: Sunday, February 6, 2022 at 12:08 AM To: Eliot Kimber eliot.kimber@servicenow.com, basex-talk@mailman.uni-konstanz.de basex-talk@mailman.uni-konstanz.de Subject: Re: [basex-talk] Managing/Debugging Server Load and Performance [External Email]
On Sun, 2022-02-06 at 03:34 +0000, Eliot Kimber wrote:
- Using the JRE provided with Oxygen, allocated with 4GB (we are
also using this server to run Oxygen via scripting and it needs 8GB to handle our insanely huge DITA maps)
Make sure you have e.g. 64 gigabytes or more of swap configures; free -h will tell you this
- Set parallel to 4 (to match the number of cores, but just
guessing that this is a useful setting based on the docs)
check /proc/cpuinfo (e.g, less /proc/cpuinfo) and you'll prolly find it can run 8 threads
I’m seeing some apparent occasional slowness on pages that should not be slow (don’t reflect long-running queries or huge data volumes)
make sure there are no xml catalogs or DTDs to be fetched externally - or, if there are catalogs, e.g. used with fn:transform(), that those catalog files do NOT start with a doctype that causes a network fetch of a dtd...
but I’m not really sure how to diagnose it or even verify that I’ve succeeded in giving BaseX all the resources it needs.
maybe in an ssh/terminal window, keep "top" running while you fetch a page, and see if the system gets really busy. Note also the centos system is probably using a hard drive, not an SSD, so file access may be slower - make sure you have indexes!
hope this helps at least a little,
-- Liam Quin, https://urldefense.com/v3/__https://www.delightfulcomputing.com/__;!!N4vogdj...https://urldefense.com/v3/__https:/www.delightfulcomputing.com/__;!!N4vogdjhuJM!U0hEyXTeoTf2B18jzQmUco2XDS97VUqRet5HS3OjWp_cIEbY9gMS9UJKn8aBFs61KYmFdA$ Available for XML/Document/Information Architecture/XSLT/ XSL/XQuery/Web/Text Processing/A11Y training, work & consulting. Barefoot Web-slave, antique illustrations: https://urldefense.com/v3/__http://www.fromoldbooks.org__;!!N4vogdjhuJM!U0hE...https://urldefense.com/v3/__http:/www.fromoldbooks.org__;!!N4vogdjhuJM!U0hEyXTeoTf2B18jzQmUco2XDS97VUqRet5HS3OjWp_cIEbY9gMS9UJKn8aBFs5ZXch2EA$
On Sun, 2022-02-06 at 06:31 +0000, Eliot Kimber wrote:
Liam,
Here’s what free -h tells me:
/data/basex/git/dita-build-tools/src/main$ free -h total used free shared buff/cache available Mem: 15G 1.4G 8.3G 529M 6.0G 13G Swap: 0B 0B 0B
I read that as there being no swap. But note that this is a virtual machine (from AWS as far as I know)—I don’t know if that makes a difference.
https://aws.amazon.com/premiumsupport/knowledge-center/ec2-memory-swap-file/... apply; i don't know for sure.
If I understand what cpuinfo is telling me, I have 4 CPUs, each with 4 cores, so should be able to service 16 threads, yes?
8 - two per core.
I have DTD parsing turned off (which is the default)
It's unrelated to XML catalog processing. If the system is not heavily loaded (as reported by a tool such as top) when processing a page that's slow, it may be waiting for network access. If it's heavily loaded, and the same query runs quickly for you, it might be that you need to check indexes are applied and up to date.
I did find prof:runtime() in the docs and verified that it has 4GB of RAM available.
OK, cool.
I’m still seeing performance issues that I’m not sure how to diagnose. The symptom is:
1. Go to a page that does a query that takes 20 seconds to complete in order to build a report. This is acceptable performance. 2. Navigate to a different page. 3. Return to the first page—it either takes a very long time (minutes) to respond or never returns. In this case, going to a different browser I can get to the page with the same good initial response time
This suggests that maybe it’s an issue with browser cache or the server not invalidating a cache or something or just an implementation mistake on my part.
I will start adding more diagnostic messages in my server code to see what’s happening when but this really feels like I just have some setting not set correctly.
I take it as a given that BaseX, properly configured and using queries that are not poorly implemented should perform acceptably as a concurrent web server with not too much load (10s or 100s of users, not 1000s of users).
Cheers,
E.
_____________________________________________ Eliot Kimber Sr Staff Content Engineer O: 512 554 9368 M: 512 554 9368 servicenow.comhttps://www.servicenow.com LinkedInhttps://www.linkedin.com/company/servicenow | Twitterhttps://twitter.com/servicenow | YouTubehttps://www.youtube.com/user/servicenowinc | Facebookhttps://www.facebook.com/servicenow
From: Eliot Kimber eliot.kimber@servicenow.com Date: Sunday, February 6, 2022 at 12:31 AM To: basex-talk@mailman.uni-konstanz.de basex-talk@mailman.uni-konstanz.de Subject: Re: [basex-talk] Managing/Debugging Server Load and Performance Liam,
Here’s what free -h tells me:
/data/basex/git/dita-build-tools/src/main$ free -h total used free shared buff/cache available Mem: 15G 1.4G 8.3G 529M 6.0G 13G Swap: 0B 0B 0B
I read that as there being no swap. But note that this is a virtual machine (from AWS as far as I know)—I don’t know if that makes a difference.
If I understand what cpuinfo is telling me, I have 4 CPUs, each with 4 cores, so should be able to service 16 threads, yes?
I have DTD parsing turned off (which is the default)—if I didn’t do that it would take hours to load our content into the database, so that *shouldn’t* be a problem (I hope).
I did find prof:runtime() in the docs and verified that it has 4GB of RAM available.
_____________________________________________ Eliot Kimber Sr Staff Content Engineer O: 512 554 9368 M: 512 554 9368 servicenow.comhttps://www.servicenow.com LinkedInhttps://www.linkedin.com/company/servicenow | Twitterhttps://twitter.com/servicenow | YouTubehttps://www.youtube.com/user/servicenowinc | Facebookhttps://www.facebook.com/servicenow
From: Liam R. E. Quin liam@fromoldbooks.org Date: Sunday, February 6, 2022 at 12:08 AM To: Eliot Kimber eliot.kimber@servicenow.com, basex-talk@mailman.uni-konstanz.de basex-talk@mailman.uni-konstanz.de Subject: Re: [basex-talk] Managing/Debugging Server Load and Performance [External Email]
On Sun, 2022-02-06 at 03:34 +0000, Eliot Kimber wrote:
- Using the JRE provided with Oxygen, allocated with 4GB (we are
also using this server to run Oxygen via scripting and it needs 8GB to handle our insanely huge DITA maps)
Make sure you have e.g. 64 gigabytes or more of swap configures; free -h will tell you this
- Set parallel to 4 (to match the number of cores, but just
guessing that this is a useful setting based on the docs)
check /proc/cpuinfo (e.g, less /proc/cpuinfo) and you'll prolly find it can run 8 threads
I’m seeing some apparent occasional slowness on pages that should not be slow (don’t reflect long-running queries or huge data volumes)
make sure there are no xml catalogs or DTDs to be fetched externally - or, if there are catalogs, e.g. used with fn:transform(), that those catalog files do NOT start with a doctype that causes a network fetch of a dtd...
but I’m not really sure how to diagnose it or even verify that I’ve succeeded in giving BaseX all the resources it needs.
maybe in an ssh/terminal window, keep "top" running while you fetch a page, and see if the system gets really busy. Note also the centos system is probably using a hard drive, not an SSD, so file access may be slower - make sure you have indexes!
hope this helps at least a little,
-- Liam Quin, https://urldefense.com/v3/__https://www.delightfulcomputing.com/__;!!N4vogdj...https://urldefense.com/v3/__https:/www.delightfulcomputing.com/__;!!N4vogdjhuJM!U0hEyXTeoTf2B18jzQmUco2XDS97VUqRet5HS3OjWp_cIEbY9gMS9UJKn8aBFs61KYmFdA$ Available for XML/Document/Information Architecture/XSLT/ XSL/XQuery/Web/Text Processing/A11Y training, work & consulting. Barefoot Web-slave, antique illustrations: https://urldefense.com/v3/__http://www.fromoldbooks.org__;!!N4vogdjhuJM!U0hE...https://urldefense.com/v3/__http:/www.fromoldbooks.org__;!!N4vogdjhuJM!U0hEyXTeoTf2B18jzQmUco2XDS97VUqRet5HS3OjWp_cIEbY9gMS9UJKn8aBFs5ZXch2EA$
I take it as a given that BaseX, properly configured and using queries that are not poorly implemented should perform acceptably as a concurrent web server with not too much load (10s or 100s of users, not 1000s of users).
Absolutely. Our usual benchmark for commercial applications is to ensure that 90% (or more) of the queries take less than a second, and to only allow non-blocking queries to take more than 5 seconds.
Good to know.
I am seeing that when BaseX is doing some long-running process (constructing one of our big custom indexes) that it seems to consume all the BaseX cycles and the HTTP server becomes unresponsive.
Is this behavior something I can adjust with configuration or is it unavoidable? I see it on my mac as well so I don’t think it’s specific to the server configuration.
I should also mention that I understand these kinds of questions would normally be addressed through a consulting engagement with the BaseX team—I would definitely pursue that but at the moment time is of the essence as this use of BaseX was completely unplanned but suddenly very much desired…
Thanks,
Eliot
_____________________________________________ Eliot Kimber Sr Staff Content Engineer O: 512 554 9368 M: 512 554 9368 servicenow.comhttps://www.servicenow.com LinkedInhttps://www.linkedin.com/company/servicenow | Twitterhttps://twitter.com/servicenow | YouTubehttps://www.youtube.com/user/servicenowinc | Facebookhttps://www.facebook.com/servicenow
From: Christian Grün christian.gruen@gmail.com Date: Monday, February 7, 2022 at 8:19 AM To: Eliot Kimber eliot.kimber@servicenow.com Cc: basex-talk@mailman.uni-konstanz.de basex-talk@mailman.uni-konstanz.de Subject: Re: [basex-talk] Managing/Debugging Server Load and Performance [External Email]
I take it as a given that BaseX, properly configured and using queries that are not poorly implemented should perform acceptably as a concurrent web server with not too much load (10s or 100s of users, not 1000s of users).
Absolutely. Our usual benchmark for commercial applications is to ensure that 90% (or more) of the queries take less than a second, and to only allow non-blocking queries to take more than 5 seconds.
I am seeing that when BaseX is doing some long-running process (constructing one of our big custom indexes) that it seems to consume all the BaseX cycles and the HTTP server becomes unresponsive.
If you perform expensive write operations on databases that other users want to access, they’ll need to wait until this operation is finished.
There are various ways out: You can e.g. create temporary instances of your index databases and drop the operative database instances after your maintenance operations have finished. For high-availability use cases, you can also work with multiple BaseX instances.
[…] this use of BaseX was completely unplanned but suddenly very much desired…
Always good to hear!
I like the multiple instance case—I hadn’t consider that, but if I undertand how BaseX works with indexes, it should be as easy as making a backup from instance A and restoring it to instance B?
Or maybe just have a shared /data dir and be careful about which instance is accessing a given database so there are not concurrent writes?
I’m using node IDs as index lookup keys so I’d have to make sure that node IDs persist across backup and restore but surely that is the case?
But it certainly makes sense to do my long-running persistent index construction in one instance and then expose it through another as read-only for user access.
Will explore this.
Cheers,
E.
_____________________________________________ Eliot Kimber Sr Staff Content Engineer O: 512 554 9368 M: 512 554 9368 servicenow.comhttps://www.servicenow.com LinkedInhttps://www.linkedin.com/company/servicenow | Twitterhttps://twitter.com/servicenow | YouTubehttps://www.youtube.com/user/servicenowinc | Facebookhttps://www.facebook.com/servicenow
From: Christian Grün christian.gruen@gmail.com Date: Monday, February 7, 2022 at 12:28 PM To: Eliot Kimber eliot.kimber@servicenow.com Cc: basex-talk@mailman.uni-konstanz.de basex-talk@mailman.uni-konstanz.de Subject: Re: [basex-talk] Managing/Debugging Server Load and Performance [External Email]
I am seeing that when BaseX is doing some long-running process (constructing one of our big custom indexes) that it seems to consume all the BaseX cycles and the HTTP server becomes unresponsive.
If you perform expensive write operations on databases that other users want to access, they’ll need to wait until this operation is finished.
There are various ways out: You can e.g. create temporary instances of your index databases and drop the operative database instances after your maintenance operations have finished. For high-availability use cases, you can also work with multiple BaseX instances.
[…] this use of BaseX was completely unplanned but suddenly very much desired…
Always good to hear!
Hi Eliot -
On Mon, Feb 7, 2022 at 3:17 PM Eliot Kimber eliot.kimber@servicenow.com wrote:
I like the multiple instance case—I hadn’t consider that, but if I undertand how BaseX works with indexes, it should be as easy as making a backup from instance A and restoring it to instance B?
Or maybe just have a shared /data dir and be careful about which instance is accessing a given database so there are not concurrent writes?
You are probably already familiar with the filesystem locks, but in case
you aren't, see [1]
I’m using node IDs as index lookup keys so I’d have to make sure that node IDs persist across backup and restore but surely that is the case?
But it certainly makes sense to do my long-running persistent index construction in one instance and then expose it through another as read-only for user access.
I can't find it quickly, but I recall Marco Lettere writing about a
similar setup for some of his architecture. That may help when looking through the list archive.
Will explore this.
Cheers,
E.
Best, Bridger
[1] https://docs.basex.org/wiki/Transaction_Management#File-System_Locks
*Eliot Kimber*
Sr Staff Content Engineer
O: 512 554 9368
M: 512 554 9368
servicenow.com https://www.servicenow.com
LinkedIn https://www.linkedin.com/company/servicenow | Twitter https://twitter.com/servicenow | YouTube https://www.youtube.com/user/servicenowinc | Facebook https://www.facebook.com/servicenow
*From: *Christian Grün christian.gruen@gmail.com *Date: *Monday, February 7, 2022 at 12:28 PM *To: *Eliot Kimber eliot.kimber@servicenow.com *Cc: *basex-talk@mailman.uni-konstanz.de < basex-talk@mailman.uni-konstanz.de> *Subject: *Re: [basex-talk] Managing/Debugging Server Load and Performance
[External Email]
I am seeing that when BaseX is doing some long-running process
(constructing one of our big custom indexes) that it seems to consume all the BaseX cycles and the HTTP server becomes unresponsive.
If you perform expensive write operations on databases that other users want to access, they’ll need to wait until this operation is finished.
There are various ways out: You can e.g. create temporary instances of your index databases and drop the operative database instances after your maintenance operations have finished. For high-availability use cases, you can also work with multiple BaseX instances.
[…] this use of BaseX was completely unplanned but suddenly very much
desired…
Always good to hear!
I like the multiple instance case—I hadn’t consider that, but if I undertand how BaseX works with indexes, it should be as easy as making a backup from instance A and restoring it to instance B?
You could e.g. take advantage of db:copy and db:alter. For backups, db:create-backup and db:restore will be helpful [1].
Best, Christian
[1] https://docs.basex.org/wiki/Database_Module
Or maybe just have a shared /data dir and be careful about which instance is accessing a given database so there are not concurrent writes?
I’m using node IDs as index lookup keys so I’d have to make sure that node IDs persist across backup and restore but surely that is the case?
But it certainly makes sense to do my long-running persistent index construction in one instance and then expose it through another as read-only for user access.
Will explore this.
Cheers,
E.
Eliot Kimber
Sr Staff Content Engineer
O: 512 554 9368
M: 512 554 9368
servicenow.com
LinkedIn | Twitter | YouTube | Facebook
From: Christian Grün christian.gruen@gmail.com Date: Monday, February 7, 2022 at 12:28 PM To: Eliot Kimber eliot.kimber@servicenow.com Cc: basex-talk@mailman.uni-konstanz.de basex-talk@mailman.uni-konstanz.de Subject: Re: [basex-talk] Managing/Debugging Server Load and Performance
[External Email]
I am seeing that when BaseX is doing some long-running process (constructing one of our big custom indexes) that it seems to consume all the BaseX cycles and the HTTP server becomes unresponsive.
If you perform expensive write operations on databases that other users want to access, they’ll need to wait until this operation is finished.
There are various ways out: You can e.g. create temporary instances of your index databases and drop the operative database instances after your maintenance operations have finished. For high-availability use cases, you can also work with multiple BaseX instances.
[…] this use of BaseX was completely unplanned but suddenly very much desired…
Always good to hear!
I think I owe Liam at least a bottle of Scotch—I found some old code I had copied that was producing XHTML with the HTML DTD as the REST response. I replaced that with just HTML as the result type and it seems to have resolved my responsiveness problems.
I was also able to optimize my index creation and lookup significantly:
* Use node IDs for all lookup keys (was using URI/database paths * Eliminate one lookup by adding referencing map to my where-used table (I had just been recording the individual topicrefs, from which I can get the map, but that requires a second action to get the topicref’s containing document element) * Short-circuit long processes when I know I’m done looking for things (had a map used 700 times by 750 other maps but I know how many top-level maps there are, so when its list of containing-top-level maps is all of them, stop looking)
So my server is starting to work as I need it to…
Cheers,
E.
_____________________________________________ Eliot Kimber Sr Staff Content Engineer O: 512 554 9368 M: 512 554 9368 servicenow.comhttps://www.servicenow.com LinkedInhttps://www.linkedin.com/company/servicenow | Twitterhttps://twitter.com/servicenow | YouTubehttps://www.youtube.com/user/servicenowinc | Facebookhttps://www.facebook.com/servicenow
From: Liam R. E. Quin liam@fromoldbooks.org Date: Sunday, February 6, 2022 at 12:08 AM To: Eliot Kimber eliot.kimber@servicenow.com, basex-talk@mailman.uni-konstanz.de basex-talk@mailman.uni-konstanz.de Subject: Re: [basex-talk] Managing/Debugging Server Load and Performance [External Email]
On Sun, 2022-02-06 at 03:34 +0000, Eliot Kimber wrote:
- Using the JRE provided with Oxygen, allocated with 4GB (we are
also using this server to run Oxygen via scripting and it needs 8GB to handle our insanely huge DITA maps)
Make sure you have e.g. 64 gigabytes or more of swap configures; free -h will tell you this
- Set parallel to 4 (to match the number of cores, but just
guessing that this is a useful setting based on the docs)
check /proc/cpuinfo (e.g, less /proc/cpuinfo) and you'll prolly find it can run 8 threads
I’m seeing some apparent occasional slowness on pages that should not be slow (don’t reflect long-running queries or huge data volumes)
make sure there are no xml catalogs or DTDs to be fetched externally - or, if there are catalogs, e.g. used with fn:transform(), that those catalog files do NOT start with a doctype that causes a network fetch of a dtd...
but I’m not really sure how to diagnose it or even verify that I’ve succeeded in giving BaseX all the resources it needs.
maybe in an ssh/terminal window, keep "top" running while you fetch a page, and see if the system gets really busy. Note also the centos system is probably using a hard drive, not an SSD, so file access may be slower - make sure you have indexes!
hope this helps at least a little,
-- Liam Quin, https://urldefense.com/v3/__https://www.delightfulcomputing.com/__;!!N4vogdj...https://urldefense.com/v3/__https:/www.delightfulcomputing.com/__;!!N4vogdjhuJM!U0hEyXTeoTf2B18jzQmUco2XDS97VUqRet5HS3OjWp_cIEbY9gMS9UJKn8aBFs61KYmFdA$ Available for XML/Document/Information Architecture/XSLT/ XSL/XQuery/Web/Text Processing/A11Y training, work & consulting. Barefoot Web-slave, antique illustrations: https://urldefense.com/v3/__http://www.fromoldbooks.org__;!!N4vogdjhuJM!U0hE...https://urldefense.com/v3/__http:/www.fromoldbooks.org__;!!N4vogdjhuJM!U0hEyXTeoTf2B18jzQmUco2XDS97VUqRet5HS3OjWp_cIEbY9gMS9UJKn8aBFs5ZXch2EA$
On Tue, 2022-02-08 at 13:42 +0000, Eliot Kimber wrote:
I think I owe Liam at least a bottle of Scotch
Nah, the smile i got from this was worth far more! I'm glad you got it fixed.
Liam
basex-talk@mailman.uni-konstanz.de
-
Bridger Dyson-Smith -
Christian Grün -
Eliot Kimber -
Liam R. E. Quin