I think I owe Liam at least a bottle of Scotch—I found some old code I had copied that was producing XHTML with the HTML DTD as the REST response. I replaced that with just HTML as the result type and it seems to have resolved my responsiveness problems.
I was also able to optimize my index creation and lookup significantly:
* Use node IDs for all lookup keys (was using URI/database paths * Eliminate one lookup by adding referencing map to my where-used table (I had just been recording the individual topicrefs, from which I can get the map, but that requires a second action to get the topicref’s containing document element) * Short-circuit long processes when I know I’m done looking for things (had a map used 700 times by 750 other maps but I know how many top-level maps there are, so when its list of containing-top-level maps is all of them, stop looking)
So my server is starting to work as I need it to…
Cheers,
E.
_____________________________________________ Eliot Kimber Sr Staff Content Engineer O: 512 554 9368 M: 512 554 9368 servicenow.comhttps://www.servicenow.com LinkedInhttps://www.linkedin.com/company/servicenow | Twitterhttps://twitter.com/servicenow | YouTubehttps://www.youtube.com/user/servicenowinc | Facebookhttps://www.facebook.com/servicenow
From: Liam R. E. Quin liam@fromoldbooks.org Date: Sunday, February 6, 2022 at 12:08 AM To: Eliot Kimber eliot.kimber@servicenow.com, basex-talk@mailman.uni-konstanz.de basex-talk@mailman.uni-konstanz.de Subject: Re: [basex-talk] Managing/Debugging Server Load and Performance [External Email]
On Sun, 2022-02-06 at 03:34 +0000, Eliot Kimber wrote:
- Using the JRE provided with Oxygen, allocated with 4GB (we are
also using this server to run Oxygen via scripting and it needs 8GB to handle our insanely huge DITA maps)
Make sure you have e.g. 64 gigabytes or more of swap configures; free -h will tell you this
- Set parallel to 4 (to match the number of cores, but just
guessing that this is a useful setting based on the docs)
check /proc/cpuinfo (e.g, less /proc/cpuinfo) and you'll prolly find it can run 8 threads
I’m seeing some apparent occasional slowness on pages that should not be slow (don’t reflect long-running queries or huge data volumes)
make sure there are no xml catalogs or DTDs to be fetched externally - or, if there are catalogs, e.g. used with fn:transform(), that those catalog files do NOT start with a doctype that causes a network fetch of a dtd...
but I’m not really sure how to diagnose it or even verify that I’ve succeeded in giving BaseX all the resources it needs.
maybe in an ssh/terminal window, keep "top" running while you fetch a page, and see if the system gets really busy. Note also the centos system is probably using a hard drive, not an SSD, so file access may be slower - make sure you have indexes!
hope this helps at least a little,
-- Liam Quin, https://urldefense.com/v3/__https://www.delightfulcomputing.com/__;!!N4vogdj...https://urldefense.com/v3/__https:/www.delightfulcomputing.com/__;!!N4vogdjhuJM!U0hEyXTeoTf2B18jzQmUco2XDS97VUqRet5HS3OjWp_cIEbY9gMS9UJKn8aBFs61KYmFdA$ Available for XML/Document/Information Architecture/XSLT/ XSL/XQuery/Web/Text Processing/A11Y training, work & consulting. Barefoot Web-slave, antique illustrations: https://urldefense.com/v3/__http://www.fromoldbooks.org__;!!N4vogdjhuJM!U0hE...https://urldefense.com/v3/__http:/www.fromoldbooks.org__;!!N4vogdjhuJM!U0hEyXTeoTf2B18jzQmUco2XDS97VUqRet5HS3OjWp_cIEbY9gMS9UJKn8aBFs5ZXch2EA$