I think I owe Liam at least a bottle of Scotch—I found some old code I had copied that was producing XHTML with the HTML DTD as the REST response. I replaced that with just HTML as the result type and it seems to have resolved my responsiveness problems.

I was also able to optimize my index creation and lookup significantly:

Use node IDs for all lookup keys (was using URI/database paths
Eliminate one lookup by adding referencing map to my where-used table (I had just been recording the individual topicrefs, from which I can get the map, but that requires a second action to get the topicref’s containing document element)
Short-circuit long processes when I know I’m done looking for things (had a map used 700 times by 750 other maps but I know how many top-level maps there are, so when its list of containing-top-level maps is all of them, stop looking)

So my server is starting to work as I need it to…

Cheers,

_____________________________________________

Eliot Kimber

Sr Staff Content Engineer

O: 512 554 9368

M: 512 554 9368

servicenow.com

LinkedIn | Twitter | YouTube | Facebook

From: Liam R. E. Quin <liam@fromoldbooks.org>
Date: Sunday, February 6, 2022 at 12:08 AM
To: Eliot Kimber <eliot.kimber@servicenow.com>, basex-talk@mailman.uni-konstanz.de <basex-talk@mailman.uni-konstanz.de>
Subject: Re: [basex-talk] Managing/Debugging Server Load and Performance

[External Email]

On Sun, 2022-02-06 at 03:34 +0000, Eliot Kimber wrote:
>
> * Using the JRE provided with Oxygen, allocated with 4GB (we are
> also using this server to run Oxygen via scripting and it needs 8GB
> to handle our insanely huge DITA maps)

Make sure you have e.g. 64 gigabytes or more of swap configures;
free -h
will tell you this

> * Set parallel to 4 (to match the number of cores, but just
> guessing that this is a useful setting based on the docs)

check /proc/cpuinfo (e.g, less /proc/cpuinfo) and you'll prolly find it
can run 8 threads

>
> I’m seeing some apparent occasional slowness on pages that should not
> be slow (don’t reflect long-running queries or huge data volumes)

make sure there are no xml catalogs or DTDs to be fetched externally -
or, if there are catalogs, e.g. used with fn:transform(), that those
catalog files do NOT start with a doctype that causes a network fetch
of a dtd...

> but I’m not really sure how to diagnose it or even verify that I’ve
> succeeded in giving BaseX all the resources it needs.

maybe in an ssh/terminal window, keep "top" running while you fetch a
page, and see if the system gets really busy. Note also the centos
system is probably using a hard drive, not an SSD, so file access may
be slower - make sure you have indexes!

hope this helps at least a little,

--
Liam Quin, https://urldefense.com/v3/__https://www.delightfulcomputing.com/__;!!N4vogdjhuJM!U0hEyXTeoTf2B18jzQmUco2XDS97VUqRet5HS3OjWp_cIEbY9gMS9UJKn8aBFs61KYmFdA$
Available for XML/Document/Information Architecture/XSLT/
XSL/XQuery/Web/Text Processing/A11Y training, work & consulting.
Barefoot Web-slave, antique illustrations: https://urldefense.com/v3/__http://www.fromoldbooks.org__;!!N4vogdjhuJM!U0hEyXTeoTf2B18jzQmUco2XDS97VUqRet5HS3OjWp_cIEbY9gMS9UJKn8aBFs5ZXch2EA$