- BaseX-Talk - mailman.uni-konstanz.de

Query optimization: What can I check or measure?
by Eliot Kimber 16 Jul '24

16 Jul '24

I am loading page view data exported from Adobe as CSV as XML with a structure like: <csv timestamp="2024-01-24"> <record language="en-US" family="vancouver" bundlename="bundle-platcap-platform-capabilities" topicpath="administer/general/concept/capabilities-bundle-landingpage.dita" notviewed="viewed" viewcount="120587"> <entry name="URL">docs.servicenow.com/bundle/vancouver-servicenow-platform/page/administer/ge…</entry> <entry name="Page Views">120587</entry> <entry name="Unique Visitors">50863</entry> <entry name="Bounce Rate">0.255523003</entry> <entry name="Content Velocity">3.289644475</entry> <entry name="Docs Engagement Rate">0.289549233</entry> </record> … 50K more records … </csv> For this database I have a token index and the MAXLEN value is set to 255 to ensure that the @topicpath values will be tokenized (none should be longer than 255). I’m measuring a consistent 0.3 seconds for this query: let $docRecords as element()* := db:token($analyticsmgmt:analyticsDatabase, $docPath, 'topicpath')/.. Where $docPath is a value that will match tokens in the @topicpath attribute. Based on other token-based optimizations I’ve done, I would expect faster results, at least 10-times faster. My questions: 1. What can I look for that might be making this lookup slower than expected or, conversely, how can I prove that this is the fastest the query will return? 2. Is there a better way to manage this kind of tabular data with many 1000s of records within BaseX? For example, does it matter if the records are part of a single document or would it be better to have each record be a separate document? Thanks, Eliot _____________________________________________ Eliot Kimber Sr Staff Content Engineer Digital Content & Design O: 512 554 9368 M: 512 554 9368 servicenow.com<https://www.servicenow.com> LinkedIn<https://www.linkedin.com/company/servicenow> | Twitter<https://twitter.com/servicenow> | YouTube<https://www.youtube.com/user/servicenowinc> | Facebook<https://www.facebook.com/servicenow>

2 4

How can I reset my password for the basex website?
by Ben Engbers 16 Jul '24

16 Jul '24

I wanted to add a link in the server clients page for my C++ shared lib but iIt seems that my password has changed. When trying to login I only see a message that I should check my login data. How can I reset that password? Ben Ben Engbers

2 3

Puzzled by timing difference of two similar queries
by Bernhard Liebl 16 Jul '24

16 Jul '24

Hello, as a rather unexperienced xquery and basex user, I'm puzzled by the timings of running the same xquery on two very similar xml files. For an academic paper, I'm trying to benchmark finding certain successive (windowed) <item> tags inside a <seq> tag. This is the full query: declare function local:check-sequence($items as element(item)*) as element(item)* { if ($items[1]/@x = $items[2]/@x and $items[2]/@x = $items[3]/@x) then if (sum($items/@y) < 0.5) then $items else () else () }; for $seq in doc("data.xml")//seq return <seq>{ for $i in 1 to (count($seq/item) - 2) let $window := subsequence($seq/item, $i, 3) return <match>{local:check-sequence($window)}</match> }</seq> Here's the puzzling thing. When I run this on an xml file that has one <seq> with 10000 <item>s, i.e. <seqs><seq> <item>[*10000] </seq></seqs>, I get the following reasonable timings: - Parsing: 0.63 ms - Compiling: 0.95 ms - Optimizing: 30.59 ms - Evaluating: 11.48 ms - Printing: 0.61 ms - Total Time: 44.27 ms However, once I run the same query on an xml file with two (!) <seq> of 10000 items each, the runtime does not double, but is now more than 50 times of that of the first query: - Parsing: 0.6 ms - Compiling: 1.3 ms - Optimizing: 48.13 ms - Evaluating: 2843.67 ms - Printing: 1.12 ms - Total Time: 2894.82 ms I don't see how I induce any exponential runtimes, and esp. check-sequence should be constant. Am I missing something fundamental here? The result size of the second version is about twice the size as the result size of the first version. Thanks in advance for any pointers. Bernhard Liebl

4 5

A new C++ client
by Ben Engbers 16 Jul '24

16 Jul '24

Before I started developing the R client for BaseX several years ago, I used to program regularly in SWI-Prolog. And I always wondered afterwards about the possible benefits of using Prolog in combination with XQuery. So after I finished developing RbaseX, I started developing ProBaseX. I did not manage to set up a connection in Swipl. Therefore, my initial plan to develop ProBasex entirely in SWI-prolog was changed to a plan to first set up a connection in C++ and then use that connection. This work eventually resulted in a linux shared library that implements the full server protocol. I will now use that library for further development work. LibBasexCpp and the test program libBasexTest.cpp are now on the Internet: https://github.com/BenEngbers/libBasexCpp I am curious to see if this library fills a need and if it makes sense to continue working on a dual platform version. Ben Engbers

2 2

feature request: control environment of external processes
by Hauke Brandes 16 Jul '24

16 Jul '24

Hello list, I would like to suggest an enhancement for the proc:system/execute/fork family of functions: it should be possible to control the environment variables of the external process. At the moment, the environment is always inherited from the calling basex process, as far as I can tell. There are several reasons why it is desirable to control the environment of external processes: * set variables to values that are not known statically when starting basex * avoid information leaking (restrict environment to the minimal required subset) * control the PATH of the external process It would be great to have an additional, optional entry in the $options map (maybe named "env" or "environment") that contains a map of environment variables for the external process. If absent, the current behavior (inherited environment) should be used. If the option is given, **only** the environment variables in the map should be used for the external process. Effectively, the "env"/"environment" option would behave as if it had a default value of `map:merge( available-environment-variables() ! map:entry(., environment-variable(.)) )` The ProcessBuilder in Java should allow for a straightforward implementation. What do you think? Cheers, Hauke

2 1

How best to handle long-running REST requests?
by Eliot Kimber 15 Jul '24

15 Jul '24

(Using 10.7 for now—upgrade to 11 is planned but not scheduled yet.) In my web app I have implemented client-side on-load JavaScript that implements deferred loading of the results of long-running queries. The JS finds elements that represent callbacks to the server and makes REST calls that are then dispatched to an appropriate worker server based on worker load. These queries can take minutes to run and return a result back to the browser, where the result is added to the HTML page, replacing the original element. In order for the server to maintain the HTTP connection while the query runs I had to turn off the server time out. This works but seems like a crude solution. My question: Is there a better way to manage this kind of client-to-server-to-client communication for long-running queries? For example, is there a way to have the response timeout be more dynamic? I didn’t see anything but I haven’t had time to push on it hard. The result I was getting in the browser with the default 30-second timeout was that when the response eventually returned, the browser reported that the connection had already been closed so it couldn’t do anything with the result. I’m using fetch() to get the result: fetch(url).then(function(response) { return response.text(); }).then(function(data) { const parser = new DOMParser(); const doc = parser.parseFromString(data, 'text/html'); … update web page … } Where “url” is the REST call back to my server. My infrastructure *should* be protecting against long-running queries blocking the main web-serving worker, but it still seems dangerous to have no time out (or a timeout set to 10 minutes or something). Thanks, Eliot _____________________________________________ Eliot Kimber Sr Staff Content Engineer Digital Content & Design O: 512 554 9368 M: 512 554 9368 servicenow.com<https://www.servicenow.com> LinkedIn<https://www.linkedin.com/company/servicenow> | Twitter<https://twitter.com/servicenow> | YouTube<https://www.youtube.com/user/servicenowinc> | Facebook<https://www.facebook.com/servicenow>

2 1

XSD 1.1 validation: base-uri(.) in the assert
by ykhabins＠bellsouth.net 14 Jul '24

14 Jul '24

Here is the latest development on the subject. I communicated with Mukul Gandhi. He is a chief maintainer of the Xerces-J XSD 1.1 project. Please see below his reply with the good news. >Hi Yitzhak, > >I'm well and hoping same for you. > >I could fix this bug, and the fix for this has been committed to Xerces-J's XML Schema 1.1 dev repos branch at, https://github.com/apache/xerces-j/tree/xml-schema-1.1-dev. > >I've also created a jira issue for this at, https://issues.apache.org/jira/browse/XERCESJ-1765. > >If you wish, you may build and use Xerces-J's XML Schema 1.1 implementation from repos url cited above. > >-- >Regards, >Mukul Gandhi Regards, Yitzhak Khabinsky

1 0

XSD 1.1 validation: base-uri(.) in the assert
by ykhabins＠bellsouth.net 11 Jul '24

11 Jul '24

Hello, We are using the final release BaseX 11.0 and Xerces 2.12.2 XSD 1.1 engine. There is a need to validate XML file and check XML file name for naming convention. Unfortunately, the assert with the base-uri(.) function is not working as expected in the assert expression. Observations: The XPath expression on the XML file emits, as expected, the following: o XPath: base-uri(.) o Results: file:/C:/Users/i300179/Downloads/ATF_NESS/base-uri/input.xml The XPath expression on the XML file emits, as expected, the following: o XPath: tokenize(base-uri(.),"/")[last()] o Results: input.xml I tried multiple expressions around base-uri(.) function. None of them worked. You will find all my attempts in the XSD file. The test case XQuery/XML/XSD is below. We are using Xerces, but the following Saxon excerpt could be pointing to the root cause of the problem. For the reference: https://www.saxonica.com/html/documentation10/functions/fn/base-uri.html "It is possible for the base URI of a document to be absent (unknown). This happens when the document is supplied to Saxon with no location information: for example when the node is represented using a tree model such as DOM that has no base URI information; or when a Source is supplied for which getSystemId() returns null. The most common example of this is a Source created using new StreamSource(new FileInputStream("input.xml")). The problem can be fixed by using the second argument of the StreamSource constructor, or by using the constructor that expects a File rather than an InputStream." XSD11_Validation.xq ======================= declare variable $xml as xs:string external; declare variable $xsd as xs:string external; let $result := validate:xsd-report($xml, $xsd, map { 'http://apache.org/xml/features/validation/cta-full-xpath-checking': true(), 'cache': false() }) return $result input.xml ======================= <root> <event> <id>1</id> </event> </root> input.xsd ======================== <?xml version="1.0" encoding="UTF-8"?> <xs:schema version="1.0.1" attributeFormDefault="unqualified" elementFormDefault="qualified" xmlns:xs="http://www.w3.org/2001/XMLSchema" xmlns:vc="http://www.w3.org/2007/XMLSchema-versioning" xmlns:xerces="http://xerces.apache.org" vc:minVersion="1.1"> <xs:element name="root"> <xs:complexType> <xs:sequence> <xs:element maxOccurs="unbounded" ref="event"/> </xs:sequence>  <xs:assert test="tokenize(base-uri(.), '/')[last()] eq 'input.xml'" xerces:message="Rule #21: XML file name shall match file naming convention">   </xs:assert> </xs:complexType> </xs:element> <xs:element name="event"> <xs:complexType> <xs:sequence> <xs:element ref="id"/> </xs:sequence> </xs:complexType> </xs:element> <xs:element name="id" type="xs:integer"/> </xs:schema> Regards, Yitzhak Khabinsky

2 1

Re: [basex-talk] XSLT with multiple outputs in BaseX 11.0
by ykhabins＠bellsouth.net 03 Jul '24

03 Jul '24

I completely concur. BaseX is a standard bearer of the latest XML standards. We are mostly using BaseX with its HTTP listener for the following: 1. XSD 1.1 validations (via Xerces 12.2.2) 2. XSLT 3.0 transformations (via Saxon engine) and not just as a XML database platform with its XQuery 4.0 support. Regards, Yitzhak Khabinsky

1 0

XSLT with multiple outputs in BaseX 11.0
by ykhabins＠bellsouth.net 03 Jul '24

03 Jul '24

Hello, I am using the latest BaseX 11.0 final. I am running an XSLT 3.0 transformation, by using SaxonHE12-4J, and hit a snag with it. The goal of the XSLT is to split large input XML file into few smaller XML files. The test case files: XML, XSLT, and xq, are below. The XSLT transformation is working as expected, producing multiple output files on the file system. Alas, BaseX GUI emits the following error: Error: Stopped at C:/Program Files (x86)/basex/src/xslt_transform30.xq, 6/15: [FODC0002] "" (Line -1): Premature end of file. My XSLT is a little bit unusual. It is leveraging powerful XSLT 3.0 functionality. It doesn't have one single output, but many output files directly to the file system, IMHO, that's the reason for the BaseX error. BaseX is expecting at least some output back from the XSLT transformation. For comparison, the Oxygen XML IDE is working as expected with such XSLT. What could be done to fix it? May be BaseX behavior needs some adjustment, so it will accept blank output as normal for the xslt:transform($input, $stylesheet)? BaseX is famous for its latest standards support. Modern XSLT versions, i.e. above XSLT 1.0, support multiple outputs. _xslt_transform30.xq =========================================== xquery version "4.0"; declare variable $input as xs:string external; declare variable $stylesheet as xs:string external; xslt:transform($input, $stylesheet) =========================================== Input.xml =========================================== <doc title="example document"> <chapter title="chapter 1"> <page>page 1</page> <page>page 2</page> </chapter> <chapter title="chapter 2"> <page>page 3</page> </chapter> </doc> =========================================== SplitProcess.xslt =========================================== <?xml version="1.0" encoding="UTF-8"?> <xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform" xmlns:xs="http://www.w3.org/2001/XMLSchema" xmlns:math="http://www.w3.org/2005/xpath-functions/math" exclude-result-prefixes="xs math" version="3.0" expand-text="yes"> <xsl:output indent="yes" method="xml"/>  <xsl:param name="directory" select="'c:/Temp/XSLT30/Split up a file into several files'"/> <xsl:template match="/"> <xsl:apply-templates select="doc/chapter/page"/> </xsl:template> <xsl:template match="page"> <xsl:result-document href="file:///{$directory}/page{position()}.xml"> <xsl:copy-of select="root(snapshot())"/> </xsl:result-document> </xsl:template> </xsl:stylesheet> =========================================== Regards, Yitzhak Khabinsky

5 4