BaseX-Talk July 2024

basex-talk@mailman.uni-konstanz.de

18 participants
23 discussions

BaseX 9.6: The Summer Edition
by Christian Grün 28 Nov '24

28 Nov '24

Dear all, We provide you with a new and fresh version of BaseX, our open source XML framework, database system and XQuery 3.1 processor: https://basex.org/ Apart from our main focus (query rewritings and optimizations), we have added the following enhancements: XQUERY: MODULES, FEATURES - Archive Module, archive:write: stream large archives to file - SQL Module: support for more SQL types - Full-Text Module, ft:thesaurus: perform Thesaurus queries - Fulltext, fuzzy search: specify Levenshtein limit - UNROLLLIMIT option: control limit for unrolling loops XQUERY: JAVA BINDINGS - Java objects of unknown type are wrapped into function items - results of constructor calls are returned as function items - the standard package "java.lang." has become optional - array arguments can be specified with the middle dot notation - conversion can be controlled with the WRAPJAVA option - better support for XQuery arrays and maps WEB APPLICATIONS - RESTXQ: Server-Timing HTTP headers are attached to the response For a more comprehensive list of added and updated features, look into our documentation (docs.basex.org) and check out the GitHub issues (github.com/BaseXdb/basex/issues). Have fun, Your BaseX Team

11 43

recursively used variables
by Rob Stapper 12 Aug '24

12 Aug '24

Hi, The code[1] below and send as attachment generates a error message: “Static variable depends on itself: $Q{http://www.w3.org/2005/xquery-local-functions}test”. I use these variables to refer to my private functions in my modules so I can easyly refer to them in a inheritance situation. It’s not a big problem for me but I was wondering if the error-triggering is justified or that it should work. [1]=========================================== declare variable $local:test := local:test#1 ; declare %private function local:test( $i) { if ( $i > 0) then $local:test( $i - 1) } ; $local:test( 10) =========================================== Kind regards, Rob Stapper Sent from Mail for Windows 10 -- This email has been checked for viruses by Avast antivirus software. https://www.avast.com/antivirus

4 7

Performance & scaling
by Csaba Fekete 29 Jul '24

29 Jul '24

Hi I have read the Advanced User Guide section in the documentation and I got a lot of useful information but I'm still a little bit stuck. I have large (1 GB+) mostly single document databases and complex queries that often take too long to execute. Apart from optimizing the queries, what are my options to speed things up? Any ideas are welcome. 1) Vertical scaling - how does a single BaseX instance respond to adding more CPUs or RAM? Which one could improve performance? Apart from indexing, are there any fine-tuning options that could help improve performance? 2) Partitioning: would it make sense to break my documents into several smaller ones, run the queries against them in parallel and then merge the results? Has anyone done such optimization? Did it work? 2) Horizontal scaling: the docs read: *"The BaseX client-server architecture offers ACID-safe transactions, with multiple readers and writers."* - Where can I get a guide on how to set up a single writer - multiple readers architecture? Thank you Csaba

2 1

Duplicate IDs and function id()
by Andrew Sales 26 Jul '24

26 Jul '24

Hello, If I understand the spec for this function[1] correctly, at most one element should be returned for a given ID value: "If several elements have the same ID value, then E is the one that is first in document order." ("Rules", item 3) BaseX behaves differently from e.g. Saxon in this respect. The query: let $doc := document{<foo xml:id='a1'><bar xml:id='a1'/></foo>} return ( 'Saxon-HE 11.6:', xslt:transform($doc, <xsl:stylesheet xmlns:xsl=" http://www.w3.org/1999/XSL/Transform" xmlns:xs="http://www.w3.org/2001/XMLSchema" exclude-result-prefixes="xs" version="3.0"> <xsl:template match="/"> <xsl:sequence select="id('a1')"/> </xsl:template> </xsl:stylesheet>), 'BaseX 10.7:', $doc/id('a1') ) returns: Saxon-HE 11.6: <foo xml:id="a1"><bar xml:id="a1"/></foo> BaseX 10.7: <foo xml:id="a1"><bar xml:id="a1"/></foo> <bar xml:id="a1"/> i.e. Saxon returns only the first in document order, whereas BaseX returns both of them. Grateful for your thoughts, Andrew [1] https://www.w3.org/TR/xpath-functions-31/#func-id

2 1

BaseX 9.7 to BaseX 11: find a regex in an external file
by Josselin Morvan 19 Jul '24

19 Jul '24

Hi Everyone, I'm migrating a webapp from BaseX 9.7 to BaseX 11 and I'm having a problem with a wrapper function (just a function for adding content to an xml template file) Basically, this function looks for nodes in a template.xml file that matche regex to update the content of this file. Here is a small example : template.xml <foo> <bar>{content}</bar> </foo> Query: let $file := 'template.xml' let $wrap := fn:doc($file) let $regex := '\{(.+?)\}’ (: looking for something between curly braces :) return for $node in $data//* where $node[fn:matches(text(), $regex)] return $node This function worked with BaseX 9.7, but returns an error with BaseX 11 : [XPTY0004] Item expected, sequence found: ("
 ", "
"). However, if I put the content of template.xml in a variable, it works (even with BaseX 11): let $wrap := <foo> <bar>{{content}}</bar> (: just escaped curly braces :) </foo> let $regex := '\{(.+?)\}' return for $node in $wrap//* where $node[fn:matches(text(), $regex)] return $node (:<bar>{content}</bar>:) Do you have any idea of what I'm doing wrong or what I’m missing? Thanks for your help! Best, Josselin

2 2

Server becomes unresponsive after several large database updates (db:delete())
by Eliot Kimber 17 Jul '24

17 Jul '24

(Using 10.7 on Centos): I have some databases with many large documents (1000s of docs ranging from 6KB to 2MB). I need to clean out older files, so I wrote this simple query, which looks at the time stamps in the doc filenames: let $db := '_valrep_validation_reports_internal' let $paths := (db:open($db)/* ! db:path(.)[matches(., '2024-0[123456]')] => sort()) return for $path in $paths return db:delete($db, $path) With only the dba app deployed, I ran this query on the four databases and it ran in a few seconds and removed the files as expected, but then I ran it on one more database and the query never returned and the dba app became unresponsive (Databases view doesn’t refresh). On the server I can see that a Java process, which can only be BaseX on this server, is at 100%: PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 8376 eliot.k+ 20 0 9251688 1.1g 16124 S 101.0 3.5 14:02.42 java What would cause this kind of behavior? Cheers, E. _____________________________________________ Eliot Kimber Sr Staff Content Engineer Digital Content & Design O: 512 554 9368 M: 512 554 9368 servicenow.com<https://www.servicenow.com> LinkedIn<https://www.linkedin.com/company/servicenow> | Twitter<https://twitter.com/servicenow> | YouTube<https://www.youtube.com/user/servicenowinc> | Facebook<https://www.facebook.com/servicenow>

1 0

Query optimization: What can I check or measure?
by Eliot Kimber 16 Jul '24

16 Jul '24

I am loading page view data exported from Adobe as CSV as XML with a structure like: <csv timestamp="2024-01-24"> <record language="en-US" family="vancouver" bundlename="bundle-platcap-platform-capabilities" topicpath="administer/general/concept/capabilities-bundle-landingpage.dita" notviewed="viewed" viewcount="120587"> <entry name="URL">docs.servicenow.com/bundle/vancouver-servicenow-platform/page/administer/ge…</entry> <entry name="Page Views">120587</entry> <entry name="Unique Visitors">50863</entry> <entry name="Bounce Rate">0.255523003</entry> <entry name="Content Velocity">3.289644475</entry> <entry name="Docs Engagement Rate">0.289549233</entry> </record> … 50K more records … </csv> For this database I have a token index and the MAXLEN value is set to 255 to ensure that the @topicpath values will be tokenized (none should be longer than 255). I’m measuring a consistent 0.3 seconds for this query: let $docRecords as element()* := db:token($analyticsmgmt:analyticsDatabase, $docPath, 'topicpath')/.. Where $docPath is a value that will match tokens in the @topicpath attribute. Based on other token-based optimizations I’ve done, I would expect faster results, at least 10-times faster. My questions: 1. What can I look for that might be making this lookup slower than expected or, conversely, how can I prove that this is the fastest the query will return? 2. Is there a better way to manage this kind of tabular data with many 1000s of records within BaseX? For example, does it matter if the records are part of a single document or would it be better to have each record be a separate document? Thanks, Eliot _____________________________________________ Eliot Kimber Sr Staff Content Engineer Digital Content & Design O: 512 554 9368 M: 512 554 9368 servicenow.com<https://www.servicenow.com> LinkedIn<https://www.linkedin.com/company/servicenow> | Twitter<https://twitter.com/servicenow> | YouTube<https://www.youtube.com/user/servicenowinc> | Facebook<https://www.facebook.com/servicenow>

2 4

How can I reset my password for the basex website?
by Ben Engbers 16 Jul '24

16 Jul '24

I wanted to add a link in the server clients page for my C++ shared lib but iIt seems that my password has changed. When trying to login I only see a message that I should check my login data. How can I reset that password? Ben Ben Engbers

2 3

Puzzled by timing difference of two similar queries
by Bernhard Liebl 16 Jul '24

16 Jul '24

Hello, as a rather unexperienced xquery and basex user, I'm puzzled by the timings of running the same xquery on two very similar xml files. For an academic paper, I'm trying to benchmark finding certain successive (windowed) <item> tags inside a <seq> tag. This is the full query: declare function local:check-sequence($items as element(item)*) as element(item)* { if ($items[1]/@x = $items[2]/@x and $items[2]/@x = $items[3]/@x) then if (sum($items/@y) < 0.5) then $items else () else () }; for $seq in doc("data.xml")//seq return <seq>{ for $i in 1 to (count($seq/item) - 2) let $window := subsequence($seq/item, $i, 3) return <match>{local:check-sequence($window)}</match> }</seq> Here's the puzzling thing. When I run this on an xml file that has one <seq> with 10000 <item>s, i.e. <seqs><seq> <item>[*10000] </seq></seqs>, I get the following reasonable timings: - Parsing: 0.63 ms - Compiling: 0.95 ms - Optimizing: 30.59 ms - Evaluating: 11.48 ms - Printing: 0.61 ms - Total Time: 44.27 ms However, once I run the same query on an xml file with two (!) <seq> of 10000 items each, the runtime does not double, but is now more than 50 times of that of the first query: - Parsing: 0.6 ms - Compiling: 1.3 ms - Optimizing: 48.13 ms - Evaluating: 2843.67 ms - Printing: 1.12 ms - Total Time: 2894.82 ms I don't see how I induce any exponential runtimes, and esp. check-sequence should be constant. Am I missing something fundamental here? The result size of the second version is about twice the size as the result size of the first version. Thanks in advance for any pointers. Bernhard Liebl

4 5

A new C++ client
by Ben Engbers 16 Jul '24

16 Jul '24

Before I started developing the R client for BaseX several years ago, I used to program regularly in SWI-Prolog. And I always wondered afterwards about the possible benefits of using Prolog in combination with XQuery. So after I finished developing RbaseX, I started developing ProBaseX. I did not manage to set up a connection in Swipl. Therefore, my initial plan to develop ProBasex entirely in SWI-prolog was changed to a plan to first set up a connection in C++ and then use that connection. This work eventually resulted in a linux shared library that implements the full server protocol. I will now use that library for further development work. LibBasexCpp and the test program libBasexTest.cpp are now on the Internet: https://github.com/BenEngbers/libBasexCpp I am curious to see if this library fills a need and if it makes sense to continue working on a dual platform version. Ben Engbers

2 2

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

BaseX-Talk July 2024