BaseX-Talk December 2023

basex-talk@mailman.uni-konstanz.de

9 participants
10 discussions

BaseX 9.6: The Summer Edition
by Christian Grün 28 Nov '24

28 Nov '24

Dear all, We provide you with a new and fresh version of BaseX, our open source XML framework, database system and XQuery 3.1 processor: https://basex.org/ Apart from our main focus (query rewritings and optimizations), we have added the following enhancements: XQUERY: MODULES, FEATURES - Archive Module, archive:write: stream large archives to file - SQL Module: support for more SQL types - Full-Text Module, ft:thesaurus: perform Thesaurus queries - Fulltext, fuzzy search: specify Levenshtein limit - UNROLLLIMIT option: control limit for unrolling loops XQUERY: JAVA BINDINGS - Java objects of unknown type are wrapped into function items - results of constructor calls are returned as function items - the standard package "java.lang." has become optional - array arguments can be specified with the middle dot notation - conversion can be controlled with the WRAPJAVA option - better support for XQuery arrays and maps WEB APPLICATIONS - RESTXQ: Server-Timing HTTP headers are attached to the response For a more comprehensive list of added and updated features, look into our documentation (docs.basex.org) and check out the GitHub issues (github.com/BaseXdb/basex/issues). Have fun, Your BaseX Team

11 43

recursively used variables
by Rob Stapper 12 Aug '24

12 Aug '24

Hi, The code[1] below and send as attachment generates a error message: “Static variable depends on itself: $Q{http://www.w3.org/2005/xquery-local-functions}test”. I use these variables to refer to my private functions in my modules so I can easyly refer to them in a inheritance situation. It’s not a big problem for me but I was wondering if the error-triggering is justified or that it should work. [1]=========================================== declare variable $local:test := local:test#1 ; declare %private function local:test( $i) { if ( $i > 0) then $local:test( $i - 1) } ; $local:test( 10) =========================================== Kind regards, Rob Stapper Sent from Mail for Windows 10 -- This email has been checked for viruses by Avast antivirus software. https://www.avast.com/antivirus

4 7

xslt:transform-report result
by Andy Bunce 11 Mar '24

11 Mar '24

Hi, Using BaseX 9.7.1 and saxon9he-9.9.1.jar The documentation suggests the ?result from xslt:transform-report should be *a* document-node where possible [1] This seems not quite to be the case when there are processing instructions or comments at the top level. In these cases a sequence of document-nodes is returned. /Andy [1] https://docs.basex.org/wiki/XSLT_Module#xslt:transform-report let $xslt:=<xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform" version="3.0"> <xsl:mode on-no-match="shallow-copy"/> </xsl:stylesheet> let $xml:=document{ <?something type="compact"?>, <foo/>} return xslt:transform-report($xml,$xslt) Returns map { "messages": (), "result": (<?something type="compact"?>, <foo/>) }

3 7

Help - Regarding Performance Improvement
by Deepak Dinakara 03 Jan '24

03 Jan '24

Hi, Reaching out to get suggestions on improving performance. Using basex to store and analyze around 350,000 to 500,000 XMLs. Size of each XML varies between a few KBs to 5MB. Each day around 10k XMLs get added/patched. I have the following queries 1) What is the optimal size or number of documents in a DB? Earlier I had 1 DB with different collections but inserts were too slow, took more than 30s just to replace a document. So split it up by some category to have around 30 DBs. Inserts are fine but again if there are too many documents in a category, patching that DB slows and querying across all DBs also gets slowed down. Any optimal number for DBs? Can I create many DBs like 1 for every 10K XMLs? I read through https://www.mail-archive.com/basex-talk@mailman.uni-konstanz.de/msg06310.ht…, having 100s of DBs cause query performance degradation? Is there any better solution? 2) Query performance has degraded with more documents in a DB. I also noticed that with/without token/attribute index, there is not much difference to query performance (they are just XML attribute queries). "Optimize" flag after inserts to recreate the index takes too much time and memory. I am not running it now since I didn't find significant improvement with/without index with my tests. Any suggestions for improving this? 3) Is it possible to just run queries against specific XMLs? I will have a pre-filter based on user selection and queries need to be run against only those XMLs. There are a number of filters users can apply and every time it can result in a different set of XMLs against which analysis has to be performed (Hence not feasible to create so many collections). Right now, I am querying against all XMLs even though I am interested only in a subset of XMLs and doing post filtering. I did go through https://mailman.uni-konstanz.de/pipermail/basex-talk/2010-July/000495.html, but again having a regex to include all the interested file paths(sometimes entire set of documents) will slow it down. Thank you, Deepak

2 2

XQuery fold-left Problem
by Marco Duiker - LandGoed 19 Dec '23

19 Dec '23

Please help me with the following problem: I try to construct the current status of a series of objects along the following simple rules: * the current status of on object with a certain `identificatie` (id) is the newest version of that object * except when the newest object has the property `<status>beëindigen</status>` then the object stops existing. I use a fold-left to proces these rules with a simple function implementing those rules. This gives me unexpected results. I've reduced the problem to a single stand alone query in which the last object (`obj3`) has the property `<status>beëindigen</status>`. So I would expect this line: `let $stand4 := fold-left(($obj3, $obj2, $obj1), (), function($stand, $obj) { local:stand-totaal($stand, $obj) })` to give me an empty result as `obj3`has the property <status>beëindigen</status> but I am getting `obj1` as demonstrated in the following stand alone query. When running the stand alone query I would *expect*: 2 <-> 1 as shown in the result (`obj2`) 3 <-> 1 as shown in the result (empty as `obj3`has the property `<status>beëindigen</status>`) 3 <-> 2 as shown in the result (empty as `obj3`has the property `<status>beëindigen</status>`) 3 <-> 2 <-> 1 as 3 <-> 2 and 3 <-> 1 (empty as `obj3` has the property `<status>beëindigen</status>`) *but in the result I am getting *`obj1`. What am I doing wrong? Any help greatly appreciated. The query I use: ``` declare namespace op="http://www.geostandaarden.nl/imow/opobject"; declare namespace da="http://www.geostandaarden.nl/imow/datatypenalgemeen"; declare namespace ga="http://www.geostandaarden.nl/imow/gebiedsaanwijzing"; declare namespace gml="http://www.opengis.net/gml/3.2"; declare namespace l="http://www.geostandaarden.nl/imow/locatie"; declare namespace ow="http://www.geostandaarden.nl/imow/owobject"; declare namespace ow-dc="http://www.geostandaarden.nl/imow/bestanden/deelbestand"; declare namespace r="http://www.geostandaarden.nl/imow/regels" ; declare namespace rol="http://www.geostandaarden.nl/imow/regelsoplocatie" ; declare namespace sl="http://www.geostandaarden.nl/bestanden-ow/standlevering-generiek" ; declare namespace xlink="http://www.w3.org/1999/xlink" ; declare namespace xsi="http://www.w3.org/2001/XMLSchema-instance"; declare namespace rg="http://www.geostandaarden.nl/imow/regelingsgebied"; declare function local:id (: returns the id(s) of ow-object(s) :) ($seq as item()*) as item()* { $seq//*:owObject/*/*:identificatie | $seq/*/*:identificatie }; declare function local:stand-totaal (: returns an object by comparing new to old :) (: newest should be returned, except when <ow:status>beëindigen</ow:status> then no object should be returned :) ($new as item()*, $old as item()*) as item()* { let $new_ids := local:id($new) let $stand_deel1 := for $obj_oud in $old//*:owObject where not(local:id($obj_oud) = $new_ids) return <sl:stand>{ $obj_oud }</sl:stand> let $stand_deel2 := for $obj_new in $new//*:owObject[not(.//*:status)] return <sl:stand>{ $obj_new }</sl:stand> return ($stand_deel1, $stand_deel2) }; let $obj1 := <sl:stand> <ow-dc:owObject xmlns:ow-dc="http://www.geostandaarden.nl/imow/bestanden/deelbestand" xmlns:sl="http://www.geostandaarden.nl/bestanden-ow/standlevering-generiek" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"> <r:RegelVoorIedereen xmlns:r="http://www.geostandaarden.nl/imow/regels"> <r:identificatie>nl.imow-mnre1034.juridischeregel.OR0000000027</r:identificatie> <r:idealisatie>http://standaarden.omgevingswet.overheid.nl/idealisatie/id/concept/Exact</r:idealisatie> <r:artikelOfLid> <r:RegeltekstRef xmlns:xlink="http://www.w3.org/1999/xlink" xlink:href="nl.imow-mnre1034.regeltekst.OR0000000027"/> </r:artikelOfLid> <r:thema>http://standaarden.omgevingswet.overheid.nl/thema/id/concept/WaterEnWatersy…</r:thema> <r:locatieaanduiding> <l:LocatieRef xmlns:l="http://www.geostandaarden.nl/imow/locatie" xmlns:xlink="http://www.w3.org/1999/xlink" xlink:href="nl.imow-mnre1034.ambtsgebied.LND6030B"/> </r:locatieaanduiding> </r:RegelVoorIedereen> </ow-dc:owObject></sl:stand> let $obj2 := <sl:stand> <ow-dc:owObject xmlns:ow-dc="http://www.geostandaarden.nl/imow/bestanden/deelbestand" xmlns:sl="http://www.geostandaarden.nl/bestanden-ow/standlevering-generiek" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"> <r:Instructieregel xmlns:r="http://www.geostandaarden.nl/imow/regels"> <r:identificatie>nl.imow-mnre1034.juridischeregel.OR0000000027</r:identificatie> <r:idealisatie>http://standaarden.omgevingswet.overheid.nl/idealisatie/id/concept/Exact</r:idealisatie> <r:artikelOfLid> <r:RegeltekstRef xmlns:xlink="http://www.w3.org/1999/xlink" xlink:href="nl.imow-mnre1034.regeltekst.OR0000000027"/> </r:artikelOfLid> <r:thema>http://standaarden.omgevingswet.overheid.nl/thema/id/concept/WaterEnWatersy…</r:thema> <r:locatieaanduiding> <l:LocatieRef xmlns:l="http://www.geostandaarden.nl/imow/locatie" xmlns:xlink="http://www.w3.org/1999/xlink" xlink:href="nl.imow-mnre1034.gebied.Kustfundament"/> </r:locatieaanduiding> <r:gebiedsaanwijzing> <ga:GebiedsaanwijzingRef xmlns:ga="http://www.geostandaarden.nl/imow/gebiedsaanwijzing" xmlns:xlink="http://www.w3.org/1999/xlink" xlink:href="nl.imow-mnre1034.gebiedsaanwijzing.Kustfundament"/> <ga:GebiedsaanwijzingRef xmlns:ga="http://www.geostandaarden.nl/imow/gebiedsaanwijzing" xmlns:xlink="http://www.w3.org/1999/xlink" xlink:href="nl.imow-mnre1034.gebiedsaanwijzing.RGKustfundament"/> </r:gebiedsaanwijzing> <r:instructieregelInstrument>http://standaarden.omgevingswet.overheid.nl/instrument/id/concept/Omgevings…</r:instructieregelInstrument> </r:Instructieregel> </ow-dc:owObject></sl:stand> let $obj3 := <sl:stand> <ow-dc:owObject xmlns:ow-dc="http://www.geostandaarden.nl/imow/bestanden/deelbestand" xmlns:sl="http://www.geostandaarden.nl/bestanden-ow/standlevering-generiek" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"> <r:Instructieregel xmlns:r="http://www.geostandaarden.nl/imow/regels"> <ow:status xmlns:ow="http://www.geostandaarden.nl/imow/owobject">beëindigen</ow:status> <r:identificatie>nl.imow-mnre1034.juridischeregel.OR0000000027</r:identificatie> <r:idealisatie>http://standaarden.omgevingswet.overheid.nl/idealisatie/id/concept/Exact</r:idealisatie> <r:artikelOfLid> <r:RegeltekstRef xmlns:xlink="http://www.w3.org/1999/xlink" xlink:href="nl.imow-mnre1034.regeltekst.OR0000000027"/> </r:artikelOfLid> <r:thema>http://standaarden.omgevingswet.overheid.nl/thema/id/concept/WaterEnWatersy…</r:thema> <r:locatieaanduiding> <l:LocatieRef xmlns:l="http://www.geostandaarden.nl/imow/locatie" xmlns:xlink="http://www.w3.org/1999/xlink" xlink:href="nl.imow-mnre1034.gebied.Kustfundament"/> </r:locatieaanduiding> <r:gebiedsaanwijzing> <ga:GebiedsaanwijzingRef xmlns:ga="http://www.geostandaarden.nl/imow/gebiedsaanwijzing" xmlns:xlink="http://www.w3.org/1999/xlink" xlink:href="nl.imow-mnre1034.gebiedsaanwijzing.Kustfundament"/> <ga:GebiedsaanwijzingRef xmlns:ga="http://www.geostandaarden.nl/imow/gebiedsaanwijzing" xmlns:xlink="http://www.w3.org/1999/xlink" xlink:href="nl.imow-mnre1034.gebiedsaanwijzing.RGKustfundament"/> </r:gebiedsaanwijzing> <r:instructieregelInstrument>http://standaarden.omgevingswet.overheid.nl/instrument/id/concept/Omgevings…</r:instructieregelInstrument> </r:Instructieregel> </ow-dc:owObject></sl:stand> let $stand1 := fold-left(($obj2, $obj1), (), function($stand, $obj) { local:stand-totaal($stand, $obj) }) let $stand2 := fold-left(($obj3, $obj1), (), function($stand, $obj) { local:stand-totaal($stand, $obj) }) let $stand3 := fold-left(($obj3, $obj2), (), function($stand, $obj) { local:stand-totaal($stand, $obj) }) let $stand4 := fold-left(($obj3, $obj2, $obj1), (), function($stand, $obj) { local:stand-totaal($stand, $obj) }) return ( '2 <-> 1 ==================', $stand1, '3 <-> 1 ==================', $stand2, '3 <-> 2 ==================', $stand3, '3 <-> 2 <-> 1 ============', $stand4 ) ``` The results: ``` 2 <-> 1 ================== <sl:stand xmlns:sl="http://www.geostandaarden.nl/bestanden-ow/standlevering-generiek"> <ow-dc:owObject xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:ow-dc="http://www.geostandaarden.nl/imow/bestanden/deelbestand"> <r:Instructieregel xmlns:r="http://www.geostandaarden.nl/imow/regels"> <r:identificatie>nl.imow-mnre1034.juridischeregel.OR0000000027</r:identificatie> <r:idealisatie>http://standaarden.omgevingswet.overheid.nl/idealisatie/id/concept/Exact</r:idealisatie> <r:artikelOfLid> <r:RegeltekstRef xmlns:xlink="http://www.w3.org/1999/xlink" xlink:href="nl.imow-mnre1034.regeltekst.OR0000000027"/> </r:artikelOfLid> <r:thema>http://standaarden.omgevingswet.overheid.nl/thema/id/concept/WaterEnWatersy…</r:thema> <r:locatieaanduiding> <l:LocatieRef xmlns:xlink="http://www.w3.org/1999/xlink" xmlns:l="http://www.geostandaarden.nl/imow/locatie" xlink:href="nl.imow-mnre1034.gebied.Kustfundament"/> </r:locatieaanduiding> <r:gebiedsaanwijzing> <ga:GebiedsaanwijzingRef xmlns:xlink="http://www.w3.org/1999/xlink" xmlns:ga="http://www.geostandaarden.nl/imow/gebiedsaanwijzing" xlink:href="nl.imow-mnre1034.gebiedsaanwijzing.Kustfundament"/> <ga:GebiedsaanwijzingRef xmlns:xlink="http://www.w3.org/1999/xlink" xmlns:ga="http://www.geostandaarden.nl/imow/gebiedsaanwijzing" xlink:href="nl.imow-mnre1034.gebiedsaanwijzing.RGKustfundament"/> </r:gebiedsaanwijzing> <r:instructieregelInstrument>http://standaarden.omgevingswet.overheid.nl/instrument/id/concept/Omgevings…</r:instructieregelInstrument> </r:Instructieregel> </ow-dc:owObject> </sl:stand> 3 <-> 1 ================== 3 <-> 2 ================== 3 <-> 2 <-> 1 ============ <sl:stand xmlns:sl="http://www.geostandaarden.nl/bestanden-ow/standlevering-generiek"> <ow-dc:owObject xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:ow-dc="http://www.geostandaarden.nl/imow/bestanden/deelbestand"> <r:RegelVoorIedereen xmlns:r="http://www.geostandaarden.nl/imow/regels"> <r:identificatie>nl.imow-mnre1034.juridischeregel.OR0000000027</r:identificatie> <r:idealisatie>http://standaarden.omgevingswet.overheid.nl/idealisatie/id/concept/Exact</r:idealisatie> <r:artikelOfLid> <r:RegeltekstRef xmlns:xlink="http://www.w3.org/1999/xlink" xlink:href="nl.imow-mnre1034.regeltekst.OR0000000027"/> </r:artikelOfLid> <r:thema>http://standaarden.omgevingswet.overheid.nl/thema/id/concept/WaterEnWatersy…</r:thema> <r:locatieaanduiding> <l:LocatieRef xmlns:xlink="http://www.w3.org/1999/xlink" xmlns:l="http://www.geostandaarden.nl/imow/locatie" xlink:href="nl.imow-mnre1034.ambtsgebied.LND6030B"/> </r:locatieaanduiding> </r:RegelVoorIedereen> </ow-dc:owObject> </sl:stand> ``` -- Marco Duiker LandGoed Technisch directeur +31617115114

3 4

DBs corrupted; Unparseable date.
by France Baril 12 Dec '23

12 Dec '23

Hi, My databases are corrupted in a strange way. Everything worked yesterday and I have not upgraded my system (automatic updates are NOT set on my OS). In the WebDAV connector, all DB names except 6 appear as a date, examples: 2023-06-14T07:37:56.294Z, 2023-12-12T09:56:02.722Z. In the console, I get this error: [qtp289639718-19] INFO com.bradmcevoy.http.HttpManager - PROPFIND :: http://localhost:8972/webdav/ - http://localhost:8972/webdav/ bx_1 | Unparseable date: "app-pub-templates" bx_1 | Unparseable date: "app-pubs" bx_1 | Unparseable date: "app-tests" bx_1 | Unparseable date: "ar-eg" bx_1 | Unparseable date: "as-in" bx_1 | Unparseable date: "az-az" bx_1 | Unparseable date: "be-by" bx_1 | Unparseable date: "bg-bg" bx_1 | Unparseable date: "bn-bd" ... It seems that the names and dates of the DBs have been interchanged. I tried restoring the DBS from my backups (newer and older back ups). I also tried restarting the server. Same difference. I am using Basex 10.7 (beta) and have been for a few months. I could update BaseX to the official release, but I would prefer to upgrade with healthy DBs to avoid adding a layer of complexity to the issue. I have not had a similar problem in a decade of using BaseX, so I am a bit clueless about how else to try. Thanks in advance for your help, -- France Baril Architecte documentaire / Documentation architect france.baril(a)architextus.com

2 2

JSON number serialization
by Thompson, Timothy 10 Dec '23

10 Dec '23

Hello, Is it part of the spec that numbers in the “basic” JSON representation (of 7+ digits) be serialized using scientific notation? For example: let $direct := <json type="object"><n type="number">1339029</n></json> let $basic := <fn:map><fn:number key="n">1339029</fn:number></fn:map> let $result := ($direct, $basic) ! serialize(., map { "method": "json", "json": map { "format": if (position() eq 1) {"direct"} else {"basic"}, "indent": "yes" } }) return $result …produces two different results: { "n":1339029 } { "n":1.339029E6 } I usually prefer working with the “basic” format, but the automatic conversion to scientific notation is inconvenient because the value is not easily castable as an xs:integer. Thanks in advance, Tim -- Tim A. Thompson (he, him) Librarian for Applied Metadata Research Yale University Library www.linkedin.com/in/timathompson<http://www.linkedin.com/in/timathompson> timathom(a)protonmail.com<mailto:timothy.thompson@yale.edu>

1 1

Using ft:search() across element boundaries: possible?
by Eliot Kimber 05 Dec '23

05 Dec '23

I’m searching for short phrases where I may want to respect order or not and where the phrases may cross element boundaries. For example, I have the phrase “Amazon Alexa Spoke” and I want to find any DITA topic whose title text includes “Amazon Alexa Spoke” in that order, or maybe I want those words in any order, depending on my search requirements. When I run this query against my database I find occurrences where all three words are in the same parent element, i.e.: <title>Create a connection record for the <ph>Amazon Alexa spoke</ph> </title> <title>Create a credential record for the <ph>Amazon Alexa spoke</ph> </title> <title>Set up the <ph>Amazon Alexa spoke</ph> </title> But I do not find it where one of the words is not in the same parent: This title is *not* found (even though this is the one I actually want to have found): <title><ph id="alexa">Amazon Alexa</ph> Spoke</title> Reading the docs on ft:search(), it is clear that it is searching on text nodes: “Returns all text nodes from the full-text index…” So I think the behavior here is as documented. Short of creating a separate database that removes the subelements within <title> elements, is there a way to use full text indexing to do the search I want? In particular, I want to be able to turn the ordered/unordered check on or off. If I always wanted ordered I could just use a regular expression match—it wouldn’t be that efficient but efficiency is not a concern in this particular case (but I can see where it would be in a more general search support situation). Or am I missing a more obvious solution to this requirement? Note that in this case I don’t care about finding different word forms—for this particular search I only care about exact word matches. Cheers, E. _____________________________________________ Eliot Kimber Sr Staff Content Engineer O: 512 554 9368 M: 512 554 9368 servicenow.com<https://www.servicenow.com> LinkedIn<https://www.linkedin.com/company/servicenow> | Twitter<https://twitter.com/servicenow> | YouTube<https://www.youtube.com/user/servicenowinc> | Facebook<https://www.facebook.com/servicenow>

1 1

Code highlighting not working on Wiki
by Johan Mörén 03 Dec '23

03 Dec '23

Hi, Just discovered that the code samples on the basex wiki doesn't seem to be working fully. Noticed it a couple of days ago and thought it was temporary, but the problem is still there. Regards, Johan

2 2

Generate CSV with literal "&", not "&"
by Eliot Kimber 01 Dec '23

01 Dec '23

I’m generating CSV data that includes URLs with multiple query parameters, so “&somekey” in them. These get serialized as “&somekey” where I want “&somekey”. My CSV XML looks like this: <record> <AppID>sn_admin_center</AppID> <DocsURL>=HYPERLINK(https://docs.servicenow.com/csh?topicname=admin-center-intro&version=vancouver)</DocsURL> </record> I’m then doing: let $report := csv:serialize($csv, map{}) let $doWrite := file:write('/Users/eliot.kimber/temp/apps-to-topics.csv', $report) To write the CSV file. The resulting file looks like: sn_admin_center,”=HYPERLINK(“"https://docs.servicenow.com/csh?topicname=admin-center-intro&version=va…")" Note that the “&” is still escaped. Reviewing the docs for the CVS module and the serialize options, I don’t see any option that looks like it would control how escaping is handled. Is there a way to do what I want? Thanks, E. _____________________________________________ Eliot Kimber Sr Staff Content Engineer O: 512 554 9368 M: 512 554 9368 servicenow.com<https://www.servicenow.com> LinkedIn<https://www.linkedin.com/company/servicenow> | Twitter<https://twitter.com/servicenow> | YouTube<https://www.youtube.com/user/servicenowinc> | Facebook<https://www.facebook.com/servicenow>

2 2

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

BaseX-Talk December 2023