BaseX-Talk

basex-talk@mailman.uni-konstanz.de

12 participants
5242 discussions

Questions about the performance of bulk string matching
by Zimmel, Daniel 18 Jan '22

18 Jan '22

Hi, recently I ran into serious (as in SERIOUS) performance trouble regarding expensive regexes, and no wonder why. Here is my scenario: * XML text documents with a total of 1m text nodes * A regex search string, including a huge string dictionary list of 50.000 strings (some of them containing 50 words each) * a match must be wrapped in an element (= create a link) I could drink many cups of tea while waiting for the regex to complete... hours... I ran out of tea! Then I found the 2021 Balisage paper from Mary Holstege titled "Fast Bulk String Matching" [1] in which she explores the Aho-Corasick algorithm, implementing it with XQuery - marvellous! Following this, while I can build a data structure which gives me fast results, building the same structure is still very slow due to the amount of text to build from. So this was not fast enough for my use case - or I may simply not be smart enough to apply it correctly :-| So, I tried tinkering with maps which turned out to give me extraordinary performance gains: * build a map from the string dictionary * walk through all text nodes one by one * for each text node, put any combination of words in the text node in word order (I need to find strings, not words) into another map * strip punctuation (for word boundaries) and do some normalization of whitespaces in both maps * compare the keys of both maps * give the new reduced string dictionary to the regular regex search While comparing the maps, I do not know where in the text my strings are, but at least I know if they are in there - to find out where exactly (and how do they fit my other regex context) I can then use a massively reduced regular regex search. Fast! I am quite happy BUT I still cannot understand why this is so much faster for my sample data: * plain method : 51323.72ms * maps method: 597.94ms(!) Is this due to any optimizations done by BaseX or have I simply discovered the power of maps? How would you do it? Is there still room for improvement? Why does Aho-Corasick not help much with this scenario? Is it because the list of strings is simply too massive? Why is this so much faster with text-splitting-to-map? See below for the query examples to better understand what I am trying to do (bulk data not included) [2],[3] There is no normalization of punctuation in the examples but that is only necessary for completeness. Best, Daniel [1] http://www.balisage.net/Proceedings/vol26/print/Holstege01/BalisageVol26-Ho… [2] plain method let $textnodes := fetch:xml(file:base-dir()||file:dir-separator()||'example.xml')//text() let $strings := file:read-text(file:base-dir()||file:dir-separator()||'strings.txt') let $regex := $strings for $textnode in $textnodes where matches($textnode,$regex) = true() return $textnode [3] maps method (:~ : Create map from string : Tokenization : : @param $strings : @return Map :) declare function local:createMapFromString ( $string as xs:string ) as map(*) { let $map_words := map:merge( for $string in tokenize($string,'\|') let $key := $string let $val := $string return map:entry($key,$val), map { 'duplicates': 'use-first' }) return $map_words }; (:~ : Create map from text nodes : Write any combination of words in document order to the map : : @param $textnodes : @return Map :) declare function local:createMapFromTextnodes ( $textnodes as xs:string+ ) as map(*) { map:merge( for $node in $textnodes let $text := normalize-space($node) let $tokens := tokenize($text,' ') let $map_nodes := map:merge( for $start in 1 to fn:count($tokens) for $length in 1 to fn:count($tokens) - $start + 1 return map:entry(fn:string-join(fn:subsequence($tokens, $start, $length), ' '),'x') ) return $map_nodes) }; (:~ : Compare two maps : : @param $map1 : @param $map2 : @return xs:string* :) declare function local:reduceMaps ( $map1 as map(*), $map2 as map(*) ) as xs:string* { for $key in map:keys($map1) where map:contains($map2,$key) let $value := map:get($map1,$key) return $value }; let $textnodes := fetch:xml(file:base-dir()||file:dir-separator()||'example.xml')//text() let $strings := file:read-text(file:base-dir()||file:dir-separator()||'strings.txt') let $map_words := local:createMapFromString($strings) let $map_textnodes := local:createMapFromTextnodes($textnodes) let $matches := local:reduceMaps($map_words,$map_textnodes) let $regex := string-join(for $match in $matches group by $match return $match,'|') for $textnode in $textnodes where matches($textnode,$regex) = true() return $textnode

2 4

Configuration Option for Files to Parse in Create Database Dialog?
by Eliot Kimber 14 Jan '22

14 Jan '22

I’m using the BaseX GUI in a very informal way where I create new databases as needed to do ad-hoc queries over content that comes from various git repos. I’m trying to setup my local configuration settings to be what I need for this database creation tasks. I’ve found and successfully configured the options for skipping corrupt and parsing archives but I don’t see an option that will populate the Input file or directory field, which I need to be “*.xml, *.dita,*.ditamap”. I found the option that sets the pattern used in the main file selector field but that doesn’t apply to this dialog. I’m not seeing anything for this in the docs. This is for BaseX 9.6.3 Thanks, E. _____________________________________________ Eliot Kimber Sr Staff Content Engineer O: 512 554 9368 M: 512 554 9368 servicenow.com<https://www.servicenow.com> LinkedIn<https://www.linkedin.com/company/servicenow> | Twitter<https://twitter.com/servicenow> | YouTube<https://www.youtube.com/user/servicenowinc> | Facebook<https://www.facebook.com/servicenow>

3 5

9.6
by Bondeson, Carl 12 Jan '22

12 Jan '22

We have been using BaseX since 7.0 in a multi-threaded environment. Our configuration utilizes an architecture where child threads query a singleton BaseX server, running as a service using YAJSW. We were unable to update past 8.0, due to issues with our licensed Java tier. We recently upgraded to a RedHat version of the open JDK.11, and were able to upgrade BaseX to 9.6. The performance is much improved. We also saw better memory management by running BaseX in its own container (YAJSW). This implementation is handling the processing of Corona, and 75 other reportable diseases from over 3,000 sources (labs, hospitals, doctor's offices, etc., etc.). This platform utilizes 6 databases, which work as rule engines, to drive the processing of data as it moves through the system. Thanks again for an outstanding product. Carl R Bondeson IT Analyst 3 Department of Public Health Operation & Support Services Information Technology Phone: 860-509-7434 Carl.Bondeson(a)ct.gov<mailto:Carl.Bondeson@ct.gov> [cid:image001.jpg@01D8078F.B6177CE0][cid:image002.png@01D8078F.B6177CE0]

1 0

9.6 on Docker Hub
by Johannes Bauer 10 Jan '22

10 Jan '22

Hi, I cannot find the latest BaseX release on Docker Hub (only the 9.5 branch). Is this still planned? Thank you. Best Regards Johannes

3 4

RESTXQ design pattern question
by Hans-Juergen Rennau 07 Jan '22

07 Jan '22

Hello, I have a question concerning RESTXQ design patterns. Imagine you have a page P1 displaying an input field, and you want to provide a button enabling the user to pick the terms from a list, rather than entering them. So after pressing the button, another page P2 appears, listing the terms and enabling a selection. After selection, P1 should again appear, looking as before, except for the selected terms appearing in the field. A common pattern, isn't it. How to implement it? One approach would be P2 calling P1 (via form/@action), passing the selection to P1 as form or query parameters. But then the input of P1 must include everything required to restore the previous appearance, including the values of input fields from forms not yet submitted. There is certainly a simpler way? I suppose a standard approach handles this completely with JavaScript, updating the page. I wonder if RESTXQ and supporting modules enable some alternative? Thank you, with kind regards -Hans-Jürgen

2 1

webapp/static
by Hans-Juergen Rennau 07 Jan '22

07 Jan '22

Hello, dear BaseX team, one question concerning folder $BASEX/webapp/static. Given an HTML page created by a query via REST or RESTXQ execution. It seems to me that referenced resources (e.g. link/@href or img/@src) must be located in that folder or a subfolder of it, and the reference must be "../static/foo.css" or "../static/path/to/subfolder/foo.css", examples: <link rel="stylesheet" type="text/css" href="../static/foo.css"/><link rel="stylesheet" type="text/css" href="../static/foo/foo.css"/> At least I did not succeed with any other locations, specified by full path or by relative path, assuming different context paths (e.g. the folder containing the query producing the HTML, or $BASEX/bin, or $BASEX). My QUESTION: is that correct, if not - what are the rules? Many thanks,Hans-Jürgen PS: Apologies if this is documented, but I failed to find it, and the complete PDF contains a single occurrence of string "/static" which does not apply.

2 1

RESTXQ generated HTML - unexpected behaviour
by Hans-Juergen Rennau 07 Jan '22

07 Jan '22

Hello, I face a problem with RESTXQ of a fairly general character, as it seems to me. When the XQuery functions of a RESTXQ application construct HTML pages, I expect these pages to behave exactly as a static page with the same content would behave. But sometimes this is not the case. As an attachment I send an example. The "webapp" folder contains an XQuery module with a RESTXQ function [2], and the "html" folder contains an HTML page [3] which is identical to the page generated by the RESTXQ function [4]. This page has been copied from [1], and it demonstrates auto completion in an input field. However, auto completion works only when calling the HTML file; it does not work when calling the RESTXQ application. In both cases, the HTML page references a JS file as well as a CSS file, located in a sibling folder "static". An alert box as well as the appearance of the page prove that JS and CSS are found and used in both cases. And yet the behaviour is different ... ~~~ This problem appears to me important, as the trust in RESTXQ would be damaged if the generated HTML "may or may not" behave as it should. On the other hand, if the reason of the problem is understood and a workaround is available, all would be well. Kind regards,Hans-Jürgen [1] https://www.w3schools.com/howto/howto_js_autocomplete.asp [2] $demo-autocomplete/webapp/autocomplete/demo-autocomplete.xq [3] $demo-autocomplete/html/demo-autocomplete.html [4] The contents of the HTML file have been created by (1) entering into a browser the URL of the RESTXQ application (localhost:8984/demo-autocomplete), (2) selecting from the context menu "Show source code", (3) copying the source code into the empty HTML file, (4) replacing <script .../> with <script...></script>. [5] How to reproduce the problem:a. Unzip the archiveb. Call $demo-autocomplete/html/demo-autocomplete.html - after acknowledging the alert box which shows that the imported JS function has been called, enter into the input field letter "a" - a list of completions appears.c. Install the RESTXQ application: copy folder $demo-autocomplete/webapp/autocomplete into the webapp folder of a BaseX instance, and copy the files $demo-autocomplete/webapp/static/* into $BASEX/webapp/static.d. Enter in a browser "localhost:8984/demo-autocomplete" - after acknowledging the alert box, enter into the input field letter "a" - no list of completions appears.

2 4

Re: [basex-talk] Copy data from MariaDB into BaseX
by Christian Grün 23 Dec '21

23 Dec '21

Thanks. Does the query do what you are looking for? On Tue, Dec 21, 2021 at 12:47 PM <benengbers(a)dds.nl> wrote: > > Christian Grün schreef op 21-12-2021 10:18: > > Hi Ben, > > > return db:add($db, $doc, $table || '.xml') > > > > Could you give us little examples for <DB-name>, <DB-schema> and > > <table-name> ? > > > > Best, > > Christian > > To the best of my knowledge in MySQL and/or MariaDB DB-name and > DB-schema are identical? The schema-name I use is 'Innovate'. > Table-names are > +--------------------+ > | Tables_in_Innovate | > +--------------------+ > | Dienst | > | Mdw_Probleem | > | Mdw_Wens | > | Medewerker | > | Medewerker_dienst | > | Probleem | > | Wens | > +--------------------+ > > Ben > PS.I hope you'll see this reply. Since a few days all mail from > basex-talk is refused by Thunderbird. At least I don't see them anymore > ....

2 5

Copy data from MariaDB into BaseX
by Ben Engbers 21 Dec '21

21 Dec '21

Hi, After completing my work on the R-client, I started working on a Prolog-client. Long ago I wrote an application in SWI-Prolog which operated on data from a MySQL-database. (In the meantime I changed from MySQL to MariaDb). My goal is to write a new version of that application but now based on data which is stored in Basex. In the basexgui, I created an empty database "MariaBases" The following code can be used to select data in MariaDb: sql:init("org.mariadb.jdbc.Driver"), let $con := sql:connect('jdbc:mariadb://localhost:3306/<DB-name>', '<user>', '<password>') return sql:execute($con, "select * from Mdw_Wens") returns: <sql:row xmlns:sql="http://basex.org/modules/sql"> <sql:column name="ID">1</sql:column> <sql:column name="Medewerker_ID">5</sql:column> <sql:column name="Wens_ID">1</sql:column> </sql:row> Is it possible to change the query-statement in such a way that the results are added to MariaBases/<DB-schema>/<table-name>? -- Ben Engbers

2 1

BaseX 9.6.4: Tweaks, Fixes & Thank Yous
by Christian Grün 17 Dec '21

17 Dec '21

Dear all, We conclude this year with yet another release of BaseX, our XML framework, database and XQuery processor [1]. The new version contains new performance tweaks and minor bug fixes, the details of which can be examined on our GitHub page [2]. As we’ve got numerous inquiries regarding Log4j: I can comfort you that BaseX is not affected by the vulnerabilities. We have our own logging mechanism, which has no built-in code execution features. The same applies to the Jetty web server. Log4j once again demonstrates that Open Source is everywhere, and that it takes valuable time and resources to build professional and secure software. We send out a big thank you to those of you who are (and have been) supporting us by paying for maintenance, sponsoring new features and giving donations. It’s you who is keeping BaseX both alive and up to date! Have a good time, Christian [1] https://basex.org [2] https://github.com/BaseXdb/basex/commits/master

4 4

Jump to page:

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

BaseX-Talk