- BaseX-Talk - mailman.uni-konstanz.de

Doc Loaded by db:create() has invalid URL according to base-uri()
by Eliot Kimber 04 Feb '22

04 Feb '22

I think this is a bug: I’m using db:create() to initialize a database with files from the file system. The OS is linux (but I see the same issue with macOS). The docs all load without error, but then when processing the loaded content with code that uses base-uri() to get the base URI of a document, I get this failure: URI '/rome/product/human-resources/task/predict-assignment-group[1].dita' is invalid Which is correct—the square brackets must be escaped. However, I didn’t create the URI, BaseX did when it loaded the data, which suggests that there’s a URI escaping action that isn’t being performed when the data is loaded: I would expect the URI to be “predict-assignment-group%5B1%5D.dita” as returned by base-uri() (and as stored in the <resource> entry for the document). If I do: collection('/rome/product/human-resources/task/predict-assignment-group[1].dita')/* I also get invalid collection URI: '/rome/product/human-resources/task/predict-assignment-group[1].dita' Using db:dir to examine the <resource> entry for the file I see: <resource raw="false" content-type="application/xml" modified-date="2022-02-03T22:44:29.107Z">predict-assignment-group[1].dita</resource> So the brackets are not escaped there. I can use the db:* functions to correct the paths (or just delete the nodes) so not a hard-stop problem. Cheers, E. _____________________________________________ Eliot Kimber Sr Staff Content Engineer O: 512 554 9368 M: 512 554 9368 servicenow.com<https://www.servicenow.com> LinkedIn<https://www.linkedin.com/company/servicenow> | Twitter<https://twitter.com/servicenow> | YouTube<https://www.youtube.com/user/servicenowinc> | Facebook<https://www.facebook.com/servicenow>

4 5

Re: [basex-talk] BaseX-Talk Digest, Vol 130, Issue 4
by Rob Stapper 04 Feb '22

04 Feb '22

Hi Christian, Referring to an old issue of 10-10-2020. Is there any change the solution to the issue as mentioned below is part BaseX 10? I would realy like to address every module-functions through a public variable even in a cyclic module-construction. Another point that I would very much be concidered is that the "at"-clause of the import-command could also handle database locations. I would like my functionality to be stored as close as possible to its relevant dataStructures. Is that something tat cn be considered? Love to hear from you, Rob Stapper > Op 09-10-2020 12:00 schreef basex-talk-request(a)mailman.uni-konstanz.de: > > > Send BaseX-Talk mailing list submissions to > basex-talk(a)mailman.uni-konstanz.de > > To subscribe or unsubscribe via the World Wide Web, visit > https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk > or, via email, send a message with subject or body 'help' to > basex-talk-request(a)mailman.uni-konstanz.de > > You can reach the person managing the list at > basex-talk-owner(a)mailman.uni-konstanz.de > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of BaseX-Talk digest..." > > > Today's Topics: > > 1. recursively used variables (Rob Stapper) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Thu, 8 Oct 2020 14:17:48 +0200 > From: Rob Stapper <r.stapper(a)lijbrandt.nl> > To: "basex-talk(a)mailman.uni-konstanz.de" > <basex-talk(a)mailman.uni-konstanz.de> > Subject: [basex-talk] recursively used variables > Message-ID: <48217f7a-0960-11eb-811f-00505699b758(a)smtp.kpnmail.nl> > Content-Type: text/plain; charset="utf-8" > > Hi, > > The code[1] below and send as attachment generates a error message: ?Static variable depends on itself: $Q{http://www.w3.org/2005/xquery-local-functions}test?. > I use these variables to refer to my private functions in my modules so I can easyly refer to them in a inheritance situation. > It?s not a big problem for me but I was wondering if the error-triggering is justified or that it should work. > > [1]=========================================== > declare variable $local:test := local:test#1 ; > declare %private function local:test( $i) { if ( $i > 0) then $local:test( $i - 1) } ; > > $local:test( 10) > =========================================== > > Kind regards, > > Rob Stapper > > > Sent from Mail for Windows 10 > > > > -- > This email has been checked for viruses by Avast antivirus software. > https://www.avast.com/antivirus >

2 1

Optimizing Lookup from Custom Indexes
by Eliot Kimber 03 Feb '22

03 Feb '22

In my content set (DITA maps and topics) I construct an index that maps each map or topic to the names of the root maps that ultimately use that topic. My index structure is: <doc-to-bundle-index> <doc-to-bundle-index-entry key="product/customer-communities/reference/gamification-components-badges.dita"> <filename>gamification-components-badges.dita</filename> <bundles> <bundle>No-bundle-found</bundle> </bundles> </doc-to-bundle-index-entry> </doc-to-bundle-index> I then want to get, for all the topics, the bundle names for each topic, grouped by bundle name (i.e., construct a map of bundle names to topics in that bundle). (This is in the service of a report that relates Oxygen map validation reports to the documents associated with the incidents in the report, grouped by bundle.) I have 10K topics in my test set. Getting the set of topic elements and the index keys for each topic is fast: about 0.1 seconds total. However, using the keys to do a lookup of the bundles for each topic takes about 2 minutes, i.e.: let $bundlesForDocs as xs:string* := for $key in $keysForDocs return $dtbIndex/doc-to-bundle-index-entry[@key eq $key]/bundles/bundle ! string(.) return $bundlesForDocs (I would really be building a map of bundles-to-docs but I used this loop just to gather timing info and take map construction out of the equation, not that I would expect map construction itself to be slow.) An obvious solution would be to capture the bundle-to-document mapping at the time I construct the index, which I will do. But my larger question is: Am I doing anything wrong or inefficient in this initial approach that is making this lookup of index entries by key slower than it should be? Or is this just an inherently slow operation that I should just not try to do if at all possible? That is, is there a way to either construct the content of the index or configure BaseX that will make this kind of bulk lookup faster? Or am I thinking about this particular use case all wrong? Thanks, Eliot _____________________________________________ Eliot Kimber Sr Staff Content Engineer O: 512 554 9368 M: 512 554 9368 servicenow.com<https://www.servicenow.com> LinkedIn<https://www.linkedin.com/company/servicenow> | Twitter<https://twitter.com/servicenow> | YouTube<https://www.youtube.com/user/servicenowinc> | Facebook<https://www.facebook.com/servicenow>

2 2

Issue with HTTP client and authentication
by Tim Thompson 03 Feb '22

03 Feb '22

Hello, I'm trying to post a SPARQL query to an endpoint using Digest authentication with the HTTP client. The query works fine using curl: curl --digest --user user:pass -X POST -d@'test.rq' \ -H "Content-type: application/sparql-query" \ 'http://example.org/sparql' But the equivalent request in BaseX fails with 401 Unauthorized: let $endpoint := "http://example.org/sparql" let $user := "user" let $pass := "pass" let $type := "application/sparql-query" let $response := ( http:send-request( <http:request method="POST" href="{$endpoint}" username="{$user}" password="{$pass}" auth-method="Digest" send-authorization="true"> <http:header name="Content-Type" value="{$type}; charset=utf-8"/> <http:body media-type="{$type}">{ ``[ select * where {?s ?p ?o} limit 1 ]`` }</http:body> </http:request> ) ) return $response Any ideas about what might be causing the BaseX HTTP client to be denied here? Thanks in advance, Tim -- Tim A. Thompson Metadata Librarian Yale University Library

5 20

Techniques for Unit Testing Updating Operations
by Eliot Kimber 02 Feb '22

02 Feb '22

I’m setting up unit tests for my code that creates various custom indexes. I have content on the file system that serves as my known input. With that data I then need to create the content database and run the processes that create the various indexes over the content database. Thus I need to create the databases, populate them, then verify the populated result. As you can’t create a database and query it in the same XQuery, I don’t see a way to use %unit:before-module to initialize my databases before running unit tests in the same module. The solution seems to be to use a BaseX script to do the database initialization, which seems easy enough: # Run unit tests with fresh database # Make sure the databases exist so we can then drop them check pce-test-01 check _lrc_pce-test-01_link_records # Drop them drop db pce-test-01 drop db _lrc_pce-test-01_link_records # Now create them fresh check pce-test-01 check _lrc_pce-test-01_link_records # # Now run the tests that use these databases in the # required order # test ./test-db-from-git.xqy test ./test-link-record-keeping.xqy However, in running this from the BaseX GUI, it appears that the test commands are trying to find files relative to the location of the basexgui command rather than relative to the script being run: Resource "/Users/eliot.kimber/apps/basex/bin/test-db-from-git.xqy" not found. I don’t see anything in the commands documentation that suggests a way to parameterize the values passed to commands. Am I missing a way to have this kind of setup script be portable from within the GUI or is there a better/different way to initialize the databases for unit tests? Thanks, E. _____________________________________________ Eliot Kimber Sr Staff Content Engineer O: 512 554 9368 M: 512 554 9368 servicenow.com<https://www.servicenow.com> LinkedIn<https://www.linkedin.com/company/servicenow> | Twitter<https://twitter.com/servicenow> | YouTube<https://www.youtube.com/user/servicenowinc> | Facebook<https://www.facebook.com/servicenow>

3 12

Bug (?) with dynamic namespace constructor
by Hans-Juergen Rennau 31 Jan '22

31 Jan '22

Dear BaseX people, is this a bug: basex "<foo>{namespace {''}{'bar'}}</foo>"=>[XQDY0102] Duplicate namespace declaration: ''.? This works as expected:basex "<foo>{namespace {'xxx'}{'bar'}}</foo>"=><foo xmlns:xxx="bar"/> With kind regards,Hans-Jürgen

2 4

Techniques for Parallelizing Updating Operations?
by Eliot Kimber 30 Jan '22

30 Jan '22

I’ve worked out how to optimize my process that indexes DITA topics based on what top-level maps they are ultimately used from (turned out I needed to first index the maps in ref count order from least to most, which meant I could then just look up the top-level maps used by any direct-reference maps that reference a given topic—with that in place each topic only requires a single index lookup). However, on my laptop these lookups still take about 0.1 second/topic so for 1000s of topics it’s a long time (relatively speaking). But the topic index process is 100% parallelizable, so I would be able to have at least 2 or 3 ingestion threads going on my 4-CPU server machine. Note that my ingestion process is two-phased: Phase 1: Construct an XQuery map with the index details for the input topics (the topics already exist in the database, only the index is new). Phrase 2: Persist the map to the database as XML elements. I do the map construction in order to both take advantage of map:merge() and because it’s the only way I can do indexing of the DITA maps and topics in one transaction: build the doc-to-root-map for the DITA maps and then use that data to build the doc-to-root-map entries for all the topics, then persist the lot to the database for future use. This is in the context of a one-time mass load of content from a new git work tree. Subsequent changes to the content database will be on individual files and the index can be easily updated incrementally. So I’m just trying to optimize the startup time so that it doesn’t take two hours to load and index our typical content set. I can also try to optimize the low-level operations, although they’re pretty simple so I don’t see much opportunity for significant improvement, but I also haven’t had time to try different options and measure them. I must also say how useful the built-in unit testing framework is—that’s really made this work easier. Cheers, Eliot _____________________________________________ Eliot Kimber Sr Staff Content Engineer O: 512 554 9368 M: 512 554 9368 servicenow.com<https://www.servicenow.com> LinkedIn<https://www.linkedin.com/company/servicenow> | Twitter<https://twitter.com/servicenow> | YouTube<https://www.youtube.com/user/servicenowinc> | Facebook<https://www.facebook.com/servicenow>

1 0

Managing Interactions with Long-Running Processes
by Eliot Kimber 26 Jan '22

26 Jan '22

I’m making good progress on our BaseX-based validation dashboard application. The basic process here is we use Oxygen’s scripting support to do DITA map validation and then ingest the result into a database (along with the content documents that were validated) and provide reports on the validation result. The practical challenge here seems to be running the Oxygen process successfully from BaseX—because our content is so huge it can take 10s of minutes for the Oxygen process to run. I set the command time-out to be much longer than the process should run but running it from the HTTP app’s query panel it eventually failed with an error that wasn’t a time out (my code had earlier reported legit errors so I know errors will be properly reported). As soon as the Oxygen process ends I want to ingest the resulting XML file, which is why I started with doing it from within BaseX. But I’m wondering if this is a bad idea and I should really be doing it with i.e., a shell script run via cron or some such? I was trying to keep everything in BaseX as much as possible just to keep it simple. Any general tips or cautions for this time of integration of BaseX with the outside world? Thanks, E. _____________________________________________ Eliot Kimber Sr Staff Content Engineer O: 512 554 9368 M: 512 554 9368 servicenow.com<https://www.servicenow.com> LinkedIn<https://www.linkedin.com/company/servicenow> | Twitter<https://twitter.com/servicenow> | YouTube<https://www.youtube.com/user/servicenowinc> | Facebook<https://www.facebook.com/servicenow>

2 2

Re: [basex-talk] Unable to Make My Custom EXPath Module Work
by Christian Grün 24 Jan '22

24 Jan '22

I guess you’re right. We haven’t revised EXPath packaging for a long time now. Actually, I’m not sure how many people use it at all ;) Anyone out there? On Mon, Jan 24, 2022 at 4:08 PM Eliot Kimber <eliot.kimber(a)servicenow.com> wrote: > > I did confirm that if the package @name URI matches the URI of a module, then the module is resolved, i.e.: > > <package xmlns="http://expath.org/ns/pkg" > name="http://servicenow.com/xquery/module/now-dita-utils" > abbrev="now-xquery" > version="0.1" spec="1.0"> > <title>XQuery modules for ServiceNow Product Content processing and support</title> <xquery> > <namespace>http://servicenow.com/xquery/module/database-from-git</namespace> > <file>database-from-git.xqm</file> > </xquery> > > … > > </package> > > That implies that each module needs to be in a separate XAR file in order to also be in a separate namespace. > > I don’t think that is consistent with the EXPath packaging spec. > > In the description of the XQuery entry it says: > > An XQuery library module is referenced by its namespace URI. Thus the xquery element associates a namespace URI to an XQuery file. An importing module just need to use an import statement of the form import module namespace xx = "<namespace-uri>";. > An XQuery main module is associated a public URI. Usually an XQuery package will provide functions through library modules, but in some cases one can want to provide main modules as well. > This implies to me that the value of the <namespace> element in the <xquery> is what should be used to resolve the package reference, not the package’s @name value, which simply serves to identify the package. > > Is my analysis correct or have I misunderstood the package mechanism? > > Cheers, > E.

3 6

Strategy for Persisting Maps that Contain Nodes: db:node-id()
by Eliot Kimber 24 Jan '22

24 Jan '22

I have large maps that include nodes as entry values. I want to persist these maps in the DB for subsequent retrieval. I believe the best/only strategy is: 1. Construct a new map where each node in a value is replaced by its node-id() value. 2. Serialize the map to JSON and store as a blob To reconstitute it, do the reverse (parse JSON into map, replace node-ids() with nodes). Is my analysis correct? Have I missed any detail or better approach? Thanks, Eliot _____________________________________________ Eliot Kimber Sr Staff Content Engineer O: 512 554 9368 M: 512 554 9368 servicenow.com<https://www.servicenow.com> LinkedIn<https://www.linkedin.com/company/servicenow> | Twitter<https://twitter.com/servicenow> | YouTube<https://www.youtube.com/user/servicenowinc> | Facebook<https://www.facebook.com/servicenow>

2 6

Unable to Make My Custom EXPath Module Work
by Eliot Kimber 24 Jan '22

24 Jan '22

I’m packaging some XQuery modules into an EXPath XAR archive and then installing them using the repo install command. The command succeeds and my repo is listed by repo list but I’m unable to get the modules to import and I can’t see where I’ve gone wrong. I must have made some non-obvious (to me) error but so far I have not found it. I’ve checked that my expath-pkg.xml against the working examples I have (functx and Schematron BaseX) and I don’t see any difference. I’ve also carefully checked all my namespace URIs to make sure they match. Here is my expath-pkg.xml: <package xmlns="http://expath.org/ns/pkg" name="http://servicenow.com/xquery" abbrev="now-xquery" version="0.1" spec="1.0"> <title>XQuery modules for ServiceNow Product Content processing and support</title> <xquery> <namespace>http://servicenow.com/xquery/module/database-from-git</namespace> <file>database-from-git.xqm</file> </xquery> <xquery> <namespace>http://servicenow.com/xquery/module/now-dita-utils</namespace> <file>now-dita-utils.xqm</file> </xquery> <xquery> <namespace>http://servicenow.com/xquery/module/now-relpath-utils</namespace> <file>now-relpath-utils.xqm</file> </xquery> </package> And here is the structure of the resulting module directory after installation: main % ls ~/apps/basex/repo/http-servicenow.com-xquery-0.1 content expath-pkg.xml main % ls ~/apps/basex/repo/http-servicenow.com-xquery-0.1/content database-from-git.xqm now-dita-utils.xqm now-relpath-utils.xqm main % And the prolog for now-relpath-utils.xqm: module namespace relpath="http://servicenow.com/xquery/module/now-relpath-utils"; Trying to import in a trivial XQuery script, e.g.: import module namespace relpath="http://servicenow.com/xquery/module/now-relpath-utils"; count(/*) Produces this error: Error: Stopped at /Users/eliot.kimber/git/dita-build-tools/src/main/xquery/file, 1/88: [XQST0059] Module not found: http://servicenow.com/xquery/module/now-relpath-utils. So clearly something isn’t hooked up correctly but I don’t see any obvious breakage or violation of a required convention. Any idea where I’ve gone wrong or tips on debugging the resolution failure? This is 9.6.4 on macOS. Thanks, Eliot _____________________________________________ Eliot Kimber Sr Staff Content Engineer O: 512 554 9368 M: 512 554 9368 servicenow.com<https://www.servicenow.com> LinkedIn<https://www.linkedin.com/company/servicenow> | Twitter<https://twitter.com/servicenow> | YouTube<https://www.youtube.com/user/servicenowinc> | Facebook<https://www.facebook.com/servicenow>

2 1

Response from proc:execute() for git Command Not Coming Through as Expected
by Eliot Kimber 22 Jan '22

22 Jan '22

[BaseX 9.6.4 on Mac through BaseX GUI] I’m trying to automate syncing a database with content from a git repo that is locally cloned. My approach is to use proc:exec() to pull the repo and then use git diff-tree to see what files changed: let $pullResult := proc:execute($gitCmd,('pull', '-v'), map{'dir' : $repoPath}) let $changeList := proc:execute($gitCmd, ('diff-tree', '--no-commit-id', '--name-status', 'HEAD'), map{'dir' : $repoPath}) However, the result returned for “diff-tree” appears to get truncated. Here’s a typical result, where I’m echoing out the $pullResult and $changeList values created above: Pull result: <result> <output>Already up to date. </output> <error>From code.devsnc.com:doc/dita-now = [up to date] master -> origin/master = [up to date] dita-now -> origin/dita-now = [up to date] rtprn -> origin/rtprn = [up to date] sanfransokyo -> origin/sanfransokyo = [up to date] scodefreeze -> origin/scodefreeze = [up to date] scratc/table_issues_fix -> origin/scratc/table_issues_fix = [up to date] scratch/fix_canvas_issue -> origin/scratch/fix_canvas_issue = [up to date] scratch/newDitavals2022 -> origin/scratch/newDitavals2022 = [up to date] scratch/simplifyDitavals -> origin/scratch/simplifyDitavals = [up to date] scratch/table_name_issue -> origin/scratch/table_name_issue </error> <code>0</code> </result> Change list: <result> <output>M doc </output> <code>0</code> </result> Note that the pull response looks as expected but the change list response is just “M\tdoc” where “doc” is the first directory in what should be a path to a file. Here’s the same result from the command line: dita-now % git diff-tree --no-commit-id --name-status -r HEAD M doc/source/product/rpa-studio/task/use-datareader-queryexcel.dita M doc/source/product/rpa-studio/task/use-datetime-add.dita M doc/source/product/rpa-studio/task/use-datetime-compare.dita … I can’t see anything I’m doing wrong or options to the execute() or system() functions that would affect the result. Any idea what might be causing this or how I could work around it? Thanks, Eliot _____________________________________________ Eliot Kimber Sr Staff Content Engineer O: 512 554 9368 M: 512 554 9368 servicenow.com<https://www.servicenow.com> LinkedIn<https://www.linkedin.com/company/servicenow> | Twitter<https://twitter.com/servicenow> | YouTube<https://www.youtube.com/user/servicenowinc> | Facebook<https://www.facebook.com/servicenow>

3 3

Outdated milton-api version
by Matthew Dziuban 21 Jan '22

21 Jan '22

Hi all, While auditing the library dependencies of an application, I found that org.basex:basex-api [1] depends on a very old version of com.ettrema:milton-api -- v1.8.1.4 released in 2014 [2]. The package seems to have been migrated to the io.milton namespace and have a recently (December 2021) released v3.1.0.301 [3]. This stood out to me because my application is using the latest version of Apache commons-io, v2.11.0, while the old version of milton-api depends on Apache commons-io v1.4, released in 2008. Would it be feasible to migrate BaseX to a newer version of the milton-api library? Thanks in advance! Matt [1] https://mvnrepository.com/artifact/org.basex/basex-api/9.6.4 [2] https://mvnrepository.com/artifact/com.ettrema/milton-api/1.8.1.4 [3] https://mvnrepository.com/artifact/io.milton/milton-api/3.1.0.301

2 1

Where-Used: Performance Improvement Strategies?
by Eliot Kimber 18 Jan '22

18 Jan '22

In the context of my 40K topic DITA corpus, I’m trying to build a “where used” report that finds, for each topic, the other topics that directly refer to the topic. I can do this by looking for the target topic’s filename in the values of @href attributes in other topics (I’m taking advantage of a local rule we have where all topic filenames should be unique). My current naive approach is simply: $topics//*[tokenize(@href, '/') = $filename] Where $topics is the 40K topics. Based on profiling, the use of tokenize() is slightly faster than either matches() or contains(), but all forms take about 0.5 seconds per target topic, which is way too slow to make this practical in practice. So I’m trying to work out what my performance optimization strategies are in BaseX. In MarkLogic I would set up an index so I could do fast lookup of tokens in @href values or something similar (it’s been long enough since I had to optimize MarkLogic queries that I don’t remember the details but basically indexes for everything). I know I could do a one-time construction of the where-used table and then use that for quick lookup for subsequent queries but I’m trying to find a solution that is more appropriate for my current “create a new database with the latest files from git and run some queries quickly to get a report” mode. I suspect that using full-text indexing may be a solution here but wondering what other performance optimization options I have for this kind of look up. Thinking about it now I definitely need to see if building the where-used table would actually be slower. That is, find every @href, resolve it and construct a map of topics to href elements that point to that topic. Hmm. Anyway, any guidance on this challenge would be appreciated. Cheers, Eliot _____________________________________________ Eliot Kimber Sr Staff Content Engineer O: 512 554 9368 M: 512 554 9368 servicenow.com<https://www.servicenow.com> LinkedIn<https://www.linkedin.com/company/servicenow> | Twitter<https://twitter.com/servicenow> | YouTube<https://www.youtube.com/user/servicenowinc> | Facebook<https://www.facebook.com/servicenow>

4 5

Questions about the performance of bulk string matching
by Zimmel, Daniel 18 Jan '22

18 Jan '22

Hi, recently I ran into serious (as in SERIOUS) performance trouble regarding expensive regexes, and no wonder why. Here is my scenario: * XML text documents with a total of 1m text nodes * A regex search string, including a huge string dictionary list of 50.000 strings (some of them containing 50 words each) * a match must be wrapped in an element (= create a link) I could drink many cups of tea while waiting for the regex to complete... hours... I ran out of tea! Then I found the 2021 Balisage paper from Mary Holstege titled "Fast Bulk String Matching" [1] in which she explores the Aho-Corasick algorithm, implementing it with XQuery - marvellous! Following this, while I can build a data structure which gives me fast results, building the same structure is still very slow due to the amount of text to build from. So this was not fast enough for my use case - or I may simply not be smart enough to apply it correctly :-| So, I tried tinkering with maps which turned out to give me extraordinary performance gains: * build a map from the string dictionary * walk through all text nodes one by one * for each text node, put any combination of words in the text node in word order (I need to find strings, not words) into another map * strip punctuation (for word boundaries) and do some normalization of whitespaces in both maps * compare the keys of both maps * give the new reduced string dictionary to the regular regex search While comparing the maps, I do not know where in the text my strings are, but at least I know if they are in there - to find out where exactly (and how do they fit my other regex context) I can then use a massively reduced regular regex search. Fast! I am quite happy BUT I still cannot understand why this is so much faster for my sample data: * plain method : 51323.72ms * maps method: 597.94ms(!) Is this due to any optimizations done by BaseX or have I simply discovered the power of maps? How would you do it? Is there still room for improvement? Why does Aho-Corasick not help much with this scenario? Is it because the list of strings is simply too massive? Why is this so much faster with text-splitting-to-map? See below for the query examples to better understand what I am trying to do (bulk data not included) [2],[3] There is no normalization of punctuation in the examples but that is only necessary for completeness. Best, Daniel [1] http://www.balisage.net/Proceedings/vol26/print/Holstege01/BalisageVol26-Ho… [2] plain method let $textnodes := fetch:xml(file:base-dir()||file:dir-separator()||'example.xml')//text() let $strings := file:read-text(file:base-dir()||file:dir-separator()||'strings.txt') let $regex := $strings for $textnode in $textnodes where matches($textnode,$regex) = true() return $textnode [3] maps method (:~ : Create map from string : Tokenization : : @param $strings : @return Map :) declare function local:createMapFromString ( $string as xs:string ) as map(*) { let $map_words := map:merge( for $string in tokenize($string,'\|') let $key := $string let $val := $string return map:entry($key,$val), map { 'duplicates': 'use-first' }) return $map_words }; (:~ : Create map from text nodes : Write any combination of words in document order to the map : : @param $textnodes : @return Map :) declare function local:createMapFromTextnodes ( $textnodes as xs:string+ ) as map(*) { map:merge( for $node in $textnodes let $text := normalize-space($node) let $tokens := tokenize($text,' ') let $map_nodes := map:merge( for $start in 1 to fn:count($tokens) for $length in 1 to fn:count($tokens) - $start + 1 return map:entry(fn:string-join(fn:subsequence($tokens, $start, $length), ' '),'x') ) return $map_nodes) }; (:~ : Compare two maps : : @param $map1 : @param $map2 : @return xs:string* :) declare function local:reduceMaps ( $map1 as map(*), $map2 as map(*) ) as xs:string* { for $key in map:keys($map1) where map:contains($map2,$key) let $value := map:get($map1,$key) return $value }; let $textnodes := fetch:xml(file:base-dir()||file:dir-separator()||'example.xml')//text() let $strings := file:read-text(file:base-dir()||file:dir-separator()||'strings.txt') let $map_words := local:createMapFromString($strings) let $map_textnodes := local:createMapFromTextnodes($textnodes) let $matches := local:reduceMaps($map_words,$map_textnodes) let $regex := string-join(for $match in $matches group by $match return $match,'|') for $textnode in $textnodes where matches($textnode,$regex) = true() return $textnode

2 4

Configuration Option for Files to Parse in Create Database Dialog?
by Eliot Kimber 14 Jan '22

14 Jan '22

I’m using the BaseX GUI in a very informal way where I create new databases as needed to do ad-hoc queries over content that comes from various git repos. I’m trying to setup my local configuration settings to be what I need for this database creation tasks. I’ve found and successfully configured the options for skipping corrupt and parsing archives but I don’t see an option that will populate the Input file or directory field, which I need to be “*.xml, *.dita,*.ditamap”. I found the option that sets the pattern used in the main file selector field but that doesn’t apply to this dialog. I’m not seeing anything for this in the docs. This is for BaseX 9.6.3 Thanks, E. _____________________________________________ Eliot Kimber Sr Staff Content Engineer O: 512 554 9368 M: 512 554 9368 servicenow.com<https://www.servicenow.com> LinkedIn<https://www.linkedin.com/company/servicenow> | Twitter<https://twitter.com/servicenow> | YouTube<https://www.youtube.com/user/servicenowinc> | Facebook<https://www.facebook.com/servicenow>

3 5

9.6
by Bondeson, Carl 12 Jan '22

12 Jan '22

We have been using BaseX since 7.0 in a multi-threaded environment. Our configuration utilizes an architecture where child threads query a singleton BaseX server, running as a service using YAJSW. We were unable to update past 8.0, due to issues with our licensed Java tier. We recently upgraded to a RedHat version of the open JDK.11, and were able to upgrade BaseX to 9.6. The performance is much improved. We also saw better memory management by running BaseX in its own container (YAJSW). This implementation is handling the processing of Corona, and 75 other reportable diseases from over 3,000 sources (labs, hospitals, doctor's offices, etc., etc.). This platform utilizes 6 databases, which work as rule engines, to drive the processing of data as it moves through the system. Thanks again for an outstanding product. Carl R Bondeson IT Analyst 3 Department of Public Health Operation & Support Services Information Technology Phone: 860-509-7434 Carl.Bondeson(a)ct.gov<mailto:Carl.Bondeson@ct.gov> [cid:image001.jpg@01D8078F.B6177CE0][cid:image002.png@01D8078F.B6177CE0]

1 0

9.6 on Docker Hub
by Johannes Bauer 10 Jan '22

10 Jan '22

Hi, I cannot find the latest BaseX release on Docker Hub (only the 9.5 branch). Is this still planned? Thank you. Best Regards Johannes

3 4

RESTXQ design pattern question
by Hans-Juergen Rennau 07 Jan '22

07 Jan '22

Hello, I have a question concerning RESTXQ design patterns. Imagine you have a page P1 displaying an input field, and you want to provide a button enabling the user to pick the terms from a list, rather than entering them. So after pressing the button, another page P2 appears, listing the terms and enabling a selection. After selection, P1 should again appear, looking as before, except for the selected terms appearing in the field. A common pattern, isn't it. How to implement it? One approach would be P2 calling P1 (via form/@action), passing the selection to P1 as form or query parameters. But then the input of P1 must include everything required to restore the previous appearance, including the values of input fields from forms not yet submitted. There is certainly a simpler way? I suppose a standard approach handles this completely with JavaScript, updating the page. I wonder if RESTXQ and supporting modules enable some alternative? Thank you, with kind regards -Hans-Jürgen

2 1

webapp/static
by Hans-Juergen Rennau 07 Jan '22

07 Jan '22

Hello, dear BaseX team, one question concerning folder $BASEX/webapp/static. Given an HTML page created by a query via REST or RESTXQ execution. It seems to me that referenced resources (e.g. link/@href or img/@src) must be located in that folder or a subfolder of it, and the reference must be "../static/foo.css" or "../static/path/to/subfolder/foo.css", examples: <link rel="stylesheet" type="text/css" href="../static/foo.css"/><link rel="stylesheet" type="text/css" href="../static/foo/foo.css"/> At least I did not succeed with any other locations, specified by full path or by relative path, assuming different context paths (e.g. the folder containing the query producing the HTML, or $BASEX/bin, or $BASEX). My QUESTION: is that correct, if not - what are the rules? Many thanks,Hans-Jürgen PS: Apologies if this is documented, but I failed to find it, and the complete PDF contains a single occurrence of string "/static" which does not apply.

2 1

RESTXQ generated HTML - unexpected behaviour
by Hans-Juergen Rennau 07 Jan '22

07 Jan '22

Hello, I face a problem with RESTXQ of a fairly general character, as it seems to me. When the XQuery functions of a RESTXQ application construct HTML pages, I expect these pages to behave exactly as a static page with the same content would behave. But sometimes this is not the case. As an attachment I send an example. The "webapp" folder contains an XQuery module with a RESTXQ function [2], and the "html" folder contains an HTML page [3] which is identical to the page generated by the RESTXQ function [4]. This page has been copied from [1], and it demonstrates auto completion in an input field. However, auto completion works only when calling the HTML file; it does not work when calling the RESTXQ application. In both cases, the HTML page references a JS file as well as a CSS file, located in a sibling folder "static". An alert box as well as the appearance of the page prove that JS and CSS are found and used in both cases. And yet the behaviour is different ... ~~~ This problem appears to me important, as the trust in RESTXQ would be damaged if the generated HTML "may or may not" behave as it should. On the other hand, if the reason of the problem is understood and a workaround is available, all would be well. Kind regards,Hans-Jürgen [1] https://www.w3schools.com/howto/howto_js_autocomplete.asp [2] $demo-autocomplete/webapp/autocomplete/demo-autocomplete.xq [3] $demo-autocomplete/html/demo-autocomplete.html [4] The contents of the HTML file have been created by (1) entering into a browser the URL of the RESTXQ application (localhost:8984/demo-autocomplete), (2) selecting from the context menu "Show source code", (3) copying the source code into the empty HTML file, (4) replacing <script .../> with <script...></script>. [5] How to reproduce the problem:a. Unzip the archiveb. Call $demo-autocomplete/html/demo-autocomplete.html - after acknowledging the alert box which shows that the imported JS function has been called, enter into the input field letter "a" - a list of completions appears.c. Install the RESTXQ application: copy folder $demo-autocomplete/webapp/autocomplete into the webapp folder of a BaseX instance, and copy the files $demo-autocomplete/webapp/static/* into $BASEX/webapp/static.d. Enter in a browser "localhost:8984/demo-autocomplete" - after acknowledging the alert box, enter into the input field letter "a" - no list of completions appears.

2 4

Re: [basex-talk] Copy data from MariaDB into BaseX
by Christian Grün 23 Dec '21

23 Dec '21

Thanks. Does the query do what you are looking for? On Tue, Dec 21, 2021 at 12:47 PM <benengbers(a)dds.nl> wrote: > > Christian Grün schreef op 21-12-2021 10:18: > > Hi Ben, > > > return db:add($db, $doc, $table || '.xml') > > > > Could you give us little examples for <DB-name>, <DB-schema> and > > <table-name> ? > > > > Best, > > Christian > > To the best of my knowledge in MySQL and/or MariaDB DB-name and > DB-schema are identical? The schema-name I use is 'Innovate'. > Table-names are > +--------------------+ > | Tables_in_Innovate | > +--------------------+ > | Dienst | > | Mdw_Probleem | > | Mdw_Wens | > | Medewerker | > | Medewerker_dienst | > | Probleem | > | Wens | > +--------------------+ > > Ben > PS.I hope you'll see this reply. Since a few days all mail from > basex-talk is refused by Thunderbird. At least I don't see them anymore > ....

2 5

Copy data from MariaDB into BaseX
by Ben Engbers 21 Dec '21

21 Dec '21

Hi, After completing my work on the R-client, I started working on a Prolog-client. Long ago I wrote an application in SWI-Prolog which operated on data from a MySQL-database. (In the meantime I changed from MySQL to MariaDb). My goal is to write a new version of that application but now based on data which is stored in Basex. In the basexgui, I created an empty database "MariaBases" The following code can be used to select data in MariaDb: sql:init("org.mariadb.jdbc.Driver"), let $con := sql:connect('jdbc:mariadb://localhost:3306/<DB-name>', '<user>', '<password>') return sql:execute($con, "select * from Mdw_Wens") returns: <sql:row xmlns:sql="http://basex.org/modules/sql"> <sql:column name="ID">1</sql:column> <sql:column name="Medewerker_ID">5</sql:column> <sql:column name="Wens_ID">1</sql:column> </sql:row> Is it possible to change the query-statement in such a way that the results are added to MariaBases/<DB-schema>/<table-name>? -- Ben Engbers

2 1

BaseX 9.6.4: Tweaks, Fixes & Thank Yous
by Christian Grün 17 Dec '21

17 Dec '21

Dear all, We conclude this year with yet another release of BaseX, our XML framework, database and XQuery processor [1]. The new version contains new performance tweaks and minor bug fixes, the details of which can be examined on our GitHub page [2]. As we’ve got numerous inquiries regarding Log4j: I can comfort you that BaseX is not affected by the vulnerabilities. We have our own logging mechanism, which has no built-in code execution features. The same applies to the Jetty web server. Log4j once again demonstrates that Open Source is everywhere, and that it takes valuable time and resources to build professional and secure software. We send out a big thank you to those of you who are (and have been) supporting us by paying for maintenance, sponsoring new features and giving donations. It’s you who is keeping BaseX both alive and up to date! Have a good time, Christian [1] https://basex.org [2] https://github.com/BaseXdb/basex/commits/master

4 4

Email.jar module documentation
by Eliúd Santiago Meza y Rivera 16 Dec '21

16 Dec '21

Hi, Sorry for ask, but where I can find or read more about email.jar module documentation?, I'm working for a personal project and beside to send e-mail, I need to read some emails, or is there another options to do this task. Cheers. Eliud Meza

2 1

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

BaseX-Talk