- BaseX-Talk - mailman.uni-konstanz.de

Not sure if feature/bug/my mistake!
by Patrick Durusau 06 May '22

06 May '22

Greetings! I have a directory of files that I use to create a database, using the directory as my starting point. On Debian, the files appear in this order: 01-matthew.xml 02-mark.xml 03-luke.xml 04-john.xml If I create an empty database and then add the files one by one, in that order, then //title[@type="main"] returns: ΚΑΤΑ ΜΑΘΘΑΙΟΝ ΚΑΤΑ ΜΑΡΚΟΝ ΚΑΤΑ ΙΩΑΝΗΝ ΚΑΤΑ ΙΩΑΝΗΝ On the other hand, if I start at the directory, add xml files, the database is created and //title[@type="main"] returns: ΚΑΤΑ ΙΩΑΝΗΝ ΠΡΟΣ ΕΦΕΣΙΟΥΣ ΚΑΤΑ ΜΑΘΘΑΙΟΝ ΙΩΑΝΟΥ Α Question (sorry for the long setup): Is the sort order of manually added files an artifact of the manual addition? That is I cannot expect users across systems to experience the same load order as I do? Asking because I'm writing documentation along with example queries for several datasets and documenting random results, isn't likely to be helpful to readers. I am aware I can use db:list(database-name) to obtain and sort on the original file names but that takes me away from the file content. Ah, BaseX 9.7.1 on Debian. You know, the files don't have internal sort attributes. ntorder = "digit" would be enough to sort on that attribute and return the book names in the expected order. I'm sure there are better solutions, suggestions appreciated! Patrick -- Patrick Durusau patrick(a)durusau.net Technical Advisory Board, OASIS (TAB) Editor, OpenDocument Format TC (OASIS), Project Editor ISO/IEC 26300 Co-Editor, ISO/IEC 13250-1, 13250-5 (Topic Maps) Another Word For It (blog): http://tm.durusau.net Homepage: http://www.durusau.net Twitter: patrickDurusau

3 3

Session issue
by Bondeson, Carl 04 May '22

04 May '22

We are seeing the following exception if we run under very heavy multi-thread conditions. ERROR] 2022-05-04 08:49:41.345 [Thread-3012] - General Throwable occurred message: No buffer space available (maximum connections reached?): connect Caused by this // create BaseX session on singleton server mapperSession = new ClientSession(BaseXDictionary.BASEX_HOST, basexdict.baseXPort, basexdict.baseXUser, basexdict.baseXPass); We are using XSLT and invoking Java methods to handle the things not easily managed in XSLT. Should we be closing sessions in Java classes or is this supposed to be handled by the JVM? This is running under Windows Server 2016 , deployed using JBoss 7.3.9, with 16 GB ram, and 4 cores. Thanks Carl R Bondeson IT Analyst 3 Department of Public Health Operation & Support Services Information Technology Phone: 860-509-7434 Carl.Bondeson(a)ct.gov<mailto:Carl.Bondeson@ct.gov> [cid:image001.jpg@01D85F9D.FCF29490][cid:image002.png@01D85F9D.FCF29490]

2 2

Re: [basex-talk] [basex-announce] BaseX 9.7.1: Tweaks, Fixes, Features
by Christian Grün 03 May '22

03 May '22

Confirmed, thanks. A new snapshot is available [1]. If you want to stick with BaseX 9.7.1, you can remove the trailing slash in your URL. [1] https://files.basex.org/releases/latest/ On Tue, May 3, 2022 at 5:13 PM Majewski, Steven Dennis (sdm7g) < sdm7g(a)virginia.edu> wrote: > > I’m also seeing a new error crop up with 9.7.1 > > Could this be connected ? > > ( I’ll try to revert to previous version to make sure that there hasn’t > been a change at the source URL that would cause a change in result. ) > > *WTJU$* basex 'html:doc( "https://www.wtju.net/recent-shows/" )' > Stopped at /Users/sdm7g/Desktop/WTJU/, 1/9: > [FODC0002] URI 'https://www.wtju.net/recent-shows/' points to directory. > *WTJU$* > *WTJU$* curl -I https://www.wtju.net/recent-shows/ > HTTP/2 200 > *server*: nginx/1.10.3 > *date*: Tue, 03 May 2022 15:10:52 GMT > *content-type*: text/html; charset=UTF-8 > *x-pingback*: https://www.wtju.net/xmlrpc.php > *link*: <https://www.wtju.net/wp-json/>; rel="https://api.w.org/" > *link*: <https://www.wtju.net/?p=173>; rel=shortlink > >

1 0

Re: [basex-talk] [basex-announce] BaseX 9.7.1: Tweaks, Fixes, Features
by Majewski, Steven Dennis (sdm7g) 03 May '22

03 May '22

I still had BaseX971-20220411.133348.zip ( 9.7.1 beta) sitting around to try And I don’t get that error using it. > On May 3, 2022, at 11:12 AM, sdm7g <sdm7g(a)virginia.edu> wrote: > > > I’m also seeing a new error crop up with 9.7.1 > > Could this be connected ? > > ( I’ll try to revert to previous version to make sure that there hasn’t been a change at the source URL that would cause a change in result. ) > > WTJU$ basex 'html:doc( "https://www.wtju.net/recent-shows/ <https://www.wtju.net/recent-shows/>" )' > Stopped at /Users/sdm7g/Desktop/WTJU/, 1/9: > [FODC0002] URI 'https://www.wtju.net/recent-shows/ <https://www.wtju.net/recent-shows/>' points to directory. > WTJU$ > WTJU$ curl -I https://www.wtju.net/recent-shows/ <https://www.wtju.net/recent-shows/> > HTTP/2 200 > server: nginx/1.10.3 > date: Tue, 03 May 2022 15:10:52 GMT > content-type: text/html; charset=UTF-8 > x-pingback: https://www.wtju.net/xmlrpc.php <https://www.wtju.net/xmlrpc.php> > link: <https://www.wtju.net/wp-json/ <https://www.wtju.net/wp-json/>>; rel="https://api.w.org/ <https://api.w.org/>" > link: <https://www.wtju.net/?p=173 <https://www.wtju.net/?p=173>>; rel=shortlink >

1 0

Text index requires `/text()` in query
by Matthew Dziuban 02 May '22

02 May '22

Hi all, I was recently debugging performance of a query with an exact string comparison and discovered that it seems the query was only rewritten to use the text index [1] if I explicitly added `/text()` to the path I was comparing. My data looks like this: <data> <element><id>123</id></element> </data> And my original query was: for $el in db:open('DatabaseName')/data/element where $el/id = '123' return $el With 3 million <element> nodes in the database, this query took about 4 seconds, which made me question whether the text index was being used. I then changed the query to add `/text()` to the `where` clause, like so: for $el in db:open('DatabaseName')/data/element where $el/id/text() = '123' return $el With this change, the query only takes 0.4 seconds. Is it expected that `/text()` is required to get the text index to kick in? Thanks in advance, Matt [1] https://docs.basex.org/wiki/Indexes#Text_Index

2 6

Performance of ft:search function
by Tim Thompson 29 Apr '22

29 Apr '22

Hello, I have a largish (5.4G) file with a full-text index that I am using to reconcile names in a local dataset. I've been experimenting with splitting the file into many smaller index files to improve performance. I group the entries by initial character and create a new index file for each distinct initial character. Each smaller file then gets its own full-text index. I've been following the approach outlined in the documentation for custom index structures <https://docs.basex.org/wiki/Indexes#Custom_Index_Structures>. Using prof:track, I've noticed the following performance for different uses of ft:search. (Here, $db refers to the 5.4G file, and $index refers to a smaller 159MB subindex. Times are averaged across 10 runs of 1000 iterations for each expression.) 1. Direct lookup against large index Time: 23ms Expression: ft:search($db, $text)/../.. 2. Direct lookup against subindex Time: 3.3ms Expression: ft:search($index, $text)/../.. 3. Lookup against subindex file with reference to large index Time: 2.9ms Expression: let $s := ft:search($index, $text)/../.. return db:open-id($db, $s/id)/../.. My question is: why would the third expression be slightly faster (or at least not slower) than the second one, if it involves additional computation? Thanks in advance, Tim -- Tim A. Thompson (he, him) Librarian for Applied Metadata Research Yale University Library

2 3

xsl:transform-report message truncation
by Andy Bunce 29 Apr '22

29 Apr '22

Hi, Using 9.7.1 (: test transform :) let $xslt:=<xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform" version="3.0"> <xsl:template match="/"> <xsl:message>I want to see all of the very long message aaaaaaaaaaaaaaaaa aaaaaaaaaaaaaaaaa bbbbbbbbbbbbbbb cccccccccccc hhhhhhhhhhhhhhhhhhhhhhhhhhhh gggggg gggggggggggggggggg aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa important bit</xsl:message> </xsl:template> </xsl:stylesheet> return xslt:transform-report(<xml/>,$xslt)?messages Returns ["I want to see all of the very long message aaaaaaaaaaaaaaaaa
 aaaaaaaaaaaaaaaaa bbbbbbbbbbbbbbb cccccccccccc hhhhhhhhhhhhhhhhhhhhhhhhhhhh
 gggggg gggggggggggggggggg aaaaaaaaaaaaaaaaaaa..."] Is it BaseX truncating this? Can it be turned off for this case? /Andy

2 2

Date picture and xslt:transform()
by Zimmel, Daniel 29 Apr '22

29 Apr '22

Hi, why do I get different results with the following two queries? xslt:transform() does not respect my date picture. Expected result: <root>29. März 2022</root> Query 1: <root>{format-date(xs:date('2022-03-29'), '[D]. [MNn] [Y]', 'de', (), ())}</root> Result: <root>29. März 2022</root> Query 2: declare namespace xsl = 'http://www.w3.org/1999/XSL/Transform'; let $xslt := <xsl:stylesheet version="3.0" xmlns:xs="http://www.w3.org/2001/XMLSchema" exclude-result-prefixes="xs"> <xsl:template match="/"> <root> <xsl:sequence select="format-date(xs:date('2022-03-29'), '[D]. [MNn] [Y]', 'de', (), ())"/> </root> </xsl:template> </xsl:stylesheet> let $xml := <root/> return for $xml in $xml return $xml => xslt:transform($xslt) Result: <root>[Language: en]29. March 2022</root> Running the XSLT with Saxon EE (not in BaseX via xslt:transform) returns (correctly): <root>29. März 2022</root> Using BaseX 9.5 ? Daniel

2 3

Stemming in BaseX Full-Text
by Tim Thompson 27 Apr '22

27 Apr '22

I'm currently involved in a project that's using MarkLogic, and I noticed that its implementation of English-language stemming differs from that of BaseX: e.g., "mouse" and "mice" both stem to "mouse." In BaseX, those words are stemmed separately. Is this a known limitation of the internal English syntax parser? Example: db:create("stem-test", <data> <x>mouse</x> <y>mice</y> </data> , "data", map {"ftindex": true(), "stemming": true(), "language": "en"} ) , update:output( ft:search("stem-test", "mice") ) Thanks, Tim -- Tim A. Thompson (he, him) Librarian for Applied Metadata Research Yale University Library

3 7

Using error and catch for error paths in REST API endpoint code
by Omar Siam 27 Apr '22

27 Apr '22

Way too often I saw myself and my colleagues write huge, very hard to understand deeply nested if () then else code to handle any control path but the complete success one in RestXQ code. [1] is an example of such code even though it uses the module I want to introduce here. It seems I didn’t have time to refactor it yet. To add insult to injury the code producing a 500, 404, 403, 401, 302 or maybe even 201 response code was * neither short nor very uniform and * would respond to the browser with XML/XHTML or JSON or text but in most situations the format the browser side had the hardest time to handle For example [2] delivers an XHTML message no matter what the Accept header says and although it is an accurate message to a user, very similar XML snippets are found throughout that unit and maybe need to be adapted each separately if the HTML needs to updated. My XML snippets CRUD API started as a port of an apigility (now laminas api-tools [3]) API working with a relational backend to store XML snippets to something that does the same but just uses BaseX for any data storage and is much better at querying using sophisticated XPaths. So from that previous API I learned about RFC 7807. [4] RFC 7807 “Problem Details for HTTP APIs” [5] is one of several specification now available for reporting errors from REST APIs. This specification explicitly states how the errors should look like in XML as well as in JSON. Maybe this is a bit bold, but I used the URL of that very RFC 7807 as the resolved URI for my module. [6] Please note: I have to admit my modules have something unusual in common: I use the namespace prefix “_” within the module and only use a more talking prefix when I use a module elsewhere. So, the “_” prefix can map to numerous URIs in my code. I am not 100% sure if there are down sides to this style but I use it for a while now and no problems come to mind. I really dislike to get an error, especially during development of some service, without any indication of where that actually occurred. That is to say: I like stack traces in my errors. It also would be great if any runtime error in my code would be reported as XML or JSON, depending on the format the browser asked for, just as errors I explicitly raise. A few parts provided by BaseX greatly help in getting all of this packaged in some xqm-file. * A stack trace is always available in “$err:additional” when catching errors [7] * One catch-all error handler can be installed (“declare %rest:error('*') function”, although there are minor downsides to this catch-all handler) [8] * The XML based direct format BaseX uses to store JSON by default makes it very easy to transform RFC 7807 XML to JSON [9] * It is easy to query the request header anywhere in RestXQ XQuery code running on BaseX [10] I tried to have easy to remember function names that make the code readable as if it was a sentence. Therefore for example, I created a function wrapping the users code that says “return api-problem:or-result(user_function#x, [params, …])”[11]. Another example would be “return api-problem:result(<problem>[…]</problem>)”. I also wanted to come up with an “intuitive way” to send standard HTTP response codes. What I came up with is a special namespace and a mapping of status codes to standard messages that is part of my module. So for example a status code 404 can be returned like this: “error(xs:QName('response-codes:_404'), $api-problem:codes_to_message(404), 'A custom message')” [12]. Something similar is probably possible with “web:response-header()” but with the error function something like if let $check_file_exists := if (not(file:exists($path))) then error(xs:QName('response-codes:_404'), $api-problem:codes_to_message(404), 'A custom message') at the top of a more complex RestXQ function is possible. There is no need to wrap custom code in one or several “if () then else” blocks. This helps readability in my opinion very much, especially if you have to check quite a few things before, say, writing something to the database. The idea also works great with permission checks that use %perm:check annotations. My first use of my api-problem module predates the addition of these helpful annotations. [13] A while before we had a HTTP header parameter that gave us the execution time, I wanted to measure execution time. So the functions take an xs:integer that should be obtained using “prof:current-ns()” [14] as the very first variable in a RestXQ function and I try to execute a second “prof:current-ns()” as late as I can imagine in my module [15]. That way I think I get a reasonably accurate timing result in my outputs as long as the respective error does not come from the catch-all handler. I also incorporated a quick and dirty HTML page rendering function that displays an error in detail if someone needs that [16]. As a small addition this error page can link to error descriptions in the W3C standards. [17] By the way: developing this started on BaseX but then we had similar needs in another open-source XML database existing today so I tried to port the code. Not only worked that rather well (probably also because I know the XQuery needed there rather well too), I also had a few new ideas and added them back to the BaseX version. May be there is some left over code in the module at the moment that reimplements some helpful, non-standard XQuery functions for this reason. The code is somewhat portable. A few thoughts: * Some say that it is necessary for security purposes to disable any stack traces in production environments. I am not really believing this does much good. But if one does not want to hard code a Boolean switch in the modules source code: What would be the fastest external source one could use in terms of compile, optimizing and execution time? Are stack traces not available as “$err:additional” when RESTXQERRORS are switched off? * Is there any sane way to get a QName with an unknown prefix of an error as a string like in the catch all handler and resolve it against all prefix-URI mappings known in some XQuery program? [1] https://github.com/acdh-oeaw/vleserver_basex/blob/main/vleserver/users.xqm#… [2] https://github.com/acdh-oeaw/vicav-app/blob/master/http.xqm#L24-L55 [3] https://api-tools.getlaminas.org/ [4] https://api-tools.getlaminas.org/documentation/modules/api-tools-api-problem [5] https://datatracker.ietf.org/doc/html/rfc7807 [6] https://github.com/acdh-oeaw/api-problem4restxq/blob/master/api-problem/api… [7] https://docs.basex.org/wiki/XQuery_3.0#Try.2FCatch [8] https://docs.basex.org/wiki/RESTXQ#Catch_XQuery_Errors [9] https://docs.basex.org/wiki/JSON_Module#Direct [10] https://docs.basex.org/wiki/Request_Module#request:header [11] https://github.com/acdh-oeaw/api-problem4restxq/blob/master/tests/api-probl… [12] https://github.com/acdh-oeaw/api-problem4restxq/blob/master/tests/http.xqm#… [13] https://github.com/acdh-oeaw/api-problem4restxq/blob/master/tests/http.xqm#… [14] https://github.com/acdh-oeaw/vleserver_basex/blob/main/vleserver/dicts.xqm#… [15] https://github.com/acdh-oeaw/api-problem4restxq/blob/master/api-problem/api… [16] https://github.com/acdh-oeaw/api-problem4restxq/blob/master/tests/api-probl… [17] https://github.com/acdh-oeaw/api-problem4restxq/blob/master/api-problem/api… Best regards -- Mag. Ing. Omar Siam Austrian Center for Digital Humanities and Cultural Heritage Österreichische Akademie der Wissenschaften | Austrian Academy of Sciences Stellvertretende Behindertenvertrauensperson | Deputy representative for disabled persons Wohllebengasse 12-14, 1040 Wien, Österreich | Vienna, Austria T: +43 1 51581-7295 omar.siam(a)oeaw.ac.at | www.oeaw.ac.at/acdh

3 9

Integers as attribute values
by Giuseppe G. A. Celano 27 Apr '22

27 Apr '22

Hi Everyone, I have an xml document with elements such as <div n=“21”>. If I run the query doc(“file.xml")//div[@subtype="chapter"]//*/parent::div[@n=21], I get the relevant div element, even if 21 is passed as an integer. On the other hand, if I type doc(“file.xmll")//div[@n=21], I get the error "Cannot convert to xs:double”, which can be solved by writing doc(“myfile.xmll")//div[@n=“21”]. Is this due to the fact that BaseX tries to convert the values of @n of all div elements into a number and, if it happens that the @n values returned are all numbers, then an error is not raised (the comparison is then possible), otherwise it is? Is this BaseX specific? Thanks. Best, Giuseppe

3 4

Free Text - understanding
by Hans-Juergen Rennau 25 Apr '22

25 Apr '22

Dear BaseX people, it would be kind if you could check my understanding of Free Text @ BaseX. (1) Tokenization ignores element borders. If correct, I suggest documentation of the fact, seehttps://www.w3.org/TR/xpath-full-text-10/#TokenizationSec"In the absence of an implementation-definedway to differentiate, element markup (start tags, end tags, andempty-element tags) creates token boundaries." (2) Function ft:search can only find individual text nodes, it is not possible to apply scope "phrase" or "all words" beyond the boundaries of an individual text node. So, for example, given a document<doc> <t1>Stand </t1> <t2>der Information. Siehe unten.</t2></doc> there is no way of searching for "Stand der Information" *and* obtain information about the location of the match (in other words - search via ft:search). (3) The unit "sentence" (as for example used in the qualifier same sentence) is exclusively defined by the occurrences of "." (dot) characters. In particular, it is unrelated to text node boundaries. For example:$doc contains text "Stand der Information siehe" same sentence yields false.$doc contains text "Stand der Information" same sentence yields true. (4) The unit "paragraph" (as for example used in the qualifier "same paragraph") is not delimited - "same paragraph" always applies. A check would be highly appreciated! Kind regards,Hans-Jürgen PS: I think there is a bug concerning "different sentence": basex "'base.x' contains text 'base x' same sentence" false basex "'base.x' contains text 'base x' different sentence" false basex "'base x' contains text 'base x' same sentence" true basex "'base x' contains text 'base x' different sentence" false PPS: Thank you very much for the excellent implementation of Free Text - for several years, it has been in productive use by a mission critical service mapping format markup to semantic markup.

2 3

Interrupted queries - partial results?
by Patrick Durusau 21 Apr '22

21 Apr '22

Greetings! Is it possible to capture partial results from an interrupted query? When I realize a query is taking too long for the data involved, it might be helpful in correcting the mistake. Thanks! Patrick -- Patrick Durusau patrick(a)durusau.net Technical Advisory Board, OASIS (TAB) Editor, OpenDocument Format TC (OASIS), Project Editor ISO/IEC 26300 Co-Editor, ISO/IEC 13250-1, 13250-5 (Topic Maps) Another Word For It (blog): http://tm.durusau.net Homepage: http://www.durusau.net Twitter: patrickDurusau

3 6

Performing extra data conversion for double values (by XQuery)
by Markus Elfring 21 Apr '22

21 Apr '22

Hello, I would like to perform a sanity check for some input data which represent numbers that are using the comma as the decimal separator. I stumbled on the message “[FORG0001] Cannot convert to xs:double: …” then according to the evaluation of another XQuery script by the software “BaseX 9.7”. https://docs.basex.org/wiki/XQuery_Errors#Functions_Errors Can any functions help with the safe handling of currency values for my use case? https://docs.basex.org/wiki/Conversion_Module Regards, Markus

2 1

Joining a varying number of information sources (with XQuery)?
by Markus Elfring 19 Apr '22

19 Apr '22

Hello, It is supported to specify a fixed number of information sources for a for clause. https://www.w3.org/TR/xquery-31/#id-xquery-for-clause I imagine that the selection of for bindings can occasionally vary so much and might become so big that it would be needed to construct FLWOR expressions by more sophisticated data structures. I guess that the specification of consistent join conditions (and related constraints) would become more challenging. Can any programming interfaces help further to work with such use cases? https://docs.basex.org/wiki/XQuery_Module Regards, Markus

5 32

Improving the understanding for counting of entries in data groups
by Markus Elfring 19 Apr '22

19 Apr '22

Hello, I constructed the following XML file for another test of the software “BaseX 9.7”. <?xml version="1.0" encoding="UTF-8" standalone="yes"?> <test_data> <info> <id>12</id> <topics> <topic>Demo1</topic> <topic>Demo2</topic> </topics> </info> <info> <id>23</id> <topics> <topic>Demo1</topic> <topic>Demo2</topic> </topics> </info> <info> <id>34</id> <topics> <topic>Test1</topic> <topic>Test2</topic> <topic>Test3</topic> </topics> </info> <info> <id>45</id> <topics> <topic>Test1</topic> <topic>Test2</topic> <topic>Test3</topic> </topics> </info> <info> <id>56</id> <topics> <topic>Test1</topic> <topic>Test2</topic> <topic>Test3</topic> </topics> </info> <info> <id>67</id> <topics> <topic>Probe1</topic> </topics> </info> </test_data> I tried the following XQuery script out accordingly. declare option output:method "csv"; declare option output:csv "header=yes, separator=|"; for $x in //test_data/info group by $topics := string-join($x/topics/topic/data(), "*") let $incidence := count($topics) order by $incidence descending return <csv> <record> <topic_combination>{$topics}</topic_combination> <incidence>{$incidence}</incidence> </record> </csv> Corresponding test result: topic_combination|incidence Demo1*Demo2|1 Test1*Test2*Test3|1 Probe1|1 I would like to see the numbers “2” and “3” instead at the end of two rows for such a data analysis approach. I would appreciate further advices for this use case. Regards, Markus

2 1

CVE-2022-22965: Spring framework 0-day remote code execution vulnerability
by Bülow, Gilles 14 Apr '22

14 Apr '22

Hi all, to exclude issues with BaseX and the topic in this email-subject (CVE-2022-22965: Spring framework 0-day remote code execution vulnerability<https://securityboulevard.com/2022/04/spring4shell-vulnerability-cve-2022-2…>) I would like to know if BaseX could possibly be affected by the problem discussed and if so, when there is an update scheduled which can solve this problem. Thanks in advance for replies Regards Gilles other links: https://www.dfn-cert.de/aktuell/0day-schwachstelle-spring-framework-apache-… https://tanzu.vmware.com/security/cve-2022-22965 -- Dr.-Ing. Gilles Bülow Project Manager Digitalization Production IT DKE German Commission for Electrical, Electronic & Information Technologies Merianstraße 28 63069 Offenbach am Main Phone: +49 69 6308-294 Fax: +49 69 6308-9294 Mobile: +49 170 5769243 gilles.buelow(a)vde.com<mailto:gilles.buelow@vde.com> http://www.dke.de<http://www.dke.de/> [cid:VDE_Email_Abbinder_VDEHiglights2021_aaf1e897-893b-444a-a4a4-f03d3e97ff22.jpg]<https://www.vde.com/highlights> VDE Verband der Elektrotechnik Elektronik Informationstechnik e. V. VDE Association for Electrical, Electronic & Information Technologies Vorstand / Executive Board: Ansgar Hinz (Vorsitzender / CEO); Dr. Beate Mand (stellv. Vorsitzende / CFO) Rechtsform / Legal form: Eingetragener Verein / Registered association Amtsgericht Vereinsregister / Local court register of associations: Frankfurt am Main / Nr. / No: VR 4884 Sitz der Gesellschaft / Association's registered office: Frankfurt am Main, Germany USt-IdNr. / VAT-ID: DE 114 235 367 Datenschutzinformationen<https://www.vde.com/dsh-vde> Data Protection Information<https://www.vde.com/dpn-vde> Der Inhalt dieser E-Mail (sowie der ggf. enthaltenen Anhänge) ist vertraulich und kann darüber hinaus durch besondere Bestimmungen geschützt sein. Sollten Sie nicht der vorgesehene Empfänger sein, bitten wir Sie, den Versender zu informieren und die Nachricht zu löschen. Die Weitergabe sowie Vervielfältigung, Verwertung und Mitteilung seines Inhalts ist nur mit unserer ausdrücklichen Genehmigung gestattet. Alle Rechte vorbehalten, insbesondere für den Fall der Schutzrechtsanmeldung. This e-mail and any attachments are confidential and may also be privileged. If you are not the intended recipient, please immediately notify the sender and delete this message. Its contents are not to be passed on, duplicated, exploited or disclosed without our express permission. All rights reserved, especially the right to apply for protective rights.

2 1

Closing a socketconnection
by Ben Engbers 14 Apr '22

14 Apr '22

Hi, While reading the basexdbc.c code from Alexander Holupirek, I saw that he explicitly sends the 'exit' command to the server before closing the socket. I couldn't find anything on this command in the client server protocol. Is it necessary to send this command to the server? If so what is the effect of sending this command? Ben Engbers

2 1

BaseX Query Performance
by Darryl Pretto 12 Apr '22

12 Apr '22

Hi, I have a BaseX 9.0 instance running on Jetty 9.4.8 on RHEL 7.9. I run many different types of queries on several thousand XML files, about 10gb total source files. These queries take several hours to complete. While I have been able to rewrite queries here and there to optimize them, I am wondering what is the best way to increase BaseX query performance overall. When I check the server CPU and memory neither of them have high utilization. The CPU use varies but mostly is at 12%. The memory use is similar. I have allocated 8gb of memory to Jetty via the Java options settings: -Xms8g -Xmx8g Are there any other resources I can check or change to increase overall query speed? Thanks, Darryl

3 2

Re: [basex-talk] Checking the display of line breaks in the info view
by Markus Elfring 12 Apr '22

12 Apr '22

> What do you mean with space character? I suggest to take a closer look at the differences from a display like “``["line 1 line 2 line 3"]``” in comparison to the mentioned test example. Would you find any other representation more appropriate here? Regards, Markus

4 8

Basexgui Feature suggestion
by Majewski, Steven Dennis (sdm7g) 11 Apr '22

11 Apr '22

It would be handy if there was a way to bind results to current context for additional inspection/investigation/query of results. Currently, it seems, you would have to save results to a file and then create database from that file to make an additional query on results. That binding would be available from editor or input bar. Or perhaps alternatively, it could include an option to base visualizations on results instead of open database — although I’m guessing the former would be easier to implement than the latter. — Steve Majewski

6 11

User group meeting in Prague?
by Imsieke, Gerrit, le-tex 10 Apr '22

10 Apr '22

Fellow BaseX Users! You might have heard that XML Prague 2022 will take place in June. (Unless the then-prevalent Greek letter makes it impossible even in that time of year, of course.) I asked Christian whether the BaseX team will organize a user group meeting after it had not happened for years now. Christian didn’t seem to be very fond of organizing such a meeting. I asked him whether he would be available to present new features, the roadmap, and for a Q&A session if the users themselves organized such a meeting. He agreed, and therefore I hereby ask the list members whether anyone will join me in organizing this. The plan looks as follows: We will apply for one or two 90-minute slots via the CFP process (https://www.xmlprague.cz/cfp/). We don’t need to have a fixed schedule yet by Dec. 20 (end of CFP date as currently announced – it will be extended anyway). Christian was so kind as to create a new repo, user-group, on Github. We will use one or more of its Wiki pages [1] in order to plan the event. The page will eventually evolve into an agenda if you agree. Looking forward to meeting many of you in Prague in June. And another organizing volunteer (or other volunteers), please come forward. Maybe we can also deal with it in the Wiki [2]. Gerrit [1] https://github.com/BaseXdb/user-group/wiki/2022-06-XML-Prague [2] https://github.com/BaseXdb/user-group/wiki/Members

3 3

Checking the display of line breaks in the info view
by Markus Elfring 10 Apr '22

10 Apr '22

Hello, I tried the following XQuery script out with the software “BaseX 9.7”. let $t1 := ``[line 1 line 2]``, $t2 := ``[line 3 line 4]`` return ('"' || $t1 || '"', "---", '"' || $t2 || '"') I observed the following display then in the info view. “… Optimized Query: ("""line 1
line 2""", "---", """line 3
line 4""") Query: let $t1 := ``[line 1 line 2]``, $t2 := ``[line 3 line 4]`` return ('"' || $t1 || '"', "---", '"' || $t2 || '"') …” May I expect that original line breaks will be preserved in the shown query display? Regards, Markus

2 4

html document retrieval runs out of main memory
by Graydon Saunders 08 Apr '22

08 Apr '22

Hello -- I'm using the basexgui to run (minus some identifying actual values defined previously in the query) (: for each path, retrieve the document :) for $remote in $paths let $name as xs:string := file:name($remote) let $target as xs:string := file:resolve-path($name,$targetBase) let $fetched := http:send-request(<http:request method='get' override-media-type='application/octet-stream' username='{$id}' password='{$pass}' />, $remote)[2] let $use as item() := try { html:parse($fetched) } catch * { $fetched } return if ($use instance of document-node()) then file:write($target,$use) else file:write-binary($target,$use) It works, in that I get exactly 100 documents retrieved. (There are unfortunately 140+ documents in the list.) However, the query fails with an "out of main memory" error when using a recent 10.0 beta or 9.7 with Xmx set to 2g. Setting Xmx to 16g with 9.7 produces the same "out of memory" error in the same length of time (about 5 minutes). java -version says 20:27 test % java -version openjdk version "11.0.14.1" 2022-02-08 OpenJDK Runtime Environment 18.9 (build 11.0.14.1+1) OpenJDK 64-Bit Server VM 18.9 (build 11.0.14.1+1, mixed mode, sharing) It's entirely possible I'm going about fetching files off a web server the wrong way; it's possible there's something there that's rather large, but I doubt it's that large. What should I be doing instead? Thanks! Graydon

2 4

I would like to share some code I use for easier RestXQ development
by Omar Siam 08 Apr '22

08 Apr '22

Hi list, I am using BaseX here at my institution a lot as an XML database that provides data using RESTful APIs [1] or serves HTML, JS and CSS with RestXQ which in turn uses HTTP requests to fetch more data [2]. Our main use of BaseX databases is about TEI/XML encoded dictionaries of various sizes. That means we mainly want to search for, get and change small tei:entry parts in larger TEI/XML documents. BaseXwas amazingly stable (compared to some other open source XML database existing today) over the last few years and has a very solid set of built in functions that almost always are sufficient to get a job done. I also like BaseX because I am pretty sure I always could understand why things did not work and what I can do about it even without doing any Java programming (although I saw some weirdness or the other over the last few years). I created or ported some XQuery modules [3][4][6][7] and a containerization environment [5] that make my life easier and I would like to share them here and maybe have a discussion about my implementations and if others can make use of them and how (for example can these [6][7] be expath packages, I see some obstacles in the way RestXQ annotations work). I will try to give an introduction to each of the modules in some separate mails. Finally, I now write to the list as I have a performance problem with a CRUD API I created [1] for the task mentioned above when using it with a dataset that is about 7GB looking at the BaseX databases that make it up. This API uses many of the modules and techniques I came up with so I thought it might be helpful to first talk about those parts. I hope that they may be useful to others as well. I tried to get creative at finding a way to optimize querying the data without having many long lasting global lock situations and having BaseX using indexes as much as possible (this started before the db:enforceindex pragma was introduced and still works for me as expected without it) while still writing the RESTful API in BaseX' implementation of RestXQ. That is why I created [7]. It heavily uses (abuses?) BaseX' jobs module. It allows me to query in smaller BaseX databases in parallel and present them as if they were one big XML DB, which vastly improves performance on update and reindex, to a point. Still there is a file based lock (or is it even class based?) [8], I think the JVM profiling tells me, that severely limits the number of (read) operations that can be done over the API without the user having to wait so long they think the operation failed. This is a multi threading problem as I see it. Or maybe I overlooked something that would solve my problems without all the creative stuff I tried? That probably will be obvious when I have explained the current implentation in more detail which I intend to do in the next few days. [1] https://vle-curation.acdh.oeaw.ac.at/openapi/, https://github.com/acdh-oeaw/vleserver_basex [2] https://vicav.acdh.oeaw.ac.at/, https://github.com/acdh-oeaw/vicav-app [3] https://github.com/acdh-oeaw/openapi4restxq [4] https://github.com/acdh-oeaw/api-problem4restxq [5] https://github.com/simar0at/heroku-buildpack-basex [6] https://github.com/acdh-oeaw/vicav-app/blob/master/http.xqm, https://github.com/acdh-oeaw/openapi4restxq/blob/master_basex/swagger-ui.xqm [7] https://github.com/acdh-oeaw/vleserver_basex/blob/main/vleserver/util.xqm [8] https://github.com/BaseXdb/basex/blob/master/basex-core/src/main/java/org/b… and other read mehods there Best regards -- Mag. Ing. Omar Siam Austrian Center for Digital Humanities and Cultural Heritage Österreichische Akademie der Wissenschaften | Austrian Academy of Sciences Stellvertretende Behindertenvertrauensperson | Deputy representative for disabled persons Wohllebengasse 12-14, 1040 Wien, Österreich | Vienna, Austria T: +43 1 51581-7295 omar.siam(a)oeaw.ac.at |www.oeaw.ac.at/acdh

2 1

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

BaseX-Talk