- BaseX-Talk - mailman.uni-konstanz.de

I can’t use character maps with RESTXQ
by Imsieke, Gerrit, le-tex 22 Jun '22

22 Jun '22

For HTML with embedded Javascript, I need to output > as > rather than as > In XSLT I’m doing <xsl:character-map name="a">  <xsl:output-character character=">" string=">"/> </xsl:character-map> and <xsl:output method="xhtml" indent="no" encoding="UTF-8" use-character-maps="a" omit-xml-declaration="no" html-version="5.0" doctype-public="" doctype-system=""/> In RESTXQ I tried the following: I created a document called html-charmap.xml with the following content: <?xml version="1.0" encoding="UTF-8"?> <serialization-parameters xmlns="http://www.w3.org/2010/xslt-xquery-serialization"> <use-character-maps> <character-map character=">" map-string=">"/> </use-character-maps> </serialization-parameters> It resides in the directory where the xqm is located. The xqm contains this function: module namespace my = 'my'; declare %rest:GET %rest:path("/charmap/test") %output:method("xhtml") %output:html-version("5.0") %output:parameter-document("html-charmap.xml") %rest:produces("text/html") function my:html() { <html> <head> <title>Charmap Test</title> <meta charset="UTF-8"/> <script> alert('You are in ' + (document.compatMode==='CSS1Compat'?'Standards':'Quirks') + ' mode and 1 > 0 is ' + (1 > 0)</script> </head> <body> <h1>Charmap Test</h1> <p>></p> <p>></p> </body> </html> }; When I retrieve /charmap/test, all > characters are still escaped as > When I change %output:parameter-document to %output:parameter-document("tml-charmap.xml"), BaseX doesn’t complain about the missing parameter document. What am I doing wrong? In XSLT 3 with Saxon, I can also do <xsl:output method="xhtml" indent="yes" include-content-type="no" encoding="UTF-8" parameter-document="html-charmap.xml" omit-xml-declaration="no" html-version="5.0" doctype-public="" doctype-system=""/> and this will use html-charmap.xml and it will complain when it’s the wrong file name. Gerrit

2 3

Something is off with db:enforceindex in 9.7.2.
by Omar Siam 20 Jun '22

20 Jun '22

Consider this XQuery: declare namespace _="urn:_"; declare function _:query-index___($dict as xs:string, $ind as xs:string, $str as xs:string) { let $rs := (# db:enforceindex #) { switch ($ind) case "any" return collection($dict)//index/w[text() contains text { $str } using wildcards] default return if ($str = '*') then collection($dict)//index[@id = $ind]/w else collection($dict)//index[@id = $ind]/w[text() contains text { $str } using wildcards] } return $rs }; _:query-index___('dc_arz_eng_publ__ind', 'any', 'beer.*') where dc_arz_eng_publ__ind contians a single XML which looks like (probably irrelevant): <indices> <index id="lem"> <w>-</w> <w>mafṛama</w> <w>without</w> </index> <index id="other"> </index> </indices> There is a java.lang.ArrayIndexOutOfBoundsException: Version: BaseX 9.7.2 Java: Azul Systems, Inc., 11.0.14.1 OS: Windows 10, amd64 Stack Trace: java.lang.ArrayIndexOutOfBoundsException: Index -1 out of bounds for length 8 at org.basex.query.var.QueryStack.get(QueryStack.java:99) at org.basex.query.QueryContext.get(QueryContext.java:553) at org.basex.query.var.VarRef.value(VarRef.java:46) at org.basex.query.expr.ParseExpr.atomValue(ParseExpr.java:61) at org.basex.query.expr.Expr.atomItem(Expr.java:116) at org.basex.query.func.fn.Docs.collection(Docs.java:36) at org.basex.query.func.fn.FnCollection.value(FnCollection.java:15) at org.basex.query.expr.index.IndexDynDb.data(IndexDynDb.java:86) at org.basex.query.expr.index.IndexDb.data(IndexDb.java:35) at org.basex.query.expr.ft.FTWords.iter(FTWords.java:153) at org.basex.query.expr.ft.FTIndexAccess.iter(FTIndexAccess.java:43) at org.basex.query.expr.ft.FTIndexAccess.value(FTIndexAccess.java:63) at org.basex.query.expr.path.CachedPath.nodes(CachedPath.java:43) at org.basex.query.expr.path.AxisPath.value(AxisPath.java:41) at org.basex.query.expr.Extension.value(Extension.java:59) at org.basex.query.expr.ParseExpr.iter(ParseExpr.java:46) at org.basex.query.expr.gflwor.For$1.next(For.java:112) at org.basex.query.expr.gflwor.Let$LetEval.next(Let.java:144) at org.basex.query.expr.gflwor.OrderBy$1.sort(OrderBy.java:73) at org.basex.query.expr.gflwor.OrderBy$1.next(OrderBy.java:54) at org.basex.query.expr.gflwor.GFLWOR$1.next(GFLWOR.java:68) at org.basex.query.QueryContext.next(QueryContext.java:359) at org.basex.query.expr.constr.Constr.add(Constr.java:73) at org.basex.query.expr.constr.CElem.item(CElem.java:149) at org.basex.query.expr.constr.CElem.item(CElem.java:1) at org.basex.query.expr.ItemMap.item(ItemMap.java:29) at org.basex.query.expr.ParseExpr.value(ParseExpr.java:51) at org.basex.query.expr.ParseExpr.iter(ParseExpr.java:46) at org.basex.query.scope.MainModule.iter(MainModule.java:60) at org.basex.query.QueryContext.iter(QueryContext.java:338) at org.basex.http.restxq.RestXqResponse.serialize(RestXqResponse.java:85) at org.basex.http.web.WebResponse.create(WebResponse.java:58) at org.basex.http.restxq.RestXqServlet.run(RestXqServlet.java:72) at org.basex.http.BaseXServlet.service(BaseXServlet.java:69) at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) at org.eclipse.jetty.servlet.ServletHolder$NotAsync.service(ServletHolder.java:1450) at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:550) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:600) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624) at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1440) at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:501) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594) at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1355) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) at org.eclipse.jetty.server.Server.handle(Server.java:516) at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:487) at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:732) at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:479) at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277) at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311) at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105) at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131) at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:409) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883) at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034) at java.base/java.lang.Thread.run(Thread.java:829) If I don't use (# db:enforceindex #) {} it works as expected Best regards -- Mag. Ing. Omar Siam Austrian Center for Digital Humanities and Cultural Heritage Österreichische Akademie der Wissenschaften | Austrian Academy of Sciences Stellvertretende Behindertenvertrauensperson | Deputy representative for disabled persons Wohllebengasse 12-14, 1040 Wien, Österreich | Vienna, Austria T: +43 1 51581-7295 omar.siam(a)oeaw.ac.at | www.oeaw.ac.at/acdh

2 2

unexpected behavior from prof:track()
by C. M. Sperberg-McQueen 20 Jun '22

20 Jun '22

Short version: When I call prof:track(...) with an expression as argument, I get the results I would expect. When I assign prof:track#1 to a variable and then call it with an expression by referring to that variable, the time shown is always 0. Is this the expected behavior? Details: Consider the following XQuery module: let $f := function($s as xs:string) { trace('hi'), prof:sleep(2000), trace($s) }, $pt := (function-lookup( QName('http://basex.org/modules/prof', 'track'), 1), function($item) { map {'time':'?', 'value': $item} } )[1] return (prof:track($f('world')), $pt($f('friends!')) ) It defines two variables: - $f, a function that produces output (so I can see that it ran) and calls prof:sleep (so it takes more than a couple of ms), and - $pt, which will be the prof:track() function if it exists (as it will in BaseX) or a substitute function which just evaluates the expression but does not provide any timing information. It then calls function $f twice, once wrapped in prof:track() and once wrapped in $pt(). The output follows: map { "memory": 0, "time": 2.00069e3, "value": ("hi", "world") } map { "memory": 0, "time": 0.0e0, "value": ("hi", "friends!") } As may be seen, the 'time' value in the map is 0. Conjecture: I suspect that this has to do with the fact that prof:track() is not a normal function -- in Lisp or Scheme, it would have to be implemented as a macro or 'special form', meaning that the arguments to the call are not evaluated before calling the function. I wonder whether what is happening here is that when the call to $pt($f('friends')) is handled, the argument is being evaluated (producing the sequence ('hi', 'friends!')) and then prof:track() is being called. Since it has no work to do, prof:track() then just returns the result, accurately showing a time cost of 0. Background: In a test harness I use to run tests for an XQuery program I am writing, I initially used prof:track() to keep track of how long each test takes to run. Recently, as part of an effort to ensure that the program is implementation-neutral, I rewrote the test harness so that it could run under other XQuery engines as well -- hence the indirection and the definition of $pt. I realize that I could get the BaseX version to work by saying something like let $prof-track := function-lookup(...) let $fall-back := function(...) if (exists($prof-track)) then prof:track(...) else $fall-back(...) but my recollection is that I tried this first and it provoked fatal errors in other engines, since prof:track() is not available in the static context in those engines. Is there any way to write a single module so that prof:track() is used (and produces useful information) if available, and the fallback function is used otherwise? I would really like to avoid generating different versions of the test harness for different XQuery engines. Many thanks for any thoughts or advice anyone on this list can provide. Michael Sperberg-McQueen -- C. M. Sperberg-McQueen Black Mesa Technologies LLC http://blackmesatech.com

2 1

Feature Preview: Caching Module
by Christian Grün 17 Jun '22

17 Jun '22

Hi there, BaseX 10 will come with a new Caching Module [1], which will allow you to store XQuery values (atomic items, nodes, sequences, maps, arrays, anything except function items) in a main-memory cache. Caches are persistent: Its contents will be written to disk at shutdown time and retrieved from disk in a new or restarted BaseX instance when accessed for the first time. The cache size is only limited by the available RAM. We started off with an XML format for representing caches in files, but we switched to a binary format to speed up the processing of large caches. It’s also possible to put large XML documents in the cache, but the classic database representation will give you better results in most cases. A first snapshot is available [2]. In addition, we’ll soon introduce new database functions, which will enable you to store XQuery values (…including maps) in a database. Have fun, Christian [1] https://docs.basex.org/wiki/Caching_Module [2] https://files.basex.org/releases/latest-10/

5 9

BaseX performance with Apple M1 Max
by Tim Thompson 15 Jun '22

15 Jun '22

Hi, all, I just had my five-year workstation upgrade for my work laptop. Since BaseX is my main productivity tool, I was curious to test performance on the new hardware. Old machine: 2016 MacBook Pro 2.6 GHz Quad-Core Intel Core i7 16 GB 2133 MHz LPDDR3 New machine: 2021 MacBook Pro Apple M1 Max 32 GB unified memory With BaseX 9.7.3 beta, I tested a 4.11 GB XML document that has a series of *entry *elements, each representing a name from our library catalog. On the old machine, it took 9 minutes to create a database from this file, with text and full-text indexes. On the new machine, it took 3.5 minutes. I ran the following query on both machines: <data>{ for tumbling window $entry in /*/entry start $s next $sn when starts-with($sn/key, $s/key) end $e next $en when not(starts-with($en/key, $e/key)) return <group id="{$entry/@id}">{$entry/key}</group> }</data> On the old machine, execution time ranged from 40s to 47s. On the new machine, it ranged from 11.5s to 18s. In short, it looks as though I'll be a lot more productive now! :) Tim -- Tim A. Thompson (he, him) Librarian for Applied Metadata Research Yale University Library

1 0

Term Frequency/inverse Document Frequency Calculation?
by Eliot Kimber 13 Jun '22

13 Jun '22

We’d like to report tf/idf for our DITA content set (https://en.wikipedia.org/wiki/Tf%E2%80%93idf) Of course this is possible using BaseX and basic full-text processing. My question: has anyone done this or is there somewhere I can look to at least get an idea of the level of effort? Having thought about it for not much time at all I’m thinking it’s an application of the basic “make an index over the words for each doc” technique that others have discussed recently. Thanks, E. _____________________________________________ Eliot Kimber Sr Staff Content Engineer O: 512 554 9368 M: 512 554 9368 servicenow.com<https://www.servicenow.com> LinkedIn<https://www.linkedin.com/company/servicenow> | Twitter<https://twitter.com/servicenow> | YouTube<https://www.youtube.com/user/servicenowinc> | Facebook<https://www.facebook.com/servicenow>

2 2

random initialization vector (IV) - "As the IV has to be passed along with the encrypted message somehow"
by Patrick Durusau 13 Jun '22

13 Jun '22

Greetings! The recent noise from MongoDB about encrypted querying made me look at the encryption documentation, here: https://docs.basex.org/wiki/Cryptographic_Module The signatures seem straight forward enough: crypto:encrypt|($data as xs:anyAtomicType, $type as xs:string, $key as xs:anyAtomicType, $algorithm as xs:string) as xs:base64Binary||| ||||*crypto:decrypt*($data as xs:anyAtomicType, $type as xs:string, $key as xs:anyAtomicType, $algorithm as xs:string) as xs:string|| ||OK, but where is the random initialization vector (IV)?|| ||The documentation says:|| ||"As the IV has to be passed along with the encrypted message somehow, data which has been encrypted by the |crypto:encrypt| function in BaseX can only be decrypted by calling the |crypto:decrypt| function."|| ||I suppose as a standards editor it's the "somehow" that I find troubling. Exactly what "somehow?" || ||Asking because I would want to invoke a recorded IV to change:|| ||"The result of an encryption using the same message, algorithm and key looks different each time it is executed."|| ||to:|| ||The result of an encryption using the same message, algorithm, key and *IV looks the same* each time it is executed.|| ||Such that I can search by delivering an encrypted string in an XQuery statement, to be match against the same encrypted string in BaseX, with the results of the query being returned and the result being decrypted upon delivery to the user.|| ||That is the data in BaseX and the query, are never visible during the query or response to the query.|| ||Does that make sense?|| ||It's not an immediate need but I thought I should ask before it is.|| ||Thanks for all you do in this wonderful project!|| ||Patrick || -- Patrick Durusau patrick(a)durusau.net Technical Advisory Board, OASIS (TAB) Editor, OpenDocument Format TC (OASIS), Project Editor ISO/IEC 26300 Co-Editor, ISO/IEC 13250-1, 13250-5 (Topic Maps) Another Word For It (blog): http://tm.durusau.net Homepage: http://www.durusau.net Twitter: patrickDurusau

2 1

Encoding hassle ....
by Marco Lettere 13 Jun '22

13 Jun '22

Dear all, after wrapping our heads around this for hours today, we don't know how to get rid of this inconsistency. Thus I ask for help ... SSCE: BaseX 9.6.4 [Standalone] Try 'help' to get more information. > xquery file:write-text("a1.txt", "°" || out:nl()) (: Same with codepoints-to-string(176) instead of "°" :) Query executed in 183.94 ms. > xquery file:read-text("a1.txt") ° Query executed in 1.49 ms. > xquery file:write-text("a2.txt", file:read-text("a1.txt")) Query executed in 3.4 ms. > xquery file:read-text("a2.txt") [file:io-error] Decoding error: xb0 Testing the files with linux command-line tool "file", this is the output: > file a1.txt a1.txt: Unicode text, UTF-8 text > file a2.txt a2.txt: ISO-8859 text Reading the file after "copying" it seems to change the encoding. How is this supposed to be handled? Regards, Marco.

3 5

Lock avoidance using inspect:functions?
by Andy Bunce 13 Jun '22

13 Jun '22

Following Omar's message on lock management [1] I thought I would share a technique that I have tried. It makes use of the function inspect:functions [2]. The code below shows two versions of an eval-update function. After running each version the info.result view reports: - local:eval-update => a global write lock. - local:eval-update2 => no locks at all! I am pretty sure this is a bug/oversight in inspect:functions. I suspect running it risks data integrity, so it is not a good idea. However, I do make use of a non-updating version of this technique, as a way to do string evaluation without triggering a global read lock. If I know that no lockable resources are referenced then it seems safe to me? Note: inspect:functions is similar to the unimplemented standard XQuery 3.1 function fn:load-xquery-module[3] /Andy [1] https://mailman.uni-konstanz.de/pipermail/basex-talk/2022-May/017207.html [2] https://docs.basex.org/wiki/Inspection_Module#inspect:functions [3] https://www.w3.org/TR/xpath-functions-31/#func-load-xquery-module (:~ evaluate XQuery string, string is enriched with external variable definitions for convenience :) declare %updating function local:eval-update($exp as xs:string,$binds as map(*)){ let $decl:= map:keys($binds)!``[declare variable $`{ . }` external;]``=>string-join("") return xquery:eval-update($decl || $exp, $binds) }; declare %updating function local:eval-update2($exp as xs:string,$binds as map(*)){ let $decl:= map:keys($binds)!``[let $`{ . }` :=$binds("`{ . }`") ]``=>string-join("") let $mod:=("module namespace _='abuse'; declare %updating function _:foo($binds){" , $decl, "return " , $exp , "};")=>string-join("") let $p:=file:create-temp-file("APB", ".xqm") let$_:=file:write-text($p,$mod) return updating inspect:functions($p)($binds) }; (: note: the bind functionality is not used in this example) let $binds:=map{"p1":42} let $exp:="db:replace('factbook', 'abuse.xml', '<a/>')" return local:eval-update2($exp,$binds)

2 2

Cannot file:write when path has chinese characters...
by France Baril 10 Jun '22

10 Jun '22

[file:invalid-path] Invalid file path: 'Temp/pdfs/ e5-轻奢版-80-global-en-us-dh9.fo <http://xn--e5--80-global-en-us-dh9-3195b4226a2w0g.fo> Any chance this limit can be overcome? -- France Baril Architecte documentaire / Documentation architect france.baril(a)architextus.com

4 8

java.lang.ArrayIndexOutOfBoundsException: 559483324
by Sebastian Guerrero 08 Jun '22

08 Jun '22

Hi BaseX team, Is this a bug or an improper use? *Unexpected error: Improper use? Potential bug? Your feedback is welcome:* *Contact: basex-talk(a)mailman.uni-konstanz.de <basex-talk(a)mailman.uni-konstanz.de>* *Version: BaseX 9.7.2* *Java: Oracle Corporation, 1.8.0_281* *OS: Windows 10, amd64* *Stack Trace: * *java.lang.ArrayIndexOutOfBoundsException: 559483324* * at org.basex.io.random.TableDiskAccess.read4(TableDiskAccess.java:174)* * at org.basex.data.Data.id <http://org.basex.data.Data.id>(Data.java:302)* * at org.basex.data.Data.pre(Data.java:286)* * at org.basex.query.func.db.DbOpenId.pre(DbOpenId.java:47)* * at org.basex.query.func.db.DbOpenId.value(DbOpenId.java:28)* * at org.basex.query.func.StaticFuncCall.evalArgs(StaticFuncCall.java:147)* * at org.basex.query.func.FuncCall.value(FuncCall.java:53)* * at org.basex.query.expr.ParseExpr.iter(ParseExpr.java:46)* * at org.basex.query.expr.Switch.iter(Switch.java:198)* * at org.basex.query.expr.IterMap$1.next(IterMap.java:72)* * at org.basex.query.QueryContext.next(QueryContext.java:359)* * at org.basex.query.expr.gflwor.GFLWOR$1.next(GFLWOR.java:66)* * at org.basex.query.QueryContext.next(QueryContext.java:359)* * at org.basex.query.expr.constr.Constr.add(Constr.java:73)* * at org.basex.query.expr.constr.CElem.item(CElem.java:149)* * at org.basex.query.expr.constr.CElem.item(CElem.java:1)* * at org.basex.query.expr.ParseExpr.value(ParseExpr.java:51)* * at org.basex.query.expr.ParseExpr.iter(ParseExpr.java:46)* * at org.basex.query.expr.constr.Constr.add(Constr.java:72)* * at org.basex.query.expr.constr.CElem.item(CElem.java:149)* * at org.basex.query.expr.constr.CElem.item(CElem.java:1)* * at org.basex.query.expr.ItemMap.item(ItemMap.java:37)* * at org.basex.query.expr.ParseExpr.value(ParseExpr.java:51)* * at org.basex.query.expr.ParseExpr.iter(ParseExpr.java:46)* * at org.basex.query.expr.gflwor.GFLWOR$1.next(GFLWOR.java:72)* * at org.basex.query.scope.MainModule$1.next(MainModule.java:67)* * at org.basex.http.restxq.RestXqResponse.serialize(RestXqResponse.java:87)* * at org.basex.http.web.WebResponse.create(WebResponse.java:58)* * at org.basex.http.restxq.RestXqServlet.run(RestXqServlet.java:72)* * at org.basex.http.BaseXServlet.service(BaseXServlet.java:69)* * at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)* * at org.eclipse.jetty.servlet.ServletHolder$NotAsync.service(ServletHolder.java:1450)* * at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799)* * at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:550)* * at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)* * at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:600)* * at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)* * at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)* * at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624)* * at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)* * at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1440)* * at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)* * at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:501)* * at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594)* * at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)* * at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1355)* * at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)* * at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)* * at org.eclipse.jetty.server.Server.handle(Server.java:516)* * at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:487)* * at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:732)* * at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:479)* * at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277)* * at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)* * at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)* * at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)* * at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338)* * at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315)* * at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173)* * at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)* * at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:409)* * at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)* * at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)* * at java.lang.Thread.run(Unknown Source)* I'm getting this error with a call to a complex ( more or less ) function. The more results it needs to handle, the more errors I get. Any help is welcome!

1 1

Faceted searching and browsing ?
by Majewski, Steven Dennis (sdm7g) 06 Jun '22

06 Jun '22

Is it possible to do faceted browsing with BaseX ? I think the answer is no, however I may have been initially fooled into trying because my first test case was far simpler than my other use cases. The simple case was getting counts of publisher from an EAD collection and sorting by greatest number. Here’s my working code: declare function local:normalize( $s ) { translate( replace( lower-case($s), '^(us-)+', '' ), '-', '' ) }; declare variable $orgs := doc('ead-inst/ead-inst.xml'); declare variable $orgcodes := collection('published')/ead/eadheader/eadid/@mainagencycode ! local:normalize(.) => distinct-values() ; declare function local:countpubfacets( $c ) { for $x in ( for $ead in $c let $ORG := local:normalize($ead/*:ead/*:eadheader/*:eadid/@mainagencycode) group by $ORG order by $ORG let $inst := if ($ORG != "") then ($orgs/list/inst[@prefix=$ORG],$orgs/list/inst[lower-case(@oclc) = $ORG] ) return array{ count($ead), $ORG, $inst/@orgcode/string(), $inst/string() } ) order by $x(1) descending return $x }; local:countpubfacets( collection('published')) In this case: (1) The number of unique @mainagencycode’s are less than 100, and (2) There is only one location for those codes. and performance is acceptable (or at least it seems to be in my tests). My other unsuccessful attempts have been with trying to rank //subject or //persname ’s. In this case, there are many thousands of unique subjects and names, and the subject and persname elements can occur in multiple locations in the file. Attempts to search similar to the above method ( as well as a couple of other variations I’ve tried ), even on a smaller subset of categories take entirely too much time — often I have to kill the search before it manages to complete. I have tried looking at index:facets() https://docs.basex.org/wiki/Index_Module#index:facets <https://docs.basex.org/wiki/Index_Module#index:facets> Which has only reinforced my notion that it’s not possible. So for now, I’m resigned to deferring that functionality, and exploring building a specialized index along side the BaseX indexes - either using Solr and querying the Solr index from BaseX, or else building some other index structure DB in BaseX along side my document DB. Eager to hear any tips or feedback on this problem or alternate solutions, and also general info about BaseX index structure and what useful info can be caught by introspection by those index module functions. Aside from the faceting, search by //subject (or other fields) is quite acceptable performance, even chaining several filters together with => declare function eadsearch:findBySubj( $ctx, $subj as xs:string?, $opt ) { if ( $subj ) then $ctx/*[ft:contains( .//subject, ft:tokenize($subj), $opt )] else $ctx }; — Steve M.

4 4

XML Catalog and xslt:transform()
by Zimmel, Daniel 02 Jun '22

02 Jun '22

Hi, after reading https://docs.basex.org/wiki/Catalog_Resolver and digging in the list archives (https://mailman.uni-konstanz.de/pipermail/basex-talk/2019-March/014199.html ) I still have trouble understanding catalog files. Is this supposed to work with xslt:transform() and BaseX GUI 9.7.2? The default option (DTD = false) is ignored by xslt:transform() because the function is definitely requesting the external DTD. This prevents transforming XML with DTD declarations that are not available (if I understand correctly, a problem that the DTD option is trying to solve in general). When I try to solve this via catalog files (actually I do not need the DTD), I do not have success. Here are my mini examples: Saxon HE 10.3 resides in the lib folder .basex setting: # Local Options SERIALIZER = indent=no DTD = true XML in local folder "C:/temp/catalog": <!DOCTYPE dokument SYSTEM "http://www.blahblahblah.info/dtd/dokument.dtd"> <dokument> <doknr>01</doknr> </dokument> catalog.xml in local folder "C:/temp/catalog": <catalog prefer="system" xmlns="urn:oasis:names:tc:entity:xmlns:xml:catalog"> <rewriteSystem systemIdStartString="http://www.blahblahblah.info/dtd/" rewritePrefix="file:///C:/temp/catalog/dtd/"/> </catalog> dokument.dtd in local folder "C:/temp/catalog/dtd": <!ELEMENT dokument (doknr)> <!ELEMENT doknr (#PCDATA)> XQuery query.xq in local folder "C:/temp/catalog": (# db:catfile catalog.xml #) { xslt:transform('dokument.xml', 'transform.xsl') } With or without pragma, this always results in a java.net.UnknownHostException (because the system ID is not available, that's true), but I would be expecting this would resolve to "file:///C:/temp/catalog/dtd/dokument.dtd" Not working in GUI nor via CCL. What am I getting wrong? Thanks, Daniel

6 8

Constructing and querying map structures in a dynamic way
by Markus Elfring 27 May '22

27 May '22

Hello, I would like to construct a map structure based on a dynamic selection by the means of a FLWOR expression. https://www.w3.org/TR/2017/REC-xquery-31-20170321/#id-maps The keys and associated values should be presented as a CSV-like file by a subsequent data processing step. Can it become easier to find existing XQuery script examples for such an use case? Regards, Markus

6 12

Executing optimized XQuery in RestXQ without having to deal with global lock situations
by Omar Siam 25 May '22

25 May '22

What I like about BaseX is that it is very good at optimizing self-contained queries about the size a user can read and understand [1] [2] and that it has a DB locking system for transaction management [3] that is robust and easy to understand. What I don’t like so much about BaseX is that these two mechanisms don’t work very well with complex code that is split into various modules. I use modules for code that may be shared among projects or just as a means of grouping common concerns in one module. That I don’t like this behaviour does not mean I know (or have any hope) that this can be solved in a better way without at least make unpleasant sacrifices elsewhere. It is just the setting I have to deal with. When BaseX cannot determine anymore which DBs are used in a query and which are not, it falls back to assuming there are no indexes, so automatic optimization in this regard is stopped, and it assumes that just all DBs known to BaseX are used in that query so it acquires a global lock. [4] When doing only reading queries this is not much of a problem. Using indexes in queries can be forced with functions or with the db:enforceindex pragma [5]. Problems start showing when trying to implement a CRUD RestXQ application. Create, update and delete can be implemented using the XQuery update standard but of course now this will get slow and cumbersome when for many read operations it cannot be determined which DBs they use and so a global read lock is held. That of course means that no global write lock can be acquired until all read operations are finished on all DBs known to a BaseX instance. This is especially problematic when one instance of BaseX with a RestXQ application is used to serve data from independent databases. Say one instance of BaseX has a RestXQ API that servers a lot of different dictionaries for different natural languages. This is my use case. Although the content of dictionary entries is different, the parts in the TEI/XML I try to manipulate, that are created, read, updated or deleted, are the same. So, a common API should handle many independent dictionaries, edited by many users, using one instance of BaseX. Also, when working with my biggest XML database of several GB I ran into problems when reindexing after an update. Reindexing all those GB of data takes too long and makes small updates in there impossible. Why not multiple instances of BaseX? Well because for better or worse BaseX runs in a JVM and even after I tried to minimize the memory footprint of an idle BaseX it is still a little less than 300 MB and we run a lot of services here on shared servers so RAM usage matters. Also, RAM usage is a part of the costs when using commercial cloud services. But of course, not running BaseX at all if not used is best if you pay per minute. And also: as recently discussed on the list: BaseX as any Java program gets optimized while running by the JVM and then those optimizations as well as caching will benefit all the data hosted in one instance but would be less efficient with multiple instances I assume. So how do I achieve four goals: * Keep the XQuery short and concise because that is what the optimizer can handle best? * Keep the code separated into Modules that deal with one particular aspect? * Use RestXQ and not another technique to actually implement the RESTful API? * All this while being able to split GB of XML data into portions that can be reindexed in a reasonable amount of time? The two thing that help here a lot is eval functions like xquery:eval [6] and String Constructors [7]. Say, I want to run a query but on different collections (databases). I can do this by having a list of collections and executing the actual query in a for loop with the concrete collection as a variable. If I just write the XQuery code down like this the problem is that the optimizer would need to evaluate the query to find out which databases to lock and what indexes can be used. BaseX is not built to do this (yet). It does not mock run the query. So, it decides that a global lock needs to be used. Depending on the use of XQuery Update either a global write lock or a global read lock is acquired. Easy to understand but does not help with performance here. If I want to make the situation worse for the optimizer I can use xquery:eval. That of course makes the XQuery code totally opaque to the optimizer. A global lock is guaranteed. Still another eval function is a solution here. There is the jobs module jobs:eval [8]. If I break up my code into jobs only these jobs hold locks for as long as they run. This can be a much shorter period of time than what it takes to run a whole RestXQ request. It is also possible to find a place that needs to be changed in a number of databases and then only write lock one of them to change something. So, if my data is stored in not one but several database files I can make them look like one big XML for API purposes, but still have small enough independent parts that can be indexed separately so updates with reindexing are relatively fast. If I have a search I want to perform on parts of databases that are in principle independent, like dictionary entries in a large dictionary, I can do this in parallel on each database. I tried to implement this idea with jobs:eval and it actually worked very well. Only the interface of the function was cumbersome to use the way I wanted to make use of this functionality. So, I wrote a wrapper around jobs:eval and jobs:wait that makes it easy to generate small self-contained XQuery code [9] using String Constructors [10] and some other functions used for querying the structure of the data stored in BaseX like listing and filtering databases by name [11]. Another other goal for this util:eval(s) function was to make it still easy to see errors [12]. A typical use is something like: run a filter query in all databases [13] that are found using a database name filter in some settings database [14] and use a string for comparison from a request URL parameter [15]. Find an entry out of a few million and replace it with an updated version. Of course, with reindexing [16]. What were some (unexpected) problems? Because now jobs and especially write jobs lock databases while the RestXQ code is running the RestXQ code itself cannot hold any read or write lock. That is possible in BaseX but some functions force a global lock. For example db:list. I think there are good reasons why you want to have a global lock and therefore atomicity during a query when you ask for the list of databases. Of course, my code happens to need to list databases quite often. And my code should not hold a global lock here after getting the list. My list of databases may change during a RestXQ call but I don’t care yet about that situation. I think it does not matter to me. There is also a now simple solution: Outsource db:list to its own job [17]. I also remember there was a problem with an automatic conversion of RestXQ parameters creating a randomly named lock. But it was no problem to do the conversion explicitly in XQuery code and so have the RestXQ code not hold any lock again. Now there already was a question on the mailing list about BaseX behaviour in a multithreaded environment. I don’t use that BaseX.jar in such a way with my own Java code but jobs are (Java) threads. And the interesting thing here is now that with a lot of threads (say 700) that don’t lock each other, a bottleneck shows in the way BaseX handles file access. At least the Java profiler showed me this as a primary source of wasted time [18]. If I get it correctly then file access is as usual done in 4KB portions which are read into a buffer and smaller parts are accessed from there. This way is by far the most efficient way to do this on any current operating system and file system. But now this buffer’s handling needs protection from the buffer being manipulated in different threads. All I found in the JDK for this is a performance nightmare was the Jave nio streams systems [19], which tries to guarantee quite a few threads related consistencies [20] and seems really slow. This seems to be a well-known fact documented numerous times on the internet [21]. I also tried with one of the tests BaseX contains [22] and an attempt to use FileChannel instead of the current RandomAccessFile base implementation and found the documented behaviour: Java nio file classes are no replacement for the current implementation when it comes to performance. Looking at other databases I saw they implement something OS dependent but it is hard to compare [23]. [1] https://docs.basex.org/wiki/Indexes [2] https://docs.basex.org/wiki/XQuery_Optimizations [3] https://docs.basex.org/wiki/Transaction_Management [4] https://docs.basex.org/wiki/Transaction_Management#Limitations [5] https://docs.basex.org/wiki/Indexes#Enforce_Rewritings [6] https://docs.basex.org/wiki/XQuery_Module#xquery:eval [7] https://www.w3.org/TR/2017/REC-xquery-31-20170321/#id-string-constructors [8] https://docs.basex.org/wiki/Jobs_Module#jobs:eval [9] https://github.com/acdh-oeaw/vleserver_basex/blob/main/vleserver/util.xqm#L… [10] https://github.com/acdh-oeaw/vleserver_basex/blob/main/vleserver/data/acces… [11] https://github.com/acdh-oeaw/vleserver_basex/blob/main/vleserver/data/profi… [12] https://github.com/acdh-oeaw/vleserver_basex/blob/main/vleserver/util.xqm#L… [13] https://github.com/acdh-oeaw/vleserver_basex/blob/main/vleserver/data/acces… [14] https://github.com/acdh-oeaw/vleserver_basex/blob/main/vleserver/data/acces… [15] https://github.com/acdh-oeaw/vleserver_basex/blob/main/vleserver/entries.xq… [16] https://github.com/acdh-oeaw/vleserver_basex/blob/main/vleserver/data/acces… [17] https://github.com/acdh-oeaw/vleserver_basex/blob/main/vleserver/dicts.xqm#… [18] https://github.com/BaseXdb/basex/blob/master/basex-core/src/main/java/org/b… and other read methods there [19] https://blogs.oracle.com/javamagazine/post/java-nio-nio2-buffers-channels-a… [20] https://docs.oracle.com/javase/7/docs/api/java/nio/channels/FileChannel.html The view of a file provided by an instance of this class is guaranteed to be consistent with other views of the same file provided by other instances in the same program. [21] https://www.mathematik.uni-marburg.de/~alexmaurer/files/NioVsIo.pdf as an example. May be more recent evaluations of Java 11 or 17 nio or nio.2 performance is better? [22] https://github.com/BaseXdb/basex/blob/master/basex-core/src/test/java/org/b… [23] https://github.com/neo4j/neo4j/blob/4.4/community/native/src/main/java/org/… Best regards -- Mag. Ing. Omar Siam Austrian Center for Digital Humanities and Cultural Heritage Österreichische Akademie der Wissenschaften | Austrian Academy of Sciences Stellvertretende Behindertenvertrauensperson | Deputy representative for disabled persons Wohllebengasse 12-14, 1040 Wien, Österreich | Vienna, Austria T: +43 1 51581-7295 omar.siam(a)oeaw.ac.at | www.oeaw.ac.at/acdh

2 2

Date output in French ?
by Maud Ingarao 24 May '22

24 May '22

Hi everybody ! We need to convert dates in "human" language and therefore we use the following : *let $date := 1723-11-25T12:00:00 **fn:format-dateTime(xs:dateTime($date), '[D] [Mn] [Y]', 'fr', (), ())* But we get :*[Language: en]25 november 1723* The date remains in english with *[Language: en] *before, whereas *fn:format-dateTime(xs:dateTime($date), '[D] [Mn] [Y]', 'de', (), ()) *works and we get the date in German. We checked *java:java.util.Locale* amongst which we find "fr" :*"nn, ar, bg, ke, an, ds, zu, am, _fr_, ti, bo..."* And we have downloaded the *lucene-snowball-2.9.4.jar* file in lib/ Thank you so much for your help ! Maud & Nathalie :-)

2 1

Adding another document from XQuery evaluation
by Markus Elfring 21 May '22

21 May '22

Hello, The software “BaseX” is providing direct support for file formats like “XML” and “CSV”. Corresponding data can be generated by customised XQuery scripts. I got the impression that additional documents can be integrated into selected databases so far by specifying file names. Does this mean that data (which would be determined by a script evaluation) need to be exported into a file for a subsequent file import? I imagine that another function combination for the available modules can make such an operation more convenient. Regards, Markus

9 21

BaseX, RESTXQ, and React?
by Eliot Kimber 20 May '22

20 May '22

I’m starting to learn React in the context of enhancing a React-based site created by others (lightstep.com/incident-reponse/docs). I’m also developing a non-trivial web application in BaseX (the Project Mirabel system I’ve mentioned before). For Mirabel I just hacked some pages using RESTXQ and the least CSS and JavaScript I could get away with, cribbing liberally from the out-of-the-box BaseX web apps. But as I learn about React and its companion, Material UI, I’m thinking that there’s probably a lot of power to combining React with RESTXQ as a way to build a top-notch site with lots of interactivity at relatively low cost (React and MUI provide all the interaction implementation details you might want). But I’m not sure what this would mean in practice. I’m wondering if anyone else has done this or at least thought about it enough to have some ideas about what to do or not do? In addition, at the most recent DITA conference, Convex 2022, George Bina presented a very interesting experiment in generating React components for DITA topics where the components did the dynamic content resolution at delivery time rather than at HTML building time, enabling practical incremental update of DITA-based content through a React-based site. The same technique could be applied to RESTXQ by generating XQuery modules for individual topics. But an advantage of a React approach is that it’s more general and could be used with any React-based site. But in any case it’s gotten me thinking more about the generation of information I see React being used mostly in statically-generated sites that are published through something like Netlify but my Mirabel application is very much almost entirely dynamic in that most of the content is the result of queries. One could treat BaseX as a supplier of microservices via a REST API and that’s probably how most users of BaseX with React do it, if for no other reason than separate groups are probably responsible for the web stuff and the BaseX stuff. But in the case of my Mirabel application it’s a single team doing everything with limited web development knowledge but access to talented web designers. So having all the web stuff also in BaseX makes development and maintenance easier with the team we have. We’re not JavaScript programmers but we can manage composing React components together and doing the minimal programming needed to make them behave appropriately. Clearly I can just have RESTXQ-generated pages import the React libraries and supporting components directly and I can generate JSX code as inline scripting and maybe that’s the best approach for an all-RESTXQ site but maybe there’s something better? Or does stepping up to a proper React site that uses BaseX as a source of microservices make more sense, even for this small project? Thanks, Eliot _____________________________________________ Eliot Kimber Sr Staff Content Engineer O: 512 554 9368 M: 512 554 9368 servicenow.com<https://www.servicenow.com> LinkedIn<https://www.linkedin.com/company/servicenow> | Twitter<https://twitter.com/servicenow> | YouTube<https://www.youtube.com/user/servicenowinc> | Facebook<https://www.facebook.com/servicenow>

4 4

ODBC support for BaseX?
by Markus Elfring 20 May '22

20 May '22

Hello, The software “BaseX” is providing direct support for document databases. Is any support available for data management by interfaces like the following? * Open Database Connectivity * Java Database Connectivity How are corresponding drivers evolving? Regards, Markus

5 6

Selecting nodes in a db by source file name
by Charles Bearden 19 May '22

19 May '22

Let's say I have a database built from 40+ XML documents, each with about 4 million nodes for a total of about 404 million nodes, of which 30k are high-level records of interest. Let's say that a given node of interest (a PubMed record, actually) may occur once in this database for each time it has been updated, and I can identify the most recent version of that record by the PubMed ID and the filename (an ascending sort of the filenames sorts them from least recent to most recent). Is there a way in XQuery I can identify a record in this database not only by its PubMed ID but also limit the result set by source document name? I know that `db:path($elem)` gives me the document name for an element, but I want to use the document name to help pick out the element from among several that may share its PMID. Context: I have 26 such databases, each of which contains 43-44 source files, each of which contains 30000 records. I have built one index database that, for each record, includes PubMed ID, source filename, and database name. For a given PMID I can very quickly retrieve the db name and the file name, but I need to search the db not only by PMID but also by source filename in order to select the most recent version of a record. Thank you & all the best, Chuck Bearden

5 5

Checking run time characteristics according to the handling of customised indexes
by Markus Elfring 17 May '22

17 May '22

Hello, Some indexes are generally supported also by the software “BaseX”. https://docs.basex.org/wiki/Indexes Some data processing approaches would usually depend on indexes for primary and foreign keys. https://en.wikipedia.org/wiki/Primary_key How much do run time characteristics influence data transfers (and conversion of corresponding data structures) for the available indexing technologies? Regards, Markus

1 0

Repeating data based on varying items in an array
by Markus Elfring 16 May '22

16 May '22

Hello, It seems that I am still not familiar enough with the programming language “XQuery” for the quick specification of a bit of code for a general data processing task. Thus I hope that the following use case can be clarified better. A sequence like “(["f" [1 2 3]], ["g" [4, 5]])” might be an interesting test example. It contains two nested arrays. * One part refers to fixed data. * Another part refers to some items with a varying number. I would like to convert the varying items into a data representation like the following. f,1 f,2 f,3 g,4 g,5 Can any customised functions help for the desired transformation? Regards, Markus

3 11

Add a comment to a backup?
by Jonathan Robie 13 May '22

13 May '22

I have been making backups before doing particularly complex things to my treebanks, and I find myself writing down information about what stage of processing a given backup corresponds to. "after replacing subtrees for missing compounds" I wish I could associate these strings with backups in BaseX so I can more easily know which one I would restore if something went wrong. Jonathan

4 8

Developments for address books
by Markus Elfring 12 May '22

12 May '22

Hello, Various data models and corresponding applications can be constructed also with the help of the software “BaseX”. One popular data management tool is the address book (which can be supported by customised hardware and software), isn't it? Would you like to point any development repositories and further information sources out for such an application domain? Regards, Markus

1 0

Oxygen and BaseX Integrations?
by Eliot Kimber 06 May '22

06 May '22

I’m looking at ways to better integrate BaseX with Oxygen, both for XQuery editing and as a way to augment Oxygen’s DITA authoring features (i.e., using BaseX to provide more comprehensive link and search database features). Radu pointed me to Project Argon https://github.com/axxepta/project-argon but it looks like it hasn’t been updated in a few years (last commit was 2017) and my initial attempt to run the gradle build failed on the Java version detection. I’ll reach to the Project Argon team but I’m wondering if anyone else is doing anything with Oxygen integration. I’m looking for a couple of things: * Enable editing XQuery modules in Oxygen with validation, either using Saxon or BaseX, whichever is easier. While the BaseX editor is decent I really miss the Oxygen features I’ve come to know and love, like outline view. My initial attempt to set up an entity resolution catalog to make this work did not succeed, but I didn’t have a lot of time to spend on it. * Embedded BaseX server in an Oxygen add-on. This shouldn’t be too hard and we have a ServiceNow-specific add-on I could do this in, but I’m always looking for solutions I can crib from… Thanks, Eliot _____________________________________________ Eliot Kimber Sr Staff Content Engineer O: 512 554 9368 M: 512 554 9368 servicenow.com<https://www.servicenow.com> LinkedIn<https://www.linkedin.com/company/servicenow> | Twitter<https://twitter.com/servicenow> | YouTube<https://www.youtube.com/user/servicenowinc> | Facebook<https://www.facebook.com/servicenow>

1 0

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

BaseX-Talk