BaseX-Talk

basex-talk@mailman.uni-konstanz.de

4 participants
5268 discussions

Feature Preview: Caching Module
by Christian Grün 17 Jun '22

17 Jun '22

Hi there, BaseX 10 will come with a new Caching Module [1], which will allow you to store XQuery values (atomic items, nodes, sequences, maps, arrays, anything except function items) in a main-memory cache. Caches are persistent: Its contents will be written to disk at shutdown time and retrieved from disk in a new or restarted BaseX instance when accessed for the first time. The cache size is only limited by the available RAM. We started off with an XML format for representing caches in files, but we switched to a binary format to speed up the processing of large caches. It’s also possible to put large XML documents in the cache, but the classic database representation will give you better results in most cases. A first snapshot is available [2]. In addition, we’ll soon introduce new database functions, which will enable you to store XQuery values (…including maps) in a database. Have fun, Christian [1] https://docs.basex.org/wiki/Caching_Module [2] https://files.basex.org/releases/latest-10/

5 9

BaseX performance with Apple M1 Max
by Tim Thompson 15 Jun '22

15 Jun '22

Hi, all, I just had my five-year workstation upgrade for my work laptop. Since BaseX is my main productivity tool, I was curious to test performance on the new hardware. Old machine: 2016 MacBook Pro 2.6 GHz Quad-Core Intel Core i7 16 GB 2133 MHz LPDDR3 New machine: 2021 MacBook Pro Apple M1 Max 32 GB unified memory With BaseX 9.7.3 beta, I tested a 4.11 GB XML document that has a series of *entry *elements, each representing a name from our library catalog. On the old machine, it took 9 minutes to create a database from this file, with text and full-text indexes. On the new machine, it took 3.5 minutes. I ran the following query on both machines: <data>{ for tumbling window $entry in /*/entry start $s next $sn when starts-with($sn/key, $s/key) end $e next $en when not(starts-with($en/key, $e/key)) return <group id="{$entry/@id}">{$entry/key}</group> }</data> On the old machine, execution time ranged from 40s to 47s. On the new machine, it ranged from 11.5s to 18s. In short, it looks as though I'll be a lot more productive now! :) Tim -- Tim A. Thompson (he, him) Librarian for Applied Metadata Research Yale University Library

1 0

Term Frequency/inverse Document Frequency Calculation?
by Eliot Kimber 13 Jun '22

13 Jun '22

We’d like to report tf/idf for our DITA content set (https://en.wikipedia.org/wiki/Tf%E2%80%93idf) Of course this is possible using BaseX and basic full-text processing. My question: has anyone done this or is there somewhere I can look to at least get an idea of the level of effort? Having thought about it for not much time at all I’m thinking it’s an application of the basic “make an index over the words for each doc” technique that others have discussed recently. Thanks, E. _____________________________________________ Eliot Kimber Sr Staff Content Engineer O: 512 554 9368 M: 512 554 9368 servicenow.com<https://www.servicenow.com> LinkedIn<https://www.linkedin.com/company/servicenow> | Twitter<https://twitter.com/servicenow> | YouTube<https://www.youtube.com/user/servicenowinc> | Facebook<https://www.facebook.com/servicenow>

2 2

random initialization vector (IV) - "As the IV has to be passed along with the encrypted message somehow"
by Patrick Durusau 13 Jun '22

13 Jun '22

Greetings! The recent noise from MongoDB about encrypted querying made me look at the encryption documentation, here: https://docs.basex.org/wiki/Cryptographic_Module The signatures seem straight forward enough: crypto:encrypt|($data as xs:anyAtomicType, $type as xs:string, $key as xs:anyAtomicType, $algorithm as xs:string) as xs:base64Binary||| ||||*crypto:decrypt*($data as xs:anyAtomicType, $type as xs:string, $key as xs:anyAtomicType, $algorithm as xs:string) as xs:string|| ||OK, but where is the random initialization vector (IV)?|| ||The documentation says:|| ||"As the IV has to be passed along with the encrypted message somehow, data which has been encrypted by the |crypto:encrypt| function in BaseX can only be decrypted by calling the |crypto:decrypt| function."|| ||I suppose as a standards editor it's the "somehow" that I find troubling. Exactly what "somehow?" || ||Asking because I would want to invoke a recorded IV to change:|| ||"The result of an encryption using the same message, algorithm and key looks different each time it is executed."|| ||to:|| ||The result of an encryption using the same message, algorithm, key and *IV looks the same* each time it is executed.|| ||Such that I can search by delivering an encrypted string in an XQuery statement, to be match against the same encrypted string in BaseX, with the results of the query being returned and the result being decrypted upon delivery to the user.|| ||That is the data in BaseX and the query, are never visible during the query or response to the query.|| ||Does that make sense?|| ||It's not an immediate need but I thought I should ask before it is.|| ||Thanks for all you do in this wonderful project!|| ||Patrick || -- Patrick Durusau patrick(a)durusau.net Technical Advisory Board, OASIS (TAB) Editor, OpenDocument Format TC (OASIS), Project Editor ISO/IEC 26300 Co-Editor, ISO/IEC 13250-1, 13250-5 (Topic Maps) Another Word For It (blog): http://tm.durusau.net Homepage: http://www.durusau.net Twitter: patrickDurusau

2 1

Encoding hassle ....
by Marco Lettere 13 Jun '22

13 Jun '22

Dear all, after wrapping our heads around this for hours today, we don't know how to get rid of this inconsistency. Thus I ask for help ... SSCE: BaseX 9.6.4 [Standalone] Try 'help' to get more information. > xquery file:write-text("a1.txt", "°" || out:nl()) (: Same with codepoints-to-string(176) instead of "°" :) Query executed in 183.94 ms. > xquery file:read-text("a1.txt") ° Query executed in 1.49 ms. > xquery file:write-text("a2.txt", file:read-text("a1.txt")) Query executed in 3.4 ms. > xquery file:read-text("a2.txt") [file:io-error] Decoding error: xb0 Testing the files with linux command-line tool "file", this is the output: > file a1.txt a1.txt: Unicode text, UTF-8 text > file a2.txt a2.txt: ISO-8859 text Reading the file after "copying" it seems to change the encoding. How is this supposed to be handled? Regards, Marco.

3 5

Lock avoidance using inspect:functions?
by Andy Bunce 13 Jun '22

13 Jun '22

Following Omar's message on lock management [1] I thought I would share a technique that I have tried. It makes use of the function inspect:functions [2]. The code below shows two versions of an eval-update function. After running each version the info.result view reports: - local:eval-update => a global write lock. - local:eval-update2 => no locks at all! I am pretty sure this is a bug/oversight in inspect:functions. I suspect running it risks data integrity, so it is not a good idea. However, I do make use of a non-updating version of this technique, as a way to do string evaluation without triggering a global read lock. If I know that no lockable resources are referenced then it seems safe to me? Note: inspect:functions is similar to the unimplemented standard XQuery 3.1 function fn:load-xquery-module[3] /Andy [1] https://mailman.uni-konstanz.de/pipermail/basex-talk/2022-May/017207.html [2] https://docs.basex.org/wiki/Inspection_Module#inspect:functions [3] https://www.w3.org/TR/xpath-functions-31/#func-load-xquery-module (:~ evaluate XQuery string, string is enriched with external variable definitions for convenience :) declare %updating function local:eval-update($exp as xs:string,$binds as map(*)){ let $decl:= map:keys($binds)!``[declare variable $`{ . }` external;]``=>string-join("") return xquery:eval-update($decl || $exp, $binds) }; declare %updating function local:eval-update2($exp as xs:string,$binds as map(*)){ let $decl:= map:keys($binds)!``[let $`{ . }` :=$binds("`{ . }`") ]``=>string-join("") let $mod:=("module namespace _='abuse'; declare %updating function _:foo($binds){" , $decl, "return " , $exp , "};")=>string-join("") let $p:=file:create-temp-file("APB", ".xqm") let$_:=file:write-text($p,$mod) return updating inspect:functions($p)($binds) }; (: note: the bind functionality is not used in this example) let $binds:=map{"p1":42} let $exp:="db:replace('factbook', 'abuse.xml', '<a/>')" return local:eval-update2($exp,$binds)

2 2

Cannot file:write when path has chinese characters...
by France Baril 10 Jun '22

10 Jun '22

[file:invalid-path] Invalid file path: 'Temp/pdfs/ e5-轻奢版-80-global-en-us-dh9.fo <http://xn--e5--80-global-en-us-dh9-3195b4226a2w0g.fo> Any chance this limit can be overcome? -- France Baril Architecte documentaire / Documentation architect france.baril(a)architextus.com

4 8

java.lang.ArrayIndexOutOfBoundsException: 559483324
by Sebastian Guerrero 08 Jun '22

08 Jun '22

Hi BaseX team, Is this a bug or an improper use? *Unexpected error: Improper use? Potential bug? Your feedback is welcome:* *Contact: basex-talk(a)mailman.uni-konstanz.de <basex-talk(a)mailman.uni-konstanz.de>* *Version: BaseX 9.7.2* *Java: Oracle Corporation, 1.8.0_281* *OS: Windows 10, amd64* *Stack Trace: * *java.lang.ArrayIndexOutOfBoundsException: 559483324* * at org.basex.io.random.TableDiskAccess.read4(TableDiskAccess.java:174)* * at org.basex.data.Data.id <http://org.basex.data.Data.id>(Data.java:302)* * at org.basex.data.Data.pre(Data.java:286)* * at org.basex.query.func.db.DbOpenId.pre(DbOpenId.java:47)* * at org.basex.query.func.db.DbOpenId.value(DbOpenId.java:28)* * at org.basex.query.func.StaticFuncCall.evalArgs(StaticFuncCall.java:147)* * at org.basex.query.func.FuncCall.value(FuncCall.java:53)* * at org.basex.query.expr.ParseExpr.iter(ParseExpr.java:46)* * at org.basex.query.expr.Switch.iter(Switch.java:198)* * at org.basex.query.expr.IterMap$1.next(IterMap.java:72)* * at org.basex.query.QueryContext.next(QueryContext.java:359)* * at org.basex.query.expr.gflwor.GFLWOR$1.next(GFLWOR.java:66)* * at org.basex.query.QueryContext.next(QueryContext.java:359)* * at org.basex.query.expr.constr.Constr.add(Constr.java:73)* * at org.basex.query.expr.constr.CElem.item(CElem.java:149)* * at org.basex.query.expr.constr.CElem.item(CElem.java:1)* * at org.basex.query.expr.ParseExpr.value(ParseExpr.java:51)* * at org.basex.query.expr.ParseExpr.iter(ParseExpr.java:46)* * at org.basex.query.expr.constr.Constr.add(Constr.java:72)* * at org.basex.query.expr.constr.CElem.item(CElem.java:149)* * at org.basex.query.expr.constr.CElem.item(CElem.java:1)* * at org.basex.query.expr.ItemMap.item(ItemMap.java:37)* * at org.basex.query.expr.ParseExpr.value(ParseExpr.java:51)* * at org.basex.query.expr.ParseExpr.iter(ParseExpr.java:46)* * at org.basex.query.expr.gflwor.GFLWOR$1.next(GFLWOR.java:72)* * at org.basex.query.scope.MainModule$1.next(MainModule.java:67)* * at org.basex.http.restxq.RestXqResponse.serialize(RestXqResponse.java:87)* * at org.basex.http.web.WebResponse.create(WebResponse.java:58)* * at org.basex.http.restxq.RestXqServlet.run(RestXqServlet.java:72)* * at org.basex.http.BaseXServlet.service(BaseXServlet.java:69)* * at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)* * at org.eclipse.jetty.servlet.ServletHolder$NotAsync.service(ServletHolder.java:1450)* * at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799)* * at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:550)* * at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)* * at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:600)* * at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)* * at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)* * at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624)* * at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)* * at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1440)* * at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)* * at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:501)* * at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594)* * at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)* * at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1355)* * at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)* * at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)* * at org.eclipse.jetty.server.Server.handle(Server.java:516)* * at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:487)* * at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:732)* * at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:479)* * at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277)* * at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)* * at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)* * at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)* * at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338)* * at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315)* * at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173)* * at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)* * at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:409)* * at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)* * at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)* * at java.lang.Thread.run(Unknown Source)* I'm getting this error with a call to a complex ( more or less ) function. The more results it needs to handle, the more errors I get. Any help is welcome!

1 1

Faceted searching and browsing ?
by Majewski, Steven Dennis (sdm7g) 06 Jun '22

06 Jun '22

Is it possible to do faceted browsing with BaseX ? I think the answer is no, however I may have been initially fooled into trying because my first test case was far simpler than my other use cases. The simple case was getting counts of publisher from an EAD collection and sorting by greatest number. Here’s my working code: declare function local:normalize( $s ) { translate( replace( lower-case($s), '^(us-)+', '' ), '-', '' ) }; declare variable $orgs := doc('ead-inst/ead-inst.xml'); declare variable $orgcodes := collection('published')/ead/eadheader/eadid/@mainagencycode ! local:normalize(.) => distinct-values() ; declare function local:countpubfacets( $c ) { for $x in ( for $ead in $c let $ORG := local:normalize($ead/*:ead/*:eadheader/*:eadid/@mainagencycode) group by $ORG order by $ORG let $inst := if ($ORG != "") then ($orgs/list/inst[@prefix=$ORG],$orgs/list/inst[lower-case(@oclc) = $ORG] ) return array{ count($ead), $ORG, $inst/@orgcode/string(), $inst/string() } ) order by $x(1) descending return $x }; local:countpubfacets( collection('published')) In this case: (1) The number of unique @mainagencycode’s are less than 100, and (2) There is only one location for those codes. and performance is acceptable (or at least it seems to be in my tests). My other unsuccessful attempts have been with trying to rank //subject or //persname ’s. In this case, there are many thousands of unique subjects and names, and the subject and persname elements can occur in multiple locations in the file. Attempts to search similar to the above method ( as well as a couple of other variations I’ve tried ), even on a smaller subset of categories take entirely too much time — often I have to kill the search before it manages to complete. I have tried looking at index:facets() https://docs.basex.org/wiki/Index_Module#index:facets <https://docs.basex.org/wiki/Index_Module#index:facets> Which has only reinforced my notion that it’s not possible. So for now, I’m resigned to deferring that functionality, and exploring building a specialized index along side the BaseX indexes - either using Solr and querying the Solr index from BaseX, or else building some other index structure DB in BaseX along side my document DB. Eager to hear any tips or feedback on this problem or alternate solutions, and also general info about BaseX index structure and what useful info can be caught by introspection by those index module functions. Aside from the faceting, search by //subject (or other fields) is quite acceptable performance, even chaining several filters together with => declare function eadsearch:findBySubj( $ctx, $subj as xs:string?, $opt ) { if ( $subj ) then $ctx/*[ft:contains( .//subject, ft:tokenize($subj), $opt )] else $ctx }; — Steve M.

4 4

XML Catalog and xslt:transform()
by Zimmel, Daniel 02 Jun '22

02 Jun '22

Hi, after reading https://docs.basex.org/wiki/Catalog_Resolver and digging in the list archives (https://mailman.uni-konstanz.de/pipermail/basex-talk/2019-March/014199.html ) I still have trouble understanding catalog files. Is this supposed to work with xslt:transform() and BaseX GUI 9.7.2? The default option (DTD = false) is ignored by xslt:transform() because the function is definitely requesting the external DTD. This prevents transforming XML with DTD declarations that are not available (if I understand correctly, a problem that the DTD option is trying to solve in general). When I try to solve this via catalog files (actually I do not need the DTD), I do not have success. Here are my mini examples: Saxon HE 10.3 resides in the lib folder .basex setting: # Local Options SERIALIZER = indent=no DTD = true XML in local folder "C:/temp/catalog": <!DOCTYPE dokument SYSTEM "http://www.blahblahblah.info/dtd/dokument.dtd"> <dokument> <doknr>01</doknr> </dokument> catalog.xml in local folder "C:/temp/catalog": <catalog prefer="system" xmlns="urn:oasis:names:tc:entity:xmlns:xml:catalog"> <rewriteSystem systemIdStartString="http://www.blahblahblah.info/dtd/" rewritePrefix="file:///C:/temp/catalog/dtd/"/> </catalog> dokument.dtd in local folder "C:/temp/catalog/dtd": <!ELEMENT dokument (doknr)> <!ELEMENT doknr (#PCDATA)> XQuery query.xq in local folder "C:/temp/catalog": (# db:catfile catalog.xml #) { xslt:transform('dokument.xml', 'transform.xsl') } With or without pragma, this always results in a java.net.UnknownHostException (because the system ID is not available, that's true), but I would be expecting this would resolve to "file:///C:/temp/catalog/dtd/dokument.dtd" Not working in GUI nor via CCL. What am I getting wrong? Thanks, Daniel

6 8

Jump to page:

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

BaseX-Talk