- BaseX-Talk - mailman.uni-konstanz.de

Re: [basex-talk] XQJ status
by Marc van Grootel 06 Apr '16

06 Apr '16

Hi all, Thanks both Charles and Jürgen for your comprehensive and enlightening answers. It is very good to hear that there is hope for XQJ even in an XQuery 3 world ;-) Beefing up the spec to XQuery 3.1 would be fantastic. For the shorter term I'm thinking along the lines of a XDM transport serialization that includes type info and aims at efficiency rather than being human readable. As my host language of choice would be Clojure I am currently looking at Transit[1] which has both a Java and Clojure library and seems like a nice candidate but there are many others such as Avro, Thrift, Protocol buffers, Hessian etc etc. I might read up a little on those too[2]. [1] https://github.com/cognitect/transit-format On Tue, Apr 5, 2016 at 9:04 PM, Charles Foster <charles(a)cfoster.net> wrote: > Hi Marc, > > I plan to open source all the XQJ drivers I’ve written, (in the long-run). > > The XQJ specification was designed to support XQuery 1.0 only, however I don’t see why it can’t be made to support XQuery 3.0 and 3.1 syntax to some extent. > > Regarding XDM Maps and Arrays being bound to external variables and returned in result sequences along with the conversion between XDM and Java types wouldn’t be too difficult to achieve in practice. > > Java’s List and Map classes spring to mind, along with XQItemAccessor getObject(...) [1] to retrieve and XQDynamicContext.bindObject(...) [2] to bind where you would of course have to do manual casting. The datatypes of Java's Map and Array entries and conversions between their XDM counterparts could be determined by following the rules already laid out in the XQJ spec itself [3]. > > One would of course be going above and beyond the original XQJ specification here, but I think technically it would still be within the rules and providing it was well documented I’m sure user’s could cope. I would love Oracle to update their existing XQJ specification to cater for XQuery 3.1 properly, I would even help them get there if they wished, but I don’t know if they would be up for something like this. > > I think once the drivers are open sourced, the world of XQJ will be in healthier place for the community to update going forward. > > Specifically regarding BaseX XQJ - for the moment, please assume that any of the driver’s provided by BaseX themselves are the latest and greatest. > > Kind Regards, > > Charles > > P.S. Please come to XML London in June - it will be great! > > [1] http://xqj.net/javadoc/javax/xml/xquery/XQItemAccessor.html > [2] http://xqj.net/javadoc/javax/xml/xquery/XQDynamicContext.html > [3] https://en.wikipedia.org/wiki/XQuery_API_for_Java#Default_data_type_mapping > > > On 5 Apr 2016, at 14:41, Marc van Grootel <marc.van.grootel(a)gmail.com> wrote: > >> Hi Christian, >> >> Thanks. Very curious to hear about XQJ's fate. >> >> I am currently using BaseX for data integration which got me thinking >> about integration of XQuery in other web applications >> (node/python/clojure etc.). I was triggered by the recent thread about >> node-basex[1] and Hans Jürgen and your Balisage paper [2]. I was >> considering what the most performant way would be to return XQuery >> results as native types. Eg. function returns an array with maps or >> XML nodes and it gets converted to the host application's native data >> types so the application can use the results as if they were native >> data types. >> >> Maybe the only viable way to tackle this *is* via HTTP and a smart >> serialization (EDN[3] for example is such a, non-JSON, serialization >> that includes type info) but I was thinking if there was an existing >> lower-level way (without the BaseX HTTP layer). XQJ would enable this >> (at least for XQuery < 3) but I have doubts if it would deserialize >> non-atomic XDM values (like nested maps/arrays) as this is new to >> XQuery 3, I didn't check but I figure that the BaseXClient on which it >> is based also doesn't provide the XDM metadata for each nested item. >> >> Just pondering how to bridge from XQuery to other languages. >> >> [1]: http://www.mail-archive.com/basex-talk%40mailman.uni-konstanz.de/msg07882.h… >> [2]: http://www.balisage.net/Proceedings/vol15/html/Rennau01/BalisageVol15-Renna… >> [3]: https://github.com/edn-format/edn >> >> >> On Tue, Apr 5, 2016 at 11:58 AM, Christian Grün >> <christian.gruen(a)gmail.com> wrote: >>> Hi Marc, >>> >>>> Is it right that the current XQJ driver for BaseX doesn't support >>>> XQuery 3 syntax? Everytime I try to use a query that includes some of >>>> the newer features it barks at me with Invalid XQuery syntax, syntax >>>> does not pass static validation. Is there way to get around that, say, >>>> when the result would not use these XQuery 3 data types? >>>> >>>> I'm wondering what the fate is of XQJ in general. Will it remain relevant? >>> >>> I decided to pass on your question directly to Charles Foster; he >>> might be able to give you some outlook! >>> >>>> Also I am confused why files.basex /maven repo has XQJ driver 1.6.0 >>>> and xqj.net has 1.4.0 as the latest or what the difference is. >>> >>> I have sent a pull request to Charles’ repo; it may be included in the >>> next version there. >>> >>>> The reason I was looking into XQJ was that it provides ways to get XDM >>>> type info with the result. I would like to be able to deal with the >>>> result from a query at a higher level than just strings. I've briefly >>>> looked into BaseXClient.java which seems to be able to return XDM >>>> metadata but then I saw that XQJ could do this out of the box. >>> >>> Right, that’s only possible with XQJ. >>> >>>> In general I would be interested to hear what the best way would be to >>>> get a client to deal with query responses as native (client-side) >>>> types (including hashmaps and arrays) instead of strings. Would it be >>>> better to focus on using BaseXClient and forget about XQJ? >>> >>> From my perspective, it’s good to use XQuery whenever possible and use >>> our Client Module to address BaseX server instances [1] ;) >>> >>> Cheers, >>> Christian >>> >>> [1] http://docs.basex.org/wiki/Client_Module >> >> >> >> -- >> --Marc >> > -- --Marc

1 0

XQJ status
by Marc van Grootel 05 Apr '16

05 Apr '16

Hi, Is it right that the current XQJ driver for BaseX doesn't support XQuery 3 syntax? Everytime I try to use a query that includes some of the newer features it barks at me with Invalid XQuery syntax, syntax does not pass static validation. Is there way to get around that, say, when the result would not use these XQuery 3 data types? Also I am confused why files.basex /maven repo has XQJ driver 1.6.0 and xqj.net has 1.4.0 as the latest or what the difference is. I'm wondering what the fate is of XQJ in general. Will it remain relevant? The reason I was looking into XQJ was that it provides ways to get XDM type info with the result. I would like to be able to deal with the result from a query at a higher level than just strings. I've briefly looked into BaseXClient.java which seems to be able to return XDM metadata but then I saw that XQJ could do this out of the box. In general I would be interested to hear what the best way would be to get a client to deal with query responses as native (client-side) types (including hashmaps and arrays) instead of strings. Would it be better to focus on using BaseXClient and forget about XQJ? Cheers, --Marc

3 4

XQuery Introduction in French
by Dirk Kirsten 05 Apr '16

05 Apr '16

Bonjour, just a little pointer: If you speak french and are new to XQuery (and especially if you are in the digital humanities) the following chapters from Farid Djaïdjas thesis might be of interest to you: "XQuery pour les Humanités Numériques" http://files.basex.org/publications/dummies/XQuery%20pour%20les%20Humanit%e… This introduction and some others introductions and further material to get you started can be found at http://docs.basex.org/wiki/Getting_Started#Tutorials_and_Slides Bien cordialement Dirk -- Dirk Kirsten, BaseX GmbH, http://basexgmbh.de |-- Firmensitz: Blarerstrasse 56, 78462 Konstanz |-- Registergericht Freiburg, HRB: 708285, Geschäftsführer: | Dr. Christian Grün, Dr. Alexander Holupirek, Michael Seiferle `-- Phone: 0049 7531 91 68 276, Fax: 0049 7531 20 05 22

1 0

BaseX optimizer performance on REx-generated parser
by Gunther Rademacher 05 Apr '16

05 Apr '16

Hi, in a recent update of REx parser generator (v5.37), some type specifications were removed from generated XQuery and XSLT functions. This affects tail-recursive functions, and it was done because it turned out that MarkLogic fails to optimize tail calls, presumably due to an explicit type check caused by the type specification (see http://markmail.org/message/gxi26da4crk2v5ge ). Now it was reported that this change causes performance problems with BaseX (see https://twitter.com/apb1704/status/714219874441146368 ). I reproduced the problem as follows: - download the XQuery 3.1 grammar from http://bottlecaps.de/rex/CR-xquery-31-20151217.ebnf - generate a parser from it on http://bottlecaps.de/rex/ using command line options: -xquery -tree -main - run the generated parser, using this command line basex -binput={42} CR-xquery-31-20151217.xquery This works OK, but it takes about 80 seconds. Some analysis showed that the time can be influenced by putting back the type specification to function p:map2, i.e. declaring 'as xs:integer' as the result type: declare function p:map2($c as xs:integer, $lo as xs:integer, $hi as xs:integer) as xs:integer This variant completes in less than 3 seconds. But even when declaring a return type of as xs:integer? which might be the type that can be inferred statically, it completes fast. Is this possibly a problem with the optimizer? Which variant of generated code would be preferable for BaseX - typed or untyped? Thanks, Gunther

3 21

Problem with module-import via relative / absolute path (WEBPATH, REST-Interface)
by Günter Dunz-Wolff 04 Apr '16

04 Apr '16

Hi all, in my queries I have to import module namespace, for example in the file test.xq something like import module namespace kleist = "http://kleist-digital.de/ns/kleist" at "../modules/kleist-functions.xqm"; I'm calling the query via REST like http://localhost/rest?run=xqs/test.xq The webpath is configured like # HTTP Services WEBPATH = /Users/xxx/Documents/kleist/BaseXWeb RESTPATH = When I'm using the absolute path, everything works fine: import module namespace kleist = "http://kleist-digital.de/ns/kleist" at "file:///Users/xxx/Documents/kleist/BaseXWeb/modules/kleist-functions.xqm"; When I'm using the relative path for importing the module namespace like import module namespace kleist = "http://kleist-digital.de/ns/kleist" at "../modules/kleist-functions.xqm"; I'll get the following error: "Could not retrieve module: /Users/modules/kleist-functions.xqm" The correct address should be: Users/xxx/Documents/kleist/BaseXWeb/modules/kleist-functions.xqm What's wrong in my configuration? Btw, I need the relative path for portability to my remote server. Any help would be great, thanks a lot, Best regards, Günter

4 6

Async Module
by Christian Grün 03 Apr '16

03 Apr '16

Hi everyone, As some of you know, James Wright has presented his excellent work on the Promise Module in Prague [1,2]. The promise:fork-join function is now part of BaseX, along with some additional functions for asynchronous query processing [3,4]. Your feedback is welcome as usual, Christian [1] https://github.com/james-jw/xq-promise/ [2] http://archive.xmlprague.cz/2016/files/xmlprague-2016-proceedings.pdf [3] http://docs.basex.org/wiki/Async_Module [4] http://files.basex.org/releases/latest/

3 3

[ANN] Openshift BaseX quick start v0.6
by Andy Bunce 03 Apr '16

03 Apr '16

Hi, A new version is available at https://github.com/Quodatum/openshift-basex-quick-start. New features: - Enable REST and WebDAV - support for setting initial users - support for setting initial data - Use Java 8 Thanks to Michael Sperberg-McQueen and Günter Dunz-Wolff for helpful comments on the previous version. /Andy

3 6

Expected behaviour with full disk
by James Ball 03 Apr '16

03 Apr '16

Hello, I’m having some problems with BaseX in situations where there it runs out of disk space. I’m delivering XML files to BaseX 8.4 using RestXQ. A request is made and the XML is added to a database depending on some criteria. I receive many hundreds of thousands of requests and everything works well unless I run out of disk space. What I would expect to happen is that BaseX would try to add the document, find it couldn’t due to the disk being full and return a server error (e.g. 500) and the transaction would be aborted leaving the database as it was before the request. Any further requests would all get errors returned. What I’m seeing is that when a request is made that causes the disk to become full BaseX gets part way through adding the document, fails and returns an error but also leaves the database corrupted. Usually the server has to be restarted and then any database that has been affected has to be deleted - it’s rare to be able to save anything from it. The errors vary but it is usually a NullPointer exception and then java.nio.channels.OverlappingFileLockException. for further requests. I’ve tried turning off logging and setting parallel to 1 to see if that made any difference but it doesn’t. Is what I’m seeing unique to my setup or do others see this? Is there anything I can do to make BaseX just stop accepting requests when disk space runs out to protect my databases? Many thanks, James

4 5

Internal optimization
by Gabriel Tessarolli 30 Mar '16

30 Mar '16

Hi, First of all, thank you for the excellent software you produce and maintain! Keep up the good work. I've been using BaseX for some academic experiments on XQuery processing, and I got this situation that you guys can probably explain. Here is some context: - I am using version 8.2.3. - Database 'expdb' was created with default options of that version, using 'auction.xml' document generated from Xmark benchmark [1]. - BaseX is running with default options too. - What the query does is irrelevant. When I execute this query: for $pe in doc('expdb/auction.xml')/site/people/person for $cat in doc('expdb/auction.xml')/site/categories/category[position() >= 1 and position() < 101] where count($pe/profile/interest) > 3 and $pe/profile/interest/@category = $cat/@id return <match> <person>{$pe/name}</person> <category>{$cat/name}</category> </match> the resulting optimized query (in 'Query Info' window, on GUI) is this: for $pe_0 in *db:open-pre*("expdb",0)/*:site/*:people/*:person[3.0 < count(*:profile/*:interest)] for $cat_1 in *db:open-pre*("expdb",0)/*:site/*:categories/*:category[position() = 1 to 100][(@*:id = $pe_0/*:profile/*:interest/@*:category)] return element match { (element person { ($pe_0/*:name) }, element category { ($cat_1/*:name) }) } If I change the original query to (note that I am only switching the position of 'for' clauses): for $cat in doc('expdb/auction.xml')/site/categories/category[position() >= 1 and position() < 101] for $pe in doc('expdb/auction.xml')/site/people/person where count($pe/profile/interest) > 3 and $pe/profile/interest/@category = $cat/@id return <match> <person>{$pe/name}</person> <category>{$cat/name}</category> </match> the optimized query changes to: for $cat_0 in *db:open-pre*("expdb",0)/*:site/*:categories/*:category[position() = 1 to 100] for $pe_1 in *db:attribute*("expdb", $cat_0/@*:id)/self::*:category/parent::*:interest/parent::*:profile/parent::*:person[3.0 < count(*:profile/*:interest)] return element match { (element person { ($pe_1/*:name) }, element category { ($cat_0/*:name) }) } You can see that BaseX was able to use attribute index to optimize last query, reducing its execution time (by a lot!). My question is: what is the explanation for this behavior? Thank you in advance! Gabriel Tessarolli --- [1] http://www.xml-benchmark.org/

2 2

detection of external variables
by Andy Bunce 30 Mar '16

30 Mar '16

Hi, I am looking for a way to determine for a module what external variables are defined. Neither inspect:module [1] or inspect:xqdoc seem to return this information. I wonder if adding an *external *attribute to the inspect:module o/p for variables (and functions?) would be a solution. e.g. <variable name="samples:test-string" uri="http://basex.org/modules/samples" type="xs:string" *external="true"*> <description>This is a sample string.</description> </variable> /Andy [1] http://docs.basex.org/wiki/Inspection_Module#inspect:module

2 4