- BaseX-Talk - mailman.uni-konstanz.de

Re: [basex-talk] Converting to UTF-8 in SQL module
by Lizzi, Vincent 17 Aug '15

17 Aug '15

Thanks for sharing your solution! Vincent From: Tim Thompson [mailto:timathom@gmail.com] Sent: Monday, August 17, 2015 5:00 PM To: Lizzi, Vincent <Vincent.Lizzi(a)taylorandfrancis.com> Subject: Re: [basex-talk] Converting to UTF-8 in SQL module Thanks, Christian, Vincent. Following Christian's suggestion, I used the RAWTOHEX() function in my SQL query, then cast it to an xs:hexBinary and applied BaseX's convert:binary-to-string(). Seems to work perfectly. --Tim -- Tim A. Thompson Metadata Librarian (Spanish/Portuguese Specialty) Princeton University Library On Mon, Aug 17, 2015 at 4:48 PM, Lizzi, Vincent <Vincent.Lizzi(a)taylorandfrancis.com<mailto:Vincent.Lizzi@taylorandfrancis.com>> wrote: Hi Tim, Oracle should able to convert its output to Unicode before returning query results to the client (BaseX). Are you using Oracle's JDBC driver? It might be helpful to look into Oracle's NLS_LANG setting or the 'convert' function. Vincent -----Original Message----- From: basex-talk-bounces(a)mailman.uni-konstanz.de<mailto:basex-talk-bounces@mailman.uni-konstanz.de> [mailto:basex-talk-bounces@mailman.uni-konstanz.de<mailto:basex-talk-bounces@mailman.uni-konstanz.de>] On Behalf Of Christian Grün Sent: Monday, August 17, 2015 3:23 PM To: Tim Thompson <timathom(a)gmail.com<mailto:timathom@gmail.com>> Cc: BaseX <basex-talk(a)mailman.uni-konstanz.de<mailto:basex-talk@mailman.uni-konstanz.de>> Subject: Re: [basex-talk] Converting to UTF-8 in SQL module Hi Tim, > I am using the BaseX SQL module to query the Oracle database of a > library catalog. Its character data is encoded as ISO 2709 (MARC-8)[1] > and is stored in Oracle as US7ASCII. > > The data contains diacritics with combining characters like this: > http://www.fileformat.info/info/unicode/char/0301/index.htm I don't know enough about the encodings of Oracle, but "US7ASCII" sounds to me as if only 7 bits are used for each character. Do you know how non-ASCII characters, such as combining characters, are stored in that format? Maybe convert:string-to-hex and convert:binary-to-string [1] could be used to convert the result to the correct encoding. Basically, all we do in BaseX is using standard JDBC functionality [2]. If there is an easy way to fix the issue with JDBC, it should be easy to also get it working in XQuery. Christian [1] http://docs.basex.org/wiki/Conversion_Module [2] https://github.com/BaseXdb/basex/blob/master/basex-core/src/main/java/org/b…

2 1

Converting to UTF-8 in SQL module
by Tim Thompson 17 Aug '15

17 Aug '15

Hello, I am using the BaseX SQL module to query the Oracle database of a library catalog. Its character data is encoded as ISO 2709 (MARC-8)[1] and is stored in Oracle as US7ASCII. The data contains diacritics with combining characters like this: http://www.fileformat.info/info/unicode/char/0301/index.htm When I run the query in BaseX, all characters with combining accents are output as something like Ì Has anyone had experience handling this kind of encoding issue in BaseX, or does anyone have any solutions/approaches to recommend as a way to convert this data to UTF-8? Thanks in advance, Tim [1] https://en.wikipedia.org/wiki/ISO_2709 [2] http://www.fileformat.info/info/unicode/char/00e8/index.htm -- Tim A. Thompson Metadata Librarian (Spanish/Portuguese Specialty) Princeton University Library <tat2(a)princeton.edu>

2 1

Concurrency Issue while loading documents.
by ankit kumar 17 Aug '15

17 Aug '15

Hi, We are adding number of documents concurrently to the Basex database using basex java api. Our application runs three threads in which each thread is adding documents to it by executing following instructions in java as given below. new Add("",new File(source).getAbsolutePath()).execute(context); while running our application sometimes throws the following error Java: Oracle Corporation, 1.7.0_71 OS: Windows 7, amd64 Stack Trace: java.lang.ArrayIndexOutOfBoundsException: 8 at org.basex.util.hash.TokenSet.index(TokenSet.java:161) at org.basex.util.hash.TokenSet.put(TokenSet.java:97) at org.basex.data.Namespaces.open(Namespaces.java:319) at org.basex.data.NSScope.open(NSScope.java:72) at org.basex.data.Data.insert(Data.java:775) at org.basex.query.up.atomic.Insert.apply(Insert.java:44) at org.basex.query.up.atomic.AtomicUpdateCache.applyUpdates(AtomicUpdateCache.java:297) at org.basex.query.up.atomic.AtomicUpdateCache.execute(AtomicUpdateCache.java:281) at org.basex.core.cmd.Add.run(Add.java:66) at org.basex.core.Command.run(Command.java:398) at org.basex.core.Command.execute(Command.java:100) at org.basex.core.Command.execute(Command.java:123) at com.myapp.MyDataBase.addSource(MyDataBase.java:85) at com.myapp.ProcessorLoader$1.run(ProcessorLoader.java:132) at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.util.concurrent.FutureTask.run(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) thanks, Anky

2 2

Resolving includes
by cmarchand＠oxiane.com 17 Aug '15

17 Aug '15

Hi, back to my questions on resolving modules includes. I want to produce a .XQ file that includes all the code contained in imported modules (via import module clause). My purpose is to produce a maven plugin to do this. I think such a code already exists in basex, and I would like to reuse it. Any entry-point will be much appreciated. Thanks in advance, Christophe

2 4

Trouble with wildcard lookup operator on maps?
by Marc van Grootel 15 Aug '15

15 Aug '15

Hi, Unless I'm not reading the spec[1] correctly then the following two snippets should have the same result: Example 1: let $map := map { 'a': (1,2,3), 'b': (4,5,6)} for $k in map:keys($map) return array { $map($k) } Example 2: let $map := map { 'a': (1,2,3), 'b': (4,5,6)} for $k in $map?* return array { $k } However, this is not the case. The first outputs, as I expected: ([1,2,3], [4,5,6]) The second, counter intuitively returns (on 8.2.3) this: ([1],[2],[3],[4],[5],[6]) The spec indicates that both examples result should be identical. Didn't check if the same is going on with arrays. Hmmm? [1] http://www.w3.org/TR/xquery-31/#id-lookup

2 2

HTTP response and raising error()
by Marc van Grootel 14 Aug '15

14 Aug '15

Hi, Nothing major but I was wondering what the thinking was behind returning a 400 when fn:error is raised. Basically that says "client, your mistake not mine" where in most cases I guess a 500 would be more appropriate. Where is this decided (RESTXQ?) and is there an easy way to change the status when fn:error is used? Of course I can always catch at the highest level and then decide which HTTP status to return but I'm not sure if I want to wrap all REST calls in try / catch. -- --Marc

3 6

BaseX "subcollections"
by Tim Thompson 13 Aug '15

13 Aug '15

Hello, I see that I can create subcollections or paths within a database if I create the database then use the "ADD TO" command to add a resource at a specified path. Is this the only way to create hiearchical structure within a database? I couldn't find many details about this in the documentation. I see that the changelog for BaseX 6.3.1 includes, [ADD] REST: support for hierarchical collections added; details will follow, but I don't see anything about this in the wiki page for the REST module. Thanks in advance, Tim -- Tim A. Thompson Metadata Librarian (Spanish/Portuguese Specialty) Princeton University Library

3 3

client:info
by Andy Bunce 12 Aug '15

12 Aug '15

Hi, Not really a problem for me but I noticed this in the GUI: let $c := client:connect('localhost', 1984, 'admin', 'admin') let $d:= client:query($c, "2+ 2") return client:info($c) Improper use? Potential bug? Your feedback is welcome: Contact: basex-talk(a)mailman.uni-konstanz.de Version: BaseX 8.2.3 Java: Oracle Corporation, 1.8.0_45 OS: Windows 7, amd64 Stack Trace: java.lang.NullPointerException at org.basex.query.func.client.ClientInfo.item(ClientInfo.java:17) at org.basex.query.expr.ParseExpr.iter(ParseExpr.java:43) at org.basex.query.expr.gflwor.GFLWOR$1.next(GFLWOR.java:99) at org.basex.query.MainModule$1.next(MainModule.java:114) at org.basex.query.QueryContext.cache(QueryContext.java:660) at org.basex.query.QueryProcessor.cache(QueryProcessor.java:103) at org.basex.core.cmd.AQuery.query(AQuery.java:83) at org.basex.core.cmd.XQuery.run(XQuery.java:22) at org.basex.core.Command.run(Command.java:398) at org.basex.core.Command.execute(Command.java:100) at org.basex.gui.GUI.exec(GUI.java:472) at org.basex.gui.GUI.access$400(GUI.java:43) at org.basex.gui.GUI$7.run(GUI.java:412) And slightly related.. In the DBA query window let $c := client:connect('localhost', 1984, 'admin', 'admin') return client:query($c, "2+ 2") will block the basex server for all http users and basexclient (s). Sad, but I can see why this might be :-), Another way to achieve this is on the dba login page enter "localhost:1984" as a *remote *host. but is there any way to programmatically detect this situation has occurred or cancel these queries without restarting the server? /Andy

2 3

basex limits: #ENames
by Simon Görke 12 Aug '15

12 Aug '15

Hello, according to [1], basex supports a maximum of 2^15 #ENames per database. Would there be any feedback (warning / error message / exit from processing) if this limit was exceed? If not: Could this be an explaination for turning the output of a xslt (xml => xml) to be not well-formed (The xslt itself is mature and has been used in exactly the same environment in several projects for > 2 years without problems.)? Now, with the latest project, an export of the database after the (seemingly issue-free) xslt-transform delivers e.g. elements at positions where they should not appear / opening elements not fitting their closing counterparts / missing content at the end? All other projects are ok with the current setup, but have significantly less data. The output dialect is somewhat "creative" and prone to resulting in a diversity of ENames, so this triggered my suspicion here... I am using basex 8.2.3 on Java 1.8.51 / 64bit. Also tried with 8.0.2, but same issue there. The transformations are triggered by a small snippet of xquery sent from a basexclient to the server. Thanks, Simon [1] http://docs.basex.org/wiki/Statistics

2 3

Command history in basex client
by adj7388 12 Aug '15

12 Aug '15

Hello, Does the basex command line client keep no command history? I am using the Linux version. In most Linux command line programs, the up- and down-arrows allow you to scroll through past commands so you don't have to retype them every time. I see "Save command-line history" in the change log for version 8.0.1 ( and I have version 8.2.3 installed) but command history does not appear to work. Thanks, Alan

3 4

HTML parsing in the DBA
by Kristian Kankainen 12 Aug '15

12 Aug '15

Hello! I tried simple cut and paste from the wiki and came up with the following query for importing an HTML file. The DBA says "Query was successful", but no resources are never added to the db. declare option db:parser "html"; declare option db:htmlparser "html=false,nodefaults=true"; db:replace("dbname", "filename.html", "/file/path/filename.html") I'm running the latest war file on Tomcat. Best wishes Kristian Kankainen

2 1

Using BaseX With oXygenXML: Validate XQuery 3.1 Syntax
by Eliot Kimber 09 Aug '15

09 Aug '15

I've started using oXygenXML rather than the DBA Web app to run my test queries (mostly so I can handle large result sets in the output). If I have a statement like: Let $foo as map(*) := map {} Oxygen reports a static validation error. If I remove the "as" clause, then it's happy. My question: is it possible to configure oXygenXML so it validates the 3.1 expressions BaseX supports? I'm not sure if this an oXygenXML or a BaseX issue. Thanks, Eliot ---- Eliot Kimber, Owner Contrext, LLC http://contrext.com

3 2

Scope of options set with SET
by Amanda Galtman 06 Aug '15

06 Aug '15

Hi, I'm not sure if the following is a bug or if I misunderstand the scope of options set with SET. Suppose I have a) A DTD that defines a text entity b) A catalog file that points to the DTD c) An XML file with a DOCTYPE line that references the DTD, and XML content that references the text entity If I do the following in a standalone BaseX session SET DTD true SET CATFILE [path to my catalog file...] xquery fetch:xml([path to my XML file...]) then the text entities are missing, presumably because the catalog or DTD is not being read. Either of the following works as expected, with the text entities resolved: xquery fetch:xml([path to my XML file...],map {'dtd': true(), 'catfile': [path to my catalog file...]}) xquery doc([path to my XML file...]) But why do I have to use the 2nd input in fetch:xml if I used SET in the same session? I thought the SET command would suffice, and the 2nd input argument in fetch:xml was available if I either wanted to override an option I had previously set with SET or did not use SET previously. Thanks, Amanda

2 3

BaseX Error
by Rogers, Paul 06 Aug '15

06 Aug '15

Good afternoon, we're getting an error on BaseX rest and DBA connections: Improper use? Potential bug? Your feedback is welcome: Contact: basex-talk(a)mailman.uni-konstanz.de<mailto:basex-talk@mailman.uni-konstanz.de> Version: BaseX 8.2.3 Java: Oracle Corporation, 1.7.0_01 OS: Linux, amd64 Stack Trace: java.lang.ArrayIndexOutOfBoundsException Is there an issue with the service? Best Regards, Paul Rogers DevOps DBA Engineer Kuoni Global Travel Services Ueberlandstrasse 360 CH-8051 Zurich, Switzerland T +41 44 325 21 39 paul.rogers(a)ch.kuoni.com<mailto:paul.rogers@ch.kuoni.com> www.kuoniglobaltravelservices.com<http://www.kuoniglobaltravelservices.com/>

2 1

SIGPIPE error
by mar＠centrum.cz 06 Aug '15

06 Aug '15

Hi, I am having difficulties with populating BASEX database. I have plenty of XML files (~ half a million, with various sizes ranging from several kilobytes up to hundred of kilobytes). I use BASEX Java API and finally I call for each file org.basex.core.cmd.Add. I am trying to import them into BASEX database, in fact there are 22 types of files (22 XSD definitions) the files conform to, so I have 22 different databases in a single BASEX server. I have plenty of RAM and CPU power and I monitor the process (both -- the BASEX server and my client program) from within JVisualVM, the JVM reaches the CPU boundaries, but RAM is never exhausted. Before importing, I need to enhance the XML data with some additional information taken from SQL database. I have written a Groovy multithreaded program that uses BASEX Java API with heavy use of GPars library. Simply put, the program: 1. has several producer threads -- each producer reads given portion of the database and provides those additional information 2. has several consumer threads -- each consumer takes the original files, wraps it with additional information and finally calls org.basex.core.cmd.Add command. Various testing with less data (upto ~ several thousands of files) provides good results -- no loss of data, BASEX server and my client program behaves as it should. Unfortunately when trying to import all of the files, the program starts fine, but when it gets "warm" I got SIGPIPE errors in log from time to time (as I said, there is plenty of RAM and CPU available) (see attachment please). Comments to picture: 1. I am adding document with ID ISPOP_166007 -- this ID is indeed missing in the final database 2. just simple call to Add: Closure add = { session -> def cmd = new org.basex.core.cmd.Add(dsn, enhancedXml) session.execute(cmd) } 3. I am reusing the session, the session is bound to current thread and never gets closed until the thread (consumer) finishes There is nothing wrong in BASEX server log, other documents are added just fine, there is no trace about document ISPOP_166007. Just for reference the complete stack trace follows: - - - - ERROR basex.support.AddResourcesSupport - unable to consume ISPOP_166007 java.net.SocketException: Roura přerušena (SIGPIPE) at java.net.SocketOutputStream.socketWrite0(Native Method) ~[na:1.8.0_45] at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109) ~[na:1.8.0_45] at java.net.SocketOutputStream.write(SocketOutputStream.java:153) ~[na:1.8.0_45] at org.basex.io.out.BufferOutput.flush(BufferOutput.java:60) ~[basex-8.2.jar!/:8.2] at org.basex.io.out.BufferOutput.write(BufferOutput.java:54) ~[basex-8.2.jar!/:8.2] at org.basex.io.out.PrintOutput.write(PrintOutput.java:66) ~[basex-8.2.jar!/:8.2] at java.io.OutputStream.write(OutputStream.java:116) ~[na:1.8.0_45] at java.io.OutputStream.write(OutputStream.java:75) ~[na:1.8.0_45] at org.basex.api.client.ClientSession.send(ClientSession.java:238) ~[basex-8.2.jar!/:8.2] at org.basex.api.client.ClientSession.execute(ClientSession.java:160) ~[basex-8.2.jar!/:8.2] at org.basex.api.client.ClientSession.execute(ClientSession.java:167) ~[basex-8.2.jar!/:8.2] at org.basex.api.client.Session.execute(Session.java:36) ~[basex-8.2.jar!/:8.2] at org.basex.api.client.Session$execute.call(Unknown Source) ~[na:na] at basex.support.AddResourcesSupport$_consume_closure9$_closure17.doCall(AddResourcesSupport.groovy:255) ~[basex-1.0.jar!/:na] at sun.reflect.GeneratedMethodAccessor368.invoke(Unknown Source) ~[na:na] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_45] at java.lang.reflect.Method.invoke(Method.java:497) ~[na:1.8.0_45] at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93) [groovy-2.4.4.jar!/:2.4.4] at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325) [groovy-2.4.4.jar!/:2.4.4] at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:294) [groovy-2.4.4.jar!/:2.4.4] at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1019) [groovy-2.4.4.jar!/:2.4.4] at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.call(PogoMetaClassSite.java:42) [groovy-2.4.4.jar!/:2.4.4] at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:125) [groovy-2.4.4.jar!/:2.4.4] at basex.BasexSessionRegistry.withThreadBoundSession(BasexSessionRegistry.groovy:79) ~[basex-1.0.jar!/:na] at basex.BasexSessionRegistry$withThreadBoundSession$0.call(Unknown Source) ~[na:na] at basex.support.AddResourcesSupport$_consume_closure9.doCall(AddResourcesSupport.groovy:257) ~[basex-1.0.jar!/:na] at sun.reflect.GeneratedMethodAccessor327.invoke(Unknown Source) ~[na:na] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_45] at java.lang.reflect.Method.invoke(Method.java:497) ~[na:1.8.0_45] at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93) [groovy-2.4.4.jar!/:2.4.4] at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325) [groovy-2.4.4.jar!/:2.4.4] at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:294) [groovy-2.4.4.jar!/:2.4.4] at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1019) [groovy-2.4.4.jar!/:2.4.4] at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.call(PogoMetaClassSite.java:42) [groovy-2.4.4.jar!/:2.4.4] at org.codehaus.groovy.runtime.callsite.BooleanReturningMethodInvoker.invoke(BooleanReturningMethodInvoker.java:51) ~[groovy-2.4.4.jar!/:2.4.4] at org.codehaus.groovy.runtime.callsite.BooleanClosureWrapper.call(BooleanClosureWrapper.java:53) ~[groovy-2.4.4.jar!/:2.4.4] at org.codehaus.groovy.runtime.DefaultGroovyMethods.find(DefaultGroovyMethods.java:3908) ~[groovy-2.4.4.jar!/:2.4.4] at org.codehaus.groovy.runtime.dgm$191.invoke(Unknown Source) ~[groovy-2.4.4.jar!/:2.4.4] at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite$PojoMetaMethodSiteNoUnwrapNoCoerce.invoke(PojoMetaMethodSite.java:274) ~[groovy-2.4.4.jar!/:2.4.4] at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite.call(PojoMetaMethodSite.java:56) ~[groovy-2.4.4.jar!/:2.4.4] at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:125) [groovy-2.4.4.jar!/:2.4.4] at basex.support.AddResourcesSupport.consume(AddResourcesSupport.groovy:251) ~[basex-1.0.jar!/:na] - - - - Best Regards, Martin

3 9

Resolving http file with local repository.
by ankit kumar 06 Aug '15

06 Aug '15

Hi, I want to load document having a http url from local cache directory. Is there a way i can resolve http url to my local cache directory. For example let $file := doc("http://abc.com/example.xml") should resolve example.xml from c:\documents\repo\http\abc.com\example.xml Thanks, Anky

2 2

Running BaseX In A Docker Container?
by Eliot Kimber 05 Aug '15

05 Aug '15

As part of my DITA For Small Teams project we're setting up a set of coordinated Docker containers to manage the various components (git server, Jenkins server, BaseX server). We're setting up our own custom containers so that they are appropriately pre-configured. In the case of BaseX that means configured with the DFST Web app and supporting modules, configuration details, etc. Just curious if anyone has experience putting BaseX in a container and if there's anything we should look out for or anything we can contribute back out of this effort. Cheers, E. ---- Eliot Kimber, Owner Contrext, LLC http://contrext.com

2 1

BaseX Package and Jetty
by Eckey, Florian 05 Aug '15

05 Aug '15

3 5

Destination of result-document from XSLT module
by Lizzi, Vincent 05 Aug '15

05 Aug '15

I'm trying to use the XSLT Module in BaseX 8.2.3 with Saxon 9.6 to run an XSLT that produces several output documents using xsl:result-document. I'm having trouble setting the location of the output documents. I want to have xsl:result-document create the output documents in a temporary folder because the documents need to be zipped together. According to Saxon's documentation, the a relative path in the href attribute of xsl:result-document will be resolved using either the path of the Destination, or the current directory. The XSLT Module does not appear have a way to provide a path for a destination document. What I'm seeing is that the result documents are created in BaseX's home directory. The XSLT works as expected when run using Saxon from the command line, where it's possible to set a destination path. Is there a way to specify a Base Output URI to the XSLT Module? Or, would it be possible to specify a file URI output location to a method like xslt:transform? One possible workaround is to provide an absolute path as a parameter to the XSLT, and use that parameter in the xsl:result-document href location. Here is a self-contained example code: declare function local:example($in, $xsl, $zipPath) { let $tempDir := file:create-temp-dir('test', 'example') let $x := xslt:transform-text($in, $xsl) return let $zip := archive:create-from($tempDir) return ( file:write-binary($zipPath, $zip) (: , file:delete($tempDir, true()) :) ) }; let $xsl := <xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform" xmlns:xs="http://www.w3.org/2001/XMLSchema" version="2.0"> <xsl:template match="/"> <xsl:result-document href="doc1.xml"> <test>this is a test 1 <xsl:apply-templates/></test> </xsl:result-document> <xsl:result-document href="doc2.xml"> <test>this is a test 2 <xsl:apply-templates/></test> </xsl:result-document> </xsl:template> </xsl:stylesheet> let $doc := <test>this is input</test> let $zipPath := 'report.zip' return local:example($doc, $xsl, $zipPath) The expected output is a zip file report.zip that contains doc1.xml and doc2.xml. However, what I'm seeing is that report.zip is created as an empty zip file and doc1.xml and doc2.xml are placed in BaseX's home directory. Thanks, Vincent

5 7

Performance with joins?
by Tim Thompson 04 Aug '15

04 Aug '15

Hello, I have a database containing two resources/documents: they both represent the same set of library catalog records (7728 "records" in each), but they each contain different data that I want to join. The first resource looks like this: <marc:collection xmlns:marc="http://www.loc.gov/MARC21/slim"> <marc:record> <marc:leader>01225cam a2200373Mi 4500</marc:leader> <marc:controlfield tag="001">5323084</marc:controlfield> <marc:datafield ind1="1" ind2="4" tag="245"> <marc:subfield code="a">Els teleclubs a les illes Balears :</marc:subfield> </marc:datafield> </marc:record> <marc:record> <marc:leader>01225cam a2200373Mi 4500</marc:leader> <marc:controlfield tag="001">5323084</marc:controlfield> <marc:datafield ind1="1" ind2="4" tag="245"> <marc:subfield code="a">Els teleclubs a les illes Balears :</marc:subfield> </marc:datafield> </marc:record> <marc:record> <marc:leader>00818cam a2200241Mi 4500</marc:leader> <marc:controlfield tag="001">6310976</marc:controlfield> <marc:datafield ind1="0" ind2="0" tag="245"> <marc:subfield code="a">Diccionari manual de sinònims i antònims de la llengua catalana /</marc:subfield> </marc:datafield> </marc:record> </marc:collection> The second one looks like this: <root> <row> <LANGUAGE>cat</LANGUAGE> <ITEM_ID>5912416</ITEM_ID> <BIB_ID>5323084</BIB_ID> <VENDOR_CODE>MXBKSMX</VENDOR_CODE> </row> <row> <LANGUAGE>cat</LANGUAGE> <ITEM_ID>5912416</ITEM_ID> <BIB_ID>5323084</BIB_ID> <VENDOR_CODE>PUVILL</VENDOR_CODE> </row> <row> <LANGUAGE>cat</LANGUAGE> <ITEM_ID>5935043</ITEM_ID> <BIB_ID>6310976</BIB_ID> <VENDOR_CODE>PUVILL</VENDOR_CODE> </row> </root> I have a simple query that joins the two using the value of the marc:controlfield[@tag = '001'] from resource 1 and the BIB_ID from resource 2. The query: declare namespace marc="http://www.loc.gov/MARC21/slim"; for $m in collection( "latin_hold_20150730" )/marc:collection/marc:record, $r in collection( "latin_hold_20150730" )/root/row[BIB_ID = $m/marc:controlfield[@tag = '001']] group by $key := $r/ITEM_ID return <test n="{ $key }">{ $m/marc:datafield[@tag = '245']/marc:subfield[@code = 'a']/string() }</test> When I run this using Saxon (substituting fn:doc for fn:collection), it only takes a second to execute. In BaseX 8.2.3 (using the GUI), however, it takes a very long time to execute (around 6 minutes!). The BaseX database has both attribute and text indexes enabled. Any idea what is causing it to take so long? Here is the BaseX Query Info: Total Time: 383756.3 ms Compiling: - pre-evaluating collection("latin_hold_20150730") - pre-evaluating collection("latin_hold_20150730") Query: declare namespace marc="http://www.loc.gov/MARC21/slim"; for $m in collection( "latin_hold_20150730" )/marc:collection/marc:record, $r in collection( "latin_hold_20150730" )/root/row[BIB_ID = $m/marc:controlfield[@tag = '001']] group by $key := $r/ITEM_ID return <test n="{ $key }">{ $m/marc:datafield[@tag = '245']/marc:subfield[@code = 'a']/string() }</test> Optimized Query: for $m_0 in (db:open-pre("latin_hold_20150730",0), ...)/marc:collection/marc:record for $r_1 in (db:open-pre("latin_hold_20150730",0), ...)/root/row[(BIB_ID = $m_0/marc:controlfield[(@tag = "001")])] let (: post-group :) $m_4 := $m_0 group by $key_2 := $r_1/ITEM_ID return element test { (attribute n { ($key_2) }, $m_4/marc:datafield[(@tag = "245")]/marc:subfield[(@code = "a")]/string()) } Result: - Hit(s): 7587 Items - Updated: 0 Items - Printed: 505 KB - Read Locking: local [latin_hold_20150730] - Write Locking: none Timing: - Parsing: 0.43 ms - Compiling: 1.96 ms - Evaluating: 383737.5 ms - Printing: 16.41 ms - Total Time: 383756.3 ms Query plan: <QueryPlan compiled="true"> <GFLWOR> <For> <Var name="$m" id="0"/> <IterPath> <DBNodeSeq size="2"> <DBNode name="latin_hold_20150730" pre="0"/> <DBNode name="latin_hold_20150730" pre="365692"/> </DBNodeSeq> <IterStep axis="child" test="marc:collection"/> <IterStep axis="child" test="marc:record"/> </IterPath> </For> <For> <Var name="$r" id="1"/> <IterPath> <DBNodeSeq size="2"> <DBNode name="latin_hold_20150730" pre="0"/> <DBNode name="latin_hold_20150730" pre="365692"/> </DBNodeSeq> <IterStep axis="child" test="root"/> <IterStep axis="child" test="row"> <CmpG op="="> <CachedPath> <IterStep axis="child" test="BIB_ID"/> </CachedPath> <IterPath> <VarRef> <Var name="$m" id="0"/> </VarRef> <IterStep axis="child" test="marc:controlfield"> <CmpG op="="> <CachedPath> <IterStep axis="attribute" test="tag"/> </CachedPath> <Str value="001" type="xs:string"/> </CmpG> </IterStep> </IterPath> </CmpG> </IterStep> </IterPath> </For> <GroupBy> <Spec> <Var name="$key" id="2"/> <IterPath> <VarRef> <Var name="$r" id="1"/> </VarRef> <IterStep axis="child" test="ITEM_ID"/> </IterPath> </Spec> </GroupBy> <CElem> <QNm value="test" type="xs:QName"/> <CAttr> <QNm value="n" type="xs:QName"/> <VarRef> <Var name="$key" id="2"/> </VarRef> </CAttr> <MixedPath> <VarRef> <Var name="$m" id="4"/> </VarRef> <IterStep axis="child" test="marc:datafield"> <CmpG op="="> <CachedPath> <IterStep axis="attribute" test="tag"/> </CachedPath> <Str value="245" type="xs:string"/> </CmpG> </IterStep> <IterStep axis="child" test="marc:subfield"> <CmpG op="="> <CachedPath> <IterStep axis="attribute" test="code"/> </CachedPath> <Str value="a" type="xs:string"/> </CmpG> </IterStep> <FnString name="string([item])"/> </MixedPath> </CElem> </GFLWOR> </QueryPlan> Thanks in advance! Tim -- Tim A. Thompson Metadata Librarian (Spanish/Portuguese Specialty) Princeton University Library

2 4

Round tripping JSON
by Marc van Grootel 04 Aug '15

04 Aug '15

Hi, I am working on some REST API code and I'm struggling with POSTing of JSON. What I want to achieve is to have the function receive an XDM map (from JSON using fn:parse-json). I also want to output from such a map (in effect roundtripping some JSON). The test POST body I'm using is: {"foo": "bar"} When reading the docs I'm led to believe that this should achieve it: declare %rest:POST("{$body}") %rest:path("/json") %rest:consumes("application/json;lax=yes;format=map") %rest:produces("application/json;format=map") function api:json($body) { $body }; No matter what I do I always get this as response: <json type="object"> <foo>bar</foo> </json> Somehow I cannot force the function to receive a map(*) as $body argument. When I wrap $body in a trace() I see that it gets XML and not a map(*). What am I missing? Or do I really have to go JSON > XML > map {....}? -- --Marc

3 18

File permissions on .basex files
by Amanda Galtman 03 Aug '15

03 Aug '15

Hi, It seems that the commands and functions that operate on the .basex files, such as ADD and db:add(...), make the .basex files writable by the creator but not by anyone else. On the other hand, executing a query requires write permission for at least some of the .basex files, even if the query does not attempt to update the database. As a result, if I create a database and want anyone other than me to be able to use it (via the GUI, for example), I have to change permissions on the .basex files after creating the database. Apparently, I have to change the permissions again if I later update the database and rebuild the indexes. Do folks on this list have a slick way to keep the permissions as open as needed? I tried searching the list archive but didn't find anything that looked relevent. Would it be a reasonable enhancement request to make this a bit smoother? For example, an option could let you specify that the .basex files being created or modified should be writable by all. Alternatively or in addition, perhaps it is possible to stop requiring write permission for non-updating queries; I'm not familiar with the reason why write permission is needed. Thanks, Amanda

2 4

Slow query
by Menashè Eliezer 03 Aug '15

03 Aug '15

Hello, I wonder if the attached query can be optimised. I'm attaching all relevant information. Basex 7.9, Debian, powerful server. This is just an example. The queries will be built based on a compilation of a search form. So reordering the conditions for having smaller subset right from the begging isn't relevant. Any help would be appreciated. 40 seconds are not acceptable. -- With kind regards, Menashè

2 45

Applying indexes
by Menashè Eliezer 03 Aug '15

03 Aug '15

Hello, 1. In which cases the "applying index" phrase should I see in the query plan? I have an old query (the one I've sent here at 06/22/2015) where only for one condition I've seen "applying index", and even this one doesn't appear anymore using a newer (8.2.2) version of BaseX and a larger database. I do see the right properties of the db: Indexes Up-to-date: true TEXTINDEX: true ATTRINDEX: true 2. Only once a value which is mentioned in the query is found in the index, in case of "=" condition? 3. Always, in case of "<" or "<" condition, assuming the specific text/attribute has been indexed? 4. Everything I see as a reply to index:facets is indexed, right? -- With kind regards, Menashè

2 4

No debug info
by Menashè Eliezer 03 Aug '15

03 Aug '15

Hello, I don't know why I don't see anymore query log events like "admin OK QUERY" in the .logs folder. Not even when I use the -d flag. BaseX 8.2.2 -- With kind regards, Menashè

2 2

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

BaseX-Talk