BaseX-Talk January 2015

basex-talk@mailman.uni-konstanz.de

36 participants
62 discussions

Feature request
by Johan Mörén 27 Jan '15

27 Jan '15

Hello! We have been using the hashing module to calculate md5 checksums on binary files successfully for a while. But last week we received our first really large file (4.3 gb) and our script threw a *java.lang.OutOfMemoryError: Requested array size exceeds VM limit* We are currently using the 7.8 version of BaseX. I suspect that BaseX materialize the stream returned by file:read-binary as a byte-array when we call the hash:md5 function. This is a snippet of our script where the problem arises ... let $binary := file:read-binary($filePath) let $checksum := lower-case(xs:string(xs:hexBinary(hash:md5($binary)))) ... I think a nice feature to add to BaseX could either be a new function in the file-module called file-checksum($algorithm) that calculates checksum on files in a streaming fashion. Or perhaps an option to the hashing functions that indicates that you want them to use streaming. Regards, Johan Mörén

2 12

Re: [basex-talk] How to associate catalog.xml with validate function
by Marc van Grootel 26 Jan '15

26 Jan '15

Sorry, for the no subject, lost it somewhere .... On Mon, Jan 26, 2015 at 1:53 PM, Marc van Grootel <marc.van.grootel(a)gmail.com> wrote: > Hi, > > I'm trying to run validations on XML files that use DTD doctype > declarations (DITA), I have a catalog.xml that contains the correct > references for the entity resolver. DITA schemas refer to other > entities but I have no control over where it loads them from so I get > "Failed to read schema document > 'urn:oasis:names:tc:dita:xsd:commonElementGrp.xsd:1.2', because 1) > could not find the document; ...." > > btw I also have Saxon present. > > I have no clue what's the easiest way of using this catalog from > XQuery code. In Ant I always used <xmlcatalog> element as part of the > <xmlvalidate> task. > > Or, maybe I could set some system property that forces the use of a > specific catalog (I hope to avoid using one fixed catalog though). > > Ideas? > > > --Marc -- --Marc

2 3

(no subject)
by Marc van Grootel 26 Jan '15

26 Jan '15

Hi, I'm trying to run validations on XML files that use DTD doctype declarations (DITA), I have a catalog.xml that contains the correct references for the entity resolver. DITA schemas refer to other entities but I have no control over where it loads them from so I get "Failed to read schema document 'urn:oasis:names:tc:dita:xsd:commonElementGrp.xsd:1.2', because 1) could not find the document; ...." btw I also have Saxon present. I have no clue what's the easiest way of using this catalog from XQuery code. In Ant I always used <xmlcatalog> element as part of the <xmlvalidate> task. Or, maybe I could set some system property that forces the use of a specific catalog (I hope to avoid using one fixed catalog though). Ideas? --Marc

1 0

Library imports in latest snapshot BaseX 8.0 beta b39b77d
by Andy Bunce 26 Jan '15

26 Jan '15

Hi, Imports to library modules are giving modules not found errors for me. E.g. add a reference to the xquery module to restxq.xqm. /Andy (:~ : This module contains some basic examples for RESTXQ annotations : @author BaseX Team :) module namespace page = 'http://basex.org/modules/web-page'; *import module namespace xquery = "http://basex.org/modules/xquery <http://basex.org/modules/xquery>";* .... Stopped at C:/Users/Bunceax/Desktop/basex/webapp/restxq.xqm, 6/1: [XQST0059] Module 'http://basex.org/modules/xquery' not found.

2 1

DB inconsistency
by Marco Lettere 26 Jan '15

26 Jan '15

Hi all, we have an installation that has been running in production for nearly two months now. It's very insertion intensive and from time to time we need to run a db:optimize to keep it on track because the time it takes to serve a request turns from about 2 seconds to 100k seconds. But after the optimize it again runs with the expected performance. Today we noticed again a strange difficulty in serving requests so we tried the usual approach but we got this exception during the optimize: [SENR0001] Attributes cannot be serialized:attribute determinerCode {"INSTA.... Upon running an inspect command, we got a long list of EOF exceptions followed by the resume stating about 85K wrong parent/descendants relationships. Is there any idea on what could have gone wrongly and if possible any way to recover from this? Thank you very much. Regards, Marco. java.io.EOFException at java.io.RandomAccessFile.readFully(RandomAccessFile.java:446) at java.io.RandomAccessFile.readFully(RandomAccessFile.java:424) at org.basex.io.random.TableDiskAccess.readBlock(TableDiskAccess.java:525) at org.basex.io.random.TableDiskAccess.readPage(TableDiskAccess.java:489) at org.basex.io.random.TableDiskAccess.cursor(TableDiskAccess.java:468) at org.basex.io.random.TableDiskAccess.read1(TableDiskAccess.java:148) at org.basex.data.Data.kind(Data.java:290) at org.basex.core.cmd.Inspect.inspect(Inspect.java:50) at org.basex.core.cmd.Inspect.run(Inspect.java:27) at org.basex.core.Command.run(Command.java:360) at org.basex.core.Command.execute(Command.java:94) at org.basex.api.client.LocalSession.execute(LocalSession.java:122) at org.basex.api.client.Session.execute(Session.java:36) at org.basex.core.CLI.execute(CLI.java:106) at org.basex.core.CLI.execute(CLI.java:90) at org.basex.BaseX.console(BaseX.java:197) at org.basex.BaseX.<init>(BaseX.java:171) at org.basex.BaseX.main(BaseX.java:42) Checking main table (43626334 nodes): - 0 invalid node kinds - 0 invalid parent references - 84663 wrong parent/descendant relationships (pre: 6299342,..) Warning: Database is inconsistent. 'db' inspected in 21789.24 ms.

2 3

Automatic replication and fail-over scenario
by Andreas Jung 25 Jan '15

25 Jan '15

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hi there, we have a customer requirement that an application server will talk to an XML database. The production system (app server + db) will reside in a data center near the customer. However the customer requires having a fail-over installation in a different data center (in a different geographical location) in the case of disaster or whatever. As far as I see BaseX does not support multi-master replication. What are the best approaches for dealing with a fail-over where the fail-over database should have a copy of the production database that is as recent as possible? Andreas -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (Darwin) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iQGUBAEBAgAGBQJUxKtJAAoJEADcfz7u4AZjCjULwN2BnpIqhiBnaJc7dPVtbY+v aaeCYmbuCpACMZyY7qLpYBVpBFghAvEA9VIf7HSmNspGBIf+818vOIj2/Ure9keU jcgWIM53qa30Kv28TSt4R6sMPKFStrmWU1EqeDfklQo7bncU9XEL+iH4Z+CdsSZj FrIejK0abVgcHhgRluM80pSeJGkjelmgzMld67UQOv9tOPJueYezb72TtUyajchN EsObhAge6dk4oHwdIi+N2Ek4V9f/GqIvVrgWbXKpSapQxM07mX/HN+yEDWbX0oUZ SxcWzwJ+JS1vzKJtlI9+FZJpnP/Ta8pkqxFHKldWAt/lofdt1p7CxNxFdl9RZPQr yhchWZOs1lYHuYhN8iUCe8kO04TjDE7eoOKmZuc9AmgUZcd/pzLaVuiy4nyjLECZ gVkQpuDe1A2NtICBlh7Kjq4Wr+/OiTz+dnjdaZhbjJ9M2EN0I9SZIrTjq/VOjHD2 rdh0sD+vRwq6OeDM/WIYSx2cuV1riIk= =h79D -----END PGP SIGNATURE-----

2 1

infelicity in the default web.xml settings? and failure logging?
by C. M. Sperberg-McQueen 24 Jan '15

24 Jan '15

Experimenting a bit with the RESTXQ interface yesterday, using basexhttp I found that the default web.xml file shipped with BaseX 7.9 has an unfortunate property. The first servlet mapping listed is <servlet-mapping> <servlet-name>RESTXQ</servlet-name> <url-pattern>/*</url-pattern> </servlet-mapping> and the last servlet mapping is the default servlet, used for static data: <servlet-mapping> <servlet-name>default</servlet-name> <url-pattern>/static/*</url-pattern> </servlet-mapping> If I understand the Servlet specification correctly, this pattern of mappings ensures that a request for (say) a static resource will invariably be passed to the RESTXQ interface and not to RESTXQ. Note that there is some doubt as to whether I do understand the spec correctly: on my reading, a request for /rest/ should also be passed to RESTXQ, and it's clearly not. But I was unable to get Jetty to serve any static content with the default web.xml. I began to succeed when I moved the servlet mappings and changed their patterns, along the lines of the web.xml file I found with Andy Bunce's GraphXQ example, so the first and last are: <servlet-mapping> <servlet-name>default</servlet-name> <url-pattern>/</url-pattern>  </servlet-mapping> <servlet-mapping> <servlet-name>RESTXQ</servlet-name> <url-pattern>/restxq/*</url-pattern>  </servlet-mapping> There may be other ways to set the web.xml file up so that static resources are served (in particular, I don't think it's required to change "/static/*" to "/" -- but I have not experimented systematically with all options, because I'm interested in making my web application work, not in figuring out what relationship holds between the Servlet spec and Jetty's behavior. Is this a problem others have run into? Is there a better solution? And a related question -- before I changed the web.xml file (and after several unsuccessful changes), requests for any of /test.xhtml /mydirectory/test.xhtml /static/test.xhtml /static/mydirectory/test.xhtml elicited the response "Not found" (or sometimes "no function found"), but I found no entries for these requests in the BaseX logs. (And I was unsuccessful in attempting to turn Jetty's stderr or request logging on.) Are failures of this kind not written to the BaseX logs? If they aren't written to the BaseX logs, but would be written to Jetty logs, then I wonder if it would be helpful to have a few hints on turning Jetty logging on (and finding out where the log files get written) either in the BaseX documentation wiki or in comments in the default jetty.xml and web.xml. Since Jetty's documentation mostly assumes that its readers are Java programmers actively developing Java web apps, I for one found it unhelpful when trying to solve my problem. -- **************************************************************** * C. M. Sperberg-McQueen, Black Mesa Technologies LLC * http://www.blackmesatech.com * http://cmsmcq.com/mib * http://balisage.net ****************************************************************

3 5

Fw: Extracting a collection from BaseX
by Alex R. Loayza 23 Jan '15

23 Jan '15

Hello, I'm finding no easy way to extract the contents of a collection to a filesystem directory. For example, In database "myDB", I have a number of XML files inside a collection named "myCollection" (a subfolder of "myDB"). I'm looking for a command or function to extract those files to "myHardDriveDirectory" in my filesystem. I tried file:write(), but it works only with a single file. I also tried db:export(), but it outputs the entire database, which is overkill. So, I ended up doing this: for $doc in collection('myDB/myCollection')return file:write('D:\myHardDriveDirectory\'||$doc/pathToADateAttribute||' Report.xml', $doc) It works (I get a bunch of "2015-01-20 Report.xml" files), but I had to use a FLWOR and dynamically construct each of the names of the files to be written to disk. This could be made much simpler with something like: db:export('myDB/myCollection', 'D:\myHardDriveDirectory') I may be missing some other way to do this. But, in case there is no alternative, would you please consider enhancing db:export() to support outputting collections (or any equivalent solution you may deem better)? It would be great to have that shipped in BaseX 8. I had also set up Webdav to try the quick drag-and-drop way, but unfortunately it has several bugs and had to stop using it. Bonus question: Is there a way to do the reverse operation with some function/command? That is, get the XML files in a filesystem directory to replace any namesake XML files in a specific collection? db:replace() works with a single file only, according to the documentation. db:add() works with filesystem directories, but would create duplicate resources in the database, instead of replacing the existing ones. The only ways I've found so far are: First by using the GUI; I have to delete the entire collection and rebuild it with the "Add resources" dialog box. The second way is to use db:delete() on the collection and then db:add(). But both are lengthy/verbose ways of doing this. Is there already a better way? If not, do you think it may be possible to also enhance db:replace() for it to work with filesystem directories and database collections? Thank you very much for any help. Alex

2 1

BaseX 8 and oXygen via XQJ
by Lizzi, Vincent 23 Jan '15

23 Jan '15

After upgrading to BaseX 8.0 I'm no longer able to run queries on BaseX from oXygen. Connecting to BaseX from oXygen using WebDav still works, but trying to run a transformation in BaseX using the XQJ driver fails with a message "Access denied". It appears that the XQJ driver needs to be updated for the new authentication mechanism. Will an updated version of the BaseX XQJ driver be available soon? Thank you, Vincent

2 2

Pre-compilled Query in Base-X
by ankit kumar 23 Jan '15

23 Jan '15

Hi, I was exploring BaseX for my XML based product. I was wondering whether there is Prepared Statement Like facility incorporated in BaseX, With the same QueryProcessor instance, is it possible to execute it multiple times with different binding values for given variable. I am executing following code. QueryProcessor pr = new QueryProcessor("declare variable $a external ; $a",context); pr.context(itr); pr.bind("a","ABC"); Resultfinalr=pr.execute(); System.out.println("Final Execution Result 1 ::" + finalr); // rebinding $a with different value pr.bind("a", "XYZ"); Result out1 = pr.execute(); System.out.println("Final Execution Result 2 ::" + out1); ******************************************************************************* I am getting "ABC" when I am executing second time. I need this because I would be executing the same query multiple times. Thanks Anky

2 7

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

BaseX-Talk January 2015