BaseX-Talk

basex-talk@mailman.uni-konstanz.de

6 participants
5269 discussions

Expected behaviour with full disk
by James Ball 03 Apr '16

03 Apr '16

Hello, I’m having some problems with BaseX in situations where there it runs out of disk space. I’m delivering XML files to BaseX 8.4 using RestXQ. A request is made and the XML is added to a database depending on some criteria. I receive many hundreds of thousands of requests and everything works well unless I run out of disk space. What I would expect to happen is that BaseX would try to add the document, find it couldn’t due to the disk being full and return a server error (e.g. 500) and the transaction would be aborted leaving the database as it was before the request. Any further requests would all get errors returned. What I’m seeing is that when a request is made that causes the disk to become full BaseX gets part way through adding the document, fails and returns an error but also leaves the database corrupted. Usually the server has to be restarted and then any database that has been affected has to be deleted - it’s rare to be able to save anything from it. The errors vary but it is usually a NullPointer exception and then java.nio.channels.OverlappingFileLockException. for further requests. I’ve tried turning off logging and setting parallel to 1 to see if that made any difference but it doesn’t. Is what I’m seeing unique to my setup or do others see this? Is there anything I can do to make BaseX just stop accepting requests when disk space runs out to protect my databases? Many thanks, James

4 5

Internal optimization
by Gabriel Tessarolli 30 Mar '16

30 Mar '16

Hi, First of all, thank you for the excellent software you produce and maintain! Keep up the good work. I've been using BaseX for some academic experiments on XQuery processing, and I got this situation that you guys can probably explain. Here is some context: - I am using version 8.2.3. - Database 'expdb' was created with default options of that version, using 'auction.xml' document generated from Xmark benchmark [1]. - BaseX is running with default options too. - What the query does is irrelevant. When I execute this query: for $pe in doc('expdb/auction.xml')/site/people/person for $cat in doc('expdb/auction.xml')/site/categories/category[position() >= 1 and position() < 101] where count($pe/profile/interest) > 3 and $pe/profile/interest/@category = $cat/@id return <match> <person>{$pe/name}</person> <category>{$cat/name}</category> </match> the resulting optimized query (in 'Query Info' window, on GUI) is this: for $pe_0 in *db:open-pre*("expdb",0)/*:site/*:people/*:person[3.0 < count(*:profile/*:interest)] for $cat_1 in *db:open-pre*("expdb",0)/*:site/*:categories/*:category[position() = 1 to 100][(@*:id = $pe_0/*:profile/*:interest/@*:category)] return element match { (element person { ($pe_0/*:name) }, element category { ($cat_1/*:name) }) } If I change the original query to (note that I am only switching the position of 'for' clauses): for $cat in doc('expdb/auction.xml')/site/categories/category[position() >= 1 and position() < 101] for $pe in doc('expdb/auction.xml')/site/people/person where count($pe/profile/interest) > 3 and $pe/profile/interest/@category = $cat/@id return <match> <person>{$pe/name}</person> <category>{$cat/name}</category> </match> the optimized query changes to: for $cat_0 in *db:open-pre*("expdb",0)/*:site/*:categories/*:category[position() = 1 to 100] for $pe_1 in *db:attribute*("expdb", $cat_0/@*:id)/self::*:category/parent::*:interest/parent::*:profile/parent::*:person[3.0 < count(*:profile/*:interest)] return element match { (element person { ($pe_1/*:name) }, element category { ($cat_0/*:name) }) } You can see that BaseX was able to use attribute index to optimize last query, reducing its execution time (by a lot!). My question is: what is the explanation for this behavior? Thank you in advance! Gabriel Tessarolli --- [1] http://www.xml-benchmark.org/

2 2

detection of external variables
by Andy Bunce 30 Mar '16

30 Mar '16

Hi, I am looking for a way to determine for a module what external variables are defined. Neither inspect:module [1] or inspect:xqdoc seem to return this information. I wonder if adding an *external *attribute to the inspect:module o/p for variables (and functions?) would be a solution. e.g. <variable name="samples:test-string" uri="http://basex.org/modules/samples" type="xs:string" *external="true"*> <description>This is a sample string.</description> </variable> /Andy [1] http://docs.basex.org/wiki/Inspection_Module#inspect:module

2 4

Out of memory getting document from Client API
by buddyonweb-software＠yahoo.com 29 Mar '16

29 Mar '16

Using the client API I am trying to retrieve a 600MB file from BaseX. Version I am using is BaseX 8.3. I have successfully stored this document using the client API without issue. As an aside, I have update the basexserver script to include 4GB of memory as Java command line option. Here is the snippet from that script:# Options for virtual machine (can be extended by global options) BASEX_JVM="-Xmx4g $BASEX_JVM" In my Java client side application I have memory set to 3 GB of memory (script snippet below):exec "$JAVACMD" -Xms1g -Xmx3g .....rest of args..... Here is a snippet of my Java code. Note that this works just fine when operation on smaller documents, but I wanted to stress test the code. String query = "doc('theCollectionName/extract.xml')";ByteArrayOutputStream baos = new ByteArrayOutputStream(); ClientSession session = new ClientSession(host, port, user, pwd);session.setOutputStream(baos);ClientQuery cq = session.query(query); cq.execute(); However, I keep getting out of memory with the below stack trace. I would think that the amount of space I am providing in both thebasexserver script (4GB) and my client side app (3GB) would be plenty to read in a 600MB file from BaseX. Is there something I am overlookingor something I am not doing correctly? Any insights/feedback is appreciated. Exception in thread "queue" java.lang.OutOfMemoryError: Java heap space at java.lang.StringCoding$StringDecoder.decode(StringCoding.java:149) at java.lang.StringCoding.decode(StringCoding.java:193) at java.lang.StringCoding.decode(StringCoding.java:254) at java.lang.String.<init>(String.java:534) at java.io.ByteArrayOutputStream.toString(ByteArrayOutputStream.java:221) at org.basex.api.client.ClientSession.exec(ClientSession.java:261) at org.basex.api.client.ClientQuery.execute(ClientQuery.java:102)

1 0

REST-Interface /Openshift/ Security
by Günter Dunz-Wolff 29 Mar '16

29 Mar '16

Hi all, I'm in the beginning of a BaseX basexhttp installation on OpenShift. I'm using Openshift quick start for BaseX from Andy Bunce. For my tests, I'm working with a simple factbook-db. Everything is working, but I can't figure out, how to protect the database to be deleted via the REST-Interface: With a simple curl -i -X DELETE "http://xxx-xxx.rhcloud.com/rest/factbook" the Database is dropped. In the database, there is only the user admin. In web.xml the REST-servlet is configured with <servlet> <servlet-name>REST</servlet-name> <servlet-class>org.basex.http.rest.RESTServlet</servlet-class> <init-param> <param-name>org.basex.user</param-name> <param-value>admin</param-value> </init-param> <init-param> <param-name>org.basex.password</param-name> <param-value>xxx</param-value> </init-param> </servlet> <servlet-mapping> <servlet-name>REST</servlet-name> <url-pattern>/rest/*</url-pattern> </servlet-mapping> What part of the security-management am I missing? Thanks a lot for any help. Best regards, Günter Dunz-Wolff

2 2

async module vs xq-promise
by Maximilian Gärber 28 Mar '16

28 Mar '16

Hi all, I just started to use the async module and I guess I am out of sync about the relation to xq-promise: Basex now has asyn:fork-join which exectues in parallel and comes back with all results xq-promise has p:fork-join which seems to be the previous implementation Could I swap one for the other? Side note: async:eval which works like xquery:eval is great, but I have to do something like this for functions: ``` let $ids := async:eval("declare variable $f external; declare variable $name external; $f($name)", map { 'f': '_:get-meta', 'name' : $name }) return $ids ``` With get-meta being a local function that accepts a single parameter. Is there an easier way to do this? Regards, Max

2 1

Re: [basex-talk] using Xerces2 with schema-validation
by Christian Grün 22 Mar '16

22 Mar '16

Hi Rob (cc to the list), You may be better off using the ZIP distribution. Usually, it’s mostly the basex.jar and basex-api.jar files that need to be updated. Hope this helps, Christian On Tue, Mar 22, 2016 at 3:04 PM, Rob Stapper <r.stapper(a)lijbrandt.nl> wrote: > HI Christian, > > Oops, not too clever. But to my defense, in my original code I use an xsd-file starting with the xml-processor for defining the version: > <?xml version="1.1" encoding="UTF-8"?> > > It turns out that my ignorance of java's classpath and BaseX's installation procedure are the cause of this trouble: > - Environment-variable classpath is ignored when java is started with the -cp option > - Manually copying the Xerces-file to the BaseX-library and changes to the basexgui.bat-file[1] are getting undone when installing a new BaseX-version. > > My solution for the moment is to keep an edited basexgui.bat-file in my document directory which I copy to BaseX-bin directory every time I install a new Basex-version. > > What is the best strategy for preserving for example classpath-extensions when installing a new version of BaseX? > Is it an idea preserving all non BaseX batch-file in BaseX/bin-directory, in my case: "myBasexgui.bat", when executing the BaseX-installation-procedure? > > [1] line added to basexgui.bat: "for %%a in (L:\xerces-2_11_0-xml-schema-1.1-beta\*.jar) do set CP=!CP!;%%a" > > Take care, > Rob > > PS, calling Xerces2's validation works now as does XML-schema's assert. Thanx. > > -----Oorspronkelijk bericht----- > Van: Christian Grün [mailto:christian.gruen@gmail.com] > Verzonden: maandag 21 maart 2016 18:11 > Aan: Rob Stapper > CC: BaseX > Onderwerp: Re: [basex-talk] using Xerces2 with schema-validation > > Hi Rob, > > it seems you forgot the version argument? > > Cheers, > Christian > > > On Mon, Mar 21, 2016 at 12:06 PM, Rob Stapper <r.stapper(a)lijbrandt.nl> wrote: >> Hi, >> >> >> >> I would like to use xerces2 for schema-validation. See attached file, >> but I can’t get it activated from within BaseX. >> >> I’ve read the chapter on validation [1] and followed the instructions : >> copying the 4 files to the library. >> >> I can see that these 4 file are part of the –cp option when calling java. >> >> But the use of assert-element while validating gives an error: >> >> >> >> Stopped at L:/Program Files/BaseX/demo-xsd-validation.xq, 1/14: >> >> [bxerr:BXVA0001] Validation failed: >> file:/C:/Users/Rob/AppData/Local/Temp/BaseX-375999151462208291.tmp, 6:33: >> s4s-elt-invalid-content.1: The content of '#AnonType_assert-demo' is >> invalid. Element 'assert' is invalid, misplaced, or occurs too often. >> >> >> >> [1] http://docs.basex.org/wiki/Validation_Module#XML_Schema_Validation >> >> >> >> What am I doing wrong? >> >> >> >> TIA for your response, >> >> Rob >> >> >> >> PS I’m using the latest or one of the latest snapshot and java 1.8 >> runtime on a windows-Vista machine >

2 1

Calling XLST Module from Java
by Taschner, Jan 22 Mar '16

22 Mar '16

Hello Guys, I´ve been messing around with BaseX for a few days and now I am kind of stuck. I have a XSL 2.0 Transformation sheet which works perfectly fine for a standalone application using Saxon HE. Now I tried to use the XSLT module of BaseX with Java. Here is my code: import java.io.IOException; import org.basex.core.Context; import org.basex.core.cmd.CreateDB; import org.basex.core.cmd.List; import org.basex.core.cmd.XQuery; public class App { public static void main(String[] args) { try { // initialize timer final long time = System.nanoTime(); // create session final BaseXClient session = new BaseXClient("localhost", 1984, "admin", "admin"); System.out.println(session.execute("info")); // create db context Context ctx = new Context(); // create db new CreateDB("stammdaten", "stammdaten2.xml").execute(ctx); // show all databases System.out.print(new List().execute(ctx)); // xls transformation System.setProperty("javax.xml.transform.TransformerFactory", "net.sf.saxon.TransformerFactoryImpl"); String query = "declare option output:method 'xml';" + " declare option db:chop 'no';" + " let $in := db:open('stammdaten')" + " let $style := doc('template.xsl')" + " return xslt:transform($in, $style)"; String result = new XQuery(query).execute(ctx); System.out.println("Result: " + result); ctx.close(); // close session session.close(); // print time needed final double ms = (System.nanoTime() - time) / 1000000d; System.out.println("\n\n" + ms + " ms"); } catch(final IOException ex) { // print exception ex.printStackTrace(); } } } I have started the basexhttp.bat in the bin directory and executed the above code. I get the following console output and stack trace: General Information: Version: 8.4.1 Used Memory: 20405 KB Global options: ASYNCTIMEOUT: 3600 AUTHMETHOD: Basic CACHERESTXQ: false DBPATH: C:/Program Files (x86)/BaseX/data DEBUG: false GLOBALLOCK: false HOST: localhost HTTPLOCAL: false IGNORECERT: false KEEPALIVE: 600 LANG: English LANGKEYS: false LOG: true LOGMSGMAXLEN: 1000 NONPROXYHOSTS: PARALLEL: 8 PASSWORD: PORT: 1984 PROXYHOST: PROXYPORT: 0 REPOPATH: C:/Program Files (x86)/BaseX/repo RESTPATH: RESTXQPATH: SERVERHOST: SERVERPORT: 1984 STOPPORT: 8985 TIMEOUT: 30 USER: WEBPATH: C:\Program Files (x86)\BaseX\webapp Local options ADDARCHIVES: true ADDCACHE: false ADDRAW: false ARCHIVENAME: false ATTRINCLUDE: ATTRINDEX: true AUTOFLUSH: true AUTOOPTIMIZE: false BINDINGS: CASESENS: false CATFILE: CHECKSTRINGS: true CHOP: true COMPPLAN: true CREATEFILTER: *.xml CREATEONLY: false CSVPARSER: DEFAULTDB: false DIACRITICS: false DOTCOMPACT: false DOTPLAN: false DTD: false EXPORTER: FORCECREATE: false FTINCLUDE: FTINDEX: false HTMLPARSER: INLINELIMIT: 100 INTPARSE: false JSONPARSER: LANGUAGE: en LSERROR: 0 MAINMEM: false MAXCATS: 100 MAXLEN: 96 MAXSTAT: 30 MIXUPDATES: false PARSER: xml QUERYINFO: false QUERYPATH: RUNQUERY: true RUNS: 1 SERIALIZE: true SERIALIZER: SKIPCORRUPT: false SPLITSIZE: 0 STEMMING: false STOPWORDS: STRIPNS: false TAILCALLS: 256 TEXTINCLUDE: TEXTINDEX: true TEXTPARSER: TOKENINCLUDE: TOKENINDEX: false UPDINDEX: false WRITEBACK: false XINCLUDE: true XMLPLAN: false Name Resources Size Input Path ----------------------------------------------------------------------------------------------------------------------------- stammdaten 1 812176 D:/Projekte/XYZ/workspaces/workspace_poc/BaseX/stammdaten2.xml 1 database(s). org.basex.core.BaseXException: Stopped at ., 1/154: [FODC0002] "" (Line 1): Vorzeitiges Dateiende. at org.basex.core.Command.execute(Command.java:101) at org.basex.core.Command.execute(Command.java:123) at net.atos.epi.XMLTransformationProofOfConceptBaseX.App.main(App.java:31) Caused by: org.basex.query.QueryException: "" (Line 1): Vorzeitiges Dateiende. at org.basex.query.QueryError.get(QueryError.java:1344) at org.basex.query.func.xslt.XsltTransform.item(XsltTransform.java:39) at org.basex.query.expr.ParseExpr.iter(ParseExpr.java:44) at org.basex.query.MainModule.iter(MainModule.java:110) at org.basex.query.QueryContext.iter(QueryContext.java:342) at org.basex.query.QueryProcessor.iter(QueryProcessor.java:81) at org.basex.core.cmd.AQuery.query(AQuery.java:90) at org.basex.core.cmd.XQuery.run(XQuery.java:22) at org.basex.core.Command.run(Command.java:398) at org.basex.core.Command.execute(Command.java:100) ... 2 more Caused by: java.io.IOException: "" (Line 1): Vorzeitiges Dateiende. at org.basex.build.xml.SAXWrapper.parse(SAXWrapper.java:83) at org.basex.build.SingleParser.parse(SingleParser.java:33) at org.basex.build.Builder.parse(Builder.java:73) at org.basex.build.MemBuilder.dataClip(MemBuilder.java:77) at org.basex.build.MemBuilder.build(MemBuilder.java:66) at org.basex.build.MemBuilder.build(MemBuilder.java:61) at org.basex.build.MemBuilder.build(MemBuilder.java:50) at org.basex.build.MemBuilder.build(MemBuilder.java:40) at org.basex.query.value.node.DBNode.<init>(DBNode.java:85) at org.basex.query.func.xslt.XsltTransform.item(XsltTransform.java:37) ... 10 more Caused by: org.xml.sax.SAXParseException; systemId: file:///D:/Projekte/XYZ/workspaces/workspace_poc/BaseX/; lineNumber: 1; columnNumber: 1; Vorzeitiges Dateiende. at com.sun.org.apache.xerces.internal.util.ErrorHandlerWrapper.createSAXParseException(Unknown Source) at com.sun.org.apache.xerces.internal.util.ErrorHandlerWrapper.fatalError(Unknown Source) at com.sun.org.apache.xerces.internal.impl.XMLErrorReporter.reportError(Unknown Source) at com.sun.org.apache.xerces.internal.impl.XMLErrorReporter.reportError(Unknown Source) at com.sun.org.apache.xerces.internal.impl.XMLScanner.reportFatalError(Unknown Source) at com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl$PrologDriver.next(Unknown Source) at com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(Unknown Source) at com.sun.org.apache.xerces.internal.impl.XMLNSDocumentScannerImpl.next(Unknown Source) at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(Unknown Source) at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(Unknown Source) at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(Unknown Source) at com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(Unknown Source) at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(Unknown Source) at com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl$JAXPSAXParser.parse(Unknown Source) at org.basex.build.xml.SAXWrapper.parse(SAXWrapper.java:78) ... 19 more (Vorzeitiges Dateiende = Premature end of file) When I run my standalone application with stammdaten2.xml and my template.xsl everything is fine, so I doubt that the stylesheet is the problem. The stylesheet is rather complex and consists of several subtemplates, which are included with <xsl: include href="xyz.xsl" />. Running the XQuery in the BaseX GUI does not work, since the GUI only provides support for XLST 1.0. Is there a way to change that? The stacktrace tells me that the system tries to open and parse an empty file, but I have no idea why and which file causes the problem. My assumption is the template.xsl causes the problem, since it said that it "Stopped at ." I copied the stylesheets in every folder that was reasonable for me, but I still got this error. Please tell me what I did wrong. Best regards, Jan Taschner

2 3

using Xerces2 with schema-validation
by Rob Stapper 21 Mar '16

21 Mar '16

Hi, I would like to use xerces2 for schema-validation. See attached file, but I can't get it activated from within BaseX. I've read the chapter on validation [1] and followed the instructions : copying the 4 files to the library. I can see that these 4 file are part of the -cp option when calling java. But the use of assert-element while validating gives an error: Stopped at L:/Program Files/BaseX/demo-xsd-validation.xq, 1/14: [bxerr:BXVA0001] Validation failed: file:/C:/Users/Rob/AppData/Local/Temp/BaseX-375999151462208291.tmp, 6:33: s4s-elt-invalid-content.1: The content of '#AnonType_assert-demo' is invalid. Element 'assert' is invalid, misplaced, or occurs too often. [1] http://docs.basex.org/wiki/Validation_Module#XML_Schema_Validation What am I doing wrong? TIA for your response, Rob PS I'm using the latest or one of the latest snapshot and java 1.8 runtime on a windows-Vista machine

2 1

fn:normalize-space question, BaseX v8.4.1
by Bridger Dyson-Smith 21 Mar '16

21 Mar '16

Hi all, I'm running into what looks some odd behavior with fn:normalize-space -- or a gross user/operator error :) -- and was hoping to get some clarification. I've attached a Very Simple Sample Query, as well as an example input document. Note: I'm using BaseX as a query processor, not as a db, but maybe this has to do with a whitespace setting (thanks to the xml IRC channel for the help with this). When I run the attached xq with Saxon (9.1.0.8J), I get the following expected result: fn:normalize-space: Letter, William H. Dennis in Knoxville, TN, to his brother David, 1863 December 12 When I run the attached xq with BaseX (v8.4.1, gui), I get the following unexpected result: fn:normalize-space: Letter,William H. DennisinKnoxville, TN, to his brother David,1863 December 12 Could someone give me an idea about what is happening with this? I'm not sure why the whitespace is chopped like this. Thanks in advance for your time and trouble. Best, Bridger

2 1

Jump to page:

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

BaseX-Talk