- BaseX-Talk - mailman.uni-konstanz.de

Out of memory getting document from Client API
by buddyonweb-software＠yahoo.com 29 Mar '16

29 Mar '16

Using the client API I am trying to retrieve a 600MB file from BaseX. Version I am using is BaseX 8.3. I have successfully stored this document using the client API without issue. As an aside, I have update the basexserver script to include 4GB of memory as Java command line option. Here is the snippet from that script:# Options for virtual machine (can be extended by global options) BASEX_JVM="-Xmx4g $BASEX_JVM" In my Java client side application I have memory set to 3 GB of memory (script snippet below):exec "$JAVACMD" -Xms1g -Xmx3g .....rest of args..... Here is a snippet of my Java code. Note that this works just fine when operation on smaller documents, but I wanted to stress test the code. String query = "doc('theCollectionName/extract.xml')";ByteArrayOutputStream baos = new ByteArrayOutputStream(); ClientSession session = new ClientSession(host, port, user, pwd);session.setOutputStream(baos);ClientQuery cq = session.query(query); cq.execute(); However, I keep getting out of memory with the below stack trace. I would think that the amount of space I am providing in both thebasexserver script (4GB) and my client side app (3GB) would be plenty to read in a 600MB file from BaseX. Is there something I am overlookingor something I am not doing correctly? Any insights/feedback is appreciated. Exception in thread "queue" java.lang.OutOfMemoryError: Java heap space at java.lang.StringCoding$StringDecoder.decode(StringCoding.java:149) at java.lang.StringCoding.decode(StringCoding.java:193) at java.lang.StringCoding.decode(StringCoding.java:254) at java.lang.String.<init>(String.java:534) at java.io.ByteArrayOutputStream.toString(ByteArrayOutputStream.java:221) at org.basex.api.client.ClientSession.exec(ClientSession.java:261) at org.basex.api.client.ClientQuery.execute(ClientQuery.java:102)

1 0

REST-Interface /Openshift/ Security
by Günter Dunz-Wolff 29 Mar '16

29 Mar '16

Hi all, I'm in the beginning of a BaseX basexhttp installation on OpenShift. I'm using Openshift quick start for BaseX from Andy Bunce. For my tests, I'm working with a simple factbook-db. Everything is working, but I can't figure out, how to protect the database to be deleted via the REST-Interface: With a simple curl -i -X DELETE "http://xxx-xxx.rhcloud.com/rest/factbook" the Database is dropped. In the database, there is only the user admin. In web.xml the REST-servlet is configured with <servlet> <servlet-name>REST</servlet-name> <servlet-class>org.basex.http.rest.RESTServlet</servlet-class> <init-param> <param-name>org.basex.user</param-name> <param-value>admin</param-value> </init-param> <init-param> <param-name>org.basex.password</param-name> <param-value>xxx</param-value> </init-param> </servlet> <servlet-mapping> <servlet-name>REST</servlet-name> <url-pattern>/rest/*</url-pattern> </servlet-mapping> What part of the security-management am I missing? Thanks a lot for any help. Best regards, Günter Dunz-Wolff

2 2

async module vs xq-promise
by Maximilian Gärber 28 Mar '16

28 Mar '16

Hi all, I just started to use the async module and I guess I am out of sync about the relation to xq-promise: Basex now has asyn:fork-join which exectues in parallel and comes back with all results xq-promise has p:fork-join which seems to be the previous implementation Could I swap one for the other? Side note: async:eval which works like xquery:eval is great, but I have to do something like this for functions: ``` let $ids := async:eval("declare variable $f external; declare variable $name external; $f($name)", map { 'f': '_:get-meta', 'name' : $name }) return $ids ``` With get-meta being a local function that accepts a single parameter. Is there an easier way to do this? Regards, Max

2 1

Re: [basex-talk] using Xerces2 with schema-validation
by Christian Grün 22 Mar '16

22 Mar '16

Hi Rob (cc to the list), You may be better off using the ZIP distribution. Usually, it’s mostly the basex.jar and basex-api.jar files that need to be updated. Hope this helps, Christian On Tue, Mar 22, 2016 at 3:04 PM, Rob Stapper <r.stapper(a)lijbrandt.nl> wrote: > HI Christian, > > Oops, not too clever. But to my defense, in my original code I use an xsd-file starting with the xml-processor for defining the version: > <?xml version="1.1" encoding="UTF-8"?> > > It turns out that my ignorance of java's classpath and BaseX's installation procedure are the cause of this trouble: > - Environment-variable classpath is ignored when java is started with the -cp option > - Manually copying the Xerces-file to the BaseX-library and changes to the basexgui.bat-file[1] are getting undone when installing a new BaseX-version. > > My solution for the moment is to keep an edited basexgui.bat-file in my document directory which I copy to BaseX-bin directory every time I install a new Basex-version. > > What is the best strategy for preserving for example classpath-extensions when installing a new version of BaseX? > Is it an idea preserving all non BaseX batch-file in BaseX/bin-directory, in my case: "myBasexgui.bat", when executing the BaseX-installation-procedure? > > [1] line added to basexgui.bat: "for %%a in (L:\xerces-2_11_0-xml-schema-1.1-beta\*.jar) do set CP=!CP!;%%a" > > Take care, > Rob > > PS, calling Xerces2's validation works now as does XML-schema's assert. Thanx. > > -----Oorspronkelijk bericht----- > Van: Christian Grün [mailto:christian.gruen@gmail.com] > Verzonden: maandag 21 maart 2016 18:11 > Aan: Rob Stapper > CC: BaseX > Onderwerp: Re: [basex-talk] using Xerces2 with schema-validation > > Hi Rob, > > it seems you forgot the version argument? > > Cheers, > Christian > > > On Mon, Mar 21, 2016 at 12:06 PM, Rob Stapper <r.stapper(a)lijbrandt.nl> wrote: >> Hi, >> >> >> >> I would like to use xerces2 for schema-validation. See attached file, >> but I can’t get it activated from within BaseX. >> >> I’ve read the chapter on validation [1] and followed the instructions : >> copying the 4 files to the library. >> >> I can see that these 4 file are part of the –cp option when calling java. >> >> But the use of assert-element while validating gives an error: >> >> >> >> Stopped at L:/Program Files/BaseX/demo-xsd-validation.xq, 1/14: >> >> [bxerr:BXVA0001] Validation failed: >> file:/C:/Users/Rob/AppData/Local/Temp/BaseX-375999151462208291.tmp, 6:33: >> s4s-elt-invalid-content.1: The content of '#AnonType_assert-demo' is >> invalid. Element 'assert' is invalid, misplaced, or occurs too often. >> >> >> >> [1] http://docs.basex.org/wiki/Validation_Module#XML_Schema_Validation >> >> >> >> What am I doing wrong? >> >> >> >> TIA for your response, >> >> Rob >> >> >> >> PS I’m using the latest or one of the latest snapshot and java 1.8 >> runtime on a windows-Vista machine >

2 1

Calling XLST Module from Java
by Taschner, Jan 22 Mar '16

22 Mar '16

Hello Guys, I´ve been messing around with BaseX for a few days and now I am kind of stuck. I have a XSL 2.0 Transformation sheet which works perfectly fine for a standalone application using Saxon HE. Now I tried to use the XSLT module of BaseX with Java. Here is my code: import java.io.IOException; import org.basex.core.Context; import org.basex.core.cmd.CreateDB; import org.basex.core.cmd.List; import org.basex.core.cmd.XQuery; public class App { public static void main(String[] args) { try { // initialize timer final long time = System.nanoTime(); // create session final BaseXClient session = new BaseXClient("localhost", 1984, "admin", "admin"); System.out.println(session.execute("info")); // create db context Context ctx = new Context(); // create db new CreateDB("stammdaten", "stammdaten2.xml").execute(ctx); // show all databases System.out.print(new List().execute(ctx)); // xls transformation System.setProperty("javax.xml.transform.TransformerFactory", "net.sf.saxon.TransformerFactoryImpl"); String query = "declare option output:method 'xml';" + " declare option db:chop 'no';" + " let $in := db:open('stammdaten')" + " let $style := doc('template.xsl')" + " return xslt:transform($in, $style)"; String result = new XQuery(query).execute(ctx); System.out.println("Result: " + result); ctx.close(); // close session session.close(); // print time needed final double ms = (System.nanoTime() - time) / 1000000d; System.out.println("\n\n" + ms + " ms"); } catch(final IOException ex) { // print exception ex.printStackTrace(); } } } I have started the basexhttp.bat in the bin directory and executed the above code. I get the following console output and stack trace: General Information: Version: 8.4.1 Used Memory: 20405 KB Global options: ASYNCTIMEOUT: 3600 AUTHMETHOD: Basic CACHERESTXQ: false DBPATH: C:/Program Files (x86)/BaseX/data DEBUG: false GLOBALLOCK: false HOST: localhost HTTPLOCAL: false IGNORECERT: false KEEPALIVE: 600 LANG: English LANGKEYS: false LOG: true LOGMSGMAXLEN: 1000 NONPROXYHOSTS: PARALLEL: 8 PASSWORD: PORT: 1984 PROXYHOST: PROXYPORT: 0 REPOPATH: C:/Program Files (x86)/BaseX/repo RESTPATH: RESTXQPATH: SERVERHOST: SERVERPORT: 1984 STOPPORT: 8985 TIMEOUT: 30 USER: WEBPATH: C:\Program Files (x86)\BaseX\webapp Local options ADDARCHIVES: true ADDCACHE: false ADDRAW: false ARCHIVENAME: false ATTRINCLUDE: ATTRINDEX: true AUTOFLUSH: true AUTOOPTIMIZE: false BINDINGS: CASESENS: false CATFILE: CHECKSTRINGS: true CHOP: true COMPPLAN: true CREATEFILTER: *.xml CREATEONLY: false CSVPARSER: DEFAULTDB: false DIACRITICS: false DOTCOMPACT: false DOTPLAN: false DTD: false EXPORTER: FORCECREATE: false FTINCLUDE: FTINDEX: false HTMLPARSER: INLINELIMIT: 100 INTPARSE: false JSONPARSER: LANGUAGE: en LSERROR: 0 MAINMEM: false MAXCATS: 100 MAXLEN: 96 MAXSTAT: 30 MIXUPDATES: false PARSER: xml QUERYINFO: false QUERYPATH: RUNQUERY: true RUNS: 1 SERIALIZE: true SERIALIZER: SKIPCORRUPT: false SPLITSIZE: 0 STEMMING: false STOPWORDS: STRIPNS: false TAILCALLS: 256 TEXTINCLUDE: TEXTINDEX: true TEXTPARSER: TOKENINCLUDE: TOKENINDEX: false UPDINDEX: false WRITEBACK: false XINCLUDE: true XMLPLAN: false Name Resources Size Input Path ----------------------------------------------------------------------------------------------------------------------------- stammdaten 1 812176 D:/Projekte/XYZ/workspaces/workspace_poc/BaseX/stammdaten2.xml 1 database(s). org.basex.core.BaseXException: Stopped at ., 1/154: [FODC0002] "" (Line 1): Vorzeitiges Dateiende. at org.basex.core.Command.execute(Command.java:101) at org.basex.core.Command.execute(Command.java:123) at net.atos.epi.XMLTransformationProofOfConceptBaseX.App.main(App.java:31) Caused by: org.basex.query.QueryException: "" (Line 1): Vorzeitiges Dateiende. at org.basex.query.QueryError.get(QueryError.java:1344) at org.basex.query.func.xslt.XsltTransform.item(XsltTransform.java:39) at org.basex.query.expr.ParseExpr.iter(ParseExpr.java:44) at org.basex.query.MainModule.iter(MainModule.java:110) at org.basex.query.QueryContext.iter(QueryContext.java:342) at org.basex.query.QueryProcessor.iter(QueryProcessor.java:81) at org.basex.core.cmd.AQuery.query(AQuery.java:90) at org.basex.core.cmd.XQuery.run(XQuery.java:22) at org.basex.core.Command.run(Command.java:398) at org.basex.core.Command.execute(Command.java:100) ... 2 more Caused by: java.io.IOException: "" (Line 1): Vorzeitiges Dateiende. at org.basex.build.xml.SAXWrapper.parse(SAXWrapper.java:83) at org.basex.build.SingleParser.parse(SingleParser.java:33) at org.basex.build.Builder.parse(Builder.java:73) at org.basex.build.MemBuilder.dataClip(MemBuilder.java:77) at org.basex.build.MemBuilder.build(MemBuilder.java:66) at org.basex.build.MemBuilder.build(MemBuilder.java:61) at org.basex.build.MemBuilder.build(MemBuilder.java:50) at org.basex.build.MemBuilder.build(MemBuilder.java:40) at org.basex.query.value.node.DBNode.<init>(DBNode.java:85) at org.basex.query.func.xslt.XsltTransform.item(XsltTransform.java:37) ... 10 more Caused by: org.xml.sax.SAXParseException; systemId: file:///D:/Projekte/XYZ/workspaces/workspace_poc/BaseX/; lineNumber: 1; columnNumber: 1; Vorzeitiges Dateiende. at com.sun.org.apache.xerces.internal.util.ErrorHandlerWrapper.createSAXParseException(Unknown Source) at com.sun.org.apache.xerces.internal.util.ErrorHandlerWrapper.fatalError(Unknown Source) at com.sun.org.apache.xerces.internal.impl.XMLErrorReporter.reportError(Unknown Source) at com.sun.org.apache.xerces.internal.impl.XMLErrorReporter.reportError(Unknown Source) at com.sun.org.apache.xerces.internal.impl.XMLScanner.reportFatalError(Unknown Source) at com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl$PrologDriver.next(Unknown Source) at com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(Unknown Source) at com.sun.org.apache.xerces.internal.impl.XMLNSDocumentScannerImpl.next(Unknown Source) at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(Unknown Source) at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(Unknown Source) at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(Unknown Source) at com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(Unknown Source) at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(Unknown Source) at com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl$JAXPSAXParser.parse(Unknown Source) at org.basex.build.xml.SAXWrapper.parse(SAXWrapper.java:78) ... 19 more (Vorzeitiges Dateiende = Premature end of file) When I run my standalone application with stammdaten2.xml and my template.xsl everything is fine, so I doubt that the stylesheet is the problem. The stylesheet is rather complex and consists of several subtemplates, which are included with <xsl: include href="xyz.xsl" />. Running the XQuery in the BaseX GUI does not work, since the GUI only provides support for XLST 1.0. Is there a way to change that? The stacktrace tells me that the system tries to open and parse an empty file, but I have no idea why and which file causes the problem. My assumption is the template.xsl causes the problem, since it said that it "Stopped at ." I copied the stylesheets in every folder that was reasonable for me, but I still got this error. Please tell me what I did wrong. Best regards, Jan Taschner

2 3

using Xerces2 with schema-validation
by Rob Stapper 21 Mar '16

21 Mar '16

Hi, I would like to use xerces2 for schema-validation. See attached file, but I can't get it activated from within BaseX. I've read the chapter on validation [1] and followed the instructions : copying the 4 files to the library. I can see that these 4 file are part of the -cp option when calling java. But the use of assert-element while validating gives an error: Stopped at L:/Program Files/BaseX/demo-xsd-validation.xq, 1/14: [bxerr:BXVA0001] Validation failed: file:/C:/Users/Rob/AppData/Local/Temp/BaseX-375999151462208291.tmp, 6:33: s4s-elt-invalid-content.1: The content of '#AnonType_assert-demo' is invalid. Element 'assert' is invalid, misplaced, or occurs too often. [1] http://docs.basex.org/wiki/Validation_Module#XML_Schema_Validation What am I doing wrong? TIA for your response, Rob PS I'm using the latest or one of the latest snapshot and java 1.8 runtime on a windows-Vista machine

2 1

fn:normalize-space question, BaseX v8.4.1
by Bridger Dyson-Smith 21 Mar '16

21 Mar '16

Hi all, I'm running into what looks some odd behavior with fn:normalize-space -- or a gross user/operator error :) -- and was hoping to get some clarification. I've attached a Very Simple Sample Query, as well as an example input document. Note: I'm using BaseX as a query processor, not as a db, but maybe this has to do with a whitespace setting (thanks to the xml IRC channel for the help with this). When I run the attached xq with Saxon (9.1.0.8J), I get the following expected result: fn:normalize-space: Letter, William H. Dennis in Knoxville, TN, to his brother David, 1863 December 12 When I run the attached xq with BaseX (v8.4.1, gui), I get the following unexpected result: fn:normalize-space: Letter,William H. DennisinKnoxville, TN, to his brother David,1863 December 12 Could someone give me an idea about what is happening with this? I'm not sure why the whitespace is chopped like this. Thanks in advance for your time and trouble. Best, Bridger

2 1

Parsing XML within string constructors
by Tim Thompson 21 Mar '16

21 Mar '16

Hello, I have a string that I want to parse using fn:parse-xml-fragment (in BaseX 8.4.1). I noticed that there seemed to be a difference when wrapping the string in a string constructor versus plain quotes. Using the string constructor seems to require a second function call in order to parse the XML. The following three expressions illustrate the difference. Numbers 2 and 3 return parsed XML, whereas number 1 returns the original, unparsed string. Is this the expected behavior when using string constructors? 1. ``[Sed ut perspiciatis]`` => parse-xml-fragment() 2. ``[Sed ut perspiciatis]`` => parse-xml-fragment() => parse-xml-fragment() 3. 'Sed ut perspiciatis' => parse-xml-fragment() -- Tim A. Thompson Metadata Librarian (Spanish/Portuguese Specialty) Princeton University Library

2 1

Catalog Resolution Under Windows
by Eliot Kimber 21 Mar '16

21 Mar '16

I'm trying to make BaseX work under Windows 7 and I don't seem to be able to get catalog resolution to work. (I'm doing a workshop in Japan and the classroom only has 32-bit Windows machines available--since Docker requires 64-bit Windows I'm having to scramble to make the same code work directly under Windows 7 32-bit--ugh.) I'm using BaseX 8.4.1 with Java 8 (the Java supplied with the 32-bit version of oXygenXML). In my .basex file I have these entries: CATFILE = C:/workspace/DITA-OT2.x/catalog-dita.xml" DTD = true SKIPCORRUPT = true CHOP = false Using the DBA Web app I can see that the CATFILE property is set to that value, DTD is checked, CHOP is unchecked, and SKIPCORRUPT is checked, so my settings are clearly being used. However, if I create a database and use the DBA app to load a document that uses a DTD mapped by the catalog (e.g., a DITA document), load fails with a "Can't resolve DTD" message. The document is valid according to oXygen (and it's the same catalog--this is the OT oXygen is using) and of course my OS X and Docker-based versions of the same setup work fine, so it looks like a Windows-specific issue. Is there any known issue with catalog resolution under Windows? Is there anything I can do to try to debug the problem? Thanks, Eliot ---- Eliot Kimber, Owner Contrext, LLC http://contrext.com

2 3

Standalone RUN for .bxs file not working in version 8.4
by Amanda Galtman 21 Mar '16

21 Mar '16

Hi, I'm unable to use RUN in standalone mode in BaseX 8.4, including the latest downloadable snapshot dated yesterday. The same thing in BaseX 8.3 works fine. java -cp BaseX842-20160310.184258.jar org.basex.BaseX C:/Users/username/.basex: writing new configuration file. BaseX 8.4.2 beta 9b9f74a [Standalone] Try 'help' to get more information. > RUN blah.bxs Stopped at ., 1/20: [XPST0003] Incomplete expression. The reference to "1/20" comes up no matter which .bxs file I try to run, even if line 1 is a comment. Did I miss a deliberate change in configuration or syntax? Thanks, Amanda

2 4