- BaseX-Talk - mailman.uni-konstanz.de

OutOfMemoryError at Query#more()
by Simon Chatelain 22 Sep '17

22 Sep '17

Hello, I am facing an issue while retrieving some big amount of XML documents from a BaseX collection. Each document (as an XML file) is around 10 KB, and in the problematic case I must retrieve around 70000 of them. I am using Session#query(String query) then Query#more() and Query#next() to iterate through the result of my query. try (final Query query = l_Session.query(“query”)) { while (query.more()) { String xml = query.next(); } } If there is more than a certain amount of XML document in the result of my query I get a OutOfMemoryError (full stack trace in attached file) when executing query.more(). I did the test with BaseX 8.6.6 and 8.6.7, Java 8, VM arguments –Xmx1024m Increasing the Xmx value is not a solution as I don’t know what the maximum amount of data I will have to retrieve in the future. So what I need is a reliable way of executing such queries and iterate through the result without exploding the heap size. I also try to use QueryProcessor and QueryProcessor#iter() instead of Session#query(String query). But is it safe to use it knowing that my application is multithreaded and that each thread has its own session to query or add elements from/to multiple collections? Moreover, for now all access to BaseX are done through a session, so my application can run with an embedded BaseX or with a BaseX server. If I start using QueryProcessor, then it will be embedded BaseX only, right? I also attached a simple example showing the problem. Any advice would be much appreciated Thanks Simon

3 6

Server Variables, cached vars, etc
by Erik Peterson 21 Sep '17

21 Sep '17

How can I create a variable that is evaluated only once but accessed across many RestXQ requests and sessions. I'm trying to cache data that comes from an integration with an expensive operation. Does BaseX support something similar to server variables like Mark Logic?

6 15

BaseX 8.6.7
by Christian Grün 21 Sep '17

21 Sep '17

Hi all, As we’ll tackle BaseX 8.7 in the next weeks, we’ve released yet another patch release today [1]. As usual, it contains minor performance tweaks (better typing in FLWOR expressions) and bug fixes (pre-evaluation of fn:tail, GUI XQuery highlighting). Have fun, Christian [1] http://basex.org/download/

1 0

DBA deployed on JBoss (Wildfly)
by Marco Lettere 21 Sep '17

21 Sep '17

Hi all, We created an application web app based on the BaseX .WAR distribution. If we deploy it under tomcat we can access all DBA related functionalities after the login prompt. If we try to do the same under WildFly 10, the application starts without errors but trying to access the DBA pages the browser is just bounced back to the login page. Please check the snippets at [1] and [2] to observe the behaviour reflected in the log files. After arriving to the final page one more request to login?path=databases is received. Is there anyone having experience with running BaseX under Jboss / Wildfly? Thanks a lot, Marco. [1] Tomcat DBA login 18:49:13.754 0:0:0:0:0:0:0:1:55352 admin 302 76.47 ms 18:49:13.754 0:0:0:0:0:0:0:1:55352 admin 200 0.1 ms 18:49:13.756 0:0:0:0:0:0:0:1:55352 admin REQUEST [GET] http://localhost:8080/mywebapp/dba/databases 18:49:13.778 0:0:0:0:0:0:0:1:55352 admin 302 21.5 ms 18:49:13.778 0:0:0:0:0:0:0:1:55352 admin 200 0.08 ms 18:49:13.780 0:0:0:0:0:0:0:1:55352 admin REQUEST [GET] http://localhost:8080/mywebapp/dba/login?path=databases 18:49:13.792 0:0:0:0:0:0:0:1:55352 admin 200 11.91 ms 18:49:13.806 0:0:0:0:0:0:0:1:55352 admin REQUEST [GET] http://localhost:8080/mywebapp/dba/static/style.css 18:49:13.808 0:0:0:0:0:0:0:1:55353 admin REQUEST [GET] http://localhost:8080/mywebapp/dba/static/js.js 18:49:13.821 0:0:0:0:0:0:0:1:55352 admin 200 15.94 ms 18:49:13.824 0:0:0:0:0:0:0:1:55353 admin 200 16.12 ms 18:49:13.827 0:0:0:0:0:0:0:1:55352 admin REQUEST [GET] http://localhost:8080/mywebapp/dba/static/basex.svg 18:49:13.841 0:0:0:0:0:0:0:1:55352 admin 200 14.16 ms 18:49:50.466 0:0:0:0:0:0:0:1:55510 admin REQUEST [POST] http://localhost:8080/mywebapp/dba/login-check 18:49:50.537 0:0:0:0:0:0:0:1:55510 admin INFO DBA user was logged in: admin 18:49:50.537 0:0:0:0:0:0:0:1:55510 admin 302 71.72 ms 18:49:50.537 0:0:0:0:0:0:0:1:55510 admin 200 0.09 ms 18:49:50.540 0:0:0:0:0:0:0:1:55510 admin REQUEST [GET] http://localhost:8080/mywebapp/dba/databases 18:49:50.577 0:0:0:0:0:0:0:1:55510 admin 200 37.53 ms 18:49:50.591 0:0:0:0:0:0:0:1:55510 admin REQUEST [GET] http://localhost:8080/mywebapp/dba/static/style.css 18:49:50.594 0:0:0:0:0:0:0:1:55511 admin REQUEST [GET] http://localhost:8080/mywebapp/dba/static/js.js 18:49:50.605 0:0:0:0:0:0:0:1:55510 admin 200 14.41 ms 18:49:50.607 0:0:0:0:0:0:0:1:55511 admin 200 12.49 ms 18:49:50.613 0:0:0:0:0:0:0:1:55511 admin REQUEST [GET] http://localhost:8080/mywebapp/dba/static/basex.svg 18:49:50.622 0:0:0:0:0:0:0:1:55511 admin 200 9.8 ms [2] Wildfly DBA login 18:57:29.915 127.0.0.1:57601 admin REQUEST [GET] http://localhost:8080/mywebapp/dba/ 18:57:30.594 127.0.0.1:57601 admin 302 681.49 ms 18:57:30.594 127.0.0.1:57601 admin 200 0.18 ms 18:57:30.599 127.0.0.1:57601 admin REQUEST [GET] http://localhost:8080/mywebapp/dba/databases 18:57:30.660 127.0.0.1:57601 admin 302 61.28 ms 18:57:30.660 127.0.0.1:57601 admin 200 0.1 ms 18:57:30.663 127.0.0.1:57601 admin REQUEST [GET] http://localhost:8080/mywebapp/dba/login?path=databases 18:57:30.684 127.0.0.1:57601 admin 200 20.65 ms 18:57:30.699 127.0.0.1:57602 admin REQUEST [GET] http://localhost:8080/mywebapp/dba/static/js.js 18:57:30.699 127.0.0.1:57601 admin REQUEST [GET] http://localhost:8080/mywebapp/dba/static/style.css 18:57:30.718 127.0.0.1:57601 admin 200 19.23 ms 18:57:30.721 127.0.0.1:57602 admin 200 21.71 ms 18:57:30.723 127.0.0.1:57601 admin REQUEST [GET] http://localhost:8080/mywebapp/dba/static/basex.svg 18:57:30.745 127.0.0.1:57601 admin 200 22.75 ms 18:57:50.379 127.0.0.1:57601 admin REQUEST [POST] http://localhost:8080/mywebapp/dba/login-check 18:57:50.442 127.0.0.1:57601 admin INFO DBA user was logged in: admin 18:57:50.442 127.0.0.1:57601 admin 302 63.28 ms 18:57:50.442 127.0.0.1:57601 admin 200 0.1 ms 18:57:50.446 127.0.0.1:57601 admin REQUEST [GET] http://localhost:8080/mywebapp/dba/databases 18:57:50.463 127.0.0.1:57601 admin 302 17.4 ms 18:57:50.463 127.0.0.1:57601 admin 200 0.09 ms 18:57:50.466 127.0.0.1:57601 admin REQUEST [GET] http://localhost:8080/mywebapp/dba/login?path=databases 18:57:50.476 127.0.0.1:57601 admin 200 10.89 ms 18:57:50.490 127.0.0.1:57602 admin REQUEST [GET] http://localhost:8080/mywebapp/dba/static/js.js 18:57:50.493 127.0.0.1:57601 admin REQUEST [GET] http://localhost:8080/mywebapp/dba/static/style.css 18:57:50.502 127.0.0.1:57602 admin 200 11.72 ms 18:57:50.505 127.0.0.1:57601 admin 200 12.17 ms 18:57:50.509 127.0.0.1:57602 admin REQUEST [GET] http://localhost:8080/mywebapp/dba/static/basex.svg 18:57:50.523 127.0.0.1:57602 admin 200 13.51 ms

2 2

Library for BaseX
by Eliúd Santiago Meza y Rivera 20 Sep '17

20 Sep '17

Hi! I've been using BaseX for a while mainly for personal purpose and I have written this library to read and make simple updates for Open Office Open Workbook files (Excel files), I hope someone could find it useful, and all comments are welcome. I leave the link below: https://github.com/eliudmeza/OOXML-Library-XQuery-BaseXdb Regards! -- LSC Eliud Santiago Meza y Rivera

2 1

db:text() vs XPath
by Giuseppe Celano 19 Sep '17

19 Sep '17

I am using BaseX 8.6.4 and I am trying to do a group-by/order-by operation, and I see that two logically equivalent queries perform very differently: one cannot see the end, while the other can (and fast). I can provide further details if necessary, but these are the queries (look at the last line of both, where the difference is): (This does not work:) declare variable $p := db:open("db2")/text/line/text(); (: returns a list of values like ΠΕΡΙ#n-s---mv- :) for $u in db:open("db1")/text/s/t let $a := $u/f/text() (: returns one value like ΠΕΡΙ :) let $b := $u/(a)o/data(.) (: returns one value like n-s---mv- :) group by $k := $a || "#" || $b (: builds a value like ΠΕΡΙ#n-s---mv- :) let $n := count($u) order by $n descending return if ($p = $k) then $k || " " || $n else () (This works:) for $u in db:open("db1")/text/s/t let $a := $u/f/text() (: returns one value like ΠΕΡΙ :) let $b := $u/(a)o/data(.) (: returns one value like n-s---mv- :) group by $k := $a || "#" || $b (: builds a value like ΠΕΡΙ#n-s---mv- :) let $n := count($u) order by $n descending return if (db:text("db2", $k)) then $k || " " || $n else () The problem is with directly accessing the database with db:text() or indirectly using XPATH. I would tend to use XPATH and expect to get an underlying translation into db:text(), but this seems not to happen: why? Best, Giuseppe

4 6

java.lang.NoClassDefFoundError: Could not initialize class java.nio.file.FileSystems$DefaultFileSystemHolder
by Liam R. E. Quin 18 Sep '17

18 Sep '17

It seems with the latest Java 1.8 - java-1.8.0-openjdk-headless-1.8.0.144-0.b01.el7_4.x86_64 on Centos 7, I can no longer drop a database, any ideas? This is with both 8.5.3 and 8.6.6, and also with the latest snapshot, BaseX867-20170824.195627.zip [[ $ bin/basexclient -p 1994 Username: admin Password: BaseX 8.6.6 [Client] Try 'help' to get more information. > open rdf Database 'rdf' was opened in 1.61 ms. > close Database 'rdf' was closed. > drop db rdf Improper use? Potential bug? Your feedback is welcome: Contact: basex-talk(a)mailman.uni-konstanz.de Version: BaseX 8.5.3 Java: Oracle Corporation, 1.8.0_131 OS: Linux, amd64 Stack Trace: java.lang.NoClassDefFoundError: Could not initialize class java.nio.file.FileSystems$DefaultFileSystemHolder at java.nio.file.FileSystems.getDefault(FileSystems.java:176) at java.nio.file.Paths.get(Paths.java:84) at org.basex.io.IOFile.toPath(IOFile.java:335) at org.basex.io.IOFile.delete(IOFile.java:243) at org.basex.io.IOFile.delete(IOFile.java:240) at org.basex.core.cmd.DropDB.drop(DropDB.java:77) at org.basex.core.cmd.DropDB.run(DropDB.java:46) at org.basex.core.Command.run(Command.java:253) at org.basex.core.Command.execute(Command.java:99) at org.basex.server.ClientListener.run(ClientListener.java:136) > open rdf Database 'rdf' was opened in 1.56 ms. > xquery count(//image) 7370 Query executed in 68.2 ms. ]] -- Liam Quin, W3C, http://www.w3.org/People/Quin/ Staff contact for Verifiable Claims WG, XQuery WG Web slave for http://www.fromoldbooks.org/

2 3

SQL2XML
by iCloud 18 Sep '17

18 Sep '17

Hello, I wrote a small app for my own needs to convert any kind of MS-SQL tables to a XML file. Feel free to download it, use it, distribute it. FREE OF CHARGE ! Still be a BETA for the moment !! Thanks for your feedback This is the download area : https://rink.hockeyapp.net/apps/71fcf4452594429d8cf81e5f2569e8e8 Have fun Michel

1 0

A few general questions about BaseX
by Anastasiou A. 18 Sep '17

18 Sep '17

Hello everyone I am trying to load BaseX with a large number of XML files (~500), each one a few hundreds of MBs big. BaseX fails with a message along the lines "This is too big for one database". Can I please ask: 1) Are there any logs, beyond the DB logs? If yes, where can I find them? a. The reason I am asking is because once basexgui gives the message, there is no indication about the error. Ideally, I would like to know if this is a limitation on memory amount or number of items (?). 2) The parser options include reading XML files from archives, which is very convenient, but once the file has been parsed, does BaseX require the "originals" for queries / returning results? 3) Is it possible to do federation with BaseX? In other words, let's say I split a database in two large parts (as per #1), is it possible to launch two baseX servers and then have them talk to each other so that ultimately I just query one of them and get back unified results? All the best

5 5

retrieving the name of the archive?
by Graydon Saunders 18 Sep '17

18 Sep '17

Hello -- BaseX will happily consume zip archives; this is just splendid for loading up a bunch of docx files. Now I find myself wanting the name of the docx file -- the original name of the archive -- and I don't know how to retrieve that. (or if it can be!) But I think it must be there somewhere because db:path repeats the standard OOXML file paths: [Content_Types].xml word/document.xml word/footnotes.xml word/footer1.xml word/endnotes.xml word/theme/theme1.xml word/settings.xml docProps/custom.xml customXml/itemProps2.xml docProps/app.xml customXml/item2.xml customXml/itemProps1.xml word/fontTable.xml customXml/item1.xml customXml/item3.xml customXml/itemProps3.xml customXml/item4.xml customXml/itemProps4.xml word/numbering.xml word/styles.xml word/webSettings.xml docProps/core.xml word/people.xml over and over; if they were all going to exactly that there'd be one copy, and all several hundred docx files are there by content. (db:list-details tells me about > 4000 individual xml files.) If I can get the name of the original archive, how do I do that? Thanks! Graydon

2 3

Basex Inner Workings
by Anastasiou A. 18 Sep '17

18 Sep '17

Hello everyone Quick question: Is there any document / URL where I could find out more about how does Basex access the disk during its operation? For example, are there any reads to be expected during executing a query? Through iotop, I can see 3-4 processes reading during startup, then another 2, very briefly firing when opening the database and then during querying there are periodic writes (?) but of very brief duration. I was wondering if there is anything that could be done from the point of view of the hardware to speed up queries (?) (except a more powerful machine at the moment) All the best Athanasios Anastasiou

5 16

Is there any documentation on the narrow limits of XQuery index optimization in BaseX?
by Omar Siam 18 Sep '17

18 Sep '17

Hi list! Recently I started to wonder why functions in my XQuery modules make no use of indexes unless I force them to by using the respective function like db:text(). Now I just did some I think minimal changes to the example for text index at: http://docs.basex.org/wiki/Indexes#Text_Index. If I just change it like below I loose almost all index optimization. The only optimizations left are: * Using db:open or collection with a string but not one in a variable of a for FLOWR expression or the simple map operator. * Using xquery:eval and passing a context with db:open or collection Is there any chance that this will change any time soon? Is this a fundamental restriction? Best regards Omar xquery version "3.1"; declare namespace _ = "urn:local:namespace:_"; import module namespace functx = "http://www.functx.com"; declare function _:_1st_example($ctx as document-node()) { $ctx//*[text() = 'Germany'] }; declare function _:_2nd_example($file as xs:string) { doc($file)//name[. = 'Germany'] }; declare function _:_3rd_example($dbname as xs:string+) { ( for $c in ($dbname!collection(.))//country where $c//city/name = 'Hanoi' return $c/name, $dbname!xquery:eval("//*[text() = 'Vietnam']", map {'': db:open(.)})) }; ( (: 1st example :) _:_1st_example(.), (: 2nd example :) _:_2nd_example('factbook.xml'), (: 3rd example :) _:_3rd_example(('factbook', 'factbook')), xs:string(_:_1st_example) ) Optimized Query: (let $ctx_314 := . return $ctx_314/descendant::*[(text() = "Germany")], db:open-pre("factbook",0)/descendant::name[(. = "Germany")], (for $c_317 in (("factbook", "factbook") ! collection(.))/descendant::country[(descendant::city/name = "Hanoi")] return $c_317/name, (("factbook", "factbook") ! xquery:eval("//*[text() = 'Vietnam']", map { "":db:open(.) }))), _:_1st_example cast as xs:string?) Compiling: [...] - RUNTIME: pre-evaluate root() to document-node() - RUNTIME: rewrite descendant-or-self step(s) - RUNTIME: apply text index for "Vietnam" - RUNTIME: pre-evaluate root() to document-node() - RUNTIME: rewrite descendant-or-self step(s) - RUNTIME: apply text index for "Vietnam"

2 3

Re: [basex-talk] Simple Map Operator in oXygen XML (XQJ)
by Omar Siam 18 Sep '17

18 Sep '17

Ok. Interesting. Please someone note this in the Wiki so there are no surprises. For the ! syntax I can just stick with for in return but missing map {} is a reason to switch to dba. Best regards Omar Am 04.09.2017 um 17:14 schrieb Charles Foster: > Hi Michael, > > BaseX XQJ supports XQuery 3.0 syntax I believe, but not XQuery 3.1 > specific syntax. > > It would be possible to update the XQJ driver to support XQuery 3.1 > syntax, but it would mean upgrading the internal XQuery parser and > re-writer. > > Omar, as a work around you could potentially use equivalent functions > in the standard F&O library (fn). > > Kind Regards, > > Charles > > >> On 4 Sep 2017, at 22:02, Michael Seiferle <ms(a)basex.org >> <mailto:ms@basex.org>> wrote: >> >> Hi Omar, >> /cc Hi @Charles, >> >> you are right, this has nothing to with oXygen in particular! >> Yet: I am not even sure if XQJ is meant to support anything but >> XQuery 1.0. >> Charles might correct me if I am wrong, but I don’t think there’s an >> easy way to add 3.1 features to the current XQJ implementation. >> >> Best from Konstanz >> >> Michael >> >> >> >> >> >>> Am 04.09.2017 um 16:42 schrieb Omar Siam <Omar.Siam(a)oeaw.ac.at >>> <mailto:Omar.Siam@oeaw.ac.at>>: >>> >>> Good idea but I was pretty sure that this comes from within BaseX >>> not oXygen. You can configure the parser and the executor of your >>> XQuery in oXygen und I used BaseX for both. >>> >>> So I found in basex-examples a XQJ tutorial. And just by changing >>> Part1.java a little and filling in the example query from the Wiki I >>> also get >>> >>> XQJQS001 - Invalid XQuery syntax, syntax does not pass static >>> validation. >>> Root Cause: >>> net.xqj.basex.bin.bB: Lexical error at line 1, column 12. >>> Encountered: " " (32), after : "!" >>> at >>> org.basex.examples.xqj.tutorial.simplemaptest.main(simplemaptest.java:34) >>> at >>> org.basex.examples.xqj.tutorial.simplemaptest.main(simplemaptest.java:34) >>> >>> So most probably BaseX'es XQJ parser does not understand the simple >>> map operator. >>> >>> XQSequence xqjs = xqje.executeQuery("xquery version \"3.1\"; (1 to 10) ! element node { . }"); >>> >>> Am 04.09.2017 um 15:37 schrieb Michael Seiferle: >>>> Hi Omar, >>>> >>>> Looks like that error message is generated by oXygen; you might want to crosspost to their list. >>>> >>>> Best >>>> Michael >>>> >>>>> Am 04.09.2017 um 14:31 schrieb Omar Siam<Omar.Siam(a)oeaw.ac.at>: >>>>> >>>>> Hi! >>>>> >>>>> I just tried to use ther Simple Map Operator while writing an XQuery in oXygen XML and executing the query using a client/server BaseX 8.6.6 data source. I configured it as described here:http://docs.basex.org/wiki/Integrating_oXygen >>>>> >>>>> When I use the Simple Map Operator (!) I only get "Invalid XQuery syntax, syntax does not pass static validation.". For example using "(1 to 10) ! element node { . }" >>>>> >>>>> If I run that XQuery in DBA for example it works. >>>>> >>>>> Any ideas? >>>>> >>>>> Best regards >>>>> >>>>> Omar Siam >>>>> >>> >> >

2 1

json-to-xml and xpath
by Günter Dunz-Wolff 18 Sep '17

18 Sep '17

Dear all, I’m trying to get data out of json via xquery. This is my scenario: 1. Fetching data with fetch:text() calling a REST-API 2. Transforming the data with json-to-xml, so I’ll get something like <array xmlns="http://www.w3.org/2005/xpath-functions“> <map> <string key=„word1“>gehen</string> ... </map> ... </array> 3. In my xqm-file I have: let $json_xml := json-to-xml(my_fetched_json) 4. I try to get „gehen“ with $json_xml//string[@key='word1‘], but it yields nothing, also $json_xml//string yields nothing. Only $json_xml//text() yields all string-values into a long string. What am I doing wrong? Or is there another way, to work with JSON-Data to get values of a given key? Thanks in advance, Günter

9 14

Startup hooks or persisting jobs
by Kendall Shaw 18 Sep '17

18 Sep '17

Am I missing an existing way to run xquery at startup (basex web service running under tomcat)? I have jobs that I schedule, but I have to schedule them again if basex is shutdown. I can test for basex being started outside of basex and then execute queries, but if there is already a way to do this within basex, I would rather do that. Kendall

3 3

upgrade of BaseX from 8.6.5 to 8.6.6 how to migrate the data?
by Stefania Axo 18 Sep '17

18 Sep '17

hi I want to upgrade from 8.6.5 to 8.6.6. How do i migrate the data? do i just copy over the Basex\data folder? thanks Stefania

3 2

Possible Bug in BaseX 8.2.3 when importing XML (Was RE: A few general questions about BaseX)
by Anastasiou A. 14 Sep '17

14 Sep '17

Hello everyone Many thanks to Alexander, Bridger, Fabrice, Michael for getting back to me with very detailed responses, these have been really helpful. A few notes: 1) The name is Athanasios :D. Sorry, just couldn’t help it, it seemed incredibly formal to be addressed via the surname in our communications. Our mail server advertises the “Surname. Initial” pattern, so I can see where the confusion came from. 2) I think that there is scope for adding some sort of “logging” to all actions of the server in general because I think I may have hit a bug but I cannot provide any more illuminating comments. Here is what is happening: a. During import, I get an error that file somethingsomething140.xml has an incredibly long element that cannot be imported at line (blahblah). The whole process just dies there. b. This is a bug, because if I simply imported JUST the offending file itself, a single file database is created without any problems and I can query it and all. So, maybe, the error is caused because of the previous file OR because of the way the files are loaded. But I have absolutely no way of knowing the “load history” of the files or the exception that was caught or anything else. In fact, once you press “OK” in the error dialog box, any database files that have been created are lost. In addition to this, the XML files to import are enumerated in a random order. So, I had to run the import again and stay there looking at each one of the files loading, to witness that the system “breaks” after 254 files (which is suspiciously close to 256). None of the files around the vicinity of the offending file caused any problems, so this may be a more difficult to catch bug (but it is thrown with both the internal and external parsers). Following this, I created smaller databases with 250 XML files and then got “predictable” errors on running out of memory and not creating indexes which I can solve more easily. 3) It’s good to know that I don’t need the original files because that’s a lot of space I can get rid of. Thank you. 4) Seems like the ADDCACHE would have saved me some trouble here, many thanks for that, but of course, if you don’t know the file enumeration order, you are still stuck in not knowing which files have already been imported. 5) Michael, logging won’t help with the internal import procedure, except of course if you were implying writing a quick script to do the import “manually”? 6) Michael, the fork-join and “client connect” are really interesting and worth a try before I start connecting things together via Hadoop. Are these modules already available to BaseX? Do I simply import their namespace or is it not even needed? Many thanks again. All the best From: Bridger Dyson-Smith [mailto:bdysonsmith@gmail.com] Sent: 12 September 2017 16:53 To: Anastasiou A. Cc: basex-talk(a)mailman.uni-konstanz.de Subject: Re: [basex-talk] A few general questions about BaseX Hi Anastasiou, Hopefully some of these answers are somewhat helpful. On Tue, Sep 12, 2017 at 4:54 AM, Anastasiou A. <a.anastasiou(a)swansea.ac.uk<mailto:a.anastasiou@swansea.ac.uk>> wrote: Hello everyone I am trying to load BaseX with a large number of XML files (~500), each one a few hundreds of MBs big. BaseX fails with a message along the lines “This is too big for one database”. Can I please ask: 1) Are there any logs, beyond the DB logs? If yes, where can I find them? a. The reason I am asking is because once basexgui gives the message, there is no indication about the error. Ideally, I would like to know if this is a limitation on memory amount or number of items (?). I'm not sure how to enable more verbose logging with the GUI -- hopefully one of the devs or power users can weigh in on this. 2) The parser options include reading XML files from archives, which is very convenient, but once the file has been parsed, does BaseX require the “originals” for queries / returning results? AFAIK, no it does not. BaseX will query and return results from the internal database(s). 3) Is it possible to do federation with BaseX? In other words, let’s say I split a database in two large parts (as per #1), is it possible to launch two baseX servers and then have them talk to each other so that ultimately I just query one of them and get back unified results? AFAIK, the preferred method is to split your files across many databases, then query multiple databases from a single expression[1]. Others will be able to speak to this better, but I don't think there's a straightforward way to run multiple BaseX servers in a single JVM. All the best Best, Bridger [1] http://docs.basex.org/wiki/Databases

2 3

Re: [basex-talk] BaseX 7.9 unstable
by Alexander Holupirek 13 Sep '17

13 Sep '17

> On 13. Sep 2017, at 11:28, Steinhauer, Dorothea <Dorothea.Steinhauer(a)proalpha.de> wrote: > >>> On 13. Sep 2017, at 10:06, Steinhauer, Dorothea <Dorothea.Steinhauer(a)proalpha.de> wrote: >>> >>> Hi, >>> >>> we use BaseX 7.9 and we have the problem that in some (non-replicable) situations no more data is written to the database. >>> The database is still readable. >>> Logfiles are currently not available. >>> >>> If the problem is known, can changes to properties help? >>> >>> For us, it is not an option to switch to a newer version of the baseX-DB because API changes would be a huge effort. >>> >>> >>> Thanks, >>> Dorothea >> >> I think, you would have to detect the pattern that leads to the unwanted behaviour. >> Maybe the documentation on Transaction Management [1] can help to reason about the problem? >> >> By the way, what do you mean with 'no more data is written to the database'. Never ever again after some situation or during a phase (and later on it possible again)? >> >> Thanks, >> Alex >> >> [1] http://docs.basex.org/wiki/Transaction_Management#Concurrency_Control > > > Hi, > > we use the database in production, so we can't wait if anything changed. You wanted to know with the question whether the database is permanently locked? I have problems to understand what 'no more data is written to the database' actually means. - Is it there a permanent write-lock on the database? - Or is the database corrupt after a write operation? - Or can't you write at some point in time and then, at later point, once again write to the database? > When the database is "write-locking", is it possible to back up our data? When a write operation alters the database you can not read (or write) the database [1], so a regular backup is not possible. What you nevertheless can do, is, on the OS-level and only for debugging and failure analysis, copy the database directory to another location and, for instance, inspect [2] the state of the database. Perhaps you can elaborate a bit on your whole scenario. How does your application operate? What do you do, if you encounter a situation in which you can not write to the database? Can you make a guess what is happening in your application when the situation occurs? Cheers, Alex [1] http://docs.basex.org/wiki/Transaction_Management#Concurrency_Control [2] http://docs.basex.org/wiki/Commands#INSPECT

1 0

BaseX 7.9 unstable
by Steinhauer, Dorothea 13 Sep '17

13 Sep '17

Hi, we use BaseX 7.9 and we have the problem that in some (non-replicable) situations no more data is written to the database. The database is still readable. Logfiles are currently not available. If the problem is known, can changes to properties help? For us, it is not an option to switch to a newer version of the baseX-DB because API changes would be a huge effort. Thanks, Dorothea

2 1

How do create XML for a one byte hex field that is used as a flag byte
by Dave Day 07 Sep '17

07 Sep '17

Greetings list, I don't know if I am asking this question correctly. I'm new to the XML world. I can google this, get some hits, but when I read the posts, it is not really clear to me. Any help is appreciated. On an IBM mainframe, I've got a one byte hex field defined as follows: PCDTFLAG DS XL1 FLAG TO INDICATE ORIGIN OF INFO DTFLENTP EQU B'10000000' INFO CAME FROM MODULE ENTRY POINT DTFLMEPS EQU B'01000000' DB2 MEPL LIST HAS BEEN SEARCHED DTFLMEPL EQU B'00100000' INFO CAME FROM DB2 MEPL LIST DTFLDB2S EQU B'00010000' DSNMODS HAS BEEN SEARHCED DTFLDSNM EQU B'00001000' INFO CAME FROM DSNMODS SEARCH DTFLSYST EQU B'00000100' INFO CAME FROM SYSTEM TABLE DTFLSYSS EQU B'00000010' SYSTEM TABLE HAS BEEN SEARCHED DTFLCMNC EQU B'00000001' COMMON AND NUC HAS BEEN SEARCHED I need to be able to create XML for the field PCDTFLAG. OK, define an element as hexBinary. But then how do I define the bits within the byte. On the mainframe side, with the equated values, I can OR the flag byte with the equated value, and this will set the correct bit on in the byte. OI PCDTFLAG,DTFLENTP will turn on bit 0 in the 1 byte field if it is not on already. I can test for the bit on or off. TM PCDTFLAG,DTFLENTP JZ some-other-area-of-code will branch if the bit is off JO another-area-of-code will branch if the bit is on So, I guess I am asking two questions. One, how to give definition to bits within a byte, and two, how to manipulate and test those defined bits. Regards, -- Dave Day

3 2

XML import issue (header related)
by Ron Katriel 06 Sep '17

06 Sep '17

Hi, I ran into a peculiar issue with importing an XML file from DrugBank into BaseX. It involves the file header, which looks like this: <?xml version="1.0" encoding="UTF-8"?> <drugbank xmlns="http://www.drugbank.ca" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.drugbank.ca http://www.drugbank.ca/docs/drugbank.xsd" version="5.0" exported-on="2017-08-31"> The XML file (which is quite large ~3GB) loads in cleanly but queries executed against it produce no output (no errors reported). When I change the header to the following and re-import the file everything works fine: <?xml version="1.0" encoding="UTF-8"?> <drugbank> I had this issue for a while (go thought this every month) so thought it might be good to share. I am currently using BaseX 8.6.4. Any idea why this is happening? Attached is the console trace for both cases (they show subtle differences in the query plans). Thanks, Ron Ron Katriel, Ph.D. | Principal Data Scientist | Medidata Solutions 350 Hudson Street, 7th Floor, New York, NY 10014 rkatriel(a)mdsol.com | direct: +1 201 337 3622 | mobile: +1 201 675 5598 | main: +1 212 918 1800

5 14

Re: [basex-talk] Fetching Images
by Günter Dunz-Wolff 06 Sep '17

06 Sep '17

Sorry about the question, I only had to put the url to <img/> source. Fetching wasn’t necessary.

1 0

Fetching Images
by Günter Dunz-Wolff 06 Sep '17

06 Sep '17

Hi all, in my RESTXQ-App I’m trying to fetch images (for example via „https://maps.googleapis.com/maps/api <https://maps.googleapis.com/maps/api>“): return fetch:binary("https://maps.googleapis.com/maps/api <https://maps.googleapis.com/maps/api>/blabla“) With fetch:content-type I get the right type: image/png. But with fetch:binary I only get the binary as large string but the browser doesn’t render the picture. If I load the url in browser directly, the image renders as expected. What am I doing wrong? Thanks for any help, Guenter

1 0

Re: [basex-talk] New basex-version -> broken paths
by Günter Dunz-Wolff 04 Sep '17

04 Sep '17

Hi Michael, thanks for the information. I changed let $tokens := doc("tokens.xml")//entry to let $tokens := fetch:xml("tokens.xml")//entry (tokens.xml is in the same directory as the xq-file.) The problem is still the same: [bxerr:BXFE0001] Resource "/Users/User/tokens.xml (No such file or directory)" not found. The path should be: /Users/User/Documents/project/BaseXWeb/tokens.xml Same problems with RESTPATH empty or path to project-folder. As I wrote, I have only problems with the REST-Service, not with RESTXQ. Best, Guenter

2 2

Simple Map Operator in oXygen XML (XQJ)
by Omar Siam 04 Sep '17

04 Sep '17

Hi! I just tried to use ther Simple Map Operator while writing an XQuery in oXygen XML and executing the query using a client/server BaseX 8.6.6 data source. I configured it as described here: http://docs.basex.org/wiki/Integrating_oXygen When I use the Simple Map Operator (!) I only get "Invalid XQuery syntax, syntax does not pass static validation.". For example using "(1 to 10) ! element node { . }" If I run that XQuery in DBA for example it works. Any ideas? Best regards Omar Siam

3 3

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

BaseX-Talk