BaseX-Talk

basex-talk@mailman.uni-konstanz.de

5251 discussions

Improving performance in a 40GB database
by James Sears 11 Jul '16

11 Jul '16

Hi - I'm wondering what sort of recommendations anyone has for making my queries "go faster". I created a database by loading 21GB of xml data - that represents the OCR'd contents of 5000 books - into a new basex 8.4.4 instance. I run optimize after the import. basexgui> Properties> Information says it's sized at 40GB and contains around 1.5 billion nodes; there are no binaries. The sorts of query I'm interested in running are those that search each article on each page of each book - for example: for $book in //book return <result> <book id="{$book/id/text()}"/> { for $page in $book/page return <page id="{$page/id/text()}"> { for $article in $page/article return <article id="{$article/id/text()}"/> } </page> } </result> On a reasonably powerful i5 laptop with SSD + plenty of RAM, this query takes around 148550.00 ms - I'd like to significantly reduce this. Individual OCR'd words on pages maybe comprise around 85% of the data - and I don't actually care about this data. So maybe if I just don't load these OCR'd words it will help? I haven't tried that yet, but ideally I'd like not to have to do it. So, if anyone has any tips or ideas for reducing the query time then I'd be very interested in hearing what you have to say.

4 8

db:attribute and eq unexpected behaviour
by Alex Muir 09 Jul '16

09 Jul '16

Hi, I have XML data with attributes that I've set in attribute indexes with the following consistent format but with different values form13FFileNumber="028-17473" form13FFileNumber="028-16573" When I query the data with following without indexes let $data := db:open('13F')//data[@form13FFileNumber eq $form13FFileNumber] or with the db:attribute method using indexes let $data := db:attribute("13F", $form13FFileNumber, "form13FFileNumber")/.. with both queries I get extra data and the same extra data So if I try to extract form13FFileNumber 028-11185 for example I get 028-11185 as the first result but also 028-13947 and others... Similarly for 028-10601 I get data for 028-10601 and 028-17017 and others does the dash with numbers cause a problem with the indexing? XQUERY Getting result xquery version "3.0"; declare namespace db="http://basex.org/modules/db"; declare namespace file="http://expath.org/ns/file"; declare variable $form13FFileNumber as xs:string external; (:let $data := db:open('13F')//data[@form13FFileNumber eq $form13FFileNumber] :) let $data := db:attribute("13F", $form13FFileNumber, "form13FFileNumber")/.. let $fields := <form13FFile> { for $c in $data return $c } </form13FFile> return file:write(concat('../OUTPUT/form13Restatment3rd_' , $form13FFileNumber , '.xml'), $fields) BD CREATE COMMANDS: <?xml version="1.0" encoding="UTF-8"?> <commands> <set option='attrinclude'>xml:id,filingDate,type,cik,sequence,form13FFileNumber,periodOfReport</set> <create-db name='13F'/> <info-db/> </commands> The xml element that has all the indexes <data xml:id="0001534270-16-000006" filingDate="2016-06-08" type="13F-HR" cik="1534270" sequence="000006" form13FFileNumber="028-17473" periodOfReport="03-31-2016"> The basex 744 db is around 8gb and data was added with this xquery let $docs := collection($path) for $doc in $docs return let $path := concat($doc/data/@periodOfReport,'/',$doc/data/@form13FFileNumber,'/',$doc/data/@xml:id,'.xml') return db:add('13F', $doc, $path) Regards Alex tech.jahtoe.com bafila.jahtoe.com

2 2

Removing the xmlns attribute and/or adding the prefix
by Mike Engledew 09 Jul '16

09 Jul '16

Hi, We are using BaseX as our backend xml database using the rest interface. Responses to our queries includ an xmlns attribute e.g. <items xmlns="http://server.my.org/xyz"> <item>Stuff1</item> <item>Stuff2</item> </items> We are attempting to unmarshall the response using the JAXB client e.g. ItemList itemList = response.readEntity(new GenericType<List<Item>>(){}); the inclusion of the xmlns attribute prevents JAXB from doing the unmarshall successfully, firstly because it lacks the actual prefix defined for our application eg. xmlns:xyz="http://server.my.org/xyz" and secondly because it forces us to define our own unmarshaller and set up a package-info.java definition with overrides for xml interactions outside our application. a) Is it possible to disable the addition of the xmlns attribute? b) Is it possible to include the prefix in the xmlns attribute being appended to the root element? I think we would favor option a) if possible because it causes us the least amount of rework :-) Thanks in advance for any help and/or suggestions, MikeE.

4 13

NVDL validation
by Andy Bunce 09 Jul '16

09 Jul '16

Hi, Just an observation.. It seems like the RelaxNG Validation functions [1] can be overloaded/abused to provide NVDL validation [2] As in: validate:rng-report("data.xml","test.nvdl") This support comes from the underlying Trang library. It looks like any schema format Trang accepts can be used. [3] /Andy [1] http://docs.basex.org/wiki/Validation_Module#RelaxNG_Validation [2] http://www.nvdl.org/ [3] https://github.com/BaseXdb/basex/blob/master/basex-core/src/main/java/org/b…

2 2

Happy Birthday Christian!
by Maximilian Gärber 08 Jul '16

08 Jul '16

Hi Christian, since you are spending most of your free time on this list, I figured my wishes would reach you here fastest ;-) Thank you for your paramount effort and for being such a nice guy! Regards, Max

1 0

correction function to retrieve images and .class files
by Wang, Yvonne 08 Jul '16

08 Jul '16

Hi, We are using bin:decode-string(db:retrieve to retrieve binary file such as text and json files. However, it does not work for images and .class files. Any suggestion on the correction functions to use to retrieve images and .class files? Thanks Yvonne

2 1

Seeing this error a lot lately (since 8.5-snapshots)
by Marc van Grootel 07 Jul '16

07 Jul '16

Hi, With the 8.5 branch i've started seeing the following error a lot lately. [XUDY0016] Documents in path 'docbook/chapters/mu.xml' can only be replaced once. This is the example function I used to trigger this. declare %updating function local:import() { for $file in file:list($srcdir, true(), '*.xml') let $path := $srcdir || $file where not(file:is-dir($path)) return db:replace('docbook', $file, doc(trace($path,'X: '))) }; (lifted from the BaseX documentation example on file:list) Note that the path exists and contains several XML files. It happened a lot too on a work project so I've seen this in several situations now. On the latter project it went away after going back to a 8.4 build. Sorry for not reporting sooner as I was under severe time pressure at the time. I tried switching MIXUPDATES on and off. That doesn't make a difference. I have two XML files in this $srcdir and when I put a trace around $path I see that both of them are listed before it stops with an error message that includes the path of the first one. X: "/Users/marcvangrootel/projects/xml/book/chapters/mu.xml" X: "/Users/marcvangrootel/projects/xml/book/chapters/test.xml" Stopped at /Users/marcvangrootel/projects/xml/c4/test/demo.xq, 30/20: [XUDY0016] Documents in path 'docbook/chapters/mu.xml' can only be replaced once. Odd! --Marc

3 5

German Article about BaseX at informatik-aktuell.de
by Maximilian Gärber 07 Jul '16

07 Jul '16

Hi, for our german readers this might be another helpful high-level intro to BaseX http://www.informatik-aktuell.de/betrieb/datenbanken/nosql-basex-eine-xml-d… Regards, Max

1 0

ArrayIndexOutOfBoundsException withe BaseX version 8.5
by Simon Chatelain 07 Jul '16

07 Jul '16

Hello, I just updated BaseX to the last version (8.5), and I am now facing some issues, while with the previous version (8.4.4) my application was working flawlessly. My application receive XML documents from a TCP socket and store them in BaseX, in parallel and upon user request the same collection is queried. For both adding new documents and querying the collection, I am using a LocalSession, each one in its own thread. The problem is that the following exception occurs very often, and when it happens, it seems to corrupt definitely the DB. I wrote 2 simple test cases. The ArrayExceptionTest.java exhibits the problem while NoArrayExceptionTest.java does not. The only difference between the 2 is: · ArrayExceptionTest.java uses LocalSession for adding documents to and querying the collection · NoArrayExceptionTest.java uses LocalSession only for querying the collection, adding documents in the collection is done by using directly the Context. org.basex.core.BaseXException: Improper use? Potential bug? Your feedback is welcome: Contact: basex-talk(a)mailman.uni-konstanz.de Version: BaseX 8.5 Java: Oracle Corporation, 1.7.0_80 OS: Windows 7, x86 Stack Trace: java.lang.ArrayIndexOutOfBoundsException: 2147483647 at org.basex.io.random.TableDiskAccess.fpre(TableDiskAccess.java:501) at org.basex.io.random.TableDiskAccess.cursor(TableDiskAccess.java:452) at org.basex.io.random.TableDiskAccess.read2(TableDiskAccess.java:155) at org.basex.data.Data.nameId(Data.java:360) at org.basex.data.Data.name(Data.java:375) at org.basex.query.value.node.DBNode.name(DBNode.java:169) at org.basex.query.expr.path.NameTest.eq(NameTest.java:92) at org.basex.query.func.db.DbAccess$1.next(DbAccess.java:67) at org.basex.query.func.db.DbAccess$1.next(DbAccess.java:62) at org.basex.query.iter.Iter.value(Iter.java:53) at org.basex.query.expr.ParseExpr.value(ParseExpr.java:68) at org.basex.query.QueryContext.value(QueryContext.java:417) at org.basex.query.expr.path.CachedPath.nodeIter(CachedPath.java:36) at org.basex.query.expr.path.AxisPath.iter(AxisPath.java:78) at org.basex.query.expr.path.AxisPath.iter(AxisPath.java:51) at org.basex.query.expr.gflwor.For$1.next(For.java:109) at org.basex.query.expr.gflwor.OrderBy$1.sort(OrderBy.java:76) at org.basex.query.expr.gflwor.OrderBy$1.next(OrderBy.java:57) at org.basex.query.expr.gflwor.GFLWOR$1.next(GFLWOR.java:96) at org.basex.query.func.fn.FnCount.item(FnCount.java:27) at org.basex.query.expr.ParseExpr.iter(ParseExpr.java:44) at org.basex.query.MainModule.iter(MainModule.java:118) at org.basex.query.QueryContext.iter(QueryContext.java:340) at org.basex.query.QueryProcessor.iter(QueryProcessor.java:91) at org.basex.core.cmd.AQuery.query(AQuery.java:94) at org.basex.core.cmd.XQuery.run(XQuery.java:22) at org.basex.core.Command.run(Command.java:253) at org.basex.core.Command.execute(Command.java:99) at org.basex.api.client.LocalSession.execute(LocalSession.java:132) at org.basex.api.client.Session.execute(Session.java:36) at basextest.ArrayExceptionTest$1.run(ArrayExceptionTest.java:57) at org.basex.core.Command.execute(Command.java:100) at org.basex.api.client.LocalSession.execute(LocalSession.java:132) at org.basex.api.client.Session.execute(Session.java:36) at basextest.ArrayExceptionTest$1.run(ArrayExceptionTest.java:57) Regards Simon

2 1

Collation in BaseX and Java
by Kristian Kankainen 06 Jul '16

06 Jul '16

Hello! In a lexicographic application I need a custom collation for ordering. The documentation for XML, Xquery and XSLT all somehow state that system available collations are to be used and all seem to support custom collations by specifying the URI. Is this possible also for custom created Java collators? I have been looking at the Java docs [1] and it looks easy enough for me, but I would like to get an affirmative from this list that it does work with BaseX. My initial guesses is that I'd need to package my collator class as a jar and install it in the Xquery module repository [2]. But how to use it from there I'm not sure. Is BaseX able to automatically use it correctly if I specify the correct URI as collation in the prolog "|declare| |default| |collation 'URI';|"? I'd me more than glad to write about it in the documentation/wiki if I get it working. Cheers! Kristian [1] http://docs.oracle.com/javase/7/docs/api/java/text/RuleBasedCollator.html [2] http://docs.basex.org/wiki/Repository

2 4

Jump to page:

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

BaseX-Talk