BaseX-Talk

basex-talk@mailman.uni-konstanz.de

9 participants
5241 discussions

Problems with bigger amounts of data
by Mathias K 28 Jun '11

28 Jun '11

Hello everyone! My name is Mathias. I'm using BaseX for an university project where we are creating a publication database. Right now we have 25 GB xml data spread over 180k documents. Ultimately I want to be able to perform Xquery searches on this data, possibly even full text. I'd like to know whether you think that BaseX is at all suitable for this amount of data. If yes, how would I add these files to the database optimally? If I use the BaseX GUI to add the folder an OutOfMemoryException is produced shortly after starting the process. Even providing more RAM (~7 GB via -Xmx7000M) only delays this. I haven't looked at the code but it appears as though all file contents are stored in RAM and are only written to hard disk at the end, which would at least explain the huge amounts of memory BaseX consumes. Since the GUI can't handle the files I wrote an importer myself which consecutively adds single files via the "Add" command. This seems to work without memory excess. However, it is taking ages to add all 180.000 files this way (several hours, haven't completed it yet). Maybe it's just further delaying the overflow since it's so slow. Also, this might just be my subjective feeling, but adding files seems to get slower as the database grows. Is there some kind of duplicate check going an that could be in the way? If yes, is there a way just to bulk insert all the data I got without checks? I'd be grateful for any thoughts on this! Thanks in advance, Mathias

2 4

Packaging API in BaseX
by Christian Grün 25 Jun '11

25 Jun '11

Dear all, we'd like to give you a preview on our upcoming EXPath Packaging API implementation, which will be released together with BaseX 6.7 next week: http://docs.basex.org/wiki/Packaging_API If you are curious enough, feel free to try our latest stable snapshot: http://files.basex.org/releases/latest/ Two example archives are found at: http://files.basex.org/xar/ Your feedback is valuable to us. If you come across any unexpected behavior, or if you have problems to get it work (..no matter if it's up to you or us..), please give us a note! Regarding the different versions of the Packaging Spec: we decided to only support the latest version (Candidate Module 11 November 2010, http://expath.org/spec/pkg), which means that out-of-dated xar archives need first to be updated. Have fun, Christian ___________________________ BaseX Team Christian Grün Uni KN, Box 188 78457 Konstanz, Germany http://www.basex.org

2 1

Running XQuery in oxygen or Eclipse with default (opened) collection
by Jan Vlčinský (CAD) 24 Jun '11

24 Jun '11

Hi I am currently trying to learn, how to use Oxygen and/or Eclipse to develop XQueries. For installation I followed just the instructions form wiki http://docs.basex.org/wiki/Integrating_oXygen and http://docs.basex.org/wiki/Integrating_Eclipse With Eclipse it worked for me a bit easier, the only problem I had was to specify in the code, what collection I am going to use as default one. My wish is that my xquery code does not have to refer explicitly to single collecion name (configuring collection name around the xquery is fine, but I want prevent having it in the xquery code itself). I did not find any method how to do this. With using explicit collection name in the xquery code it works well. I know I could employ some external parameter and use it to bear the name of the collection, but this is still not, what I would call "use the currently opened collection" and I do not like the fact, my code has to be modified for this purpose. With oxygen is the situation the same with the complication, it keeps asking for xml input, when I use XQuery Transformation - and that is not, what I need. Any advise? Jan *Ing. Jan Vlčinský* CAD programy Slunečnicová 338/3, 734 01 Karviná Ráj, Czech Republic tel: +420-597 602 024; mob: +420-608 979 040 skype: janvlcinsky; GoogleTalk: jan.vlcinsky(a)gmail.com http://cz.linkedin.com/in/vlcinsky

2 2

Command line on OS X
by Craig Girard 23 Jun '11

23 Jun '11

I have 2 db's I would like to join. I have the xq thanks to help from Michael yesterday, but my db's are large and I receive a 'out of main memory' error. I have found some info on command line options but have been unable to get them to work. I am running BaseX on OS X. Is there a way to run an xq from the terminal or have the output from the xq go to a file rather than the screen? Thank you all for your help, Craig Girard

3 2

Rebind the variables
by Oda 23 Jun '11

23 Jun '11

Hi, I want to rebind the variables, but got the result that wasn't expected. How can I get the expected result without creating new processor instance? (When I use proc.query() and don't declare $var2 then got the expected result. But I have to declare $var2.) ---------- String query = "declare variable $var1 as xs:string external; " + "declare variable $var2 as xs:string := 'BBB'; " + "($var1, $var2)"; QueryProcessor proc = new QueryProcessor(query, new Context()); // "AAA BBB" proc.bind("var1", "AAA"); System.out.println(proc.execute()); //duplicate definition error of $var2 //proc.query(query); // "AAA BBB" not "CCC BBB" proc.bind("var1", "CCC"); System.out.println(proc.execute()); proc.close(); ---------- Thanks, Oda.

2 2

NEWBIE question - Joining db's
by Craig Girard 22 Jun '11

22 Jun '11

Is there a way in basex to join 2 databases together? I have 2 databases I would like to join based on an ID. I have found examples in xquery but they don't seem to work in basex. Thank you for your time, Craig

2 1

compound documents
by Imsieke, Gerrit, le-tex 21 Jun '11

21 Jun '11

What would be your recommendation for dealing with compound documents? Suppose we have a larger work of literature that consists of chapters, and for some reason we want to store the chapters as different files. A rationale for splitting the work might be that different persons should be able to edit two chapters at the same time. There’s a skeleton document that slurps in the individual chapters, maybe using XInclude or another linking/transclusion mechanism. Suppose there’s an XML database that stores a copy of the complete work and keeps it up to date. And that’s the point: how to implement the update procedure? a) Resolve transclusions first, then store whole work as one XML (as a consequence, every node has the same base URI) b) Store skeleton and individual chapter files, provide virtual single-tree view transparently (but each node would still know its base URI) c) Store skeleton and individual chapter files, leave it to the user (= xquery programmer) to create a custom transcluder for creating intermediate compound document instances that may then be queried or exported In cases b) and c), if a content unit (chapter) was edited, I’d just delete the corresponding DB item and add the changed file to the DB. In case a), I’d either reassemble the compound work and replace the probably large XML file in the DB with a new one (probably an expensive operation), or I’d replace the corresponding content via XQUF (requires that I’m able to precisely locate where the altered snippet was in the compound document). So if fine-granular file-based storage was a requirement, what would you do in order to index the content in BaseX? I think b) is desirable, but not implemented. Maybe you even know a method d)? Gerrit

1 0

whitespace pruning while storing an XML doc to BaseX, how to disable
by software developer 21 Jun '11

21 Jun '11

Hi, Does BaseX attempt some whitespace pruning on the XMLs being stored; if yes, how to get BaseX not to do so. On the wire, at the time request reaches BaseX, the XML is like: ..<xsl:for-each select="LINK"> ...<xsl:if test="A = 'TOKEN'"> ....insert into ....MYTABLE(VERSION) ....values ('LATEST'); ...</xsl:if> . in above is a space character, note the terminating xsl:if is on a line of its own. What BaseX stores is: <xsl:for-each select="LINK"> <xsl:if test="A = 'TOKEN'">insert into MYTABLE(VERSION) values ('LATEST');</xsl:if> </xsl:for-each> As you can see, BaseX has not persisted/stored the incoming XML as-is. I do not want this intelligent pruning since it breaks stuff for me. Thanks.

2 2

BaseX and integration into <oXygen/> - which edition is needed
by Jan Vlčinský (CAD) 20 Jun '11

20 Jun '11

Hi I am considering purchase of <oXygen/> and wonder, which edition is supporting integration with BaseX: It has to be Editor and not Author (Editor cares about transformations, Author about creation of content). But do I need Enterprise or I will live happily with Professional? My feeling is that Professional will be sufficient, as http://docs.basex.org/wiki/Integrating_oXygen describes general method, not bound to some specific vendor, but I can be wrong. Thanks for any reaction. Jan *Ing. Jan Vlčinský* CAD programy Slunečnicová 338/3, 734 01 Karviná Ráj, Czech Republic tel: +420-597 602 024; mob: +420-608 979 040 skype: janvlcinsky; GoogleTalk: jan.vlcinsky(a)gmail.com http://cz.linkedin.com/in/vlcinsky

3 2

Please confirm subscription to Clinical Trials Centre's newsletter! (Action required)
by magnifier 19 Jun '11

19 Jun '11

1 0

Jump to page:

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

BaseX-Talk