BaseX-Talk

basex-talk@mailman.uni-konstanz.de

6 participants
5250 discussions

BaseX Evaluation for Big Data
by Huband, Gary W *HS 06 Dec '13

06 Dec '13

I'm evaluating BaseX to use in a project that involves 10s of TB of xml data. Our goal is to be able to return a small amount of data from a query within a few seconds. Is BaseX capable of handling this much data? My initial evaluation of 1GB of data met our requirements. But, when I evaluated BaseX using about 20GB of data I get an out of memory message when I run a query. I'm running Windows 7 Pro on a Core i7, 8GB memory, and using the BaseX GUI. Thanks for your time Gary

3 3

Re: [basex-talk] BaseX Evaluation for Big Data
by jean-marc Mercier 06 Dec '13

06 Dec '13

Hi Gary, Maybe you should wait for BaseX team members advises over this topic. Telling from my own experience, I can remember also having met some problem trying to insert 20 Gb of data into BaseX through the GUI. I had to achieve this into smaller steps, using batch commands (see http://docs.basex.org/wiki/Commands), in order to free memory. i.e. I first inserted the xml into 4 or 5 passes. Then created some indexes, etc...In that way it was possible to run the insertion, even with 4Gb machines. Hope this helps Jean-Marc 2013/12/6 Huband, Gary W *HS <GWH2SJ(a)hscmail.mcc.virginia.edu> > I still get an out of main memory when I try to create attribute indexes. > > I originally added the xml data with no indexes. > > Gary > > ________________________________ > From: jean-marc Mercier [jeanmarc.mercier(a)gmail.com] > Sent: Friday, December 06, 2013 9:37 AM > To: Huband, Gary W *HS > Cc: basex-talk(a)mailman.uni-konstanz.de > Subject: Re: [basex-talk] BaseX Evaluation for Big Data > > I meant I am NOT a baseX member ! sorry for the mistyping ! > > > 2013/12/6 jean-marc Mercier <jeanmarc.mercier(a)gmail.com<mailto: > jeanmarc.mercier(a)gmail.com>> > Hi Gary > > I am a BaseX team member, but have you sized up your JVM to run "Big Data" > ? > > See the files BaseX/basex*.bat. I am currently using set VM=-Xmx6g > > Hope this helps > > Jean-Marc > > > 2013/12/6 Huband, Gary W *HS <GWH2SJ(a)hscmail.mcc.virginia.edu<mailto: > GWH2SJ(a)hscmail.mcc.virginia.edu>> > > I'm evaluating BaseX to use in a project that involves 10s of TB of xml > data. Our goal is to be able to return a small amount of data from a query > within a few seconds. Is BaseX capable of handling this much data? > > My initial evaluation of 1GB of data met our requirements. But, when I > evaluated BaseX using about 20GB of data I get an out of memory message > when I run a query. > > I'm running Windows 7 Pro on a Core i7, 8GB memory, and using the BaseX > GUI. > > Thanks for your time > > Gary > > _______________________________________________ > BaseX-Talk mailing list > BaseX-Talk(a)mailman.uni-konstanz.de<mailto: > BaseX-Talk(a)mailman.uni-konstanz.de> > https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk > > > >

1 1

Node Insertion through XQJ and XSD Validation
by Ando Roy 06 Dec '13

06 Dec '13

Greetings, I've just started reading through the materials on XQJ and would like to ask the following: Let's say I have a simple XML file (I'll use the W3Schools Xquery tutorial): 1. Can I have an example query of how I could insert another book node into this file? 2. If I would like to perform schema validation through XSD, how should I change the code? - eg. (if I have set the maxOccurs attribute set as 1), will XQJ throw some exception? <bookstore> <book category="COOKING"> <title lang="en">Everyday Italian</title> <author>Giada De Laurentiis</author> <year>2005</year> <price>30.00</price> </book> </bookstore> Regards, Roy

2 1

inspect functions and %private
by jean-marc Mercier 06 Dec '13

06 Dec '13

Hi, To report a small issue in the inspect functions module with %private declaration. The following code runs perfectly : declare %private function local:test(){()}; for $fun in inspect:functions() return "gotcha" However, If the function test is located in another module, say test, then the following code import module namespace test="http://www.example.com/test" at test.xq"; for $fun in inspect:functions() return "gotcha" raise an exception. Cheers, Jean-Marc

2 5

the slf4j in basex is 1.5.8, compatibility issue took place
by johnny 05 Dec '13

05 Dec '13

Hello there: When I use basex with a gradle project, it failed at runtime because of a transitive dependency problem: basex used slf4j 1.5.8, and the other packages in the projects used higher version. I attempted to resolve this by following this link, it didn't work: http://forums.gradle.org/gradle/topics/how_to_exclude_transitive_dependency Do you guys have plan to upgrade the slf4j in basex? I still prefer to use the repo as opposed to building my own jar out of the source. thanks, Johnny

2 1

csv-parser
by Lars Johnsen 04 Dec '13

04 Dec '13

Hi all When importing CSV files, it seems that BaseX is parsing fields for balanced bracketing and quotes (a single quote or bracket causes trouble). Is it possible to turn that off, so that files are processed based only on delimiter? I couldn't find any information in the documentation. All the best, Lars

2 2

Error while adding a document.
by Tojo Rakotoarison 04 Dec '13

04 Dec '13

Hello to the BaseX Team ! I am new to BaseX and exploring its functionalities. I have created an empty database named 'NBS' and I have got a script in which I have the following lines : OPEN NBS SHOW USERS ADD nbs_1Chr.xml When I am trying to run it in the basexclient, it raises an error and does not want to add the document. Here is the output. Do you have any idea where the problem comes from ? The XML file nbs_1Chr.xml is in the current working directory. Thank you very much ! --- > RUN insert_nbs_test.bxs Username Read Write Create Admin ------------------------------------ admin X X X X 1 Users. Database 'NBS' was opened in 1.93 ms. Improper use? Potential bug? Your feedback is welcome: Contact: basex-talk(a)mailman.uni-konstanz.de Version: BaseX 7.6 Java: Sun Microsystems Inc., 1.6.0_22 OS: Linux, i386 Stack Trace: java.lang.RuntimeException: Data Access out of bounds: - pre value: 0 - #used blocks: 0 - #total locks: 0 - access: -1 (0 > -1] org.basex.util.Util.notexpected(Util.java:53) org.basex.io.random.TableDiskAccess.cursor(TableDiskAccess.java:488) org.basex.io.random.TableDiskAccess.read5(TableDiskAccess.java:211) org.basex.data.Data.textOff(Data.java:422) org.basex.data.DiskData.text(DiskData.java:234) org.basex.index.resource.Docs.paths(Docs.java:107) org.basex.index.resource.Docs.insert(Docs.java:142) org.basex.index.resource.Resources.insert(Resources.java:73) org.basex.data.Data.insert(Data.java:676) org.basex.core.cmd.Add.run(Add.java:118) org.basex.core.Command.run(Command.java:342) org.basex.core.cmd.Execute.run(Execute.java:49) org.basex.core.Command.run(Command.java:342) org.basex.core.Command.exec(Command.java:321) org.basex.core.Command.execute(Command.java:78) org.basex.server.ClientListener.run(ClientListener.java:145) -- Tojo Rakotoarison *@tojo_rs <https://twitter.com/#%21/tojo_rs> *

2 2

user.dir and BaseX Add Command
by Simon Görke 03 Dec '13

03 Dec '13

Hi together, I have a question regarding the baseX add command. We change the user.dir system property in the basexserver.bat as shown below: java -Duser.dir=c:/tmp -cp "%CP%" %VM% org.basex.BaseXServer %* This is done with the background of defining an export/import directory without using absolute paths with the add command itself (we deploy our setup from version control to different directory environments, and it should work "out-of-the-box"). 1. With exporting, this seems to work fine except for a minor issue: in addition to the successful export to c:\tmp, an empty folder is created in the basex/bin directory with the name of the exported db. 2. With adding, we always get the error message " Resource "C:/tmp/test.xml" not found.", even if test.xml exists in the filesystem at c:\tmp. However, if we copy test.xml to the basex/bin-folder, the add command is successful. It seems that though the error message relates to the correct (modified) path, baseX uses the default one when trying to add the file? Would it be possible to fix this issue, or is there another way to achieve the intended behavior? Best Regards Simon

2 4

Results from ft:extract()
by Lars Johnsen 02 Dec '13

02 Dec '13

Hello all When using ft:extract() on nodes, it seems to clip into the match itself too often. Is it possible to have ft:extract() leave as much before the match as after? For example, here are two results for "spise lunsj" (= eat lunch (language is Norwegian)) the first is as it should be, while the second have half of the matched string clipped . The result is obtained first as a set of hits using fulltext search [text() contains text {$terms} all], then each hit is processed through ft:extract($hit, $terms): ... gjerne komme for å spise lunsj med meg på Harrods. Da skal jeg servere hjortetestikler fra eiendommen min i Skottland. Vi trenger nemlig alle store... ... de franske VM-spillerne hjem til Paris. Der ble de mottatt av fans både på Charles de Gaulle-flyplassen og da de ankom Elysee-palasset for å spise ... Regards, Lars G Johnsen National Library of Norway

2 1

CDATA in xquery
by Erol Akarsu 02 Dec '13

02 Dec '13

I would like to generate CDATA of one xmk construct like this. This does not calculate $1,$f2 and $f3 and insert their serialized version into features tag Can we do this in xquery? let $allfs := <record> <name>{$name}</name> <features> <![CDATA[ <h3>Features:</h3> <br/> <ul> <li>{$f1}</li> <li>{$f2}</li> <li>{$f3}</li> </ul> ]]> </features> </record> Erol Akarsu

2 2

Jump to page:

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

BaseX-Talk