BaseX-Talk

basex-talk@mailman.uni-konstanz.de

6 participants
5270 discussions

baseX performance with large number of small documents (are the indexes working??)
by Shahin Roboubi 25 Jan '12

25 Jan '12

I'm trying to see if I can use baseX for a project we have. We need to store a large number of small documents (about 5,000,000 where each document is 1 to 10K). I had some performance issues and searched the mailing list and found some answers like this: https://mailman.uni-konstanz.de/pipermail/basex-talk/2012-January/002478.ht… This suggests I should be able to get good performance (query times that are around ~100ms or so). I'm running this on a linux server with fast disks and 24 GB of RAM (4 GB for JVM). By the way, I'm doing the queries through the baseX GUI... not sure if that makes any difference. I created 3 test databases, small, medium and large. The results are shown below. All databases have full text search disabled (because I don't need it) and "Path Summary", "Text Index", "Attribute index" enabled. It seems like the indexes are not doing anything or just not working, because the query times are going up linearly (up to 5 seconds for the large database!!) with the size of the database... can someone explain what is happening/why, and how I can fix it? Thanks a lot, Shahin Roboubi Software Engineer MDA Embedded Attachment: ----------------------------------------------------------------------------------- Database Properties Name: radarsat2small Size: 97 MB Nodes: 3891930 Resources: 92665 Timestamp: 05.01.2012 15:07:37 Query: /metadata/Radarsat2Signal[Acquisition[orbit_number<200]] Compiling: - adding text() step - rewriting orbit_number/text() < 200 Result: root()/metadata/Radarsat2Signal[Acquisition[orbit_number/text() < 200.0]] Timing: - Parsing: 0.25 ms - Compiling: 0.37 ms - Evaluating: 530.21 ms - Printing: 5.09 ms - Total Time: 535.94 ms Result: - Results: 165 Items - Updated: 0 Items - Printed: 145 KB Query plan: <IterPath> <Root/> <IterStep axis="child" test="metadata"/> <IterStep axis="child" test="Radarsat2Signal"> <AxisPath> <IterStep axis="child" test="Acquisition"> <CmpR min="-INF" max="200"> <AxisPath> <IterStep axis="child" test="orbit_number"/> <IterStep axis="child" test="text()"/> </AxisPath> </CmpR> </IterStep> </AxisPath> </IterStep> </IterPath> ----------------------------------------------------------------------------------- Database Properties Name: radarsat2medium Size: 194 MB Nodes: 7777056 Resources: 185168 Timestamp: 05.01.2012 15:26:07 Query: /metadata/Radarsat2Signal[Acquisition[orbit_number<100]] Compiling: - adding text() step - rewriting orbit_number/text() < 100 Result: root()/metadata/Radarsat2Signal[Acquisition[orbit_number/text() < 100.0]] Timing: - Parsing: 0.25 ms - Compiling: 0.56 ms - Evaluating: 1079.27 ms - Printing: 5.99 ms - Total Time: 1086.08 ms Result: - Results: 185 Items - Updated: 0 Items - Printed: 163 KB Query plan: <IterPath> <Root/> <IterStep axis="child" test="metadata"/> <IterStep axis="child" test="Radarsat2Signal"> <AxisPath> <IterStep axis="child" test="Acquisition"> <CmpR min="-INF" max="100"> <AxisPath> <IterStep axis="child" test="orbit_number"/> <IterStep axis="child" test="text()"/> </AxisPath> </CmpR> </IterStep> </AxisPath> </IterStep> </IterPath> ----------------------------------------------------------------------------------- Database Properties Name: radarsat2large Size: 873 MB Nodes: 34999986 Resources: 833333 Timestamp: 05.01.2012 16:32:29 Query: /metadata/Radarsat2Signal[Acquisition[orbit_number<20]] Compiling: - adding text() step - rewriting orbit_number/text() < 20 Result: root()/metadata/Radarsat2Signal[Acquisition[orbit_number/text() < 20.0]] Timing: - Parsing: 0.28 ms - Compiling: 2.16 ms - Evaluating: 5296.87 ms - Printing: 5.71 ms - Total Time: 5305.04 ms Result: - Results: 174 Items - Updated: 0 Items - Printed: 153 KB Query plan: <IterPath> <Root/> <IterStep axis="child" test="metadata"/> <IterStep axis="child" test="Radarsat2Signal"> <AxisPath> <IterStep axis="child" test="Acquisition"> <CmpR min="-INF" max="20"> <AxisPath> <IterStep axis="child" test="orbit_number"/> <IterStep axis="child" test="text()"/> </AxisPath> </CmpR> </IterStep> </AxisPath> </IterStep> </IterPath>

2 1

referring to the database root node from a function
by Piotr Bański 25 Jan '12

25 Jan '12

Hi all, This is, I'm afraid, a basic question, that in my hasty searches I just must have missed a solution to. I would like to find a clean way of referencing the default context of my main module, where I can easily do "let $x := //whatever", but after I move this line to a function declaration. Is there any kind of database-root() function that would provide such a reference? I realise that I can pass the relevant node as one of the function's arguments, or use a doc() there with the appropriate path, or maybe even use the "declare context item" functionality that i haven't been successful in putting to work so far, but maybe there exists a much simpler way out? I'll be grateful for a hint. Thanks in advance, Piotr

3 7

python client library
by info＠arjenvanelteren.com 23 Jan '12

23 Jan '12

Hi All, I've been accessing the BaseX library from python for a while now. Unfortunately the client library for python does not provide the full client library (returning per item result and the event code) I've updated the python client library for basex. It contains: *) a small bugfix in buffered reading (plus some refactoring), *) support for iterating over the result of a query *) and watching events. If you like I can create some examples for these... I've attached the updated file. Kind regards, Arjen van Elteren

2 1

(no subject)
by Adrian Hossu 23 Jan '12

23 Jan '12

Hello all, We have an issue with performance in basex. We are using basex to manage some user generated content, which has frequent updates. Each user has his own "home" within one large xml document, and they are able to update content within their own home. Our problem is that as our number of users increased, we started to notice some major slow-downs on our server. A small update of the user's content (one or two xml nodes) sometimes results in our [flush-...] process writing about 30MB of data. I hope this makes sense. I haven't actually looked at the basex source code, so I don't know what happens during these updates, but it seems to me that the amount of data written to the disk per update is relative to our database (document) size, although the amount of data being updated remains constant. Perhaps there is an index update happening? Sorry if I'm talking nonsense because I don't really understand how basex works. So basically, I have several questions: 1) Is it a bad idea to have one single large document, which can have many frequent updates? 2) Would it be better to have a document for each user? This would result in about 1000 documents being created per day. Does basex index documents within a database, for quick retreival, and is there a limit on the number of documents that you can have in a single database? 3) if we switch to one-document-per-user, would updating these documents somehow still result in large amounts of data being written per update, even if the user's documents are very small? Or would this be a possible solution to our problem? Thank you and I hope this made sense, - Adrian

2 1

[XQST0054] Stack Overflow
by Erol Akarsu 20 Jan '12

20 Jan '12

Christian, I am getting this error: [XQST0054] Stack Overflow: circular variable declaration? Do you know which can cause this? I does not happen all the time but some time Thanks Erol Akarsu On Fri, Dec 9, 2011 at 10:45 AM, Christian Grün <christian.gruen(a)gmail.com>wrote: > > Can I pass command line arguments to basex while executing a xquery > script? > > You may run a script with the "run" command and prefix it with other > commands. An example: > > basex -c "open ...; ...; run yourquery.xq" > > With 7.1 and the latest snapshot, you can use an arbitrary number of > commands, query files etc: > > basex -c "open...." yourquery.xq > > Hope this helps, > Christian >

2 1

Running BaseX commands from xquery esp REPO LIST
by Andy Bunce 19 Jan '12

19 Jan '12

Hi, I using the EXPath repository and it is going well. It would be useful to me to manage the repository from Xquery, at least to get the REPO LIST result. I see Exist DB has http://demo.exist-db.org/repo/repo.xml#d5510e705 Are there any plans to add something similar to BaseX One quick, if a bit hacky, way to enable this to be built would be a new util:command("command string") e.g. util:command("get timeout") <result>TIMEOUT: 0</result> /Andy

2 1

performance question
by Cerstin Mahlow 18 Jan '12

18 Jan '12

Hi, I would like to set up a collection of TEI-annotated texts (novels, dramas, poems, etc.). In total, it would be around 3 GB XML data in some 1000 files, the text size varies from 29 KB to 94 MB. I have a server running with Java 1.6.0_07 on CentOS 5.7 on a Virtual Machine with 1 GB RAM. I started to add files to the database and wrote a preliminary query interface (<http://oldphras.unibas.ch/cgi-bin/basex-client.pl>). Since we want to look for examples of multi-word units, I would like to use queries like: //(p|l) [text() contains text "Korb geben" using stemming using language "de"] (In the end, queries will be more complex to allow users to search for several words in different word order within a sentence using stemming or fuzzy) To make inspection of results easier, I added ft:mark. A collection with only a dozen of texts of about 71 MB with full text index for German, optimized, etc. works quite well. However, the example query needs more than 9s, which is rather slow. What is worse: Adding more files, resulting in about 323 MB, causes a timeout when running the query. I already set the memory for the Java VM to 1024, but it does not help. I tried it with the GUI on my iMac with 4 GB RAM and got a time out when the collection size is above 900 MB (which is still only a small part of my data). Is there any recommendation for size of RAM or specific settings when processing collections of about 3 GB? Is there a better way to write queries when looking for inflected forms of several words and allowing for spelling errors? Thank you in advance Cerstin -- Dr. phil. Cerstin Mahlow Universität Basel Deutsches Seminar Nadelberg 4 4051 Basel Schweiz Tel: +41 61 267 07 65 Fax: +41 61 267 34 40 Mail: cerstin.mahlow(a)unibas.ch Web: http://www.oldphras.net ---------------------------------------------------------------- This message was sent using IMP, the Internet Messaging Program.

3 12

workaround for lack of http_proxy support
by jidanni＠jidanni.org 17 Jan '12

17 Jan '12

Whilst waiting for Debian's basex to support http(s)_proxy, I found a workaround: installing the caching proxy WWWOFFLE, one then can use $ basex -q 'doc("http://localhost:8080/http/example.org/api?xyz")' without which I would very fast exceed daily API usage limits for mere repetitive tests.

1 0

BaseX 7.1, XMLPrague, Review
by Christian Grün 16 Jan '12

16 Jan '12

Hi all, thanks for your continuous feedback and support (such as contributing to our Wiki, to our code base, or giving feedback on this list). For those who have noticed that BaseX 7.1 was announced for the middle of January: we will be postponing the release a little and align it with the upcoming XMLPrague conference. Next, our XMLPrague BaseX Meetup has now been confirmed. It will take place on Feb 10 (Friday) from 09:00-11:30 am. Please find more information on the XMLPrague and on our homepage soon [1]. Last but not least, we are glad to report that 2011 was a very successful year for BaseX, both regarding the commercial branch as well as the open source development! Rest assured that 2012 will get exciting again.. Stay tuned, Christian [1] http://www.xmlprague.cz/

1 0

Re: [basex-talk] URI for xquery : collection()
by Christian Grün 16 Jan '12

16 Jan '12

Dear Francesco, in oXygen, if your databases are locally stored, you can directly address the database you are working with by using e.g. the db:open() function: for $text in db:open("lettres_20120109", "Info2085.xml")/tei:TEI .... Currently, however, you'lll not be able to traverse collections on a remote server via oXygen; in this case, you'll have to resort to one of the other APIs/tools, or wait for an updated version of our XQJ implementation, which will also support remote connections. Hope this helps, more questions are welcome, Christian On Mon, Jan 16, 2012 at 7:12 PM, Francesco Beretta <Francesco.Beretta(a)ish-lyon.cnrs.fr> wrote: > Hi, > > as a beginner with BaseX, I'm trying to execute a xquery via an OxygenXML > transformation scenario on an external server. > > I created the data source and connection as explained in the BaseX wiki. > > The xquery functions well on a single document, with following connection > syntax : > > "for $text in > doc('http://admin:*****@10.211.55.5:8984/webdav/lettres_20120109/Info2085.xml')/tei:TEI > ..." > > But it doesn't work against the whole database if I write : > "for $text in > collection('http://admin:*****@10.211.55.5:8984/webdav/lettres_20120109')/tei:TEI > ...". > > It works fine if I use the baseX GUI on the 10.211.55.5 server (which > becomes a localhost) : > "for $text in collection('lettres_20120109')/tei:TEI" > > What I'm doing wrong ? > > Many thanks and greetings > > Francesco Beretta

1 0

Jump to page:

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

BaseX-Talk