BaseX-Talk

basex-talk@mailman.uni-konstanz.de

4 participants
5268 discussions

fulltext queries that span elements / reducing result sets to most specific matches
by Imsieke, Gerrit, le-tex 10 May '10

10 May '10

Dear Team, I found that fulltext queries that do not retrieve text nodes are significantly slower than querying text nodes as in text()[. contains text 'sometext'] Example: There's some sample data that may contain both CO2 and CO2. I want a query that returns both kinds of results. With my sample data of approx. 2700 documents, 70 MB, 2.6M nodes, height 18, a query for text()[. contains text 'CO'] takes approx. 5 seconds and returns approx. 900 results. A query for *[. contains text 'CO'] takes 27 seconds (and returns 1600 results). Now *[. contains text 'CO2'] also takes 27 seconds and returns 1300 results. Since the subscript is properly marked up in the sample documents, text()[. contains text 'CO2'] returns 0 results. Can you think of a way that will speed up fulltext queries to element content, i.e., the *[. contains text '...'] scenario? -------- A related addition (thereby violating the "one topic, one mailing list posting" dogma slightly): Given a document <doc><sect>CO2</sect></doc>, the query *[. contains text 'CO2'] will contain 3 results: doc, sect, and p. But I'm only interested in the "shortest possible" or "most specific" result, i.e., a result that does not contain any other element satisfying the same query. In above example, the shortest possible result will be p. This may be achieved by the following query: let $prelim := //*[. contains text 'CO2' ] for $d in $prelim[every $c in * satisfies empty($c intersect $prelim)] return <result>{ $d }</result> The predicate [every $c in * satisfies empty($c intersect $prelim)] doesn't slow down the query significantly, at least with BaseX operating on my sample data. Thus it's important to optimize the initial //*[. contains text 'CO2' ] query. Would it help if BaseX knew that we only need the innermost element satisfying the FT query? So if there were, for example, an option //*[. contains text 'CO2' using option basex:element-scope "most-specific"], could BaseX use this in order to make use of its indexes and accelerate the query? I'm not sure, because if some text occurs within a certain element way down the tree, a distinct occurence of same text may be found higher up in the tree, immediately below an ancestor element: <a>text foo <c>text</c> bar</a> So it's important to look at all the elements in the tree, with no speed gain if you look at the bottom-most elements first. So I think it boils down to the issue of accelerating fulltext queries that span element boundaries. ------ You could argue that if I knew in advance which elements to search and use specific elements instead of *, optimization will be easier (so do the qizx people, if I remember correctly). But there are different document types in the DB (other types may be added at any time without prior notice), so it's hard to specify in advance which elements should be queried/returned. And even if you focused your query on, e.g., para elements in DocBook documents, there are situations like <para><orderedlist><listitem><para>CO<subscript>2</subscript></para></listitem></orderedlist></para> where you cannot restrict a query to paras because you will get the occuring text twice, in the absence of additional filtering. Or even worse with CALS tables in DocBook: an entry may or may not contain a para. Searching for paras that contain some text will return (1) the para that contains the whole table plus (2) a para within this table ONLY IF the author chose to mark up the inner para (which he doesn't have to). Filtering the results or using the fictitious "most-specific" scope option, the result will be narrowed down to either the entry or, if present, the para within the entry. You don't need to filter *[self::para[not(descendant::tgroup or descendant::orderedlist or ...)] or self::entry or self::simpara or ...][. contains text 'CO2'] wihch might ultimately work for DocBook but not for the general case (in our environment, there will be DocBook, XHTML, dozens of customer-specific schema variants and an unpredictable number of 3rd-party schemas). ------ This message has become quite lengthy. I hope you don't mind. Gerrit -- Gerrit Imsieke Geschäftsführer / Managing Director le-tex publishing services GmbH Weissenfelser Str. 84, 04229 Leipzig, Germany Phone +49 341 355356 110, Fax +49 341 355356 510 gerrit.imsieke(a)le-tex.de, http://www.le-tex.de Registergericht / Commercial Register: Amtsgericht Leipzig Registernummer / Registration Number: HRB 24930 Geschäftsführer: Gerrit Imsieke, Svea Jelonek, Thomas Schmidt, Dr. Reinhard Vöckler

2 1

some generic BaseX questions
by Robby Pelssers 10 May '10

10 May '10

Hi all, * I'm wondering where baseX stores it's database by default and how one can override the location? * What approach is mostly taken to backup the database /and how can I restore it? * What would a production server typically consume in memory? * Did anyone encounter database problems (e.g. corrupt) and what could be the root cause (out-of-memory, ...) ? Basically I'm looking for people who have hands-on experience with using BaseX in a high performance critical production environment and what their experience is so far. Cheers, Robby Pelssers

1 0

question about inserting new data (using JAX-RX)
by Robby Pelssers 10 May '10

10 May '10

When reading following snippet from the documentation I start wondering how-to specify that I want to add this resource to a specific collection in the database. And the same basically applies to GET, DELETE, ... How does one specifically target a resource from a collection using JAX-RX? Kind regards, Robby Pelssers ---------------------------------------- PUT Requests PUT can be used to create or update a database resource. To create a new resource called XMark, the following URL http://localhost:8984/basex/jax-rx/XMark must be sent to the server as PUT request, followed by the XML file in the HTTP body. If the process was successful, a HTTP response with status code 201 (CREATED) is sent back. Otherwise, 404 will be sent. ----------------------------------------

2 1

jax-rx and external variables
by Hans-Peter Oeri 10 May '10

10 May '10

Hi! While learning about xml databases and how to access them, I came across your jax-rx project. While most of REST is quite straightforward, I don't see any method to define external variables. I don't quite understand why the possibility to define not only the xquery, but also the values for its external variables has been left out? It would - after all - be nice... Just like parameter substitution in SQL APIs... POST could get variable values and value types in an additional <variables> block. GET could accept parameters like var.* - probably deriving the type from the variable declaration in the xquery. If the http querystring (which always is a string...) cannot be converted accordingly, it could throw. What do you think? HPO

2 3

XQuery injection and auto-increment
by Kim Stebel 10 May '10

10 May '10

Hi! I'm planning to use BaseX as the database behind a web service. While implementing this web service, I ran into these issues: 1) Is there a way to have an auto-incremented attribute added on every insert of a certain element type? 2) How do I avoid XQuery injection? Currently I just use a whitelist of characters that are allowed in user input. But what if I want users to be able to input special characters? How would I escape them in my queries? Did I overlook something in the API? regards Kim

2 1

is there a function to update attributes?
by Beatriz Sancho 10 May '10

10 May '10

Hello, I'm trying to update or create some attributes in a node. ¿how can i do that with BaseX? This is my DTD: <!ELEMENT SITUATION ((ROLES)?,(USERS)?,(GROUPS)?,(OBJECTS)?)> <!ATTLIST SITUATION id ID #REQUIRED name CDATA #IMPLIED desc CDATA #IMPLIED start CDATA #IMPLIED finish CDATA #IMPLIED> category_scope (small | large | community | other) #IMPLIED size_scope CDATA #IMPLIED environment ( synchronous | asynchronous | face-to-face | blended | others) #IMPLIED educational_level CDATA #IMPLIED experience_level (none|elementary|advanced|expert) #IMPLIED> and i want to update or create attributes in the SITUATION node. I have seen a function called insertAttributes but i don't know how to use it with baseX, with the prefix basex: it doesn't works... i am a little lost. Could you help me, please?

2 2

ways to insert and query for documents
by Robby Pelssers 07 May '10

07 May '10

Hi all, I was wondering if BaseX also supports the concept of having collections in the database. The reason why I need collections is that I have xml files with the same identifier. My customer exports product information for different states (workflow | work in progress | released) to the file system currently but we would like to export everything to an xml database. So I'd like to create 3 separate collections for the 3 states and being able to query for product 'X' in state 'Y'. Is this possible? Robby Pelssers

2 3

Re: [basex-talk] Strange query output
by Christian Grün 06 May '10

06 May '10

Thanks Toon, it's always interesting to hear about bugs that should have been encountered much earlier. This one might need some more time to be fixed, that's why I've added a sourceforge bug entry: https://sourceforge.net/tracker/?func=detail&aid=2997544&group_id=192179&at… Your query can be broken down to the following one: //a[b = //b] The general problem is that we attach text() location steps to a path if we know that the correspondent node is a leaf node. This way, we can rewrite queries to use the index more easily. I.e., the upper query is rewritten to: //a[b/text() = //b/text()] As the b elements are leaves in your example, but never have text() children, the predicate yields false… //a[ () = () ] In fact, for the given document, your test (b = //b) will result in an empty string test "" = "". This might be why no one else has stumbled over this before. The query info view in BaseX gives you some hint how queries are evaluated (…maybe you're using it anyway). Thanks, Christian PS: I've posted this to the mailing list to let others know as well. On Thu, May 6, 2010 at 11:05 AM, Calders, T.G.K. <t.calders(a)tue.nl> wrote: > Dear Christian, > > Sorry to bug you again. As you can guess, my lecture is on Thursday morning. > > The following query still gives an incorrect output in basex 6.1.1: > ----- > for $b in //b, $a in //a > where $a/b=$b > return <binding> > {$b} > {$a} > </binding> > ----- > When applied on the following document: > ----- > <doc> > <a id="1"> > > <c>1a</c> > <c>1b</c> > </a> > <a id="2"> > > <c>2a</c> > <c>2b</c> > </a> > </doc> > ----- > It returns an empty answer, whereas the correct answer creates 4 "binding"-elements (the where clause does not filter anything away as both "b"-elements in the document evaluate to "" under string()). > > On the other hand, > ----- > let $x:=<doc> > <a id="1"> > > <c>1a</c> > <c>1b</c> > </a> > <a id="2"> > > <c>2a</c> > <c>2b</c> > </a> > </doc> > > > for $b in $x//b, $a in $x//a > where $a/b=$b > return <binding> > {$b} > {$a} > </binding> > ----- > Does produce the correct output. I noticed that this error is present in some earlier versions of basex as well (e.g., in version 5.7 which I used last year). The query was evaluated correctly, though, in versions 4.0 and 5.0. > > Best regards, > Toon > > > -----Original Message----- > From: Christian Grün [mailto:christian.gruen@gmail.com] > Sent: donderdag 29 april 2010 16:51 > To: Calders, T.G.K. > Cc: basex-talk(a)mailman.uni-konstanz.de > Subject: Re: Strange query output > > Dear Toon Calders, > > thanks for your detailed bug report. Yes, this was a serious issue; > the predicate was optimized for an accessing the index, although it > was dependent on the query context. This is now fixed in the > repository version 6.1.1 of BaseX: > > http://basex.org/maven/org/basex/basex/6.1.1/ > > I've forwarded your mail to the mailing list to let others know about > it as well. > > All the best, > Christian > > > > On Thu, Apr 29, 2010 at 12:17 PM, Calders, T.G.K. <t.calders(a)tue.nl> wrote: >> Dear BaseX developers, >> >> First of all, many thanks for making this nice tool freely available. I use >> it in my Database class so the students can practice a bit with XPath and >> XQuery and they are all, without any exception, very enthusiastic about the >> tool. >> >> During class today, however, one of the students encountered a strange query >> output; I reduced the example to its essence: >> >> For the following document: >> >> <errordoc> >> <instructor fname="Toon" sname="Calders"/> >> <lecturer fname="Toon" sname="Calders"/> >> </errordoc> >> >> We ran the query "//instructor[@sname=../lecturer/@sname]" which returned >> an empty answer when executed with BaseX 6.1. (with previous versions it >> seems to work fine; e.g., 5.7). >> >> On the other hand, the query: >> >> let $x:=<errordoc> >> <instructor fname="Toon" sname="Calders"/> >> <lecturer fname="Toon" sname="Calders"/> >> </errordoc> >> >> return $x//instructor[@sname=../lecturer/@sname] >> >> does return: >> <instructor fname="Toon" sname="Calders"/> >> >> Best regards, >> Toon Calders >> TU Eindhoven ___________________________ Christian Gruen Universitaet Konstanz Department of Computer & Information Science D-78457 Konstanz, Germany Tel: +49 (0)7531/88-4449, Fax: +49 (0)7531/88-3577 http://www.inf.uni-konstanz.de/~gruen

1 0

XPath function and same node name
by Grégoire Colbert 03 May '10

03 May '10

Hi, I have a long XML document like this : <cities> <city> <name>Paris</name> <zip>75000</zip> <alt_names> <name>paris</name> </alt_names> </city> <city> <name>Cesson-Sévigné</name> <zip>35510</zip> <alt_names> <name>paris</name> </alt_names> </city> </cities> My XQuery is supposed to return the cities whose name begin with a few letters contained in "$cityname". So I use this : let $search := request:get-parameter("search","") return <cities> { for $city in collection("cities")//city[starts-with(alt_names/name,$search)] return <city> {$city/name} {$city/zip} </city> } </cities> But there is a problem with the [starts-with(alt_names/name,$search)] part. It seems that I cannot use the same tag name ("name") both in "city" and in "alt_names". The error is : [XPTY0004] No sequence (element name { ... }, element name { ... }) allowed. 1) Is this a bug? 2) What is the equivalent in BaseX to request:get-parameter, which works in "eXist db" and affects the GET parameter (here "search") to the $search variable? Thank you for your help, Regards, Grégoire

2 2

Fwd: Strange query output
by Christian Grün 29 Apr '10

29 Apr '10

Philip, as long as you don't work with our "api" and "example" packages, you don't need Maven at all. Talking about myself… I'm only using the Maven plugin to dissolve dependencies. I've posted this to the list, so maybe someone else can help you with that? Christian On Thu, Apr 29, 2010 at 5:40 PM, Philip Oliver <phil3(a)olivercomputing.com> wrote: > Christian - I was trying to update my Eclipse installation (Galileo SR2 on OS X) with the maven integration and it blew up my Eclipse installation. Not a BaseX problem but still annoying. I don't know much about Maven but I think I'd prefer to just build from the command line if possible using 'mvn' which is what I was doing with the 'mvn install'. Can you say what I should do in order to use 'mvn' at the command line (rather than Eclipse) to build BaseX to get a jar? > > Phil > > On Apr 29, 2010, at 11:37 AM, Christian Grün wrote: > >> Philip, >> >> thanks for the hint, it was work in progress. All JUnit tests should >> now run as expected. If not, just tell us (we note that many people >> have difficulties with the Eclipse/Maven integration, so I'm not sure >> if the current configuration meets all demands). >> >> Christian >> >> >> >> On Thu, Apr 29, 2010 at 5:00 PM, Philip Oliver >> <phil3(a)olivercomputing.com> wrote: >>> Christian, >>> >>> In light of the bug report, I updated my SVN repository copy and did a 'mvn install', and got the following errors in the automated tests: >>> >>> ------------------------------------------------------- >>> T E S T S >>> ------------------------------------------------------- >>> Running org.basex.test.query.NamespaceTest >>> Tests run: 4, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.418 sec <<< FAILURE! >>> Running org.basex.test.query.FTTest >>> Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.09 sec <<< FAILURE! >>> Running org.basex.test.cs.SemaphoreTest >>> Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.004 sec <<< FAILURE! >>> Running org.basex.test.cs.PermissionTest >>> BaseX 6.1.1 [Server] >>> >>> >>> >>> I don't know how significant these are, or if it's a result of me not properly building BaseX, but FYI. >>> >>> >> > >

1 0

Jump to page:

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

BaseX-Talk