BaseX-Talk

basex-talk@mailman.uni-konstanz.de

4 participants
5268 discussions

XQFT and XQUF finalized
by Christian Grün 18 Mar '11

18 Mar '11

Dear members, a quick information to everyone: the XQuery Full Text and XQuery Update Facility specifications have just been finalized: http://www.w3.org/TR/xpath-full-text-10/ http://www.w3.org/TR/xquery-update-10/ This means that our XQFT and XQUF implementation won't be subject to sudden changes anymore. This doesn't apply to XQuery 3.0 yet; we continue doing our best, however, to support all features of the latest XQuery 3.0 working draft. Best, Christian -------- Forwarded Message -------- From: Ian Jacobs <ij(a)w3.org> [...] Dear Advisory Committee Representative, I am pleased to announce two new XQuery Recommendations: XQuery and XPath Full Text 1.0 http://www.w3.org/TR/2011/REC-xpath-full-text-10-20110317/ published by the XML Query Working Group [1] XQuery Update Facility 1.0 http://www.w3.org/TR/2011/REC-xquery-update-10-20110317/ published jointly by the XML Query and XSL Working Groups [2] The W3C Advisory Committee who responded to the calls for review ([3], [4]) supported the publication of the two documents as W3C Recommendations. There was a request that some minor issues arising during the review period be addressed, and this was done. There were no Formal Objections. Please join us in congratulating the groups for their achievement. For Tim Berners-Lee, Director; Ian Jacobs, Head of W3C Communications. [1] http://www.w3.org/XML/Query/ [2] http://www.w3.org/Style/XSL/ [3] http://lists.w3.org/Archives/Member/w3c-ac-members/2011JanMar/0017 [4] http://lists.w3.org/Archives/Member/w3c-ac-members/2011JanMar/0018 ====================================== XQuery and XPath Full Text 1.0 http://www.w3.org/TR/2011/REC-xpath-full-text-10-20110317/ ------ Abstract This document defines the syntax and formal semantics of XQuery and XPath Full Text 1.0, which is a language that extends XQuery 1.0 [XQuery 1.0: An XML Query Language (Second Edition)] and XPath 2.0 [XML Path Language (XPath) 2.0 (Second Edition)] with full-text search capabilities. ------ Status of this Document This is a Recommendation of the W3C. It was jointly developed by the W3C XML Query Working Group and the W3C XSL Working Group, each of which is part of the XML Activity. This document incorporates minor changes made against the Proposed Recommendation of 25 January 2011. Changes to this document since the Proposed Recommendation are detailed in J Change Log. A Java applet that parses XQuery and XPath Full Text 1.0 expressions is available at http://www.w3.org/2010/02/qt-applets/xquery10-fulltext/ . A Test Suite has been created for this document. Implementors are encouraged to run this test suite and report their results. The Test Suite can be found at http://dev.w3.org/cvsweb/2007/xpath-full-text-10-test-suite/ . An implementation report is available at http://dev.w3.org/2007/xpath-full-text-10-test-suite/PublicPagesStagingArea… . No substantive changes have been made to this specification since its publication as a Proposed Recommendation. Please report errors in this document using W3C's public Bugzilla system (instructions can be found at http://www.w3.org/XML/2005/04/qt-bugzilla) . If access to that system is not feasible, you may send your comments to the W3C XSLT/XPath/XQuery public comments mailing list, public-qt-comments(a)w3.org . It will be very helpful if you include the string “[FT]” in the subject line of your report, whether made in Bugzilla or in email. Please use multiple Bugzilla entries (or, if necessary, multiple email messages) if you have more than one comment to make. Archives of the comments and responses are available at http://lists.w3.org/Archives/Public/public-qt-comments/ . This document has been reviewed by W3C Members, by software developers, and by other W3C groups and interested parties, and is endorsed by the Director as a W3C Recommendation. It is a stable document and may be used as reference material or cited from another document. W3C's role in making the Recommendation is to draw attention to the specification and to promote its widespread deployment. This enhances the functionality and interoperability of the Web. This document was produced by groups operating under the 5 February 2004 W3C Patent Policy. W3C maintains a public list of any patent disclosures made in connection with the deliverables of the XML Query Working Group and also maintains a public list of any patent disclosures made in connection with the deliverables of the XSL Working Group; those pages also include instructions for disclosing a patent. An individual who has actual knowledge of a patent which the individual believes contains Essential Claim(s) must disclose the information in accordance with section 6 of the W3C Patent Policy. ====================================== XQuery Update Facility 1.0 http://www.w3.org/TR/2011/REC-xquery-update-10-20110317/ ------ Abstract This document defines an update facility that extends the XML Query language, XQuery. The XQuery Update Facility provides expressions that can be used to make persistent changes to instances of the XQuery 1.0 and XPath 2.0 Data Model. ------ Status of this Document This is a Recommendation of the W3C. It was developed by the W3C XML Query Working Group, which is part of the XML Activity. This document incorporates minor changes made against the Proposed Recommendation of 25 January 2011. Changes to this document since the Proposed Recommendation are detailed in [H Revision Log]. A Java applet that parses XQuery Update Facility 1.0 expressions is available at http://www.w3.org/2010/02/qt-applets/xquery10-update/. A Test Suite has been created for this document. Implementors are encouraged to run this test suite and report their results. The Test Suite can be found at http://dev.w3.org/2007/xquery-update-10-test-suite/ . An implementation report is available at http://dev.w3.org/2007/xquery-update-10-test-suite/results/ . No substantive changes have been made to this specification since its publication as a Proposed Recommendation. Please report errors in this document using W3C's public Bugzilla system (instructions can be found at http://www.w3.org/XML/2005/04/qt-bugzilla) . If access to that system is not feasible, you may send your comments to the W3C XSLT/XPath/XQuery public comments mailing list, public-qt-comments(a)w3.org . It will be very helpful if you include the string “[UPD]” in the subject line of your report, whether made in Bugzilla or in email. Please use multiple Bugzilla entries (or, if necessary, multiple email messages) if you have more than one comment to make. Archives of the comments and responses are available at http://lists.w3.org/Archives/Public/public-qt-comments/ . This document has been reviewed by W3C Members, by software developers, and by other W3C groups and interested parties, and is endorsed by the Director as a W3C Recommendation. It is a stable document and may be used as reference material or cited from another document. W3C's role in making the Recommendation is to draw attention to the specification and to promote its widespread deployment. This enhances the functionality and interoperability of the Web. This document was produced by a group operating under the 5 February 2004 W3C Patent Policy. W3C maintains a public list of any patent disclosures made in connection with the deliverables of the group; that page also includes instructions for disclosing a patent. An individual who has actual knowledge of a patent which the individual believes contains Essential Claim(s) must disclose the information in accordance with section 6 of the W3C Patent Policy. -- Ian Jacobs (ij(a)w3.org) http://www.w3.org/People/Jacobs/ Tel: +1 718 260 9447 -- Liam Quin - XML Activity Lead, W3C, http://www.w3.org/People/Quin/ Pictures from old books: http://fromoldbooks.org/ Ankh: irc.sorcery.net irc.gnome.org www.advogato.org

1 0

Using BaseX with Scripts
by Adrien Guillon 17 Mar '11

17 Mar '11

Hello BaseX community, I am writing a script to load a RDBMS with some information from large XML files (1GB). I am using BaseX and XQuery to process the files and generate appropriate data dumps for bulk loading the database. My ultimate goal is to write a couple BASH scripts that will just load up the database, however I need some advice on how to script BaseX. I would like to load the XML file, generate a couple indexes, run my queries, then delete the indexes. It does not look like BaseX is so easily scripted from the command line, unless I initially load the database or run a BaseX server. I have tried running BaseX just using doc() inside the XQuery but this takes forever... Ideally I would want to write as script like this: #!/bin/bash basex --generate-index --dbname="hello_world" file.xml basex --query qry.xml > result.xml basex --delete --dbname="hello world" Any suggestions on how I can do this? Thanks, AJ

2 3

XQuery Update: insert or replace depending if node exists not possible ?
by Petr Vacek 17 Mar '11

17 Mar '11

Greetings, I am trying to build simple XML database (inside baseX or existdb - but baseX seems nicer for java APIs), but I have trouble figuring how to modify values in document : content is simple as this for test : <p> </pl> <id>6></id> </pl> </p> I am trying to build something like function which would insert element into <pl> if that element is not present or replace it if it is present. But Xquery is giving me troubles yet : When I try it head-on with if-then-else logic : if (exists(/p/pl[id=6]/name)=false) then insert node <name>thenname</name> into /p/pl[id=6] else replace value of node /p/pl[id=6]/name with 'elsename' I get error Error: [XUDY0027] Replace target must not be empty. - clearly I am confused, why the else part is evaluated in both cases, thus the error. When I empty out the else part : if (exists(/p/pl[id=6]/name)=true) then insert node <name>thenname</name> into /p/pl[id=6] else <dummy/> Then I get Error: [XUST0001] If expression: no updating expression allowed. When I try through declaring updating function, even then it reports error : declare namespace testa='test'; declare updating function testa:bid($a, $b) { if (exists(/p/pl[id=6]/name)=true) then insert node <name>thenname</name> into /p/pl[id=6] else <dummy/> }; testa:bid(0,0) Error: [XUST0001] If expression: no updating expression allowed. I've got these errors from baseX 6.5.1 package. So how can I modify values in a simple fashion if possible ? If I call insert straight, the there could be multiple elements of same value. If I call replace it will fail when the node does not exist. If I delete the node before insert/replace then I could destroy sub-nodes which I don't want. In most SQL databases, these are quite simple task (like MYSQL 'replace' command). Thanks in advance for any hints and tips how to get basic 'replace' functionality done. Regards Petr Vacek

3 2

Optimal query searching for maximal string value
by Jan Vlčinský (CAD) 16 Mar '11

16 Mar '11

Hi I am searching for the most effective way for returning element with highest string value. *Situation* I have heap of xml files, containing MSG elements, having attribute id and inner element TGEN, denoting time of MSG creation <MSG id="abcd"> <TGEN>2011-02-23T15:21:14-01:00<TGEN> <some-content-here/> </MSG> My aim is to select for each id value the latest version (having highest value of TGEN) For comparing TGEN it is sufficient to treat it as string value, there is no need to cast it to dateTime, which is expensive. *My question:* How to form the query, which will use existing indexes most efficiently. I am not yet very familiar with reading query plans so I am not able to be sure, it goes the most efficient way. *Background* built in max function converts values into xs:double http://www.xqueryfunctions.com/xq/functx_max-string.html declares function max-string declare namespace functx = "http://www.functx.com"; declare function functx:max-string ( $strings as xs:anyAtomicType* ) as xs:string? { max(for $string in $strings return string($string)) } ; *Sample data file* Sample XML can be created using following xquery element MJD { for $id in 1 to 10, $age_in_hours in 24 to 48 let $age := xs:dayTimeDuration(concat('PT', $age_in_hours, 'H')) let $tgen := xs:dateTime('2011-03-16T12:00:00Z') - $age return element MSG { attribute id {$id}, attribute age {$age}, element TGEN {$tgen} } } *Our existing solution* We already did some optimization, biggest speed up gained by using group by Here is my latest version of xquery let $latest_msgs := (for $msg in //MSG let $id := $msg/@id group by $id return ( (for $m in $msg let $tgen := $m//TGEN order by xs:string($tgen) descending return $m )[1]) ) return <MJD count='{count($latest_msgs)}' created='{max($latest_msgs//xs:string(TGEN))}'>{$latest_msgs}</MJD> Any comments are welcome, especially about efficient use of indexes Jan -- *Ing. Jan Vlčinský* CAD programy Slunečnicová 338/3, 734 01 Karviná Ráj, Czech Republic tel: +420-597 602 024; mob: +420-608 979 040 skype: janvlcinsky; GoogleTalk: jan.vlcinsky(a)gmail.com http://cz.linkedin.com/in/vlcinsky

1 0

Re: [basex-talk] Updates and indexing
by NewIntellectual 15 Mar '11

15 Mar '11

It's worth noting that updating indices after a database update is an essential move in the direction of making BaseX suitable for a realtime dynamic database. I hadn't considered it my earlier remarks to Charles Munat, but losing the indices after an update is, by itself, enough to make it unsuitable currently - they are essential to maintain performance.

2 1

Vastly different database creation times
by David leigh 15 Mar '11

15 Mar '11

My application starts the basex server (and only stops the server when the application closes) and attempts to open the database that it expects to find. If it doesn't find it, it will create it with the source XML files. When a user wants to update the database, it simply drops it and recreates it from the XML files.actually executing the exact same code. I see that the database creation time for those two tasks is quite different. The first one takes quite a bit longer. If, when the application starts, it DOES find the database, it opens it. If the user then clicks "update" it does the drop/recreate process. If it does it again, it does the drop/recreate process. It seems that in these cases, the first one also takes noticeably longer. Subsequent updates are faster. My conclusion is that the first CREATE DB after starting the server takes longer than all the subsequent ones before the server is stopped. Is this a "memory acquisition" issue? I.e. the server starts with a small memory footprint by default and then simply asks (frequently) for more when it needs it? If so, can it be started with a larger amount initially or tell it to grab it in larger quantities in order to reduce the time for the initial database creation? If that has nothing to do with it, any other ideas? Here is my basex log illustrating these two examples: 18:19:15.419 Server was started. 18:19:15.905 [127.0.0.1:60244] LOGIN admin OK 18:19:16.032 [127.0.0.1:60244] OPEN OpenSongSearch Error: Database 'OpenSongSearch' was not found. 32.63 ms 18:19:16.049 [127.0.0.1:60244] LOGOUT admin OK 18:19:16.469 [127.0.0.1:60245] LOGIN admin OK 18:19:16.493 [127.0.0.1:60245] DROP DB OpenSongSearch OK 0.77 ms 18:19:16.493 [127.0.0.1:60245] SET CREATEFILTER * OK 0.66 ms 18:19:16.494 [127.0.0.1:60245] SET ENTITY true OK 0.42 ms 18:19:16.494 [127.0.0.1:60245] SET FTINDEX true OK 0.36 ms 18:19:16.495 [127.0.0.1:60245] SET INTPARSE true OK 0.36 ms 18:19:16.495 [127.0.0.1:60245] SET PARSER xml OK 0.32 ms 18:19:16.496 [127.0.0.1:60245] SET WILDCARDS true OK 0.34 ms 18:20:31.458 [127.0.0.1:60245] CREATE DB OpenSongSearch C:\Users\David\DOCUME~1\OpenSong\Songs OK 74961.71 ms 18:20:31.460 [127.0.0.1:60245] LOGOUT admin OK 18:20:33.146 [127.0.0.1:60262] LOGIN admin OK 18:20:33.175 [127.0.0.1:60262] OPEN OpenSongSearch OK 9.2 ms 18:20:33.177 [127.0.0.1:60262] INFO DB OK 1.66 ms 18:20:33.177 [127.0.0.1:60262] CLOSE OK 0.38 ms 18:20:33.178 [127.0.0.1:60262] LOGOUT admin OK 18:21:10.314 [127.0.0.1:60270] LOGIN admin OK 18:21:10.340 [127.0.0.1:60270] DROP DB OpenSongSearch OK 7.85 ms 18:21:10.340 [127.0.0.1:60270] SET CREATEFILTER * OK 0.25 ms 18:21:10.341 [127.0.0.1:60270] SET ENTITY true OK 0.29 ms 18:21:10.341 [127.0.0.1:60270] SET FTINDEX true OK 0.23 ms 18:21:10.341 [127.0.0.1:60270] SET INTPARSE true OK 0.23 ms 18:21:10.342 [127.0.0.1:60270] SET PARSER xml OK 0.25 ms 18:21:10.342 [127.0.0.1:60270] SET WILDCARDS true OK 0.25 ms 18:21:17.190 [127.0.0.1:60270] CREATE DB OpenSongSearch C:\Users\David\DOCUME~1\OpenSong\Songs OK 6848.68 ms 18:21:17.195 [127.0.0.1:60270] LOGOUT admin OK 18:21:18.140 [127.0.0.1:60272] LOGIN admin OK 18:21:18.167 [127.0.0.1:60272] OPEN OpenSongSearch OK 9.26 ms 18:21:18.168 [127.0.0.1:60272] INFO DB OK 1.05 ms 18:21:18.169 [127.0.0.1:60272] CLOSE OK 0.36 ms 18:21:18.169 [127.0.0.1:60272] LOGOUT admin OK 18:21:20.636 Server was stopped. 19:11:23.757 Server was started. 19:11:24.319 [127.0.0.1:61004] LOGIN admin OK 19:11:24.899 [127.0.0.1:61004] OPEN OpenSongSearch OK 517.02 ms 19:11:24.903 [127.0.0.1:61004] INFO DB OK 3.75 ms 19:11:24.905 [127.0.0.1:61004] CLOSE OK 2.23 ms 19:11:24.906 [127.0.0.1:61004] LOGOUT admin OK 19:11:25.358 [127.0.0.1:61005] LOGIN admin OK 19:11:25.724 [127.0.0.1:61005] OPEN OpenSongSearch OK 348.05 ms 19:11:25.727 [127.0.0.1:61005] INFO DB OK 2.81 ms 19:11:25.728 [127.0.0.1:61005] CLOSE OK 0.88 ms 19:11:25.729 [127.0.0.1:61005] LOGOUT admin OK 19:11:49.961 [127.0.0.1:61013] LOGIN admin OK 19:11:49.985 [127.0.0.1:61013] DROP DB OpenSongSearch OK 6.82 ms 19:11:49.986 [127.0.0.1:61013] SET CREATEFILTER * OK 0.67 ms 19:11:49.986 [127.0.0.1:61013] SET ENTITY true OK 0.32 ms 19:11:49.987 [127.0.0.1:61013] SET FTINDEX true OK 0.37 ms 19:11:49.987 [127.0.0.1:61013] SET INTPARSE true OK 0.34 ms 19:11:49.988 [127.0.0.1:61013] SET PARSER xml OK 0.4 ms 19:11:49.988 [127.0.0.1:61013] SET WILDCARDS true OK 0.34 ms 19:12:09.591 [127.0.0.1:61013] CREATE DB OpenSongSearch C:\Users\David\DOCUME~1\OpenSong\Songs OK 19602.28 ms 19:12:09.592 [127.0.0.1:61013] LOGOUT admin OK 19:12:10.547 [127.0.0.1:61018] LOGIN admin OK 19:12:10.598 [127.0.0.1:61018] OPEN OpenSongSearch OK 8.42 ms 19:12:10.599 [127.0.0.1:61018] INFO DB OK 0.96 ms 19:12:10.600 [127.0.0.1:61018] CLOSE OK 0.39 ms 19:12:10.601 [127.0.0.1:61018] LOGOUT admin OK 19:12:20.716 [127.0.0.1:61021] LOGIN admin OK 19:12:20.742 [127.0.0.1:61021] DROP DB OpenSongSearch OK 8.13 ms 19:12:20.742 [127.0.0.1:61021] SET CREATEFILTER * OK 0.27 ms 19:12:20.743 [127.0.0.1:61021] SET ENTITY true OK 0.26 ms 19:12:20.743 [127.0.0.1:61021] SET FTINDEX true OK 0.25 ms 19:12:20.744 [127.0.0.1:61021] SET INTPARSE true OK 0.27 ms 19:12:20.744 [127.0.0.1:61021] SET PARSER xml OK 0.25 ms 19:12:20.745 [127.0.0.1:61021] SET WILDCARDS true OK 0.26 ms 19:12:26.555 [127.0.0.1:61021] CREATE DB OpenSongSearch C:\Users\David\DOCUME~1\OpenSong\Songs OK 5811.0 ms 19:12:26.557 [127.0.0.1:61021] LOGOUT admin OK 19:12:27.677 [127.0.0.1:61024] LOGIN admin OK 19:12:27.705 [127.0.0.1:61024] OPEN OpenSongSearch OK 7.94 ms 19:12:27.706 [127.0.0.1:61024] INFO DB OK 0.99 ms 19:12:27.707 [127.0.0.1:61024] CLOSE OK 0.37 ms 19:12:27.707 [127.0.0.1:61024] LOGOUT admin OK 19:12:29.820 Server was stopped. Thanks!

2 2

Re: [basex-talk] Updates and indexing
by Gary Larsen 15 Mar '11

15 Mar '11

Hi Dimitar, Thanks for the info. I see that adding a document changes the database indexing status to off. A query to locate a document by node value in a database with 54k documents went from 3 seconds to 4 minutes. I can image its difficult to balance performance in different aspects of the application and maintain database stability at the same time. For the past couple of weeks Ive tested query performance on a half dozen databases and BaseX queries were almost always quickest and never crashed even when running with lower memory. Depending on how things go with my research I may be able to help test the index updating feature when you are ready for that. Thanks, Gary >Hi Gary, >I'm the one who's working on the index updating. As Michael said it's a work >in progress, but I expect to finish my work soon. Most probably it will be >released at some point after 6.6 (which should be available on March 23). >Best regards, >Dimitar Am Dienstag 15 März 2011, 10:01:20 schrieb Michael Seiferle: > Hi Gary, > > yes this is true either way, still in 6.5.1. > We are currently working on incremental index updates, but I unfortunately > can't give you any hint on a release date. > > For most use-cases recreating the indexes (OPTIMIZE [1]) works fine in day > to day use, but we are aware that this might get a little annoying under > some circumstances. > > Thanks a lot for your feedback, feel free to discuss or ask for more > information. -- > > Kind regards > Michael From: Gary Larsen [mailto:gary.larsen@envisn.com] Sent: Monday, March 14, 2011 8:08 PM To: 'basex-talk(a)mailman.uni-konstanz.de' Subject: Updates and indexing Hi, I was reading a thread in the mail list from Oct 2010 which states updates are not indexed: https://mailman.uni-konstanz.de/pipermail/basex-talk/2010-October/000707.htm l Is this still the case with version 6.5.1? When talking of updates does that mean whole document replacement (or deletion) as well as XQuery updates? Thanks for any clarification, gary

1 0

Best practice approach to data security between orgs while maintaining ability to combine data from multiple orgs
by Charles F. Munat 15 Mar '11

15 Mar '11

Hello, I'm building a Web-based application that tracks students in various schools. Much of the data is tree-structured, and, after numerous attempts, I've given up on using an RDBMS. It's a hammer and I need a wrench. The schema for each school's data will be the same. Sitting above the schools is the "project" for which the application is being built. The project needs to be able to query across multiple or all schools in order to run various analyses. So I want all the data in a single database, if possible. But each school, I think, should be it's own "segment" of the database. That way, when a project manager logs in, he or she can look across all the segments, but when a school coordinator logs in, he or she accesses only the data for that school. I hope this makes sense. I am not really very familiar with how the BaseX DB is organized. I don't want individual XML files (do I?). I just want a database filled with data. So how do "collections" work? How would I segment the database so that schools are separate whiles still providing easy cross-school access for project personnel? I could just do something like this: <project> <project-data/> <school id="1"/> <school id="2"/> </project> Where each school is simply an element under the root "project" element (with project data stashed away in a "project-data" element, or equivalent). But is there a better way? What is the best practice here? I am running the DB unembedded. Thanks! Chas. Munat Somewhere in South America

5 7

Re: [basex-talk] Updates and indexing
by Michael Seiferle 15 Mar '11

15 Mar '11

Hi Gary, yes this is true either way, still in 6.5.1. We are currently working on incremental index updates, but I unfortunately can't give you any hint on a release date. For most use-cases recreating the indexes (OPTIMIZE [1]) works fine in day to day use, but we are aware that this might get a little annoying under some circumstances. Thanks a lot for your feedback, feel free to discuss or ask for more information. -- Kind regards Michael [1] http://docs.basex.org/wiki/Update#Indexes Am 15.03.2011 um 01:08 schrieb Gary Larsen: > Hi, > > > > I was reading a thread in the mail list from Oct 2010 which states updates are not indexed: > > > > https://mailman.uni-konstanz.de/pipermail/basex-talk/2010-October/000707.ht… > > > > Is this still the case with version 6.5.1? When talking of updates does that mean whole document replacement (or deletion) as well as XQuery updates? > > > > Thanks for any clarification, > > gary > >

2 1

Updates and indexing
by Gary Larsen 14 Mar '11

14 Mar '11

Hi, I was reading a thread in the mail list from Oct 2010 which states updates are not indexed: https://mailman.uni-konstanz.de/pipermail/basex-talk/2010-October/000707.htm l Is this still the case with version 6.5.1? When talking of updates does that mean whole document replacement (or deletion) as well as XQuery updates? Thanks for any clarification, gary

1 0

Jump to page:

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

BaseX-Talk