BaseX-Talk

basex-talk@mailman.uni-konstanz.de

6 participants
5250 discussions

xqdoc and basex system modules
by Andy Bunce 07 Apr '15

07 Apr '15

I am looking to generate documentation from the system module stubs in etc/modules.zip. I have two issues 1) There is an issue with inspect.xqm declare namespace xqdoc="http://www.xqdoc.org/1.0"; let $a:=inspect:xqdoc("inspect.xqm")//xqdoc:function[xqdoc:name='inspect:xqdoc']//xqdoc:description return parse-xml-fragment($a) Errors And xqdoc:description looks like <xqdoc:description xmlns:xqdoc="http://www.xqdoc.org/1.0"></code> and end with <code></code> , and tags start with <code>@</code> . xqDoc comments can be specified for main and library modules and variable and function declarations. <p>We have slightly extended the xqDoc conventions to do justice to the current status of XQuery (Schema: <a href=" http://files.basex.org/etc/xqdoc-1.1.30052013.xsd ">xqdoc-1.1.30052013.xsd</a>): </p> <ul> <li> an <code>&lt;xqdoc:annotations/&gt;</code> node is added to each variable or function that uses annotations. The xqdoc:annotation child nodes may have additional <code>xqdoc:literal</code> elements with <code>type</code> attributes (xs:string, xs:integer, xs:decimal, xs:double) and values. </li> <li> a single <code>&lt;xqdoc:namespaces/&gt;</code> node is added to the root element, which summarizes all prefixes and namespace URIs used or declared in the module. </li> <li> name and type elements are added to variables </li> </ul></xqdoc:description> A comment parsing/escaping issue? 2) inspect:xqdoc("array.xqm") and the same with map.xqm Function 'array:size' is in reserved namespace. I guess this is to be expected and maybe I should just ignore any modules in namespaces http://www.w3.org/... I guess it is a result of the code being compiled rather than just documented? /Andy

2 3

Path Summary order
by Cecil Tarasoff 02 Apr '15

02 Apr '15

Hi BaseX Gurus, Thanks for such a powerful reporting tool. I am still a newbie to XML and I would be lost without BaseX. Thank-you! I was wondering whether there is any way to have the path summary output it's result in alphabetical sequence of attributes and then elements? I export the path summary from different versions of the XML and use BeyondCompare to reveal the differences but the inconsistent ordering of the elements and attributes makes this a bit of a challenge. It would be really handy for me if there were an option available where I could elect to output the Path Summary with the attributes sorted alphabetically followed by the elements also in alphabetical order (within their parent). If you could consider this for a future enhancement I would be forever grateful. Of course, if there is an easier way for me to compare XML structure, I would love to know the tricks! Thanks again for your dedication to this os product. Cecil Tarasoff

2 1

require performance improvement suggestion for large size xml
by ankit kumar 31 Mar '15

31 Mar '15

Hi , I am using BaseX for my XML base application. In my application user provides various data filtration criteria. After passing the data through all the filters resultant data is processed for evaluation. For each filter we have created corresponding xquery and output of each filter is passed as input to the next filter for further filtration. What I notice is query execution time except for the first query is very high. From query plan I observed that for the first query indexing was being applied but not for the rest. This may be the case as the subsequent queries where getting fired on the data(sequence created by preceding query) passed on by the previous filter. This was not observed when the xml file was of smaller size , but as I increased the size of xml to 300 MB each query started taking times in minutes. I tried following thing I fired all the filter queries on the database to take advantage of indexing and after every filter execution , I took intersection of output sequence generated by each filter to keep track of final filtered data. The problem with above solution is , intersection query between larger sequence of data started tacking time large amount of time. Is there a way I can increase the performance. Thanks, Ankit

2 2

BaseX 8.1: The Spring Edition
by Christian Grün 28 Mar '15

28 Mar '15

Dear all, Our traditional spring edition has just been released! We are delighted to present BaseX 8.1, our XML database and XQuery 3.1 processor, which gives you the following new features: XQUERY - efficient Finger Tree algorithm for arrays - prof:variables() outputs list of currently bound variables - new higher-order functions: hof:scan-left, hof:take-while WEB APPLICATIONS - the Web Module offers convenience functions for building web apps - RESTXQ: %input annotation; input-specific content-type parameters OPTIONS - RESTPATH: specify server path for queries and command scripts - IGNORECERT: ignore untrusted certificates when connecting to servers COMMAND-LINE - Bind input strings to the query context with -I You find the new release at http://basex.org. We are looking forward to your feedback, Christian BaseX Team

1 0

Performance related query.
by ankit kumar 27 Mar '15

27 Mar '15

Hi, I am getting performance issue while using my own xquery library. I have written an xquery module which contains a single function which return all the categories belong to a set of products as given below. Also there is one constrain that i cannot pass all the category to the getCategory() function as argument. I have to take only products as input. I tried to take /products/p:category directly instead of $lib:category variable. but it is showing root not found. So i have to defined it as global variable. =====XQUERY MODULE====product_library.xq module namespace lib = "product_library"; declare namespace p="a:b:c"; declare variable $lib:category := /products/p:category; declare function lib:getCategory($products){ let $catRefs := distinct-values($products/@catid) return $lib:category[@id = $catRefs] }; Then i am including this library in another file, where i am invoking getCategory() function of the module. My code for that is given below. ======Client Code ====== product_client.xq import module 'product_library' at 'file:///C:/Users/ankumar/Desktop/product_library.xq'; declare namespace lib ="product_library"; declare namespace p="a:b:c"; let $products := /products/*[@catid] return count(lib:getCategory($products)) Executing the above code is taking too long. So, I stopped that and write the same logic of getCategory() function of module in the same Client file as given below. =====Changed Client Code========== import module 'product_library' at 'file:///C:/Users/ankumar/Desktop/product_library.xq'; declare namespace lib ="product_library"; declare namespace p="a:b:c"; let $products := /products/*[@catid] (: return count(lib:getCategory($products)) :) let $catRefs := distinct-values($products/@catid) return count(/products/p:category[@id = $catRefs]) It is executing very fast, and giving me the desired result with a second. I don't know why it is happening. My whole module is written in that way only. If you have any idea, why it is happening and how can i make this efficient then share with me. I am attaching my module file, client file,the xml instance file and the query info for with module and without module file with the mail. Is there anything to do with query optimization ?. large.xml <https://docs.google.com/file/d/0B_pB7l14skhVMkxzVjZpRWJ3blU/edit?usp=drive_…>

2 4

memory usage when writing files
by Lars Johnsen 27 Mar '15

27 Mar '15

Hi all Here is code that gradually eats up memory, whether run in GUI or as command. All it does is creating temporary collections out of folders, and writing them to file. Is there a simple way to avoid this code to eat up memory? It runs out of memory (set at 12GB for command, 18GB in GUI) after 300 folders or so, and it has to process 20 000 of them. Best Lars G Johnsen Norwegian National Library Here is the actual code (: process list of folders :) for $collections in file:list($digibooks) let $html := $htmlfiles || substring-before($collections, "_ocr") || ".html" return (: code is rerun so check if files exist :) if (not(file:exists($html))) then try { (: create a temporary collection of the files and write result to disk :) file:write( $html, db:digibok-to-html( collection($digibooks || $collections)) ) } catch * { $err:code } else ()

2 4

DITA for Small Teams Git-to-BaseX Hooks
by Eliot Kimber 26 Mar '15

26 Mar '15

I think I have my git hooks for loading XML into BaseX working well enough (there's more refinement to do but the minimum functionality is there). The code is available here: https://github.com/dita-for-small-teams/dfst-git-commit-hooks/tree/develop (That's on the develop branch) The functionality works as follows: - On checkout of a branch, see if a corresponding database is in BaseX, if not, create the database (named for the repo directory and branch), load all XML files (*.xml,*.dita,*.ditamap), and capture the git branch name and commit hash. If the database exists and the commit is not the current commit, reflect changes from the DB's commit and the branch's commit. - On commit or merge, update the database to reflect the changes between HEAD and HEAD^ I do not react (yet) to branch delete, but BaseX database maintenance is easy enough that it would be hard (e.g., just drop the database for that branch). I think this set hooks is sufficient to ensure that the BaseX databases will always be in sync with the git repo. I'm using the approach of one database per branch so that I can then implement branch-aware queries in the repo. This is essential for link management where you want to be able to view the documents through specific branch such that non-version-specific links will resolve to the version visible on that branch (this approach doesn't handle being able to resolve to older versions--that would require maintaining per-commit databases, although that might be necessary for certain use cases, hmmm). Thanks for all the help here, especially Charles Duffy's help with bash script coding. Cheers, E. ————— Eliot Kimber, Owner Contrext, LLC http://contrext.com

2 1

Failure on Second Update to Database
by Eliot Kimber 26 Mar '15

26 Mar '15

I'm getting a failure on the second commit of XML data via the basexclient command (from my git hook code). I've created a gist here: https://gist.github.com/dc9f6d55d891b06ecae9.git with relevant logs and the data/ directory. The exception reported is: ================== Contrext01:dfst-sample-project ekimber$ git add .;git commit -m "updated test file" Running: basexclient -c CHECK\ dfst_master\;\ OPEN\ dfst_master\;\ REPLACE\ docs/topic-01.dita\ /Users/ekimber/workspace-dfst/dfst-sample-project/docs/topic-01.dita -U admin -P admin -p 1984 -n localhost Running: basexclient -c CHECK\ dfst_master\;\ OPEN\ dfst_master\;\ REPLACE\ dfst/metadata.xml\ \<dfst_metadata\>\<gitstate\>\<branch\>master\</branch\>\<commit\>8d264e1e7 3d5596758a069a4355920c448a5b41d\</commit\>\</gitstate\>\</dfst_metadata\> -U admin -P admin -p 1984 -n localhost Improper use? Potential bug? Your feedback is welcome: Contact: basex-talk(a)mailman.uni-konstanz.de Version: BaseX 8.0.3 Java: Oracle Corporation, 1.7.0_65 OS: Mac OS X, x86_64 Stack Trace: java.lang.RuntimeException: Data Access out of bounds: - pre value: 21 - #used blocks: 1 - #total locks: 1 - access: 0 (1 > 0] at org.basex.util.Util.notExpected(Util.java:60) at org.basex.io.random.TableDiskAccess.cursor(TableDiskAccess.java:462) at org.basex.io.random.TableDiskAccess.read1(TableDiskAccess.java:148) at org.basex.data.Data.kind(Data.java:304) at org.basex.data.atomic.Replace.getInstance(Replace.java:42) at org.basex.data.atomic.AtomicUpdateCache.addReplace(AtomicUpdateCache.java:9 5) at org.basex.core.cmd.Replace.replace(Replace.java:95) at org.basex.core.cmd.Replace.run(Replace.java:57) at org.basex.core.Command.run(Command.java:379) at org.basex.core.Command.execute(Command.java:95) at org.basex.server.ClientListener.run(ClientListener.java:146) [master 8d264e1] updated test file 1 file changed, 1 insertion(+), 1 deletion(-) Contrext01:dfst-sample-project ekimber$ ================= The first update succeeds, the second one fails. The second one is committing literal XML on the command while the first is committing a file on the file system. This behavior is consistent: if I drop the database, restart the server, the repeat this test, I get the same failure. Is this my user error or a bug? Cheers, E. ————— Eliot Kimber, Owner Contrext, LLC http://contrext.com

2 10

Configuration File for Basexclient Connection Details?
by Eliot Kimber 25 Mar '15

25 Mar '15

Looking through the docs and trying to tests locally, it appears that the basexclient command does not use the USER or PASSWORD fields in .basex when run. That is, given this in my ~/.basex file: USER = admin PASSWORD = admin The basexclient command still prompts me for my credentials. Is this correct or am I missing a configuration option somewhere? What I'm looking for is the ability to call the basexclient command from git hooks without those scripts having to maintain their own configuration file for the connection details. Thanks, Eliot ————— Eliot Kimber, Owner Contrext, LLC http://contrext.com

2 4

Re: [basex-talk] Simple xQuery functions do not work as expected
by Goetz Heller 25 Mar '15

25 Mar '15

Hi Christian, the issue showed up again, this time outside of a function. The computer was only moderately charged, but the change of behavior occurred after I loaded a 25MB text file into Notepad++. Should I change something in the JVM settings? Kind regards, Goetz -----Ursprüngliche Nachricht----- Von: Christian Grün [mailto:christian.gruen@gmail.com] Gesendet: Mittwoch, 25. März 2015 11:27 An: Goetz Heller Betreff: Re: [basex-talk] Simple xQuery functions do not work as expected > P.S.: Please make sure that our conversation appears at BaseX-Talk since I'm receiving mails from other members of the community who do appear to not be informed about the current status. I continued to write directly to you, because your second e-mail was not directed to the list anymore. It makes perfectly sense to include the list as well. > > > -----Ursprüngliche Nachricht----- > Von: Christian Grün [mailto:christian.gruen@gmail.com] > Gesendet: Mittwoch, 25. März 2015 09:38 > An: Goetz Heller > Betreff: Re: [basex-talk] Simple xQuery functions do not work as > expected > >> However, I would not consider this issue as resolved. > > Obviously yes. > >> For me it is a hint that under heavy load (memory), such misbehavior might occur, and therefore some investigations make sense in my opinion. > > I was surprised to hear that a restart of your machine solved the problem, because the GUI is nothing else than a plain JVM instance with a low memory footprint (at least for queries like the one you sent to me). > > As I cannot remember to have encountered a similar problem, we rely on your feedback. Feel free to send us some more examples if the problem should decide to haunt you. > > Best, > Christian > > >> >> Kind regards, >> >> Goetz >> >> -----Ursprüngliche Nachricht----- >> Von: Christian Grün [mailto:christian.gruen@gmail.com] >> Gesendet: Dienstag, 24. März 2015 17:12 >> An: Goetz Heller >> Betreff: Re: [basex-talk] Simple xQuery functions do not work as >> expected >> >> I am using Windows as well. >> >> Does the problem also happen with the latest snapshot? Could you try the zip distribution? >> >> http://files.basex.org/releases/latest/ >> >> >> >> On Tue, Mar 24, 2015 at 5:07 PM, Goetz Heller <heller(a)hellerim.de> wrote: >>> Did you try it on the Windows platform already? Maybe there was a mistake when packaging. >>> >>> Kind regards, >>> >>> Goetz >>> >>> -----Ursprüngliche Nachricht----- >>> Von: Christian Grün [mailto:christian.gruen@gmail.com] >>> Gesendet: Dienstag, 24. März 2015 16:50 >>> An: Goetz Heller >>> Betreff: Re: [basex-talk] Simple xQuery functions do not work as >>> expected >>> >>> Strange. Do you possibly have an empty or out-dated database named "labels"? If yes, could you try to back up and delete it, and run the query again? >>> >>> >>> On Tue, Mar 24, 2015 at 4:45 PM, Goetz Heller <heller(a)hellerim.de> wrote: >>>> Hi Christian, >>>> >>>> here ist he complete output. Apparently, it's over-optimized: >>>> >>>> Result: >>>> >>>> <r>et | </r> >>>> >>>> >>>> Compiling: >>>> - pre-evaluating doc("C:\test\labels.xml") >>>> - applying attribute index for $key_0 >>>> - applying attribute index for "_and" >>>> - inlining local:getLabel#2 >>>> - removing redundant $key_4 as xs:string* cast. >>>> - removing redundant $lang_5 as xs:string* cast. >>>> - inlining $key_4 >>>> - inlining $lang_5 >>>> - removing unknown element/attribute TERM >>>> - pre-evaluating db:attribute("Labels", >>>> "_and")/self::*:label/parent::*:LABEL/()/text() >>>> - pre-evaluating ((: xs:string*, true :) ()) >>>> - simplifying flwor expression >>>> - inlining $retVal1_2 >>>> - inlining $retVal2_3 >>>> - simplifying flwor expression >>>> >>>> >>>> >>>> Query: >>>> declare variable $labels := doc('C:\test\labels.xml'); declare >>>> function local:getLabel( $key as xs:string*, $lang as xs:string* ) >>>> as >>>> xs:string* { $labels/LABELS/LABEL[@label = $key]/TERM[@LG = >>>> $lang]/text() }; let $retVal1 := $labels/LABELS/LABEL[@label = >>>> '_and']/TERM[@LG = 'FR']/text() let $retVal2 := >>>> local:getLabel('_and', >>>> 'FR') return <r>{$retVal1} | {$retVal2}</r> >>>> >>>> >>>> >>>> Optimized Query: >>>> element r { (db:attribute("Labels", >>>> "_and")/self::*:label/parent::*:LABEL/*:TERM[(@*:LG = "FR")]/text(), " >>>> | ", ()) } >>>> >>>> >>>> Result: >>>> - Hit(s): 1 Item >>>> - Updated: 0 Items >>>> - Printed: 12 Bytes >>>> - Read Locking: global >>>> - Write Locking: none >>>> >>>> >>>> Timing: >>>> - Parsing: 1.32 ms >>>> - Compiling: 154.5 ms >>>> - Evaluating: 0.43 ms >>>> - Printing: 0.01 ms >>>> - Total Time: 156.26 ms >>>> >>>> >>>> Query plan: >>>> <QueryPlan compiled="true"> >>>> <CElem> >>>> <QNm value="r" type="xs:QName"/> >>>> <IterPath> >>>> <ValueAccess data="Labels" type="ATTRIBUTE"> >>>> <Str value="_and" type="xs:string"/> >>>> </ValueAccess> >>>> <IterStep axis="self" test="*:label"/> >>>> <IterStep axis="parent" test="*:LABEL"/> >>>> <IterStep axis="child" test="*:TERM"> >>>> <CmpG op="="> >>>> <CachedPath> >>>> <IterStep axis="attribute" test="*:LG"/> >>>> </CachedPath> >>>> <Str value="FR" type="xs:string"/> >>>> </CmpG> >>>> </IterStep> >>>> <IterStep axis="child" test="text()"/> >>>> </IterPath> >>>> <Str value=" | " type="xs:string"/> >>>> <Empty size="0"/> >>>> </CElem> >>>> </QueryPlan> >>>> >>>> Kind regards, >>>> >>>> Goetz >>>> >>> >> >

3 6

Jump to page:

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

BaseX-Talk