- BaseX-Talk - mailman.uni-konstanz.de

Performance related query.
by ankit kumar 27 Mar '15

27 Mar '15

Hi, I am getting performance issue while using my own xquery library. I have written an xquery module which contains a single function which return all the categories belong to a set of products as given below. Also there is one constrain that i cannot pass all the category to the getCategory() function as argument. I have to take only products as input. I tried to take /products/p:category directly instead of $lib:category variable. but it is showing root not found. So i have to defined it as global variable. =====XQUERY MODULE====product_library.xq module namespace lib = "product_library"; declare namespace p="a:b:c"; declare variable $lib:category := /products/p:category; declare function lib:getCategory($products){ let $catRefs := distinct-values($products/@catid) return $lib:category[@id = $catRefs] }; Then i am including this library in another file, where i am invoking getCategory() function of the module. My code for that is given below. ======Client Code ====== product_client.xq import module 'product_library' at 'file:///C:/Users/ankumar/Desktop/product_library.xq'; declare namespace lib ="product_library"; declare namespace p="a:b:c"; let $products := /products/*[@catid] return count(lib:getCategory($products)) Executing the above code is taking too long. So, I stopped that and write the same logic of getCategory() function of module in the same Client file as given below. =====Changed Client Code========== import module 'product_library' at 'file:///C:/Users/ankumar/Desktop/product_library.xq'; declare namespace lib ="product_library"; declare namespace p="a:b:c"; let $products := /products/*[@catid] (: return count(lib:getCategory($products)) :) let $catRefs := distinct-values($products/@catid) return count(/products/p:category[@id = $catRefs]) It is executing very fast, and giving me the desired result with a second. I don't know why it is happening. My whole module is written in that way only. If you have any idea, why it is happening and how can i make this efficient then share with me. I am attaching my module file, client file,the xml instance file and the query info for with module and without module file with the mail. Is there anything to do with query optimization ?. large.xml <https://docs.google.com/file/d/0B_pB7l14skhVMkxzVjZpRWJ3blU/edit?usp=drive_…>

2 4

memory usage when writing files
by Lars Johnsen 27 Mar '15

27 Mar '15

Hi all Here is code that gradually eats up memory, whether run in GUI or as command. All it does is creating temporary collections out of folders, and writing them to file. Is there a simple way to avoid this code to eat up memory? It runs out of memory (set at 12GB for command, 18GB in GUI) after 300 folders or so, and it has to process 20 000 of them. Best Lars G Johnsen Norwegian National Library Here is the actual code (: process list of folders :) for $collections in file:list($digibooks) let $html := $htmlfiles || substring-before($collections, "_ocr") || ".html" return (: code is rerun so check if files exist :) if (not(file:exists($html))) then try { (: create a temporary collection of the files and write result to disk :) file:write( $html, db:digibok-to-html( collection($digibooks || $collections)) ) } catch * { $err:code } else ()

2 4

DITA for Small Teams Git-to-BaseX Hooks
by Eliot Kimber 26 Mar '15

26 Mar '15

I think I have my git hooks for loading XML into BaseX working well enough (there's more refinement to do but the minimum functionality is there). The code is available here: https://github.com/dita-for-small-teams/dfst-git-commit-hooks/tree/develop (That's on the develop branch) The functionality works as follows: - On checkout of a branch, see if a corresponding database is in BaseX, if not, create the database (named for the repo directory and branch), load all XML files (*.xml,*.dita,*.ditamap), and capture the git branch name and commit hash. If the database exists and the commit is not the current commit, reflect changes from the DB's commit and the branch's commit. - On commit or merge, update the database to reflect the changes between HEAD and HEAD^ I do not react (yet) to branch delete, but BaseX database maintenance is easy enough that it would be hard (e.g., just drop the database for that branch). I think this set hooks is sufficient to ensure that the BaseX databases will always be in sync with the git repo. I'm using the approach of one database per branch so that I can then implement branch-aware queries in the repo. This is essential for link management where you want to be able to view the documents through specific branch such that non-version-specific links will resolve to the version visible on that branch (this approach doesn't handle being able to resolve to older versions--that would require maintaining per-commit databases, although that might be necessary for certain use cases, hmmm). Thanks for all the help here, especially Charles Duffy's help with bash script coding. Cheers, E. ————— Eliot Kimber, Owner Contrext, LLC http://contrext.com

2 1

Failure on Second Update to Database
by Eliot Kimber 26 Mar '15

26 Mar '15

I'm getting a failure on the second commit of XML data via the basexclient command (from my git hook code). I've created a gist here: https://gist.github.com/dc9f6d55d891b06ecae9.git with relevant logs and the data/ directory. The exception reported is: ================== Contrext01:dfst-sample-project ekimber$ git add .;git commit -m "updated test file" Running: basexclient -c CHECK\ dfst_master\;\ OPEN\ dfst_master\;\ REPLACE\ docs/topic-01.dita\ /Users/ekimber/workspace-dfst/dfst-sample-project/docs/topic-01.dita -U admin -P admin -p 1984 -n localhost Running: basexclient -c CHECK\ dfst_master\;\ OPEN\ dfst_master\;\ REPLACE\ dfst/metadata.xml\ \<dfst_metadata\>\<gitstate\>\<branch\>master\</branch\>\<commit\>8d264e1e7 3d5596758a069a4355920c448a5b41d\</commit\>\</gitstate\>\</dfst_metadata\> -U admin -P admin -p 1984 -n localhost Improper use? Potential bug? Your feedback is welcome: Contact: basex-talk(a)mailman.uni-konstanz.de Version: BaseX 8.0.3 Java: Oracle Corporation, 1.7.0_65 OS: Mac OS X, x86_64 Stack Trace: java.lang.RuntimeException: Data Access out of bounds: - pre value: 21 - #used blocks: 1 - #total locks: 1 - access: 0 (1 > 0] at org.basex.util.Util.notExpected(Util.java:60) at org.basex.io.random.TableDiskAccess.cursor(TableDiskAccess.java:462) at org.basex.io.random.TableDiskAccess.read1(TableDiskAccess.java:148) at org.basex.data.Data.kind(Data.java:304) at org.basex.data.atomic.Replace.getInstance(Replace.java:42) at org.basex.data.atomic.AtomicUpdateCache.addReplace(AtomicUpdateCache.java:9 5) at org.basex.core.cmd.Replace.replace(Replace.java:95) at org.basex.core.cmd.Replace.run(Replace.java:57) at org.basex.core.Command.run(Command.java:379) at org.basex.core.Command.execute(Command.java:95) at org.basex.server.ClientListener.run(ClientListener.java:146) [master 8d264e1] updated test file 1 file changed, 1 insertion(+), 1 deletion(-) Contrext01:dfst-sample-project ekimber$ ================= The first update succeeds, the second one fails. The second one is committing literal XML on the command while the first is committing a file on the file system. This behavior is consistent: if I drop the database, restart the server, the repeat this test, I get the same failure. Is this my user error or a bug? Cheers, E. ————— Eliot Kimber, Owner Contrext, LLC http://contrext.com

2 10

Configuration File for Basexclient Connection Details?
by Eliot Kimber 25 Mar '15

25 Mar '15

Looking through the docs and trying to tests locally, it appears that the basexclient command does not use the USER or PASSWORD fields in .basex when run. That is, given this in my ~/.basex file: USER = admin PASSWORD = admin The basexclient command still prompts me for my credentials. Is this correct or am I missing a configuration option somewhere? What I'm looking for is the ability to call the basexclient command from git hooks without those scripts having to maintain their own configuration file for the connection details. Thanks, Eliot ————— Eliot Kimber, Owner Contrext, LLC http://contrext.com

2 4

Re: [basex-talk] Simple xQuery functions do not work as expected
by Goetz Heller 25 Mar '15

25 Mar '15

Hi Christian, the issue showed up again, this time outside of a function. The computer was only moderately charged, but the change of behavior occurred after I loaded a 25MB text file into Notepad++. Should I change something in the JVM settings? Kind regards, Goetz -----Ursprüngliche Nachricht----- Von: Christian Grün [mailto:christian.gruen@gmail.com] Gesendet: Mittwoch, 25. März 2015 11:27 An: Goetz Heller Betreff: Re: [basex-talk] Simple xQuery functions do not work as expected > P.S.: Please make sure that our conversation appears at BaseX-Talk since I'm receiving mails from other members of the community who do appear to not be informed about the current status. I continued to write directly to you, because your second e-mail was not directed to the list anymore. It makes perfectly sense to include the list as well. > > > -----Ursprüngliche Nachricht----- > Von: Christian Grün [mailto:christian.gruen@gmail.com] > Gesendet: Mittwoch, 25. März 2015 09:38 > An: Goetz Heller > Betreff: Re: [basex-talk] Simple xQuery functions do not work as > expected > >> However, I would not consider this issue as resolved. > > Obviously yes. > >> For me it is a hint that under heavy load (memory), such misbehavior might occur, and therefore some investigations make sense in my opinion. > > I was surprised to hear that a restart of your machine solved the problem, because the GUI is nothing else than a plain JVM instance with a low memory footprint (at least for queries like the one you sent to me). > > As I cannot remember to have encountered a similar problem, we rely on your feedback. Feel free to send us some more examples if the problem should decide to haunt you. > > Best, > Christian > > >> >> Kind regards, >> >> Goetz >> >> -----Ursprüngliche Nachricht----- >> Von: Christian Grün [mailto:christian.gruen@gmail.com] >> Gesendet: Dienstag, 24. März 2015 17:12 >> An: Goetz Heller >> Betreff: Re: [basex-talk] Simple xQuery functions do not work as >> expected >> >> I am using Windows as well. >> >> Does the problem also happen with the latest snapshot? Could you try the zip distribution? >> >> http://files.basex.org/releases/latest/ >> >> >> >> On Tue, Mar 24, 2015 at 5:07 PM, Goetz Heller <heller(a)hellerim.de> wrote: >>> Did you try it on the Windows platform already? Maybe there was a mistake when packaging. >>> >>> Kind regards, >>> >>> Goetz >>> >>> -----Ursprüngliche Nachricht----- >>> Von: Christian Grün [mailto:christian.gruen@gmail.com] >>> Gesendet: Dienstag, 24. März 2015 16:50 >>> An: Goetz Heller >>> Betreff: Re: [basex-talk] Simple xQuery functions do not work as >>> expected >>> >>> Strange. Do you possibly have an empty or out-dated database named "labels"? If yes, could you try to back up and delete it, and run the query again? >>> >>> >>> On Tue, Mar 24, 2015 at 4:45 PM, Goetz Heller <heller(a)hellerim.de> wrote: >>>> Hi Christian, >>>> >>>> here ist he complete output. Apparently, it's over-optimized: >>>> >>>> Result: >>>> >>>> <r>et | </r> >>>> >>>> >>>> Compiling: >>>> - pre-evaluating doc("C:\test\labels.xml") >>>> - applying attribute index for $key_0 >>>> - applying attribute index for "_and" >>>> - inlining local:getLabel#2 >>>> - removing redundant $key_4 as xs:string* cast. >>>> - removing redundant $lang_5 as xs:string* cast. >>>> - inlining $key_4 >>>> - inlining $lang_5 >>>> - removing unknown element/attribute TERM >>>> - pre-evaluating db:attribute("Labels", >>>> "_and")/self::*:label/parent::*:LABEL/()/text() >>>> - pre-evaluating ((: xs:string*, true :) ()) >>>> - simplifying flwor expression >>>> - inlining $retVal1_2 >>>> - inlining $retVal2_3 >>>> - simplifying flwor expression >>>> >>>> >>>> >>>> Query: >>>> declare variable $labels := doc('C:\test\labels.xml'); declare >>>> function local:getLabel( $key as xs:string*, $lang as xs:string* ) >>>> as >>>> xs:string* { $labels/LABELS/LABEL[@label = $key]/TERM[@LG = >>>> $lang]/text() }; let $retVal1 := $labels/LABELS/LABEL[@label = >>>> '_and']/TERM[@LG = 'FR']/text() let $retVal2 := >>>> local:getLabel('_and', >>>> 'FR') return <r>{$retVal1} | {$retVal2}</r> >>>> >>>> >>>> >>>> Optimized Query: >>>> element r { (db:attribute("Labels", >>>> "_and")/self::*:label/parent::*:LABEL/*:TERM[(@*:LG = "FR")]/text(), " >>>> | ", ()) } >>>> >>>> >>>> Result: >>>> - Hit(s): 1 Item >>>> - Updated: 0 Items >>>> - Printed: 12 Bytes >>>> - Read Locking: global >>>> - Write Locking: none >>>> >>>> >>>> Timing: >>>> - Parsing: 1.32 ms >>>> - Compiling: 154.5 ms >>>> - Evaluating: 0.43 ms >>>> - Printing: 0.01 ms >>>> - Total Time: 156.26 ms >>>> >>>> >>>> Query plan: >>>> <QueryPlan compiled="true"> >>>> <CElem> >>>> <QNm value="r" type="xs:QName"/> >>>> <IterPath> >>>> <ValueAccess data="Labels" type="ATTRIBUTE"> >>>> <Str value="_and" type="xs:string"/> >>>> </ValueAccess> >>>> <IterStep axis="self" test="*:label"/> >>>> <IterStep axis="parent" test="*:LABEL"/> >>>> <IterStep axis="child" test="*:TERM"> >>>> <CmpG op="="> >>>> <CachedPath> >>>> <IterStep axis="attribute" test="*:LG"/> >>>> </CachedPath> >>>> <Str value="FR" type="xs:string"/> >>>> </CmpG> >>>> </IterStep> >>>> <IterStep axis="child" test="text()"/> >>>> </IterPath> >>>> <Str value=" | " type="xs:string"/> >>>> <Empty size="0"/> >>>> </CElem> >>>> </QueryPlan> >>>> >>>> Kind regards, >>>> >>>> Goetz >>>> >>> >> >

3 6

Facing Problem With Max()
by ankit kumar 25 Mar '15

25 Mar '15

Hi, I am trying to get max value for a sequence of nodes using max(). I have a query which gives me a set of element and after that i get its max value. let $elements := (<date>10-02-2012</date>,<date>12-02-2012<date>); return max($element) I am expecting that it gives me 10-02-2012, while it is giving error "cannot cast to double '10-02-2012'. Thanks Ankit

2 1

Simple xQuery functions do not work as expected
by Goetz Heller 25 Mar '15

25 Mar '15

I tried to implement a simple Xml-based dictionary. Here is a fragment of the Xml data I’m using: <?xml version="1.0" encoding="utf-8"?> <LABELS> <LABEL label="_and"> <TERM LG="EN">and</TERM> <TERM LG="FR">et</TERM> <TERM LG="DE">und</TERM> <TERM LG="BG">и</TERM> <TERM LG="CS">a</TERM> <TERM LG="DA">og</TERM> <TERM LG="EL">και</TERM> <TERM LG="ES">y</TERM> <TERM LG="ET">ja</TERM> <TERM LG="FI">ja</TERM> <TERM LG="HU">és</TERM> <TERM LG="IT">e</TERM> <TERM LG="LT">ir</TERM> <TERM LG="LV">līdz</TERM> <TERM LG="MT">u</TERM> <TERM LG="NL">en</TERM> <TERM LG="PL">i</TERM> <TERM LG="PT">e</TERM> <TERM LG="RO">şi</TERM> <TERM LG="SK">a</TERM> <TERM LG="SL">in</TERM> <TERM LG="SV">och</TERM> </LABEL> </LABELS> Say it’s saved on C:\test\labels.xml. Now, I defined the following module: BEGIN_CODE>>> (:: default language:) declare variable $lang as xs:string := 'DE'; (:: dictionary :) declare variable $labels := doc('C:\test\Labels.xml'); (:: get localized label (default language) :) declare function local:getLabel( $key as xs:string* ) as xs:string* { $labels/LABELS/LABEL[compare(@label, $key) = 0]/TERM[compare(@LG, $lang) = 0]/text() }; (:: get localized label (specific language) :) declare function local:getLabel( $key as xs:string*, $lang as xs:string* ) as xs:string* { $labels/LABELS/LABEL[compare(@label, $key) = 0]/TERM[compare(@LG, $lang) = 0]/text() }; (:: get localized label (specific language) - debug version :) declare function local:getLabelDebug( $key as xs:string*, $lang as xs:string* ) { let $terms := $labels/LABELS/LABEL[compare(@label, $key) = 0]/TERM for $t in ($terms) return <t>{xs:string($t/@LG)}: {$t/text()} (comparison: {$t/@LG = $lang})</t> }; (: does not work as I expect: let $retVal := local:getLabel('_and', 'FR') because it returns nothing :) (: using "for $t in ($terms[compare(@LG, $lang) = 0])" yields empty result :) let $retVal := local:getLabelDebug('_and', 'FR') return <r>{$retVal}</r> <<<END_CODE The first function appears to work as expected. However, instead of “[@LG = $lang]” I had to use [compare(@LG, $lang) = 0]” – I don’t know why, but otherwise, it does not return any results. On the other hand, using “[@label = $key]” on the same line worked. The second function does not return anything at all. The debug version of the latter shows that all the needed information is in place. Obviously, usage of the same variable name globally and locally is not an issue (I’ve tried it out). What’s wrong with the code? Is there an issue with the current version (8.0.3)? My installation runs on a Windows 8.1 machine. Some help is appreciated. Kind regards, Goetz

3 2

GUI suggestion: no trace output when running tests
by Marc van Grootel 24 Mar '15

24 Mar '15

Hi, When running tests from the GUI I sometimes insert debugging code to figure out which tests fails or why. The current behavior of the GUI makes this impossible though because when tests are run this info is printed to the Query Info window but immediately after tests are run it prints the test summary overwriting the debugging info (such as trace output). Currently I then go to the command-line but it might be an idea to somehow keep the output generated during test run visible. Cheers, --Marc

2 1

Running basexclient Command From a Bash Script
by Eliot Kimber 24 Mar '15

24 Mar '15

I'm trying to implement agit commit hook that updates a BaseX database with updates committed to the git repo (the BaseX database is being used for search and link management). My script is here: https://github.com/dita-for-small-teams/dfst-git-commit-hooks/blob/develop/ post-commit The important code is: ---------- addOrUpdate="git diff HEAD^ HEAD --name-only --diff-filter=AM" # echo "Adding/Updating files:" for line in `${addOrUpdate}`; do cmd="basexclient -c \"OPEN ${basexDatabase};REPLACE ${line} ${topDir}/${line};EXIT\" ${basexOptions}" echo "Running cmd: ${cmd}" $($cmd) done --------- When I run the script I'm getting this response: Contrext01:docs ekimber$ ../../dita-for-small-teams/commit-hooks/git/post-commit Running cmd: basexclient -c "OPEN sample-project;REPLACE docs/topic-01.dita /Users/ekimber/workspace-dfst/dfst-sample-project/docs/docs/topic-01.dita;E XIT" -U admin -P admin -p 1984 -n localhost Stopped at , 1/6: Unknown command: "OPEN. Did you mean 'OPEN'? Contrext01:docs ekimber$ If I copy the command from the message and run it directly from the command line it works as expected. There must be some subtlety of using base in this context but my bash fu is weak and I have no idea what I'm doing wrong--I can't see any obvious user error but there must be one. Can anyone tell me what my bash scripting mistake is? Is there a better way to do this sort of scripted interaction with basex? It didn't appear that BaseX scripts provided a way to take parameters--if they could then I'd just call a .bxs script with the relevant parameters. Short of generating the script and then running it, I couldn't think of a simpler way to do what I want that just using the -c option on the basexclient command. Thanks, Eliot ————— Eliot Kimber, Owner Contrext, LLC http://contrext.com

3 4

get-current-user()
by Marc van Grootel 24 Mar '15

24 Mar '15

Hi, Maybe I'm missing something but I cannot find the XQuery way to get access to the logged in user from a web application using RESTXQ. The servlet request object has getRemoteUser and Exist, for example, has xmldb:get-current-user and Marklogic has xdmp:get-current-user. I don't need to know intricate user details just need the handle on the logged in / authenticated user (possibly from the servlet request object?) --Marc

2 3

BaseX GUI minor bug with Preferences dialog in v8.0.3
by Amanda Galtman 24 Mar '15

24 Mar '15

Hi, Here are reproduction steps for a bug I'm seeing in v8.0.3 on win64. The issue was present in v7.9, but not in 7.7.2. Those are the versions I happen to have installed; I don't know the exact version that introduced the issue. 1. Select Database > Open & Manage, select a database, and open it. 2. Select Options > Preferences. 3. Put the cursor in the Database Path edit field and type something. Your *first* keystroke is accepted, and then the Preferences dialog closes. The database that you opened in step 1 also seems to get closed. If you close the database before doing step 2, the issue does not occur. Amanda

2 2

Re: [basex-talk] Query Execution Taking Too Long...
by Leonard Wörteler 24 Mar '15

24 Mar '15

Hi Ankit (CC mailing list), Am 24.03.2015 um 06:57 schrieb ankit kumar: > We could have used distinct-values() But, in our case we would like to > distinguish on the basis of attributes values. also distinct-values() > will be treating values of <a>1</a> and <a>01</a> as different. . > For the later case will be using custom node-correspond function instead > of deep-equal which will be doing comparison between two nodes. the general problem is that grouping a set of elements where all you have is an equivalence predicate is always expensive to compute (i.e. quadratic in the size of the set). In order to speed up your use case you should therefore try to find some way to expose more information about the nodes to compare. If it is possible to generate an identifier for each node that can be used for comparisons, then efficient hash-based algorithms can be used (e.g. those in `group by` or `distinct-values(...)`). You could for example have a function `my:id($node)` so that `my:id($n1) eq my:id($n2)` holds iff `node-correspond($n1, $n2)` does. Then you can rewrite your query to run in linear time: declare namespace p="a:b:c"; declare namespace my="my://namespace"; declare function my:id($n) { (: your code here :) $n/string() }; for $start in /products/p:category/start group by $id := my:id($start) return $start[1] If you cannot generate such an ID, you can also use sorting to reduce running time. If you can define a comparison function `my:less-than($n1, $n2)` that sorts the nodes into an order where equivalent items are next to each other, you can first sort the sequence in O(n log n) time (e.g. using `hof:sort-with($seq, my:less-than#2)` in BaseX) and then merge the runs of equivalent nodes in linear time. Hope that helps, Leo

1 0

gui folder view delete
by Andy Bunce 24 Mar '15

24 Mar '15

Hi, In the GUI folder view I can delete nodes by selecting and using right click. But if I select a top level document name entry the UI prompts for delete confirmation but no delete occurs. /Andy

2 3

Query Execution Taking Too Long...
by ankit kumar 23 Mar '15

23 Mar '15

Hi, I am trying to get all distinct start element '/products/p:category/start' nodes of a big file. I have written a query which is given below. It is taking to long to get the result. I am attaching the query info and the xml file. After Running couple of minutes, I stopped the execution. Query is trying to get all the distinct start elements There are 3 lac category elements. declare namespace functx = "http://www.functx.com"; declare namespace p="a:b:c"; declare function functx:is-node-in-sequence( $node as node()? , $seq as node()* ) as xs:boolean {some $nodeInSeq in $seq satisfies deep-equal($nodeInSeq,$node)}; declare function functx:distinct-nodes( $nodes as node()* large.xml <https://docs.google.com/file/d/0B_pB7l14skhVLWxvZGNyMnFwc0k/edit?usp=drive_…> ) as node()* {for $seq in (1 to count($nodes))return $nodes[$seq][not(functx:is-node-in-sequence(.,$nodes[position() < $seq]))]} ; let $diff_starts := functx:distinct-nodes(/products/p:category/start) return $diff_starts Please let me know if you require further details.

2 1

Issues posting/putting JSON content with markup in it via http module.
by James Wright 23 Mar '15

23 Mar '15

Hello everyone, First, I want to say that I really like the JSON improvements in BaseX 8.0. Great job. I am however having an issue when posting JSON with markup in it. For example: let $out := map { name: 'myView', markup: "<div name='value'>Some other stuff</div>" } http:send-request(<http:request method="POST" />, uri, $out) The issue is that the resulting JSON body being posted has the 'markup' property xml escaped. IE: { "name": "myView", "markup": "<div name='value'>Some other ..... " } I had a similar issue with file:write but resolved that by using file:write-string. I have tried calling json:serialize on $out manually as well as declaring the option output:method to 'json'. I even tried changing the request object by inserting a http:body. IE: <http:request method="GET"> <http:body content-type="application/json"> {json:serialize($out)} </http> </http> Lastly, I have tried declaring the cdata-section option: declare option output:cdata-section-elements 'http:body'; Still no go... Is this not possible? One option would be to serialize the content using file:write and then call curl. This is a workaround I have used in the past however I was hoping I was behind that now with the latest improvements to XQuery 3.1. Maybe I need to send the http:request object as a map instead of an xml node? Thanks again for your time on all of this. - James

2 2

Bug in evaluating conditionals? Not short-circuiting?
by Marc van Grootel 23 Mar '15

23 Mar '15

Hi, I think I found a bug in 8.0.3 (and earlier) declare function local:shortcircuit($a) { if ($a castable as xs:double and xs:double($a) gt 0) then $a else () }; local:shortcircuit('foo') => [FORG0001] Cannot cast to xs:double: "foo". In http://www.w3.org/TR/xpath/#booleans "An and expression is evaluated by evaluating each operand and converting its value to a boolean as if by a call to the boolean function. The result is true if both values are true and false otherwise. The right operand is not evaluated if the left operand evaluates to false." Haven't tested it for other conditionals. --Marc

2 1

My BaseX GUI shows huge font and icons after upgrade
by Kristian Kankainen 21 Mar '15

21 Mar '15

Hello! After updating BaseX I got a strange looking GUI. Most fonts and icons are huge as can be seen from the screen shot. I use Fedora 21 and the package from your OpenSUSE repo. Could it be some missing fonts or other depending packages? My package information: Installed Packages Name : basex Arch : noarch Version : 8.0.2 Release : 1.1 Size : 2.8 M Repo : installed From repo : home_basex_opensuse And the repo file is the following (as you provide) [home_basex_opensuse] name=BaseX openSUSE Build (Fedora_20) type=rpm-md baseurl=http://download.opensuse.org/repositories/home:/basex:/opensuse/Fed… gpgcheck=1 gpgkey=http://download.opensuse.org/repositories/home:/basex:/opensuse/Fedo… enabled=1 /etc/yum.repos.d/home_basex_opensuse.repo (END) Best regards Kristian K

1 0

Reading standard input
by Hans-Juergen Rennau 19 Mar '15

19 Mar '15

Dear BaseX team, if I understand the documentation correctly, the processing of standard input is currently restricted to "-i-", that is, reading standard input as an XML document providing the context item, e.g. so: myCmd | basex -i- -q //foo Nice, but how about standard input which corresponds, for example, to a JSON document or a CSV document? Wouldn't it be useful to be able to apply functions to standard input,. e.g. json:parse, csv:parse or simple fn:unparsed-text? Example use case: NAGIOS commands piping the output of a command into BaseX. Whereas the context item can have any item type, the command line option -i accepts only documents, and that is the problem. I experimented with the "declare context item" declaration, but it did not work. And, at any rate, I would appreciate the conciseness of using an atomic context item without having to add the "declare context item" declaration. One solution were a new command line option (e.g. -j) which represents the context item as xs:untypedAtomic, and which also understands "-" to denote standard input. Then one could do this: myCmd | basex -j- json:parse(.)//foo That would be very nice and broaden the scope of BaseX again by a meter or two. Cheers,Hans-Jürgen

2 2

Fwd: Re: How to process very long node sequences
by Leonard Wörteler 19 Mar '15

19 Mar '15

Wrong mail account... -------- Weitergeleitete Nachricht -------- Betreff: Re: [basex-talk] How to process very long node sequences Datum: Wed, 18 Mar 2015 19:26:00 +0100 Von: Leo Wörteler <lw(a)basex.org> An: Hans-Juergen Rennau <hrennau(a)yahoo.de> Kopie (CC): basex-talk(a)mailman.uni-konstanz.de <basex-talk(a)mailman.uni-konstanz.de> Dear Hans-Jürgen, Am 18.03.2015 um 15:23 schrieb Hans-Juergen Rennau: > I want to persuade the processor to visit, process and forget the nodes > one after the other, rather than to attempt loading all nodes into > memory before proceeding to process them. BaseX already uses iterative processing as default mode. It only falls back to caching if it has to, e.g. for sorting, reversing or duplicate elimination. Calls to user-defined functions are currently also blocking, but we are currently investigating if that can be changed. > Schematically: > > declare function f:processNode($node as node()) as empty-sequence() {...}; > > for $node in doc('huge-doc')/a/b/c > return f:processNode($node) The example you gave will always be evaluated without caching the node sequence. The `for` clause requests the result of its argument iteratively, and the XPath expression you used only contains child steps, which can be evaluated in document order without duplicates. > Well, in some cases it works, in others in doesn't. Is there any safe > way how to enforce sequential visit-process-forget processing? When the XPath expression becomes more complex, it is not as easy to predict if it uses caching internally. BaseX tries quite hard to detect paths that do not need it, the algorithm can be seen in [1]. If you see a `CachedPath` in the Info View of the GUI, you can try to reformulate the query. > I call this problem general because it is a critical aspect of dealing > with huge documents. XQuery does not have a dedicated *streaming mode* like that of XSLT 3.0 [2] (yet), but it would definitely be possible to check if some part of a query (e.g. marked by a pragma) is evaluated without caching. It would however be quite some work. Hope that helps, Leo [1] https://github.com/BaseXdb/basex/blob/0b828a8/basex-core/src/main/java/org/… [2] http://www.w3.org/TR/xslt-30/#dt-guaranteed-streamable

4 4

How to process very long node sequences
by Hans-Juergen Rennau 18 Mar '15

18 Mar '15

Dear BaseX team, can you help me with the following, very general problem? I have a huge document in a database and want to iterate over many nodes which it contains, performing for each one an action which does not produce a value - for example, storing it in a SQL database. I want to persuade the processor to visit, process and forget the nodes one after the other, rather than to attempt loading all nodes into memory before proceeding to process them. Schematically: declare function f:processNode($node as node()) as empty-sequence() {...}; for $node in doc('huge-doc')/a/b/creturn f:processNode($node) ~ ~ ~ Well, in some cases it works, in others in doesn't. Is there any safe way how to enforce sequential visit-process-forget processing? I call this problem general because it is a critical aspect of dealing with huge documents. Cheers,Hans-Jürgen

1 0

Unexpected format-integer result
by Neven Jovanović 16 Mar '15

16 Mar '15

Hello, when BaseX 8.0 performs the following XQuery expression: format-integer(556, '#,##1') I expect to see just: 556 -- but I get the following: ,556 Am I misreading something in the format-integer() chapter of XPath and XQuery Functions and Operators 3.0[1], or is this a bug? Best, Neven [1] <http://www.w3.org/TR/xpath-functions-30/#func-format-integer>

2 2

Large Document Upload Performance
by Jonathan Clarke 16 Mar '15

16 Mar '15

Hi, I hope you can help me. I am using BaseX to underpin a complex distributed system, which also requires storage of xml document in soft real-time. At the moment, I'm getting storage times for a 4Mb XML file of about 500ms. Can you advise how I might be able to bring that down, please, by at least 75%? I have tried tweaking the systems for BaseX 8, including turning off autoflush, and each setting that I try actually seems to increase the processing time. I also tried to use AddCache, and that just crashed the latest production release of the server. Many thanks for your help in advance, Dr Jonathan Clarke.

3 9

DITA project
by France Baril 14 Mar '15

14 Mar '15

Hi, I'm working on a new DITA project. The DITA standard is used for technical documentation. It creates XML models with inheritance by using @class="ancestor parent child". If the child should behave like it's ancestor the XSL would say: <template match="contains(@class, 'ancestor')"> ... </template> The advantage is that the model can evolve and when new elements are added, you only need to code transformations for the differences. I am trying to figure out if I can use type switching with contains in attribute. Search gets me no syntax for something like this: typeswitch ($node) case attribute(contains(class, 'ancestor')) Maybe I should register to the xquery group to get an answer, but since I'm already here, I though I should ask, and maybe there is a BaseX specific option. I mean other that an unmanageable number of 'if then else' statements. Thanks! -- France Baril Architecte documentaire / Documentation architect france.baril(a)architextus.com

3 5

Add document performances with UPDINDEX = true
by Simon Chatelain 13 Mar '15

13 Mar '15

Hello, First let me give you the context: I have a never ending stream of XML element coming in that I want to store and then make available through a REST interface. Thus BaseX seems to be a well suited candidate. To be on the safe side I must be able to sustain an insertion rate of about 200 elements per second. The XML elements I have to store are of the type: <notification ts=”2015-03-13T10.44.25.123” nid=”type-of-data”> <name-1>value1</ name-1> <name-2>value2</ name-2> <name-3>value3</ name-3> <name-4>value4</ name-4> …. </notification> So quite simple and small. I will mainly retrieve data by selecting notifications of a specific @nid between two @ts values, thus I need an attribute index. I am using for now an embedded BaseX DB, to test the insertion of elements. Here is how I configure my DB: Context m_Context = new Context(); new Set(MainOptions.AUTOFLUSH, false).execute(m_Context); new Set(MainOptions.ADDCACHE, false).execute(m_Context); new Set(MainOptions.INTPARSE, true).execute(m_Context); new Set(MainOptions.STRIPNS, true).execute(m_Context); new Set(MainOptions.UPDINDEX, true).execute(m_Context); new Set(MainOptions.TEXTINDEX, false).execute(m_Context); new Set(MainOptions.ATTRINDEX, true).execute(m_Context); new CreateDB(_SourceId).execute(m_Context); And this is how I insert the elements: try { String l_XmlRepresentation = _Notification.getXmlRepresentation(); if (l_XmlRepresentation.isEmpty()) { return; } ByteArrayInputStream l_InputStream = new ByteArrayInputStream(l_XmlRepresentation.getBytes(m_Charset)); Add add = new Add(_Notification.getSourceId()); add.setInput(l_InputStream); add.execute(m_Context); if (_CurrentNotification % 10000 == 0) { // flush every 10000 notifications new Flush().execute(m_Context); } } catch (BaseXException ex) { s_Logger.log(Level.SEVERE, null, ex); } The performances I get are as follows Size 10'000, Speed: 1'292 Size 20'000, Speed: 625 Size 30'000, Speed: 361 Size 40'000, Speed: 248 Size 50'000, Speed: 184 Size 60'000, Speed: 148 Size 70'000, Speed: 123 Size 80'000, Speed: 104 Size 90'000, Speed: 91 Size 100'000, Speed: 77 Size 110'000, Speed: 69 Size 120'000, Speed: 61 Size 130'000, Speed: 56 Size 140'000, Speed: 46 Where “Size” is the number of elements in the collection and “Speed” is average speed of insertion [in element per second] of the last 10000 elements. My question is: do those performances seem normal or am I doing something wrong, knowing that with UPDINDEX = false, I have a steady insertion rate of 10000 elements per second. Thanks a lot Simon

2 3

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

BaseX-Talk