BaseX-Talk February 2025

basex-talk@mailman.uni-konstanz.de

13 participants
10 discussions

HTTP server performance seems very slow...
by Peter Villadsen 15 Mar '25

15 Mar '25

All, I have been using BaseX for a while, connecting to the TCP endpoint. I know the performance I typically get, and it is impressive! However, now I wanted to use the HTTP endpoint, and it seems the performance is at least 2 orders of magnitude worse! Here is the query that I am POSTing to http://localhost:8984/rest/RainFnd_6.0.10.0 <query xmlns=http://basex.org/rest> <text>/Class[@Package='ApplicationPlatform']/@Name</text> </query> This simple query will generate around 1500 results from the 13GB database (RainFnd_6.0.10.0<http://localhost:8984/rest/RainFnd_6.0.10.0>). It takes just over 7 seconds to do this. If I do this in the BaseX GUI that is self contained, it takes around 20ms. However, it seems that the time spent executing the query against the database is negligible. Please consider this query: <query xmlns=http://basex.org/rest> <text>1 + 2</text> </query> In which there there is obviously no database access. It takes almost the same amount of time as the query that accesses the database. 7 seconds to calculate 1 + 2 is too long. If I post the 1 + 2 query to the endpoint without specifying the database on the URL: http://localhost:8984/rest it takes around 7 milliseconds, close to what I expected, certainly within expectations for the time spent sending the query over the wire and serializing etc. This leads me to believe that a lot of the time (>7 seconds) may be spent opening the database each time a POST is done? Is there a way to tweak the HTTP server to "remember" the connection with the current database for a little while? This may be against the REST principles, of course. The database is guaranteed to be read-only in my case. The problem is that this makes the HTTP server inappropriate for interactive applications. I can still use the TCP server, where I get the results I need, but using the HTTP would be simpler, and have less overhead in terms of code needed to communicate with the server. Please let me know if there is a way to accomplish acceptable performance with the HTTP server. Best Regards Peter Villadsen Principal Technical Program Manager Microsoft Business Applications Group

3 7

Copy document via REST API
by Liedtke, Alexander 02 Mar '25

02 Mar '25

Hello all, I am trying to copy (duplicate) an existing document via REST API, but can not get it working. I tried via POST as commands. Can somebody please provide me a working example ? My use case is simply duplicate an existing document in a databse, as a backup file. From the documentation: <add (path='...')>[input]</add>, I translated it to xml command similar: <commands> <command>OPEN db</command> <command>ADD copy.xml source.xml</command> </commands> with diffferent variations, like adding path argument etc. However, in the end I got nothing working. I also tried it via xquery, but I would prefer xml like above. But in the end I am happy with any working example. Two more questions in that context. What is the recommended header Content-Type xml or text/plain? Does it make sense rto use the http://basex_rest_url/?command url or just the REST URL? I am pretty sure, there is an easy solution for copying documents, so any help on this is appreciated. Thank you in advance Alexander Liedtke

2 3

Re: Performance tuning
by Christian Grün 27 Feb '25

27 Feb '25

Just some quick feedback: Does it work if you specify the input along with CREATE DB? basex -c"CREATE DB taurus SPANYOLORSZÁG.xml" You can also specify a directory as input. Thanks, Christian Csaba Fekete <feketecsaba(a)gmail.com> schrieb am Do., 27. Feb. 2025, 17:36: > Hi Christian > Sorry, I thought I was sending this to the mailing list. Thanks for > answering anyway! > Now I'm trying with a smaller dataset and I am adding the documents one by > one. I also upgraded BaseX to the latest version. > The largest document is 1151M in size and it can't be imported, even if I > use attrindex and textindex. > The file is actually publicly available: > http://taurusreisen.hu/partner/v2/SPANYOLORSZAG.zip > Here is my command and the output: > /opt/basex/bin/basex -Oattrindex=true -Otextindex=true -v -V -c"OPEN > taurus; ADD ./SPANYOLORSZÁG.xml" > Database 'taurus' was opened in 18.21 ms. > Out of Main Memory. > I am thinking of solving the problem by splitting the file to several > chunks, which will be CPU-demanding but could make it work. > Any ideas are welcome. > Thank you again, and a million thanks for BaseX! It is a fantastic tool. > Regards, > Csaba > > On Thu, 27 Feb 2025 at 15:52, Christian Grün <christian.gruen(a)gmail.com> > wrote: > >> Hi Csaba, >> >> It’s difficult to give a general advice; XML documents are just too >> different. In principle, a few GB or even MB can be sufficient to create >> databases for very large collections (10 GB and more), but sometimes >> namespaces are a showstopper. See [1] for some statistics. >> >> What’s the total size of your XML documents? Can you create the database >> if you enable the text and attribute index? >> >> Best, >> Christian >> >> [1] https://docs.basex.org/main/Statistics >> >> >> >> >> On Tue, Feb 25, 2025 at 2:10 PM Csaba Fekete <feketecsaba(a)gmail.com> >> wrote: >> >>> Hi >>> I have a web server that runs Basex 11.1. The server is a VPS with 18G >>> of RAM. >>> I have a directory of documents in various sizes, ranging from a few >>> kilobytes up to 2G. >>> I am trying to import these documents with the command >>> CREATE DB mydb /path/to/docs >>> With the default jvm max heap size (2GB) I get the error: Out of main >>> memory >>> If I raise the max heap size to 4GB, I get the same error. >>> If I raise it to 8GB, the system becomes unresponsive. >>> How can I determine how much system memory I need to be able to carry >>> out this task? >>> Thanks >>> >>

2 2

Performance tuning
by Csaba Fekete 27 Feb '25

27 Feb '25

Hi I have a web server that runs Basex 11.1. The server is a VPS with 18G of RAM. I have a directory of documents in various sizes, ranging from a few kilobytes up to 2G. I am trying to import these documents with the command CREATE DB mydb /path/to/docs With the default jvm max heap size (2GB) I get the error: Out of main memory If I raise the max heap size to 4GB, I get the same error. If I raise it to 8GB, the system becomes unresponsive. How can I determine how much system memory I need to be able to carry out this task? Thanks

2 1

RESTXQ Textarea request paramater
by brian＠kennison.name 27 Feb '25

27 Feb '25

Everyone, I’m trying to build a comment form using RESTXQ that has a texarea element. I was hoping to use the new lines from the textarea for simple formatting but when the form is submitted the request parameter is striped of all the linefeeds ( it looks like normalized text). Is this the expected behavior? Thanks — Brian

3 4

Editor format adjustments
by Joel Kalvesmaki 24 Feb '25

24 Feb '25

Hi everyone, I just joined. BaseX is awesome (as y'all know). I like CTRL+SHIFT+F for formatting, but it is also messing up string values, which especially wreaks havoc with regular expressions. That is... let $regex := "(a|z)" ...becomes... let $regex := "( a|z )" I didn't see anything in the documentation or archives about micromanaging the formatting. Is the best practice to avoid formatting altogether? Or is there a hidden pragma to exempt lines from being formatted? Joel -- Joel Kalvesmaki Director, Text Alignment Network http://textalign.net

5 5

Re: Synchronizing access to store
by Andy Bunce 24 Feb '25

24 Feb '25

>so if you call store:get, store:put or store:write in the first process, a second process will not wait until the store operations are completed. In non XQuery contexts a semaphore [1] might be used to ensure that my other threads don't get between a get and put. In the spirit of blurring the XQuery Java boundaries I tried [2]. It seems to work. Is it dangerous? [1] https://docs.oracle.com/javase/8/docs/api/java/util/concurrent/Semaphore.ht… [2] declare namespace Semphore ="java:java.util.concurrent.Semaphore"; declare function local:config-update($k as xs:string,$v as item(),$sem) { Semphore:acquire($sem), try{ let $u:=store:get("config")=> map:put($k, $v) return store:put("config",$u) }catch * { trace("Errrr",$err:description) } ,Semphore:release($sem) }; let $sem:=Semphore:new(1,true()) let $s1 := store:put("config", map{}) let $s2 := xquery:fork-join( for $i in (1 to 100) return function(){ let $r:=(prof:sleep(10),$i) return local:config-update( string($i),$r,$sem) } ) return count(map:keys(store:get("config"))) On Tue, 28 Jan 2025 at 14:09, Marco Lettere <m.lettere(a)gmail.com> wrote: > Ok, thanks for the clarification. > > M. > On 28/01/25 15:08, Christian Grün wrote: > > Sorry Christian, do you mean *not* synchronized? >> > With »synchronized«, I meant to refer to a lower level: You will not end > up with a corrupt key/value store or with I/O conflicts when accessing and > updating the store via multiple threads. However, as you have already > observed, multiple operations are not executed in a well-defined order, so > if you call store:get, store:put or store:write in the first process, a > second process will not wait until the store operations are completed. > >

3 5

where instead of when in window start clause
by Martin Honnen 22 Feb '25

22 Feb '25

I mistyped a start clause in a tumbling window expression using "where" instead of "when" for tumbling window $w in (1 to 20) start at $s where $s mod 3 = 1 return <window>{$w}</window> see https://fiddle.basex.org/?share=%28%27query%21%27for+tumbling+-*w+in+%7B1+t…, but I didn't get an error but a result <window>1</window> <window>4</window> <window>7</window> <window>10</window> <window>13</window> <window>16</window> <window>19</window> Shouldn't that query give a syntax error that "when" is expected instead of "where"? Or is that a new XQuery 4 feature that the "when" clause can be omitted and the above is kind of parsed/executed as for tumbling window $w in (1 to 20) start at $s where $s mod 3 = 1 return <window>{$w}</window> ?

2 1

Install on windows fails because of JAVA_TOOL_OPTIONS
by Peter Boot 22 Feb '25

22 Feb '25

Hello, I tried to install BaseX on Windows 11 and it kept giving me the message that Java 11 was needed, even though I already was at 20 and now am at 21. Then I noticed that java --version also reported a JAVA_TOOLS_OPTIONS environment variable: java --version Picked up JAVA_TOOL_OPTIONS: -Dfile.encoding=UTF-8 openjdk 21.0.6 2025-01-21 LTS OpenJDK Runtime Environment Temurin-21.0.6+7 (build 21.0.6+7-LTS) OpenJDK 64-Bit Server VM Temurin-21.0.6+7 (build 21.0.6+7-LTS, mixed mode, sharing) After removing that variable it no longer appears in the output of the version command and the install runs fine. I suppose it messes up parsing the version output. Peter

2 1

Need confirmation on http:send-request
by Marco Lettere 13 Feb '25

13 Feb '25

Dear Christian et al, moving to basex 11.7 today I've found out that it is not necessary to serialize the body content in an http-request such as [1]. If I do (as I had to up to Basex 10.7) the body is no longer understood. Now I obtsain correct behaviour when using code as in [2]. Can you confirm this change in behaviour? And if yes, this has a huge impact on the many many http calls we have so maybe it should be documented? Thanks a lot as usual! M. [1] let $body := json:serialize(<json type="object"/>) return <http:request href="url" method="POST"> <http:body media-type="application/json">{$body}</http:body> </http:request> [2] let $body := <json type="object"/> return <http:request href="url" method="POST"> <http:body media-type="application/json">{$body}</http:body> </http:request>

2 1

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

BaseX-Talk February 2025