- BaseX-Talk - mailman.uni-konstanz.de

Understanding the path index
by Beck, Julia 29 Oct '19

29 Oct '19

Hi, first of all: thank you, the fix for [1] did the trick and in 9.2.4 the query is working as expected. Today, I come back to you with another challenge in performance which again seems to have something to do with indexing(?). So here's the situation: I have two databases "abc" and "def". "abc" contains 1 xml doc with about 150.000 nodes and "def" contains 1 xml doc with about 400.000 nodes. Both are similarly strutured and have their up-to-date text and attr indexes. The xml docs look both (simplified) like the following: <rdf:RDF> <ore:Aggregation rdf:about="123"> <edm:object rdf:resource="urn1"/> <...> </ore:Aggregation> <edm:WebResource rdf:about="urn1"> <...> </edm:WebResource> <ore:Aggregation rdf:about="124"> <edm:object rdf:resource="urn2"/> <...> </ore:Aggregation> <edm:WebResource rdf:about="urn2"> <...> </edm:WebResource> <ore:Aggregation rdf:about="125"> <edm:object rdf:resource="urn2"/> <edm:object rdf:resource="urn3"/> <...> </ore:Aggregation> <edm:WebResource rdf:about="urn3"> <...> </edm:WebResource> <...> </rdf:RDF> So one aggregation refers to one (or more) web resources. I boiled down my original query to the following purpose to keep it simple: for each aggregation give me the corresponding web resource. for $agg in db:open($db_name)/rdf:RDF/ore:Aggregation return for $urn in $agg/edm:object/@rdf:resource return (# db:enforceindex #) {db:open($db_name)/rdf:RDF/edm:WebResource[@rdf:about=$urn]} For both databases the query gives me the required result and the query info tells me that the attribute index for $urn is applied in both cases (this is also the case if I leave out the pragma). However, oddly enough, for the "larger" database "def" with a larger attribute index it takes roughly 1 second while the "smaller" database "abc" with a smaller attribute index takes 20 seconds. This is not very long but the original query is more complicated and I have bigger databases with the same structure where it starts to matter. The only (and I think important) difference between "abc" and "def" is that "abc" contains only 3 web resources that all 5.000 aggregations refer to. While in "def" each aggregation refers to a particular web resource (== 7.000 aggregations and 7.000 web resources). With index:facets I had a look at the facet values and learned that there is a "maximum number of distinct values to store per name". Is there a difference in performance because of that? Maybe I do not get the index structures but it feels strange that it takes longer to find the correct attribute in a range of 3 different values than in a range of 7.000. Maybe there is also another problem in my query, databases or my reasoning that I do not see? Either way, I need help in understanding this phenomenon :-) I hope you could follow, please don't hesitate to ask if you need anything to reproduce this situation (I am using BaseX 9.2.4). Julia [1] https://mailman.uni-konstanz.de/pipermail/basex-talk/2019-July/014511.html

3 6

http:send-request - problem with response
by Bogdan Bogucki 28 Oct '19

28 Oct '19

Hello, I encounter problem with http:send-request function. I need handle multiple forward requests (3) with cookies. First request returns cookies information which are required with next requests. Output from http:send-request dosen't contains all fields which are returned by server. Response from browser is flowing: cache-control private cf-cache-status DYNAMIC cf-ray 52b3cedc5b9bcc9f-WAW content-type text/html; charset=utf-8 date Fri, 25 Oct 2019 11:21:37 GMT expect-ct max-age=604800, report-uri="ht….com/cdn-cgi/beacon/expect-ct" location http://www.pracuj.pl/praca/jav…eloper-warszawa,oferta,7171988 server cloudflare set-cookie __cfduid=daf54110b2a87d66c2a53…; domain=.pracuj.pl; HttpOnly set-cookie _yaic=13; expires=Fri, 31-Dec-9999 23:59:59 GMT; path=/ set-cookie _urnadiam=A; domain=.pracuj.pl…3:00:00 GMT; path=/; HttpOnly x-aspnet-version 4.0.30319 x-aspnetmvc-version 5.2 X-Firefox-Spdy h2 x-powered-by ASP.NET x-ua-compatible IE=edge Request: http:send-request( <http:request method='get' follow-redirect='false'> <http:header name="User-Agent" value="Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:69.0) Gecko/20100101 Firefox/69.0"/> <http:header name="Accept" value="text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8"/> <http:header name="Accept-Languag" value="pl,en-US;q=0.7,en;q=0.3"/> <http:header name="DNT" value="1"/> <http:header name="Connection" value="keep-alive"/> <http:header name="Upgrade-Insecure-Requests" value="1"/> <http:body media-type='application/x-www-form-urlencoded; charset=UTF-8'/> </http:request>, 'https://www.pracuj.pl/praca/java-developer-warszawa,oferta,7171988') Response form http:send-request is: <http:response xmlns:http="http://expath.org/ns/http-client" status="302" message="Found"> <http:header name="Transfer-Encoding" value="chunked"/> <http:header name="CF-RAY" value="52b3d0de097fb839-WAW"/> <http:header name="Server" value="cloudflare"/> <http:header name="Connection" value="keep-alive"/> <http:header name="Date" value="Fri, 25 Oct 2019 11:23:00 GMT"/> <http:header name="X-AspNetMvc-Version" value="5.2"/> <http:header name="X-UA-Compatible" value="IE=edge"/> <http:header name="CF-Cache-Status" value="DYNAMIC"/> <http:header name="Cache-Control" value="private"/> <http:header name="X-AspNet-Version" value="4.0.30319"/> <http:header name="Set-Cookie" value="_urnadiam=B; domain=.pracuj.pl; expires=Tue, 31-Dec-2019 23:00:00 GMT; path=/; HttpOnly"/> <http:header name="Expect-CT" value="max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct…"/> <http:header name="X-Powered-By" value="ASP.NET"/> <http:header name="Content-Type" value="text/html; charset=utf-8"/> <http:body media-type="text/html"/> </http:response> Two set-cookie fields is missing. How should I invoke http:send-request to receive proper result ? Regards Bogdan

2 1

Optimization issue?
by Johannes Echterhoff 25 Oct '19

25 Oct '19

Hello, I'm experiencing some strange behavior when executing the following query in the BaseX 9.2.4 GUI (nevermind the awkward query structure - that is just to be able to inspect the results of the individual evaluation steps): let $t := <x>200.0000</x> let $num := number($t) let $gt5000 := $num gt 5000 let $ltMinus5000 := $num lt -5000 return $gt5000 or $ltMinus5000 The result is true, although it should be false. The compilation and optimization logs for this query in the GUI tell us the following: ---------- Compiling: - rewrite > comparison to range comparison: ($num_1 > 5000) -> ($num_1 >= 5000.000000000001) - rewrite 'gt' comparison to range comparison: ($num_1 gt 5000) -> ($num_1 >= 5000.000000000001) - pre-evaluate unary expression to xs:integer item: -5000 - rewrite < comparison to range comparison: ($num_1 < -5000) -> ($num_1 <= -5000.000000000001) - rewrite 'lt' comparison to range comparison: ($num_1 lt -5000) -> ($num_1 <= -5000.000000000001) - inline $gt5000_2 - inline $ltMinus5000_3 - simplify or: (true()) - pre-evaluate or to xs:boolean item: (true()) -> true() - remove unused variable $num_1 - remove unused variable $t_0 - simplify FLWOR expression: true() Optimized Query: true() ---------- When the return statement is changed to "return ($num,$gt5000,$ltMinus5000,$gt5000 or $ltMinus5000)" then the result of the boolean operation (last element of the result sequence) is false, as expected. The GUI log then is: ---------- Compiling: - rewrite > comparison to range comparison: ($num_1 > 5000) -> ($num_1 >= 5000.000000000001) - rewrite 'gt' comparison to range comparison: ($num_1 gt 5000) -> ($num_1 >= 5000.000000000001) - pre-evaluate unary expression to xs:integer item: -5000 - rewrite < comparison to range comparison: ($num_1 < -5000) -> ($num_1 <= -5000.000000000001) - rewrite 'lt' comparison to range comparison: ($num_1 lt -5000) -> ($num_1 <= -5000.000000000001) Optimized Query: let $t_0 := element x { ("200.0000") } let $num_1 := number($t_0) let $gt5000_2 := ($num_1 >= 5000.000000000001) let $ltMinus5000_3 := ($num_1 <= -5000.000000000001) return ($num_1, $gt5000_2, $ltMinus5000_3, ($gt5000_2 or $ltMinus5000_3)) ---------- Am I missing something, or could this be a bug in BaseX 9.2.4? With BaseX 9.0.2, both queries (with the different return statements) returned the expected value false for the boolean operation. Best regards, Johannes

2 3

Question about WITHDB option
by France Baril 24 Oct '19

24 Oct '19

I am looking at the documentation for WITHDB and I want to ask for further explanation 'If db:open is consistently used to access documents', does it means as opposed to having a DEFAULTDB or does it mean that we use db:open but not local queries on the document obtained (example below), or since there is a notion of lock, do you mean db:open as opposed to db:add or db:replace? In short, I lack context to interpret the explanation. let $current-doc := db:open($lang)/*[@id=x] (: Query document further without db:opened :) let $avaliable-sections := $current-doc/descendant::*[name()=''section][@status !='archived'] >From the documentation: WITHDB Introduced with Version 9.3. *Signature* WITHDB *Default* true *Summary* By default, URIs specified in the fn:doc <http://docs.basex.org/wiki/Databases#XML_Documents> and fn:collection <http://docs.basex.org/wiki/Databases#XML_Documents> functions will also be resolved against existing databases. If db:open <http://docs.basex.org/wiki/Database_Module#db:open> is consistently used to access database documents, it is recommendable to disable this option: - Access to local and external resources will be faster, as the database lookup will be skipped. - No locks will be created by the two functions (see limitations of database locking <http://docs.basex.org/wiki/Transaction_Management#Limitations> for more details). -- France Baril Architecte documentaire / Documentation architect france.baril(a)architextus.com

2 3

catalog.xml - xsd - urn
by SW-Service 22 Oct '19

22 Oct '19

Good day, Where is the catalog.xml file stored? I want to validate xml files against XSD, but the xsd is referenced via an urn. thank you very much Guten Tag, Wo wird die catalog.xml Datei abgelegt? Ich möchte xml-Dateien gegen XSD validieren, aber die XSD wird über einen urn referenziert. Herzlichen Dank <xdomea:Aussonderung.Aussonderung.0503 xmlns:xdomea="urn:xoev-de:xdomea:schema:2.3.0">

2 2

Re: [basex-talk] Backups
by Christian Grün 21 Oct '19

21 Oct '19

Hi Christoph, > db_alter() has not changed. It alters the name of the database but the corresponding backups stay unchanged. I combine it with db:drop-backup() That’s true: db:alter() will only rename databases. This has been a deliberate design choice (in analogy with db:drop, a user might want to treat backups differently than the currently active instance); but I think we should add an additional db:alter-backup function. I have just created an issue for that [1]; Thanks for the observation. > I copied the old data, restxq etc. into the new structure. > … > In both cases I lost my users. How can I upgrade without loosing them? Users are stored in the data directory in the users.xml file. Did you copy this file as well? Maybe the BaseX server was still running while you were copying the files? For databases, this shouldn’t matter (database are dynamically accessed), but the users.xml file will only be parsed at startup time. Best, Christian [1] https://github.com/BaseXdb/basex/issues/1742 On Mon, Oct 21, 2019 at 1:47 PM Christoph Gaukel <christoph.gaukel(a)gmx.de> wrote: > > Hi Christian, > > thank you for your answer. I upgraded from 9.1.2 to 9.2.4 and indeed: > > the output of db:backups('test_balance_tmp') changed from: > <backup database="test_balance_tmp" date="2019-10-18T00:05:50" size="2230">test_balance_tmp-2019-10-18-00-05-50</backup> > to: > <backup database="test_balance_tmp" date="2019-10-17T22:05:50Z" size="2230">test_balance_tmp-2019-10-18-00-05-50</backup> > > The same with db:drop(), now it deletes the database but not the backups. > > db_alter() has not changed. It alters the name of the database but the corresponding backups stay unchanged. I combine it with db:drop-backup() > > > From doing the upgrade, I got a new question. I have done two trials: > > I copied the old data, restxq etc. into the new structure. > I copied BaseX.jar and basex/lib into the old structure > > In both cases I lost my users. How can I upgrade without loosing them? > > > Thanks, > > Christoph > > Am 18.10.19 um 10:25 schrieb Christian Grün: > > Hi Christoph, > > Welcome back to the list. > > db:drop deletes both, the database and the corresponding backups > > The function should only delete database and preserve backups. For > deleting backups, db:drop-backup can be used. Maybe there was a bug in > an older version of BaseX? > > db:info($db)/resourceproperties/inputdate is a UTC dateTime with a time zone offset > db:backups($db)/@date is a local dateTime without a time zone offset > > Same here: Both returned timezones should be a UTC dateTime with the Z > time offset. > > Maybe you need to switch to the latest version of BaseX? > Feel free to give us an update if the problems persist with BaseX 9.x. > > Hope this helps, > Christian >

1 0

HTTP request
by Giuseppe G. A. Celano 18 Oct '19

18 Oct '19

Hi, I have a curl request of the kind curl --data "something" http://something <http://something/> which I would like to perform using the HTTP module, but I cannot specify the data argument in the query. Any help? The following does not work: http:send-request(<http:request method="post"></http:request>, 'http://something' , "something" ) Best, Giuseppe

2 1

simplest possible FLOWR specifying a database
by thufir 12 Oct '19

12 Oct '19

these FLWOR .xq files work as is: let $db := db:open("com.w3schools.books") for $x in $db/bookstore return $x or <new_catalog>{ let $db := db:open("com.w3schools.books") for $x in $db/bookstore/book return <new_book>{$x/title,$x/author}</new_book> }</new_catalog> Probably a silly question, but, rather than, as in the first query, using $x can I not somehow specify just "/"? thanks, Thufir

4 5

what do with treasury data?
by thufir 11 Oct '19

11 Oct '19

I randomly came across the data at: https://www.treasurydirect.gov/xml/ which is all well and good, but what might be a good starter project with this data? Perhaps join it with other data? But what other data? Not even sure what this data, except that there's mention of announcements and auctions. thanks, Thufir

1 0

Migrating ~ 3M record db from BaseX to PostgreSQL results in OOM
by first name last name 08 Oct '19

08 Oct '19

Hello, This is essentially part2 of trying to index large amounts of web data. To summarize what happened before: The initial discussion started here [1], Christian suggested some options, I dove into each of them, I realized that doing this on a low-memory system is harder than I initially thought. At Christian's suggestion, I tried to split the big db into smaller dbs and came up with a rudimentary sharding mechanism [3]. All attempts to full-text 30GB of data in BaseX, for me, resulted in OOM (do take into consideration that I only have 3.1GB of memory to allocate for BaseX). Where to? I decided to look more into what Christian said in [2] about option 2, and to pick the exact values that I want, and to transfer them to PostgreSQL (after transferring, a GiST index would have to be built there, to allow full-text searches; PostgreSQL is picked because it uses an in-memory buffer for all large operations, and several files on disk, and if it needs to combine results that exceed the available memory, it goes to disk, but at all times it never exceeds the given amount of memory). Variant 1 (see attached script pg-import.sh) All good. So, I basically started writing XQuery that would do the following: - Open up a JDBC connection to PostgreSQL - Get me all text content from each thread page of the forum, and the db it belonged to - Create a prepared statement for one such thread page, populate the prepared statement, and execute it This ended up in OOM after around 250k records. So just to be clear, 250k lines were rows in PostgreSQL, which is nice but eventually it ended up in OOM. (Perhaps it has to do with how the GC works in Java .. I don't know) Variant 2 (see attached script pg-import2.sh) I did something similar to the above: - Open up a JDBC connection to PostgreSQL - Get all posts and for each post get the author, the date, the message content, the post id, the BaseX db name (cause we're going over all shards, and each shard is a BaseX db) - Create a prepared statement for each post with the data mentioned above, and execute it This also ended up in OOM after around 340k records (my approximation would be that there were around 3M posts in the data). To summarize, I'm tempted to believe that there might be a leak in the BaseX implementation of XQuery. I will provide in the following, the relevant versions of the software used: - BaseX 9.2.4 - java version "1.8.0_151" Java(TM) SE Runtime Environment (build 1.8.0_151-b12) Java HotSpot(TM) 64-Bit Server VM (build 25.151-b12, mixed mode) - the JVM memory param value was -Xmx3100m I would be interested to know your thoughts Thanks, Stefan [1] https://mailman.uni-konstanz.de/pipermail/basex-talk/2019-September/014715.… [2] https://mailman.uni-konstanz.de/pipermail/basex-talk/2019-October/014727.ht… [3] https://mailman.uni-konstanz.de/pipermail/basex-talk/2019-October/014729.ht…

2 4

Want event, not polling....
by Peter Villadsen 07 Oct '19

07 Oct '19

Hello, I am using a client written in C#. For the sake of this discussion, imagine the app being discussed as an app just like the Java GUI app that ships with BaseX: You can basically author Xquery snippets and send them for execution by the server. We regularly have queries that tend to run for a long time, and it would be an advantage for the user to be able to cancel these long running jobs, just like he can in the GUI app. However, I am not sure how to do this; I can imagine two solutions: 1. I see that jobs that are submitted normally to the server end up as jobs, just like the ones that are submitted with jobs:eval(query): I can stop that job with jobs:stop(jobId) on another session if the jobId is known. You would need a separate session that fetches the running jobs and gets the current one (recognizing it by creating a cookie in a comment in the query) with jobs:list-details(), then getting the jobId and then calling jobs:stop(jobId). It is possible I guess, but very heavy handed. 2. The other option is just as bad: submit the query through jobs:eval(query), getting the jobId and then poll for it to finish; more polling which will not scale with many users and will not provide the quick feedback we want. I see some traces of an eventing mechanism that was deprecated (or at least no longer included in the client). That is what I would ideally have (ignoring bindings etc): Query q = new Query("1 to 10" , new Guid()); q.Ended += MyTerminationHandler(); // C# syntax for adding an event handler for termination q.Interrupted += MyInterruptedHandler(); await q.execute(); // Called when the query has terminated with a result void MyTerminationHandler(Guid g, string result) { ... update UI ... } // Called when the query has terminated with a result void MyInterruptedHandler (Guid g) { ... update UI ... } How do I do this without the polling, being able to pass massive results and queries back and forth? Best Regards Peter Villadsen Principal Architect Microsoft Business Applications Group

1 0

how to specify a database in a.xq FLOWR query?
by thufir 07 Oct '19

07 Oct '19

how do I specify a database for a .xq file to use? a simple FLOWR: thufir@dur:~/basex/w3$ thufir@dur:~/basex/w3$ basex titles.xq [warning] /usr/bin/basex: Unable to locate /usr/share/java/jing.jar in /usr/share/java <title lang="en">Learning XML</title> <title lang="en">XQuery Kick Start</title>thufir@dur:~/basex/w3$ thufir@dur:~/basex/w3$ thufir@dur:~/basex/w3$ cat titles.xq for $x in doc("books.xml")/bookstore/book where $x/price>30 order by $x/title return $x/title thufir@dur:~/basex/w3$ (from w3schools) thanks, Thufir

3 4

// versus /*/
by thufir 07 Oct '19

07 Oct '19

Do these have the same meaning? Might there be a subtle distinction, or might they be read differently but functionally identical? > > xquery /*/book[@id="bk112"] <book id="bk112"> <author>Galos, Mike</author> <title>Visual Studio 7: A Comprehensive Guide</title> <genre>Computer</genre> <price>49.95</price> <publish_date>2001-04-16</publish_date> <description>Microsoft Visual Studio 7 is explored in depth, looking at how Visual Basic, Visual C++, C#, and ASP+ are integrated into a comprehensive development environment.</description> </book> Query executed in 1.22 ms. > > xquery //book[@id="bk112"]/title <title>Visual Studio 7: A Comprehensive Guide</title> Query executed in 1.52 ms. > > xquery /*/book[@id="bk112"]/title <title>Visual Studio 7: A Comprehensive Guide</title> Query executed in 1.67 ms. > They're equally efficient, at least as used above? thanks, Thufir

2 2

Unidentified java processes keeping locks on DBs (BaseX 9.2.3)
by France Baril 07 Oct '19

07 Oct '19

Hi, We have been getting process issues on Windows installations for a while now. Has anyone had fantom locks (non-existent processes locking indices) and if so, can you point to clues that could help resolve the issue? [image: image.png] This leads to batch processes failure and DB corruption so we are eager even for the tiniest clues. Thank you -- France Baril Architecte documentaire / Documentation architect france.baril(a)architextus.com

1 0

get list of databases
by thufir 06 Oct '19

06 Oct '19

Hi, pardon the silly question, but how do I get a list of databases exactly? I'm looking at: https://github.com/BaseXdb/basex/blob/master/basex-examples/src/main/java/o… and want output like: > list Name Resources Size Input Path ---------------------------------- books 0 4570 1 database(s). > but using: http://docs.basex.org/javadoc/org/basex/core/cmd/List.html thanks, Thufir

2 2

basex OOM on 30GB database upon running /dba/db-optimize/
by first name last name 06 Oct '19

06 Oct '19

Hi, Let's say there's a 30GB dataset [3] containing most threads/posts from [1]. After importing all of it, when I try to run /dba/db-optimize/ on it (which must have some corresponding command) I get the OOM error in the stacktrace attached. I am using -Xmx2g so BaseX is limited to 2GB of memory (the machine I'm running this on doesn't have a lot of memory). I was looking at [2] for some estimates of peak memory usage for this "db-optimize" operation, but couldn't find any. Actually it would be nice to know peak memory usage because.. of course, for any database (including BaseX) a common operation is to do server sizing, to know what kind of server would be needed. In this case, it seems like 2GB memory is enough to import 340k documents, weighing in at 30GB total, but it's not enough to run "dba-optimize". Is there any info about peak memory usage on [2] ? And are there guidelines for large-scale collection imports like I'm trying to do? Thanks, Stefan [1] https://www.linuxquestions.org/ [2] http://docs.basex.org/wiki/Statistics [3] https://drive.google.com/open?id=1lTEGA4JqlhVf1JsMQbloNGC-tfNkeQt2

3 13

Does BaseX used as a command line XQuery processor build a full tree of an input document passed with -i option?
by Martin Honnen 03 Oct '19

03 Oct '19

If I use BaseX (9.2.4) simply as an XQuery 3.1 processor from the command line with e.g. basex.bat -i input.xml query.xq does BaseX then first always parse the input.xml into an XDM tree, meaning if I use huge input documents that way I can run easily run out of memory? Or does that depend on the type of query or some other settings? I was wondering whether a "tumbling window" based split algorithm like declare namespace output = "http://www.w3.org/2010/xslt-xquery-serialization"; declare option output:method 'xml'; declare option output:indent 'yes'; declare variable $chunk-size as xs:integer external := 500; for tumbling window $chunk in /*/* start at $sp when $sp mod $chunk-size = 1 count $p return put(document { element { node-name(head($chunk)/..) } { $chunk } }, 'xquery-split-result-' || $p || '.xml') would run/work without memory problems for huge inputs.

2 1

Re: [basex-talk] HTTPServer + gzip compression
by Stefan Koch 03 Oct '19

03 Oct '19

2 1

Re: [basex-talk] Connection timeout
by Marie Starck 01 Oct '19

01 Oct '19

I am using an ssl certificate and have a keystore on my production Basex instance. Are there extra steps needed to make an HTTPS request from a server to a basex server when those are in place? On Thu, Sep 26, 2019 at 6:14 PM Marie Starck <marie.compsci14(a)gmail.com> wrote: > Hey Christian, > > This error message is from doing it with the nodejs basex client. I get > the same connection time out when doing an http or https request on 8984 > and 8986 (non-ssl and ssl respectively). > > Thank you, > > Marie > > On Thu, Sep 26, 2019 at 5:54 PM Christian Grün <christian.gruen(a)gmail.com> > wrote: > >> Hi Marie, >> >> SOCKET ERROR: { Error: connect ETIMEDOUT 159.89.126.125:1984 >>> >> >> By default, the REST endpoints are available on port 8984. >> >> Maybe this helps, >> Christian >> >> >>

2 1

HTTPServer + gzip compression
by Stefan Koch 01 Oct '19

01 Oct '19

Hi, I'm struggling to get gzip compression working. I'm using the REST module. http://docs.basex.org/wiki/REST Version: BaseX 9.2.2 Tried to add the handler in jetty.xml as descriped here: https://www.eclipse.org/jetty/documentation/current/gzip-filter.html But it didn't work. Tried gzip filter via web.xml - but it is deprecated since jetty 9.3. Gzip handler is the correct way to do it. Searched the mailling list, similar problem was reported here: https://mailman.uni-konstanz.de/pipermail/basex-talk/2019-February/014160.h… Unfortunately no solution :( Not much experience with embedded jetty - but adding gzip compression is straight forward in tomcat or apache - no clue why it doesn't work.Shouldn't be that hard - and it's kinda a default feature. Did anybody get it to work? Any tipps? Thanks Stefan Koch B&IT Solutions Dipl.-Kfm. Rudolf Markus Petri Lietzenburger Str. 77 10719 Berlin Tel: 0049 30 8867 6099 Fax: 0049 30 8867 6159 Mail: koch(a)buit-solutions.com Web: www.buit-solutions.com

2 1

Database corrupted on updating with RestXQ
by Marco Lettere 26 Sep '19

26 Sep '19

Hi all, starting with 9.2.1 we experience a strange error with a RESTXQ API of ours that we have been using for years. The typical pattern is lookup a document update it and store it back. We have done this milions of time and also all the tests work neatly. But when using it from inside the web application, at around the tenth operation we get the DB corrupted with the stack trace [1]. This seems to happen only when the database is nearly empty and, for sure, it doesn't appear on 8.x series. It feels like some concurrency flaw is introduced somewhere but it's hard to say because the API is eight years in age and it never changed significantly. I know it's hard without an SSCE... but while working to insulate this byzantine behaviour (I've been trying for days now) maybe someone in the list has a clue or is able to interpret the stack trace? Thanks and have a nice weekend, Marco. [1] Improper use? Potential bug? Your feedback is welcome: Contact: basex-talk(a)mailman.uni-konstanz.de Version: BaseX 9.3 beta Java: Ubuntu, 13 OS: Linux, amd64 Stack Trace: java.lang.ArrayIndexOutOfBoundsException: Index 1 out of bounds for length 1 at org.basex.io.random.TableDiskAccess.fpre(TableDiskAccess.java:507) at org.basex.io.random.TableDiskAccess.cursor(TableDiskAccess.java:468) at org.basex.io.random.TableDiskAccess.read1(TableDiskAccess.java:156) at org.basex.data.Data.kind(Data.java:304) at org.basex.query.value.node.DBNode.<init>(DBNode.java:50) at org.basex.query.value.seq.DBNodeSeq.get(DBNodeSeq.java:139) at org.basex.query.value.seq.DBNodeSeq.get(DBNodeSeq.java:164) at org.basex.query.func.db.DbOpen.value(DbOpen.java:19) at org.basex.query.func.StandardFunc.optimize(StandardFunc.java:82) at org.basex.query.expr.Arr.inline(Arr.java:69) at org.basex.query.expr.path.Path.inline(Path.java:962) at org.basex.query.expr.gflwor.ForLet.inline(ForLet.java:66) at org.basex.query.expr.gflwor.GFLWOR.inline(GFLWOR.java:804) at org.basex.query.expr.gflwor.GFLWOR.inlineLets(GFLWOR.java:339) at org.basex.query.expr.gflwor.GFLWOR.optimize(GFLWOR.java:106) at org.basex.query.expr.gflwor.GFLWOR.inline(GFLWOR.java:785) at org.basex.query.expr.Try.inline(Try.java:106) at org.basex.query.expr.gflwor.GFLWOR.inline(GFLWOR.java:816) at org.basex.query.expr.gflwor.GFLWOR.inlineLets(GFLWOR.java:339) at org.basex.query.expr.gflwor.GFLWOR.optimize(GFLWOR.java:106) at org.basex.query.func.StaticFunc.inline(StaticFunc.java:294) at org.basex.query.func.StaticFuncCall.compile(StaticFuncCall.java:67) at org.basex.query.scope.MainModule.comp(MainModule.java:81) at org.basex.query.QueryCompiler.compile(QueryCompiler.java:114) at org.basex.query.QueryCompiler.compile(QueryCompiler.java:105) at org.basex.query.QueryContext.compile(QueryContext.java:312) at org.basex.query.QueryContext.iter(QueryContext.java:331) at org.basex.http.restxq.RestXqResponse.serialize(RestXqResponse.java:73) at org.basex.http.web.WebResponse.create(WebResponse.java:63) at org.basex.http.restxq.RestXqServlet.run(RestXqServlet.java:53) at org.basex.http.BaseXServlet.service(BaseXServlet.java:65) at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:865) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:542) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146) at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1700) at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1345) at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:480) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1667) at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1247) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) at org.eclipse.jetty.server.Server.handle(Server.java:505) at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:370) at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:267) at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305) at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103) at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126) at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:698) at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:804) at java.base/java.lang.Thread.run(Thread.java:835)

2 11

Connection timeout
by Marie Starck 26 Sep '19

26 Sep '19

Been debugging this for the past couple of days and I am a bit stumped. I am doing a simple get REST request to my BaseX server from a node server and I keep getting a connection timeout error. I also tried the basex node client and I get the same error message. All of this works fine on localhost and what's even weirder is that I can access the BaseX rest endpoint through my browser just fine. Any ideas? I have SSL installed on my BaseX server. Could that be the issue? SOCKET ERROR: { Error: connect ETIMEDOUT 159.89.126.125:1984 at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1173:14) errno: 'ETIMEDOUT', code: 'ETIMEDOUT', syscall: 'connect', address: '159.89.126.125', port: 1984 } Thank you for any help you can give me, Marie

2 1

readable encryption result
by Andre Bovy 23 Sep '19

23 Sep '19

Dear all, I try to make the result of the encryption readable by converting the result either in Base64 or in Hex. Yet I am not able to convert it back to be able to decrypt. Could you help me ? André Bovy let $encrypted := crypto:encrypt('message', 'symmetric','keykeyke','DES’) let $base64 := string(convert:string-to-base64($encrypted)) let $encrypted := convert:binary-to-string(xs:base64Binary($base64)) (: generates invalid characters :) return crypto:decrypt($encrypted, 'symmetric','keykeyke','DES')

2 2

Full-text index on mixed content
by Schopper, Daniel 23 Sep '19

23 Sep '19

Dear all, chatting after a session of the ongoing TEI conference ( https://graz-2019.tei-c.org) I was asked about plans to support fulltext indexes on mixed content nodes in BaseX – I did not know of any, so I wanted to pass the question on to this list: Is there a plan to implement this feature in the near (or not-so-near) future? If not, did somebody of the core devs estimate the effort to get this done? (needless to say that it would an awsome feature to have in BaseX ;-) Thanks in advance & best Daniel (just being curious) -- Mag. Daniel Schopper Austrian Centre for Digital Humanities Working Group Data, Resources and Standards Austrian Academy of Sciences Sonnenfelsgasse 19 1010 Vienna T: +43 1 51581-2216 E: daniel.schopper(a)oeaw.ac.at

2 1

Suggestion for concurrency of RestXQ
by Marco Lettere 23 Sep '19

23 Sep '19

Dear all, on one of our applications based on RestXQ services running on 9.0.2 we experience a strange behaviour related to concurrency. It is a stateless service running on a server with http-local set to true. So no database access. We removed all accesses to sessions and the code doesn't even write to files or other external resources. It's really only computation. When I run multiple requests, I can clearly see that they are accepted simultaneously (up to 8 which matches PARALLEL). They even complete out of order according to the session id reported in the log. [1] The strange thing is: performance seems to be as if calls were executed serially. If T is the time to complete one call then the pattern is T + T + T + T + T. I have to add that the algorithm is rather complex. Lots of functions (possibly inlined, private and the main one even recursive) which filter and transform a large XML request into a possibly large XML output. If I simplify the called RestXQ to something like [2] then I get a log like [3] which shows a pattern like the one I'm expecting T+d+d+d+d where d is just a slight overhead (orders of magnitude smaller). My question is what do I have to check in order to understand where the hard synchronization of the calls occurs? Might this be related to non-deterministic functions? What else? I've tried to isolate parts of the code but I'm unable to get predictable results. Thank you very much for all the hints you will be able to provide! Regards, Marco. [1] 17:10:19.204 127.0.0.1:58038 admin REQUEST [POST] http://localhost:8081/rate-selector/select-matching-rates 17:10:19.204 127.0.0.1:58040 admin REQUEST [POST] http://localhost:8081/rate-selector/select-matching-rates 17:10:19.205 127.0.0.1:58042 admin REQUEST [POST] http://localhost:8081/rate-selector/select-matching-rates 17:10:19.205 127.0.0.1:58044 admin REQUEST [POST] http://localhost:8081/rate-selector/select-matching-rates 17:10:19.208 127.0.0.1:58046 admin REQUEST [POST] http://localhost:8081/rate-selector/select-matching-rates 17:10:21.325 127.0.0.1:58040 admin 200 2121.01 ms 17:10:23.174 127.0.0.1:58046 admin 200 3965.74 ms 17:10:25.035 127.0.0.1:58038 admin 200 5830.63 ms 17:10:26.889 127.0.0.1:58044 admin 200 7683.76 ms 17:10:28.739 127.0.0.1:58042 admin 200 9534.46 ms [2] (:~ : Test concurrency :) declare %rest:path("/test") %rest:POST("{$body}") function rs:test($body as node()) { <a>{$body}{fold-left(1 to 10000000, 0, function($acc, $next){ $acc + math:sqrt($next)})}</a> }; [3] 17:48:02.782 127.0.0.1:48262 admin REQUEST [POST] http://localhost:8081/rate-selector/test 17:48:02.783 127.0.0.1:48252 admin REQUEST [POST] http://localhost:8081/rate-selector/test 17:48:02.783 127.0.0.1:48258 admin REQUEST [POST] http://localhost:8081/rate-selector/test 17:48:02.784 127.0.0.1:48260 admin REQUEST [POST] http://localhost:8081/rate-selector/test 17:48:02.784 127.0.0.1:48254 admin REQUEST [POST] http://localhost:8081/rate-selector/test 17:48:04.428 127.0.0.1:48262 admin 200 1646.03 ms 17:48:04.509 127.0.0.1:48260 admin 200 1725.45 ms 17:48:04.710 127.0.0.1:48254 admin 200 1926.27 ms 17:48:04.772 127.0.0.1:48258 admin 200 1988.41 ms 17:48:04.904 127.0.0.1:48252 admin 200 2120.33 ms

2 1

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

BaseX-Talk