BaseX-Talk

basex-talk@mailman.uni-konstanz.de

1 participants
5252 discussions

Migrating ~ 3M record db from BaseX to PostgreSQL results in OOM
by first name last name 08 Oct '19

08 Oct '19

Hello, This is essentially part2 of trying to index large amounts of web data. To summarize what happened before: The initial discussion started here [1], Christian suggested some options, I dove into each of them, I realized that doing this on a low-memory system is harder than I initially thought. At Christian's suggestion, I tried to split the big db into smaller dbs and came up with a rudimentary sharding mechanism [3]. All attempts to full-text 30GB of data in BaseX, for me, resulted in OOM (do take into consideration that I only have 3.1GB of memory to allocate for BaseX). Where to? I decided to look more into what Christian said in [2] about option 2, and to pick the exact values that I want, and to transfer them to PostgreSQL (after transferring, a GiST index would have to be built there, to allow full-text searches; PostgreSQL is picked because it uses an in-memory buffer for all large operations, and several files on disk, and if it needs to combine results that exceed the available memory, it goes to disk, but at all times it never exceeds the given amount of memory). Variant 1 (see attached script pg-import.sh) All good. So, I basically started writing XQuery that would do the following: - Open up a JDBC connection to PostgreSQL - Get me all text content from each thread page of the forum, and the db it belonged to - Create a prepared statement for one such thread page, populate the prepared statement, and execute it This ended up in OOM after around 250k records. So just to be clear, 250k lines were rows in PostgreSQL, which is nice but eventually it ended up in OOM. (Perhaps it has to do with how the GC works in Java .. I don't know) Variant 2 (see attached script pg-import2.sh) I did something similar to the above: - Open up a JDBC connection to PostgreSQL - Get all posts and for each post get the author, the date, the message content, the post id, the BaseX db name (cause we're going over all shards, and each shard is a BaseX db) - Create a prepared statement for each post with the data mentioned above, and execute it This also ended up in OOM after around 340k records (my approximation would be that there were around 3M posts in the data). To summarize, I'm tempted to believe that there might be a leak in the BaseX implementation of XQuery. I will provide in the following, the relevant versions of the software used: - BaseX 9.2.4 - java version "1.8.0_151" Java(TM) SE Runtime Environment (build 1.8.0_151-b12) Java HotSpot(TM) 64-Bit Server VM (build 25.151-b12, mixed mode) - the JVM memory param value was -Xmx3100m I would be interested to know your thoughts Thanks, Stefan [1] https://mailman.uni-konstanz.de/pipermail/basex-talk/2019-September/014715.… [2] https://mailman.uni-konstanz.de/pipermail/basex-talk/2019-October/014727.ht… [3] https://mailman.uni-konstanz.de/pipermail/basex-talk/2019-October/014729.ht…

2 4

Want event, not polling....
by Peter Villadsen 07 Oct '19

07 Oct '19

Hello, I am using a client written in C#. For the sake of this discussion, imagine the app being discussed as an app just like the Java GUI app that ships with BaseX: You can basically author Xquery snippets and send them for execution by the server. We regularly have queries that tend to run for a long time, and it would be an advantage for the user to be able to cancel these long running jobs, just like he can in the GUI app. However, I am not sure how to do this; I can imagine two solutions: 1. I see that jobs that are submitted normally to the server end up as jobs, just like the ones that are submitted with jobs:eval(query): I can stop that job with jobs:stop(jobId) on another session if the jobId is known. You would need a separate session that fetches the running jobs and gets the current one (recognizing it by creating a cookie in a comment in the query) with jobs:list-details(), then getting the jobId and then calling jobs:stop(jobId). It is possible I guess, but very heavy handed. 2. The other option is just as bad: submit the query through jobs:eval(query), getting the jobId and then poll for it to finish; more polling which will not scale with many users and will not provide the quick feedback we want. I see some traces of an eventing mechanism that was deprecated (or at least no longer included in the client). That is what I would ideally have (ignoring bindings etc): Query q = new Query("1 to 10" , new Guid()); q.Ended += MyTerminationHandler(); // C# syntax for adding an event handler for termination q.Interrupted += MyInterruptedHandler(); await q.execute(); // Called when the query has terminated with a result void MyTerminationHandler(Guid g, string result) { ... update UI ... } // Called when the query has terminated with a result void MyInterruptedHandler (Guid g) { ... update UI ... } How do I do this without the polling, being able to pass massive results and queries back and forth? Best Regards Peter Villadsen Principal Architect Microsoft Business Applications Group

1 0

how to specify a database in a.xq FLOWR query?
by thufir 07 Oct '19

07 Oct '19

how do I specify a database for a .xq file to use? a simple FLOWR: thufir@dur:~/basex/w3$ thufir@dur:~/basex/w3$ basex titles.xq [warning] /usr/bin/basex: Unable to locate /usr/share/java/jing.jar in /usr/share/java <title lang="en">Learning XML</title> <title lang="en">XQuery Kick Start</title>thufir@dur:~/basex/w3$ thufir@dur:~/basex/w3$ thufir@dur:~/basex/w3$ cat titles.xq for $x in doc("books.xml")/bookstore/book where $x/price>30 order by $x/title return $x/title thufir@dur:~/basex/w3$ (from w3schools) thanks, Thufir

3 4

// versus /*/
by thufir 07 Oct '19

07 Oct '19

Do these have the same meaning? Might there be a subtle distinction, or might they be read differently but functionally identical? > > xquery /*/book[@id="bk112"] <book id="bk112"> <author>Galos, Mike</author> <title>Visual Studio 7: A Comprehensive Guide</title> <genre>Computer</genre> <price>49.95</price> <publish_date>2001-04-16</publish_date> <description>Microsoft Visual Studio 7 is explored in depth, looking at how Visual Basic, Visual C++, C#, and ASP+ are integrated into a comprehensive development environment.</description> </book> Query executed in 1.22 ms. > > xquery //book[@id="bk112"]/title <title>Visual Studio 7: A Comprehensive Guide</title> Query executed in 1.52 ms. > > xquery /*/book[@id="bk112"]/title <title>Visual Studio 7: A Comprehensive Guide</title> Query executed in 1.67 ms. > They're equally efficient, at least as used above? thanks, Thufir

2 2

Unidentified java processes keeping locks on DBs (BaseX 9.2.3)
by France Baril 07 Oct '19

07 Oct '19

Hi, We have been getting process issues on Windows installations for a while now. Has anyone had fantom locks (non-existent processes locking indices) and if so, can you point to clues that could help resolve the issue? [image: image.png] This leads to batch processes failure and DB corruption so we are eager even for the tiniest clues. Thank you -- France Baril Architecte documentaire / Documentation architect france.baril(a)architextus.com

1 0

get list of databases
by thufir 06 Oct '19

06 Oct '19

Hi, pardon the silly question, but how do I get a list of databases exactly? I'm looking at: https://github.com/BaseXdb/basex/blob/master/basex-examples/src/main/java/o… and want output like: > list Name Resources Size Input Path ---------------------------------- books 0 4570 1 database(s). > but using: http://docs.basex.org/javadoc/org/basex/core/cmd/List.html thanks, Thufir

2 2

basex OOM on 30GB database upon running /dba/db-optimize/
by first name last name 06 Oct '19

06 Oct '19

Hi, Let's say there's a 30GB dataset [3] containing most threads/posts from [1]. After importing all of it, when I try to run /dba/db-optimize/ on it (which must have some corresponding command) I get the OOM error in the stacktrace attached. I am using -Xmx2g so BaseX is limited to 2GB of memory (the machine I'm running this on doesn't have a lot of memory). I was looking at [2] for some estimates of peak memory usage for this "db-optimize" operation, but couldn't find any. Actually it would be nice to know peak memory usage because.. of course, for any database (including BaseX) a common operation is to do server sizing, to know what kind of server would be needed. In this case, it seems like 2GB memory is enough to import 340k documents, weighing in at 30GB total, but it's not enough to run "dba-optimize". Is there any info about peak memory usage on [2] ? And are there guidelines for large-scale collection imports like I'm trying to do? Thanks, Stefan [1] https://www.linuxquestions.org/ [2] http://docs.basex.org/wiki/Statistics [3] https://drive.google.com/open?id=1lTEGA4JqlhVf1JsMQbloNGC-tfNkeQt2

3 13

Does BaseX used as a command line XQuery processor build a full tree of an input document passed with -i option?
by Martin Honnen 03 Oct '19

03 Oct '19

If I use BaseX (9.2.4) simply as an XQuery 3.1 processor from the command line with e.g. basex.bat -i input.xml query.xq does BaseX then first always parse the input.xml into an XDM tree, meaning if I use huge input documents that way I can run easily run out of memory? Or does that depend on the type of query or some other settings? I was wondering whether a "tumbling window" based split algorithm like declare namespace output = "http://www.w3.org/2010/xslt-xquery-serialization"; declare option output:method 'xml'; declare option output:indent 'yes'; declare variable $chunk-size as xs:integer external := 500; for tumbling window $chunk in /*/* start at $sp when $sp mod $chunk-size = 1 count $p return put(document { element { node-name(head($chunk)/..) } { $chunk } }, 'xquery-split-result-' || $p || '.xml') would run/work without memory problems for huge inputs.

2 1

Re: [basex-talk] HTTPServer + gzip compression
by Stefan Koch 03 Oct '19

03 Oct '19

2 1

Re: [basex-talk] Connection timeout
by Marie Starck 01 Oct '19

01 Oct '19

I am using an ssl certificate and have a keystore on my production Basex instance. Are there extra steps needed to make an HTTPS request from a server to a basex server when those are in place? On Thu, Sep 26, 2019 at 6:14 PM Marie Starck <marie.compsci14(a)gmail.com> wrote: > Hey Christian, > > This error message is from doing it with the nodejs basex client. I get > the same connection time out when doing an http or https request on 8984 > and 8986 (non-ssl and ssl respectively). > > Thank you, > > Marie > > On Thu, Sep 26, 2019 at 5:54 PM Christian Grün <christian.gruen(a)gmail.com> > wrote: > >> Hi Marie, >> >> SOCKET ERROR: { Error: connect ETIMEDOUT 159.89.126.125:1984 >>> >> >> By default, the REST endpoints are available on port 8984. >> >> Maybe this helps, >> Christian >> >> >>

2 1

Jump to page:

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

BaseX-Talk