BaseX-Talk August 2015

basex-talk@mailman.uni-konstanz.de

35 participants
51 discussions

Re: [basex-talk] Performance and heavy load
by Martín Ferrari 28 Aug '15

28 Aug '15

Sorry, I realized I hadn't post this to the mailing list. Hi Martin, I've just finished my test. It was a million resources in 5:31:20 hours, average of around 50 resources per second, I don't see any errors int the log. I wasn't able to get more speed as my client VM can't handle it. The only drawback I see is that when I get the list of dbs through the rest interface, the number of resources for my db is not updated until the sessions are closed. I can get the resources with no problem though. This test is perhaps similar to yours. This is what I did: static void DoInserts(string db) { const int numThreads = 10; for (int i = 0; i < threads.Length; ++i) { BaseXClient.Session baseXClientSession = new BaseXClient.Session(server, 1984, "admin", "admin"); baseXClientSession.Execute("open " + db); threads[i] = new System.Threading.Thread(Write); threads[i].Start(baseXClientSession); } } static void Write(object param) { BaseXClient.Session baseXClientSession = (BaseXClient.Session)param; const int times = 100000; for (int i = 0; i < times; ++i) { int tempId = System.Threading.Interlocked.Increment(ref id); System.IO.MemoryStream ms = new System.IO.MemoryStream(System.Text.Encoding.UTF8.GetBytes(AddMetadata(fileContents, DateTime.Now))); baseXClientSession.Replace("prueba" + tempId.ToString("D8") + ".xml", ms); } } Cheers, Martín.From: ferrari_martin(a)hotmail.com To: mar(a)centrum.cz Subject: RE: [basex-talk] Performance and heavy load Date: Thu, 20 Aug 2015 15:35:43 +0000 I haven't tried 250 thousand resources in a row. I just started a 1 million resources upload and will check. Maybe for a quick test you can just close sessions that have been opened for a couple minutes? Have you seen a noticeable difference in performance by pooling instead of opening/closing? I have, but I'm curious as it might depend on some factors (OS, network latency, processor speed maybe, etc). I'll let you know how my upload goes. Cheers, Martín. > Date: Thu, 20 Aug 2015 02:11:39 +0200 > To: basex-talk(a)mailman.uni-konstanz.de > CC: ferrari_martin(a)hotmail.com > Subject: Re: [basex-talk] Performance and heavy load > From: mar(a)centrum.cz > > Hi, thanks for a quick answer. > > I have been doing something simillar -- only each thread had its own > session (so no need to ask if it is in use) which got closed once the > thread had been done. Multiple threads producing data (reading SQL > database and filesystem producing XML) and multiple threads consuming data > (ie. storing into a BaseX database). > > Monitoring the BaseX server JVM with JVisualVM showed plenty of live > threads. Once it peeked with 600 or so live threads, I started to get > SIGPIPE errors (ie. lost connections) and BaseX server has started to slow > down. This way I was able to import about 250 thousand resources with some > random errors, than it got much worse. > > Once I started to create and close the connection for each operation > (simple Add()), everything has been working fine and I am able to import > all my resources, but with slight performance penalty. > > I have about 650 thousand resources with various sizes 2k-700k each. > > I may try to use your approach, at least just to verify that the BaseX > server behaves the same way. > > Thanks again, > Martin. > > On Wed, Aug 19, 2015 at 03:04:22PM +0000, Martín Ferrari wrote: > > Hi Martin, I'm not familiar with the Java client, I believe there's one that connects to BaseX directly without using the network?. I'm using the C# client found at https://github.com/BaseXdb/basex/blob/master/basex-api/src/main/c%23/BaseXC…. This C# client connects to the server using tcp connections. > > What I do is implement a pool of sessions. So, if a thread asks for a session and there's one already in the pool and not being currently in use, the thread gets that session, which will be marked as in use. If there's no available session, a new one is created and returned. Periodically, sessions that have been inactive for a certain amount of time are closed. This way, sending 10000 resources required only around 13 actual sessions (and corresponding tcp connections) in my tests. > > I've inserted 100000 10k resources at around 60 resources per second (this was all one client was able to handle, BaseX server was able to handle more than that) with no issues. > > I only need this as we have a huge live flow, otherwise I wouldn't have bothered :). > > > > I'm not sure if it helps, but this is my code for getting a session from the pool (I've added timeout to the BaseXClient.cs code). The whole session pool file is 380 lines, I can send it to you if you want. > > > > > > public SessionEntry GetSession(string password, int timeout) { SessionEntry sessionEntry = null; lock(sessionList) { foreach(SessionEntry se in sessionList) { if (se.InUse == false) { sessionEntry = se; sessionEntry.InUse = true; break; } } } > > if (sessionEntry == null) { sessionEntry = new SessionEntry(); sessionEntry.BaseXSession = new BaseXClient.Session(server, port, userName, password, timeout); if (dbName != null) { try { sessionEntry.BaseXSession.Execute("open " + dbName); } catch (Exception) { try { sessionEntry.BaseXSession.Close(); } catch (Exception) { } throw; } } sessionEntry.InUse = true; lock (sessionList) { sessionList.Add(sessionEntry); } } else { sessionEntry.BaseXSession.Timeout = timeout; } > > return sessionEntry; } > > Cheers, Martín. > > > > > Date: Wed, 19 Aug 2015 13:41:34 +0200 > > > To: basex-talk(a)mailman.uni-konstanz.de > > > From: mar(a)centrum.cz > > > Subject: Re: [basex-talk] Performance and heavy load > > > > > > Hi, I would like to know more about "keep the session opened" as you state > > > it -- I am using Java/Groovy client populating a large database (over half > > > a million resources) and if I keep the session opened, so it could be > > > reused within the thread, after a while it starts to cause problems. The > > > only solution I was able to come up with was to close each connection after > > > I add/replace a resource and open a new one. Than it behaves correctly. > > > > > > JVM running the BaseX server is keeping threads alive somehow not releasing > > > the resources properly (I have been monitoring the JVM through JVisualVM) > > > -- I stil plan to debug it a little, but I had no chance. > > > > > > Performance is quite important, so I would like to know more about your > > > solution, could you tell me more about your code? > > > > > > Regards, > > > Martin > >

1 0

Decode a RESTXQ path
by Kristian Kankainen 28 Aug '15

28 Aug '15

Hello! I have a simplistic search path as %rest:GET %rest:path("search/{$query}") for an application. It all works fine with ASCII query string, but I can't figure out how to decode the string when It contains utf-8 characters. Best wishes Kristian K

2 3

non-ascii collection names over WebDAV?
by Andreas Jung 25 Aug '15

25 Aug '15

Hi there, does BaseX support the creation of collections containing non-ascii (uft8-encoded) characters? In my case we have a functionality to import a ZIP file into BaseX or Exist-DB over WebDAV. File names and directory names must be preserved. BaseX seems to normalize names to ascii. Any way to change this? Andreas

2 6

Installation of RESTXQ scripts over HTTP API?
by Andreas Jung 25 Aug '15

25 Aug '15

Hi there, I know that .xqm script are installed directly on the filesystem (compared to Exist where the scripts are stored directly within the database). Is there some way to install RESTXQ through some (HTTP) API on the filesystem? Background of my question: I want to deploy RESTXQ script to a pre-installed BaseX server where I not necessarily have access to over HTTP. In Exist-DB I can directly upload the scripts over WebDAV (which is very convenient) ...now looking into a similar option in order for BaseX in order keep the deployment of applications as easy as possible ("one click" installation). Andreas

2 4

C# Client for BaseX 8.2.3
by Martín Ferrari 22 Aug '15

22 Aug '15

Hi, I'm using the BaseXClient.cs C# client found here: https://github.com/BaseXdb/basex/releases/tag/8.2.3 I wonder if it's the latest, as it's missing the nonce section in the Session constructor. It seems to work perfectly by just adding it. Thanks! Martín.

2 3

Re: [basex-talk] Looks like...
by Philipp Steiner 21 Aug '15

21 Aug '15

Hi Scoobs, unless you already know a lot about XML, XPath & XQuery (and just didn't use java much so far), i think BaseX might be a little overkill for your use-case. I would recommend an App like Gnome Contacts (allows creation of custom fields for grad dates, courses and freeform notes) for your contacts database, this would be more "fast-and-dirty" and wouldn't require you to "learn new tricks"... Cheers, Philipp On 21.08.2015 08:35, basex-talk-request(a)mailman.uni-konstanz.de wrote: > Message: 5 > Date: Thu, 20 Aug 2015 19:03:49 -0300 > From: Scoobbs<scoobbs(a)gmail.com> > To:basex-talk@mailman.uni-konstanz.de > Subject: [basex-talk] Looks like... > Message-ID: > <CALKe61jhOuAKV9-jsYO-v6DHNSyjG7SgQdzHLLkMt7481ysQ3w(a)mail.gmail.com> > Content-Type: text/plain; charset="utf-8" > > *?Looks like no one cares much for Ubuntu. I have version 7.7.2 installed, > through the Ubuntu Software Center. It was just a matter of cliking here > and there(actually no more than two clicks) and there I was. It even looks > like BaseX realy works!!!??Well, now that I have decided to try to use it, > there came a message saying a new version is available, namely > 8.2.3.Alright, I agreed I should update it. But then I was taken to a site > where something should be downloaded, in order for the installation to > happen. A 'file' or 'container' with a .jar extension should be it, I > thought. But, then again, there seemed to be no executable(or, easily > executable) file/module/app, whatever, was found. So I concluded I am not > as clever as most of the people building BaseX are. And, I ask:- Why is > there not an actually easy way to install the application? - Is it so damn > hard to build a thing like that? - Why is it so simple to implement > applications included in the cited Ubuntu Software Center, for simple > minded people like me, who can use effective tools like BaseX seems to be > but are not capable of understanding(much less to use, and retain in > memory, tens of cryptic commands???Anyway, my intent is actually to ask > someone(some good soul?) to update the Software Center so that one( I > suppose many...) can fast-and-dirtly(sorry... this is just a way to show my > frustration) use the tool.Thanks for any attention given to this request. > And, if someone does anything like what I asked above, please just tell me > where and what I must do. No need for replies nor attempts to teach me how > to do it(I inclusively think maybe it is not that difficult to do as it is > today, once one knows how to, but I am actually too old to learn new > tricks...).??Cordially, ?Scoobbs?(who just thought he could build a simple > database containing names, addresses, etc. about friends and > family...)??**?======?==========================================================================================??====?=?PS: > ** well, maybe, with some links to other variable attibutes of persons, > like, courses attended, graduation data etc. etc.)?* >

1 0

Re: [basex-talk] Looks like... (Scoobbs)
by cmarchand＠oxiane.com 21 Aug '15

21 Aug '15

Hi Scoobbs, you are perfectly right, the basex available on ubuntu repository is quite old. But, your mail smells (bad) ! Basex team is responsible on developing basex, and not on publishing packages for all OS, as explained by Dirk Kirsten. You can't fight again BaseX team if the actual packaging do not meet your requirements. BaseX distribute a package that can be fully installed on a complex environment in a very easy way - very easy. You just have to read the documentation to install BaseX on your Ubuntu server. For me, to install 8.2.3, it took exactly 12 minutes to make basex to run perfectly. If you don't understand the documentation, and still have problems to install, all list-users will be happy to help you. You could also think to a bit of sponsoring, it could help to produce a package for Ubuntu, during august, which is vacation-time for the whole north hemisphere. Please, be a little more respectful of other's work. Best regards, Christophe - which is not part of BaseX team. Le 2015-08-21 08:35, basex-talk-request(a)mailman.uni-konstanz.de a écrit : > *?Looks like no one cares much for Ubuntu. I have version 7.7.2 installed, > through the Ubuntu Software Center. It was just a matter of cliking here > and there(actually no more than two clicks) and there I was. It even looks > like BaseX realy works!!!??Well, now that I have decided to try to use it, > there came a message saying a new version is available, namely > 8.2.3.Alright, I agreed I should update it. But then I was taken to a site > where something should be downloaded, in order for the installation to > happen. A 'file' or 'container' with a .jar extension should be it, I > thought. But, then again, there seemed to be no executable(or, easily > executable) file/module/app, whatever, was found. So I concluded I am not > as clever as most of the people building BaseX are. And, I ask:- Why is > there not an actually easy way to install the application? - Is it so damn > hard to build a thing like that? - Why is it so simple to implement > applications included in the cited Ubuntu Software Center, for simple > minded people like me, who can use effective tools like BaseX seems to be > but are not capable of understanding(much less to use, and retain in > memory, tens of cryptic commands???Anyway, my intent is actually to ask > someone(some good soul?) to update the Software Center so that one( I > suppose many...) can fast-and-dirtly(sorry... this is just a way to show my > frustration) use the tool.Thanks for any attention given to this request. > And, if someone does anything like what I asked above, please just tell me > where and what I must do. No need for replies nor attempts to teach me how > to do it(I inclusively think maybe it is not that difficult to do as it is > today, once one knows how to, but I am actually too old to learn new > tricks...).??Cordially, ?Scoobbs?(who just thought he could build a simple > database containing names, addresses, etc. about friends and > family...)??**?======?==========================================================================================??====?=?PS: > ** well, maybe, with some links to other variable attibutes of persons, > like, courses attended, graduation data etc. etc.)?*

1 0

Create XQuery map from from Java
by Andy Bunce 21 Aug '15

21 Aug '15

Hi, Is there a simple example of how to create write a Java function that returns an XQuery map anywhere? /Andy

2 2

Looks like...
by Scoobbs 21 Aug '15

21 Aug '15

*Looks like no one cares much for Ubuntu. I have version 7.7.2 installed, through the Ubuntu Software Center. It was just a matter of cliking here and there(actually no more than two clicks) and there I was. It even looks like BaseX realy works!!!Well, now that I have decided to try to use it, there came a message saying a new version is available, namely 8.2.3.Alright, I agreed I should update it. But then I was taken to a site where something should be downloaded, in order for the installation to happen. A 'file' or 'container' with a .jar extension should be it, I thought. But, then again, there seemed to be no executable(or, easily executable) file/module/app, whatever, was found. So I concluded I am not as clever as most of the people building BaseX are. And, I ask:- Why is there not an actually easy way to install the application? - Is it so damn hard to build a thing like that? - Why is it so simple to implement applications included in the cited Ubuntu Software Center, for simple minded people like me, who can use effective tools like BaseX seems to be but are not capable of understanding(much less to use, and retain in memory, tens of cryptic commands?Anyway, my intent is actually to ask someone(some good soul?) to update the Software Center so that one( I suppose many...) can fast-and-dirtly(sorry... this is just a way to show my frustration) use the tool.Thanks for any attention given to this request. And, if someone does anything like what I asked above, please just tell me where and what I must do. No need for replies nor attempts to teach me how to do it(I inclusively think maybe it is not that difficult to do as it is today, once one knows how to, but I am actually too old to learn new tricks...).Cordially, Scoobbs(who just thought he could build a simple database containing names, addresses, etc. about friends and family...)**=====================================================================================================PS: ** well, maybe, with some links to other variable attibutes of persons, like, courses attended, graduation data etc. etc.)*

2 1

Same XQuery faster 2nd+ time
by Philipp Steiner 21 Aug '15

21 Aug '15

Hi, i'm a student learning XPath/XQuery/BaseX and currently experiment with XQueries on large xml files in BaseX, hope i didn't miss something obvious :) Test: 1st FLOWR query on collection: ~700ms 2nd+ identical query same collection: just ~200ms ! ? My XML-files are OPTIMIZE'd on import, but the created indexes would influence the 1st query already. This is a great speed bump, but hard on tests that assume a first query on a recently imported new collection. So i wondered how this behavior could be explained & maybe turned off for testing purposes? Cheers, Philipp

2 2

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

BaseX-Talk August 2015