BaseX-Talk

basex-talk@mailman.uni-konstanz.de

6 participants
5269 discussions

Finding document based on filename
by Mansi Sheth 01 Sep '15

01 Sep '15

Hello, I would be having 100s of databases, with each database having 100 XML documents. I want to devise an algorithm, where given a part of XML file name, i want to know which database(s) contains it, or null if document is not currently present in any database. Based on that, add current document into the database. This is to always maintain latest version of a document in DB, and remove the older version, while adding newer version. So far, only way I could come up with is: for $db in all-databases: open $db $fileNames = list $db for eachFileName in $fileNames: if $eachFileName.contains(sub-xml filename): add to ret-list-db return ret-list-db Above algorithm, seems highly inefficient, Is there any indexing, which can be done ? Do you suggest, for each document insert, I should maintain a separate XML document, which lists each file inserted etc. Once, i get hold of above list of db, I would be eventually deleting that file and inserting a latest version of that file(which would have same sub-xml file name). So, constant updating of this external document also seems painful (Map be ?). Also, would it be faster, using XQUERY script files, thru java code, or using Java API for such operations ? How do you all deal with such operations ? - Mansi

4 5

Selective indexes
by Martín Ferrari 01 Sep '15

01 Sep '15

Hi Dirk, I didn't want to hijack the other thread, so I started a new one :) I've read https://github.com/BaseXdb/basex/issues/59 and I think it's great!. I have a slightly different request though that I was holding back because I didn't want to sound abusive, but now that you brought it up... I might be mistaken as I haven't really debugged the code, but I believe that indexing could maybe take so long (for our use case anyway) because it indexes all the values in the xml documents. So I wonder whether it would be beneficial for the indexing speed to be able to state which specific fields to index instead of all of them. This would allow for having fast regularly used queries, without impacting insertions that much. I can explain our use case quickly: we store XML files that are interchanged with third parties. We have lots of them all the time, so insertion speed is critical. Query speed on the other hand is not critical, though obviously having fast queries is a lot better than having slow ones. But it becomes more important as the queries lock the db for writing too, aside from feeling "if I just could index this one field", hehe. If we could just index the fields that we run most of our queries against, it would be perfect. Right now, I think we can get away with organizing resources inside directories and that would be enough as that seems to work perfectly (we'll perform a couple more tests with high volume), but I thought it would be really cool if we could have a similar insertion speed to what's achieved without indexing, by just indexing some fields (again, *guessing* that the cause of the decrease in speed is due to indexing all fields in the documents, which may or may not be the case). Thanks! Martín.

2 2

Re: [basex-talk] BaseX and MySQL: Driver can't be initialized
by Ulrike Schaper - SideStep Business Solutions GmbH 01 Sep '15

01 Sep '15

Thanks everybody, starting the GUI from command line did the trick. Best regards, Ulrike.

1 0

Re: [basex-talk] BaseX and MySQL: Driver can't be initialized
by Ulrike Schaper - SideStep Business Solutions GmbH 01 Sep '15

01 Sep '15

Thanks everybody, starting the GUI from command line did the trick. Best regards, Ulrike.

1 0

BaseX and MySQL: Driver can't be initialized
by Ulrike Schaper - SideStep Business Solutions GmbH 31 Aug '15

31 Aug '15

Hi everybody, I know, that this issue has been discussed already, but I still have problems concerning BaseX and MySQL: Of course I read the documentation: I downloaded the Connector/J from the MySQL website and put the .jar file (mysql-connector-java-5.1.36-bin.jar) into the basex/lib folder. I closed and restarted the GUI and tried the following: import module namespace sql="http://basex.org/modules/sql"; sql:init("com.mysql.jdbc.Driver") I receive the following error message: [bxerr:BXSQ0007] Could not initialize supplied driver: 'com.mysql.jdbc.Driver' Is there something obvious I'm missing? Best regards, Ulrike.

2 1

Re: [basex-talk] Performance and heavy load
by Martín Ferrari 28 Aug '15

28 Aug '15

Sorry, I realized I hadn't post this to the mailing list. Hi Martin, I've just finished my test. It was a million resources in 5:31:20 hours, average of around 50 resources per second, I don't see any errors int the log. I wasn't able to get more speed as my client VM can't handle it. The only drawback I see is that when I get the list of dbs through the rest interface, the number of resources for my db is not updated until the sessions are closed. I can get the resources with no problem though. This test is perhaps similar to yours. This is what I did: static void DoInserts(string db) { const int numThreads = 10; for (int i = 0; i < threads.Length; ++i) { BaseXClient.Session baseXClientSession = new BaseXClient.Session(server, 1984, "admin", "admin"); baseXClientSession.Execute("open " + db); threads[i] = new System.Threading.Thread(Write); threads[i].Start(baseXClientSession); } } static void Write(object param) { BaseXClient.Session baseXClientSession = (BaseXClient.Session)param; const int times = 100000; for (int i = 0; i < times; ++i) { int tempId = System.Threading.Interlocked.Increment(ref id); System.IO.MemoryStream ms = new System.IO.MemoryStream(System.Text.Encoding.UTF8.GetBytes(AddMetadata(fileContents, DateTime.Now))); baseXClientSession.Replace("prueba" + tempId.ToString("D8") + ".xml", ms); } } Cheers, Martín.From: ferrari_martin(a)hotmail.com To: mar(a)centrum.cz Subject: RE: [basex-talk] Performance and heavy load Date: Thu, 20 Aug 2015 15:35:43 +0000 I haven't tried 250 thousand resources in a row. I just started a 1 million resources upload and will check. Maybe for a quick test you can just close sessions that have been opened for a couple minutes? Have you seen a noticeable difference in performance by pooling instead of opening/closing? I have, but I'm curious as it might depend on some factors (OS, network latency, processor speed maybe, etc). I'll let you know how my upload goes. Cheers, Martín. > Date: Thu, 20 Aug 2015 02:11:39 +0200 > To: basex-talk(a)mailman.uni-konstanz.de > CC: ferrari_martin(a)hotmail.com > Subject: Re: [basex-talk] Performance and heavy load > From: mar(a)centrum.cz > > Hi, thanks for a quick answer. > > I have been doing something simillar -- only each thread had its own > session (so no need to ask if it is in use) which got closed once the > thread had been done. Multiple threads producing data (reading SQL > database and filesystem producing XML) and multiple threads consuming data > (ie. storing into a BaseX database). > > Monitoring the BaseX server JVM with JVisualVM showed plenty of live > threads. Once it peeked with 600 or so live threads, I started to get > SIGPIPE errors (ie. lost connections) and BaseX server has started to slow > down. This way I was able to import about 250 thousand resources with some > random errors, than it got much worse. > > Once I started to create and close the connection for each operation > (simple Add()), everything has been working fine and I am able to import > all my resources, but with slight performance penalty. > > I have about 650 thousand resources with various sizes 2k-700k each. > > I may try to use your approach, at least just to verify that the BaseX > server behaves the same way. > > Thanks again, > Martin. > > On Wed, Aug 19, 2015 at 03:04:22PM +0000, Martín Ferrari wrote: > > Hi Martin, I'm not familiar with the Java client, I believe there's one that connects to BaseX directly without using the network?. I'm using the C# client found at https://github.com/BaseXdb/basex/blob/master/basex-api/src/main/c%23/BaseXC…. This C# client connects to the server using tcp connections. > > What I do is implement a pool of sessions. So, if a thread asks for a session and there's one already in the pool and not being currently in use, the thread gets that session, which will be marked as in use. If there's no available session, a new one is created and returned. Periodically, sessions that have been inactive for a certain amount of time are closed. This way, sending 10000 resources required only around 13 actual sessions (and corresponding tcp connections) in my tests. > > I've inserted 100000 10k resources at around 60 resources per second (this was all one client was able to handle, BaseX server was able to handle more than that) with no issues. > > I only need this as we have a huge live flow, otherwise I wouldn't have bothered :). > > > > I'm not sure if it helps, but this is my code for getting a session from the pool (I've added timeout to the BaseXClient.cs code). The whole session pool file is 380 lines, I can send it to you if you want. > > > > > > public SessionEntry GetSession(string password, int timeout) { SessionEntry sessionEntry = null; lock(sessionList) { foreach(SessionEntry se in sessionList) { if (se.InUse == false) { sessionEntry = se; sessionEntry.InUse = true; break; } } } > > if (sessionEntry == null) { sessionEntry = new SessionEntry(); sessionEntry.BaseXSession = new BaseXClient.Session(server, port, userName, password, timeout); if (dbName != null) { try { sessionEntry.BaseXSession.Execute("open " + dbName); } catch (Exception) { try { sessionEntry.BaseXSession.Close(); } catch (Exception) { } throw; } } sessionEntry.InUse = true; lock (sessionList) { sessionList.Add(sessionEntry); } } else { sessionEntry.BaseXSession.Timeout = timeout; } > > return sessionEntry; } > > Cheers, Martín. > > > > > Date: Wed, 19 Aug 2015 13:41:34 +0200 > > > To: basex-talk(a)mailman.uni-konstanz.de > > > From: mar(a)centrum.cz > > > Subject: Re: [basex-talk] Performance and heavy load > > > > > > Hi, I would like to know more about "keep the session opened" as you state > > > it -- I am using Java/Groovy client populating a large database (over half > > > a million resources) and if I keep the session opened, so it could be > > > reused within the thread, after a while it starts to cause problems. The > > > only solution I was able to come up with was to close each connection after > > > I add/replace a resource and open a new one. Than it behaves correctly. > > > > > > JVM running the BaseX server is keeping threads alive somehow not releasing > > > the resources properly (I have been monitoring the JVM through JVisualVM) > > > -- I stil plan to debug it a little, but I had no chance. > > > > > > Performance is quite important, so I would like to know more about your > > > solution, could you tell me more about your code? > > > > > > Regards, > > > Martin > >

1 0

Decode a RESTXQ path
by Kristian Kankainen 28 Aug '15

28 Aug '15

Hello! I have a simplistic search path as %rest:GET %rest:path("search/{$query}") for an application. It all works fine with ASCII query string, but I can't figure out how to decode the string when It contains utf-8 characters. Best wishes Kristian K

2 3

non-ascii collection names over WebDAV?
by Andreas Jung 25 Aug '15

25 Aug '15

Hi there, does BaseX support the creation of collections containing non-ascii (uft8-encoded) characters? In my case we have a functionality to import a ZIP file into BaseX or Exist-DB over WebDAV. File names and directory names must be preserved. BaseX seems to normalize names to ascii. Any way to change this? Andreas

2 6

Installation of RESTXQ scripts over HTTP API?
by Andreas Jung 25 Aug '15

25 Aug '15

Hi there, I know that .xqm script are installed directly on the filesystem (compared to Exist where the scripts are stored directly within the database). Is there some way to install RESTXQ through some (HTTP) API on the filesystem? Background of my question: I want to deploy RESTXQ script to a pre-installed BaseX server where I not necessarily have access to over HTTP. In Exist-DB I can directly upload the scripts over WebDAV (which is very convenient) ...now looking into a similar option in order for BaseX in order keep the deployment of applications as easy as possible ("one click" installation). Andreas

2 4

C# Client for BaseX 8.2.3
by Martín Ferrari 22 Aug '15

22 Aug '15

Hi, I'm using the BaseXClient.cs C# client found here: https://github.com/BaseXdb/basex/releases/tag/8.2.3 I wonder if it's the latest, as it's missing the nonce section in the Session constructor. It seems to work perfectly by just adding it. Thanks! Martín.

2 3

Jump to page:

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

BaseX-Talk