BaseX-Talk

basex-talk@mailman.uni-konstanz.de

1 participants
5252 discussions

Start a nNew thread

Running BaseX on Solaris
by Goetz Heller 25 Apr '15

25 Apr '15

Are there any experiences with running BaseX on Solaris? Kind regards Goetz

1 0

Fwd: IllegalMonitorStateException at org.basex.core.locks.DBLocking
by Simon Chatelain 25 Apr '15

25 Apr '15

Oups, forgot to put the mailing list in copy. Hello Christian, After some testing on my side, I didn't see the ConcurrentModificationException any more. That's the good news. However, when running a slightly modified version of the small test case you wrote from my sample application, I faced another problem. The test can run for a few seconds to almost an hour but eventually, the following exception is thrown. java.lang.IllegalMonitorStateException at java.util.concurrent.locks.ReentrantReadWriteLock$Sync.tryRelease(ReentrantReadWriteLock.java:374) at java.util.concurrent.locks.AbstractQueuedSynchronizer.release(AbstractQueuedSynchronizer.java:1260) at java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.unlock(ReentrantReadWriteLock.java:1131) at org.basex.core.locks.DBLocking.release(DBLocking.java:215) at org.basex.core.Context.unregister(Context.java:287) at org.basex.core.Command.execute(Command.java:105) at org.basex.api.client.LocalSession.execute(LocalSession.java:132) at org.basex.api.client.Session.execute(Session.java:36) at basextest.BaseXTestDBLocking$1.run(BaseXTestDBLocking.java:43) I attach my new test case (BaseXTestAdd.java), but the main modification is that in each created thread I also add documents to the collections instead of only opening it. I was also able to see that in the call to getOrCreateLock() in DBLocking#release(final Proc pr) (line 212) the lock is created while it should already be in the locks Map, but I really cannot understand how this is possible. It would mean that the lock was removed by another thread but for that the usage value must be wrong in the lockUsage map, and I cannot find any sequence of operation that could lead to such a situation. Trying to pin-point more precisely the problem, I wrote another test ( BaseXTestDBLocking.java) that calls directly the acquire and release methods of the DBLocking class. The problem seems to happen more quickly with this test. Any thoughts ? Regards Simon On Tue, Apr 14, 2015 at 3:31 PM, Simon Chatelain <schatela(a)gmail.com> wrote: > That was fast :-) . Thank you very much. I will test it right away. > > Regards > > Simon > > On Tue, Apr 14, 2015 at 3:28 PM, Christian Grün <christian.gruen(a)gmail.com > > wrote: > >> Simon, >> >> I am glad to report the concurrency bug in the user access code has >> been fixed [1,2]. BaseX 8.1.1 may already be released this week. Of >> course, please report back to me if you should encounter additional >> bugs. >> >> Thanks for testing, >> Christian >> >> [1] >> https://github.com/BaseXdb/basex/commit/b5d1716e662a300d484cbbb19d1bf23cbef… >> [2] http://files.basex.org/releases/latest >> >> >> >> On Tue, Apr 14, 2015 at 2:44 PM, Christian Grün >> <christian.gruen(a)gmail.com> wrote: >> > I have minimized your example a bit (attached), and I agree it seems >> > to be the user iterator that causes surprises. >> > >> > Looking at it.. >> > Christian >> > >> > >> > >> > On Tue, Apr 14, 2015 at 1:48 PM, Christian Grün >> > <christian.gruen(a)gmail.com> wrote: >> >> Hi Simon, >> >> >> >> Thanks for the updated code. It's still quite a lot code.. Is all of >> >> it required to reproduce the bug? E.g., does the bug only occur with >> >> the specified options etc? Could you possibly reduce it even more? >> >> Idealle, we can later rewrite it to a JUnit test once the problem is >> >> fixed (and we try to keep our tests as concise as possible). >> >> >> >> But I'll have some thoughts about your helpful assumptions in parallel. >> >> >> >> Thanks in advance, >> >> Christian >> >> >> >> >> >> On Tue, Apr 14, 2015 at 1:10 PM, Simon Chatelain <schatela(a)gmail.com> >> wrote: >> >>> Hello Christian, >> >>> >> >>> I checked again my code, but I am quite sure that each LocalSession is >> >>> created and used by only one thread. >> >>> I attach the test application code. >> >>> >> >>> Each LocalSession is used to execute several commands, but that >> shouldn't be >> >>> a problem, right ? >> >>> >> >>> I also had a look in BaseX code, especially in the >> org.basex.core.users.User >> >>> class. >> >>> >> >>> What happens in my test application is that I create several Writer >> that >> >>> will add XML document each in a specific collection, but using the >> same >> >>> user. >> >>> Each time I create a new Writer it will first create the collection >> then >> >>> grant read and write permission to this collection to the user >> "USER", and >> >>> then start adding XML document in this collection. >> >>> >> >>> >> >>> My suspicion is that the problem occurs when a new Writer is created, >> and it >> >>> is granting permission to the newly created collection while another >> Writer >> >>> (in another thread) is adding a new XML document in its own >> collection. >> >>> If, during the execution of the Grant command, the method >> User#perm(final >> >>> Perm prm, final String pattern) is called while the execution of a >> >>> concurrent Write command is at the same time executing the >> User#find(final >> >>> String db) method, then we have a ConcurrentModificationException as >> perm >> >>> method will put a new entry in locals field while the find method is >> >>> iterating on the locals entrySet of the same User, the same User >> instance >> >>> being shared by all subcontext created when opening a new >> LocalSession. >> >>> For test purposes, I synchronized the access to locals in the perm(), >> find() >> >>> and remove() methods of the class User, in that case I do not have the >> >>> ConcurrentModificationException any more. >> >>> >> >>> What do you think, is this possible ? >> >>> >> >>> Regards >> >>> >> >>> Simon >> >>> >> >>> On Mon, Apr 13, 2015 at 11:04 AM, Christian Grün < >> christian.gruen(a)gmail.com> >> >>> wrote: >> >>>> >> >>>> > But the java.util.ConcurrentModificationException >> >>>> > at org.basex.core.users.User.find >> >>>> > is still happening. >> >>>> >> >>>> I am pretty sure that any of the concurrent command calls are not >> >>>> moved to their own thread yet. >> >>>> >> >>>> But feel free to pass me on your code (ideally, a version that's as >> >>>> small as possible). >> >>>> >> >>>> Best, >> >>>> Christian >> >>>> >> >>>> >> >>>> >> >>>> > >> >>>> > What I did is basically, in the test application, replace all >> session >> >>>> > creations >> >>>> > >> >>>> > try (final Session session = new LocalSession(rootContext, ...)) { >> >>>> > ... >> >>>> > } >> >>>> > catch {...} >> >>>> > >> >>>> > by >> >>>> > >> >>>> > Thread t = new Thread(new Runnable() { public void run() { >> >>>> > try (final Session session = new LocalSession(rootContext, ...)) { >> >>>> > ... >> >>>> > } >> >>>> > catch {...} >> >>>> > >> >>>> > } >> >>>> > }); >> >>>> > >> >>>> > Not very elegant but should be enough to make sure every session is >> >>>> > executed >> >>>> > in its own Java thread. >> >>>> > >> >>>> > >> >>>> > Am I still missing something ? >> >>>> > >> >>>> > Thank you very much for your help, it is very appreciated. >> >>>> > >> >>>> > Regards >> >>>> > >> >>>> > Simon >> >>>> > >> >>>> > >> >>>> > On Tue, Apr 7, 2015 at 12:16 PM, Christian Grün >> >>>> > <christian.gruen(a)gmail.com> >> >>>> > wrote: >> >>>> >> >> >>>> >> Hi Simon, >> >>>> >> >> >>>> >> Thanks for the detailed description of your setup and your test >> code. >> >>>> >> >> >>>> >> In BaseX, every session needs to be executed in a separate Java >> >>>> >> thread. The exceptions will e.g. disappear if you pass on your >> admin >> >>>> >> session to the store() and updateMetaStore() functions (or if you >> run >> >>>> >> these operations in a separate thread). >> >>>> >> >> >>>> >> Hope this helps, >> >>>> >> Christian >> >>>> >> >> >>>> >> >> >>>> >> On Thu, Apr 2, 2015 at 1:58 PM, Simon Chatelain < >> schatela(a)gmail.com> >> >>>> >> wrote: >> >>>> >> > Hello all, >> >>>> >> > >> >>>> >> > I have the following problem. >> >>>> >> > >> >>>> >> > Using: >> >>>> >> > >> >>>> >> > BaseX 8.1.1 beta cf713e2 (20150328.133759) >> >>>> >> > Java 1.8.0_31 >> >>>> >> > Windows 7 >> >>>> >> > >> >>>> >> > The context: I have several sources of data from which I >> receive XML >> >>>> >> > documents. My application will create a collection for each of >> these >> >>>> >> > sources >> >>>> >> > and store each received document into the collection >> corresponding to >> >>>> >> > the >> >>>> >> > source, and then update some kind of metadata about the >> collection. >> >>>> >> > The >> >>>> >> > metadata of all collections are stored into a common >> collection. Each >> >>>> >> > source >> >>>> >> > is handled by a different thread. >> >>>> >> > I am using an embedded BaseX, and access to it through >> LocalSession. >> >>>> >> > >> >>>> >> > Now the problem is that after a certain amount of time (not >> constant, >> >>>> >> > could >> >>>> >> > be seconds or minutes) there is some exceptions. >> >>>> >> > Either: java.lang.IllegalMonitorStateException >> >>>> >> > at org.basex.core.locks.DBLocking.acquire >> >>>> >> > and org.basex.core.locks.DBLocking.release >> >>>> >> > >> >>>> >> > Or: java.util.ConcurrentModificationException >> >>>> >> > at org.basex.core.users.User.find >> >>>> >> > >> >>>> >> > I put the full stack trace in attachment. >> >>>> >> > >> >>>> >> > I build a test application that mimic what happens in the real >> >>>> >> > application >> >>>> >> > and it seems to exhibit the same problem. The source of this >> test >> >>>> >> > application is attached >> >>>> >> > >> >>>> >> > My question: Am I doing something wrong, or is there a bug in >> BaseX? >> >>>> >> > >> >>>> >> > Thanks a lot >> >>>> >> > >> >>>> >> > Simon >> >>>> > >> >>>> > >> >>> >> >>> >> > >

3 5

Basex xquery import module classpath lookup
by Karel Hovorka 24 Apr '15

24 Apr '15

Hi, I am trying to run xquery file on basex, but it depends on xquery module located in classpath. I have found parameter "org.basex.QUERYPATH", that configures module lookup on filesystem. Is there way to lookup modules in classpath? Thanks, Karel Hovorka, Semantico LTD

3 25

Best Forum for Generate XQuery How To Discussions?
by Eliot Kimber 23 Apr '15

23 Apr '15

What is the best forum for asking general how-to XQuery questions that are not BaseX specific? Thanks, Eliot ————— Eliot Kimber, Owner Contrext, LLC http://contrext.com

2 1

multi-language full-text indexing
by Goetz Heller 23 Apr '15

23 Apr '15

Here's another addendum: Even if multi-language full-text indexing is not going tob e implemented in the near future, it still would be a useful feature to be able to restrict full-text indexing to parts of a document, e.g. CREATE FULL-TEXT INDEX ON DATABASE XY STARTING WITH ( (path_a)/PART_A, (path_b)/ PART_B,… ) Kind regards, Goetz -----Ursprüngliche Nachricht----- Von: Christian Grün [mailto:christian.gruen@gmail.com] Gesendet: Mittwoch, 22. April 2015 11:03 An: Goetz Heller Cc: BaseX Betreff: Re: [basex-talk] multi-language full-text indexing > It is desirable to have > documents indexed by locale-specific parts, e.g. I can see that this would absolutely make sense, but it would be quite some effort to realize it. There are also various conceptul issues related to XQuery Full Text: If you don't specify the language in the query, we'd need to dynamically decide what stemmers to use for the query strings, depending on the nodes that are currently targeted. This would pretty much blow up the existing architecture. As there are so many other types of index structures that could be helpful, depending on the particular use case, we usually recommend users to create additional BaseX databases, which can then serve as indexes. This can all be done in XQuery. I remember there have been various examples for this on this mailing list (see e.g. [1,2]). Hope this helps, Christian [1] https://www.mail-archive.com/basex-talk@mailman.uni-konstanz.de/msg04837.ht… [2] https://www.mail-archive.com/basex-talk@mailman.uni-konstanz.de/msg06089.ht… > > > > CREATE FULL-TEXT INDEX ON DATABASE XY STARTING WITH ( > > (path_a)/LOCALIZED_PART_A[@LANG=$lang], > > (path_b)/LOCALIZED_PART_B[@LG=$lang],… > > ) FOR LANGUAGE $lang IN ( > > BG, > > DN, > > DE WITH STOPWORDS filepath_de WITH STEM = YES, > > EN WITH STOPWORDS filepath_en, > > FR, … > > ) [USING language_code_map] > > and then to write full-text retrieval queries with a clause such as > ‘FOR LANGUAGE BG’, for example. The index parts would be much smaller > and full-text retrieval therefore much faster. The language codes > would be mapped somehow to standard values recognized by BaseX in the > language_code_map file. > > Are there any efforts towards such a feature?

4 4

Creation of Full-Text-Index failed
by Goetz Heller 22 Apr '15

22 Apr '15

For the task at hand I need to create a database on a daily base from file packages I received. The language taken here is German, however the files contain lots of international characters as well. Usually this does not harm, and I don't know if this is the real cause of failure in this case. Actually, the database was created, but an error message occurred which was not very specific: "file xxx could not be parsed". File "xxx" was the last file of the package, and it was accessible for xQuery search. However, no full-text index was created as with the other packages. Trying to create the index directly resulted in a different message: "Improper use? . Stack Trace: java.lang.ArrayIndexOutOfBoundsException". The package can be downloaded from http://www.hellerim.de/downloads/BaseX/20150203_023.7z. This does not look like a problem with the data but rather like a bug in BaseX. If I'm wrong, however, I would prefer to get a message which points me to the problem so I can try to solve it. Kind regards, Goetz

2 2

Distributing queries to several on several processors
by Goetz Heller 22 Apr '15

22 Apr '15

So far I did not find any information on how BaseX can be advised how to use computing resources. The use case here is as follows: I get several megabytes of xml files each day, usually between 50 and 100 MB. These are organized in one database per day. Since most queries run on a daily base this works perfectly fine. However, there are situations when I need to run a query over a larger time span, say three or six months. (Note that I'm speaking of read-only queries here, not of transactions.) Of course I can do this in a loop (for $db in $db-list) but since the data in each database is completely independent from that in the other databases it would make perfect sense to parallelize the query. Is there a way to achieve this using xQuery? I'm aware of the possibility to split the sequence into several ones and run them in different threads on different connections using Java, for instance. But even then I still don't know what the server does (my queries run in a client-server configuration): will it occupy just one processor, or will it distribute the workload?

4 4

multi-language full-text indexing
by Goetz Heller 22 Apr '15

22 Apr '15

I'm working with documents destined to be consumed anywhere in the European Community. Many of them have the same tags multiple times but with a different language attribute. It does not make sense to create a full-text index for the whole of these documents therefore. It is desirable to have documents indexed by locale-specific parts, e.g. CREATE FULL-TEXT INDEX ON DATABASE XY STARTING WITH ( (path_a)/LOCALIZED_PART_A[@LANG=$lang], (path_b)/LOCALIZED_PART_B[@LG=$lang],. ) FOR LANGUAGE $lang IN ( BG, DN, DE WITH STOPWORDS filepath_de WITH STEM = YES, EN WITH STOPWORDS filepath_en, FR, . ) [USING language_code_map] and then to write full-text retrieval queries with a clause such as 'FOR LANGUAGE BG', for example. The index parts would be much smaller and full-text retrieval therefore much faster. The language codes would be mapped somehow to standard values recognized by BaseX in the language_code_map file. Are there any efforts towards such a feature?

3 5

Re: [basex-talk] Distributing queries to several on several processors
by Christian Grün 22 Apr '15

22 Apr '15

Hi Götz (cc @ basex-talk), > OK, I think I understand. However, I think there should be some possibilities to allow the user to give hints. In my opinion, FOR-loops would be first-class candidates to use parallel streams, in particular in the use case I described in my previous posting: > > FOR $var IN (collection) > PARALLEL RETURN (expression-list) Makes sense, in general.. XQuery pragmas could be solution: (# basex: parallel #) { ... } Higher-order functions provide functions like hof:parallel-map(...). However, it has many effects on the architecture of BaseX in terms of performance, because we'd need to create new contexts for each parallelized query, which takes additional time. See the following query as example: $x[. = "123"] The dot applies to the "current context item". If we parallelize a query, we'd have multiple current context items. The same multiplication would apply to the stack frame and other runtime variables, and the time lost for duplicating these instances is in most cases more expensive than doing stuff in a single thread. At least that's our experience so far. Once again, we are happy to see people jump into our code and show us that it can be done better.. Christian

2 1

multi-language full-text indexing
by Goetz Heller 22 Apr '15

22 Apr '15

The case you described should be made a non-issue: If a multi-language full-text index was created then it was surely intended to execute searches within the confines of a specific language. Hence, if none was specified in the query, a runtime error should be thrown in such cases. Kind regards, Goetz -----Ursprüngliche Nachricht----- Von: Christian Grün [mailto:christian.gruen@gmail.com] Gesendet: Mittwoch, 22. April 2015 11:03 An: Goetz Heller Cc: BaseX Betreff: Re: [basex-talk] multi-language full-text indexing > It is desirable to have > documents indexed by locale-specific parts, e.g. I can see that this would absolutely make sense, but it would be quite some effort to realize it. There are also various conceptul issues related to XQuery Full Text: If you don't specify the language in the query, we'd need to dynamically decide what stemmers to use for the query strings, depending on the nodes that are currently targeted. This would pretty much blow up the existing architecture. As there are so many other types of index structures that could be helpful, depending on the particular use case, we usually recommend users to create additional BaseX databases, which can then serve as indexes. This can all be done in XQuery. I remember there have been various examples for this on this mailing list (see e.g. [1,2]). Hope this helps, Christian [1] https://www.mail-archive.com/basex-talk@mailman.uni-konstanz.de/msg04837.ht… [2] https://www.mail-archive.com/basex-talk@mailman.uni-konstanz.de/msg06089.ht… > > > > CREATE FULL-TEXT INDEX ON DATABASE XY STARTING WITH ( > > (path_a)/LOCALIZED_PART_A[@LANG=$lang], > > (path_b)/LOCALIZED_PART_B[@LG=$lang],… > > ) FOR LANGUAGE $lang IN ( > > BG, > > DN, > > DE WITH STOPWORDS filepath_de WITH STEM = YES, > > EN WITH STOPWORDS filepath_en, > > FR, … > > ) [USING language_code_map] > > and then to write full-text retrieval queries with a clause such as > ‘FOR LANGUAGE BG’, for example. The index parts would be much smaller > and full-text retrieval therefore much faster. The language codes > would be mapped somehow to standard values recognized by BaseX in the > language_code_map file. > > Are there any efforts towards such a feature?

2 1

Jump to page:

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

BaseX-Talk