BaseX-Talk

basex-talk@mailman.uni-konstanz.de

4 participants
5268 discussions

Disable or control query caching
by Manuel Bernhardt 22 May '12

22 May '12

Hello, we're getting an apparent deadlock (followed by a GC overhead limit exceeded) on one machine, when starting some processing on a collection of over 800 000 records. Going after it with YourKit yields the following: application-akka.actor.default-dispatcher-110 <--- Frozen for at least 3m 7s org.basex.server.Query.cache(InputStream) org.basex.server.ClientQuery.cache() org.basex.server.Query.more() eu.delving.basex.client.Implicits$RichClientQuery.hasNext() scala.collection.Iterator$$anon$19.hasNext() scala.collection.Iterator$$anon$29.hasNext() scala.collection.Iterator$class.foreach(Iterator, Function1) scala.collection.Iterator$$anon$29.foreach(Function1) core.processing.CollectionProcessor$$anonfun$process$2.apply(ClientSession) core.processing.CollectionProcessor$$anonfun$process$2.apply(Object) core.storage.BaseXStorage$$anonfun$withSession$1.apply(ClientSession) core.storage.BaseXStorage$$anonfun$withSession$1.apply(Object) eu.delving.basex.client.BaseX$$anonfun$withSession$1.apply(ClientSession) eu.delving.basex.client.BaseX$$anonfun$withSession$1.apply(Object) eu.delving.basex.client.BaseX.withSession(Function1) eu.delving.basex.client.BaseX.withSession(String, Function1) core.storage.BaseXStorage$.withSession(Collection, Function1) core.processing.CollectionProcessor.process(Function0, Function1, Function1, Function3) core.processing.DataSetCollectionProcessor$.process(DataSet) actors.Processor$$anonfun$receive$1.apply(Object)<2 recursive calls> akka.actor.Actor$class.apply(Actor, Object) actors.Processor.apply(Object) akka.actor.ActorCell.invoke(Envelope) akka.dispatch.Mailbox.processMailbox(int, long) akka.dispatch.Mailbox.run() akka.dispatch.ForkJoinExecutorConfigurator$MailboxExecutionTask.exec() akka.jsr166y.ForkJoinTask.doExec() akka.jsr166y.ForkJoinPool$WorkQueue.runTask(ForkJoinTask) akka.jsr166y.ForkJoinPool.runWorker(ForkJoinPool$WorkQueue) akka.jsr166y.ForkJoinWorkerThread.run() In the server logs I can observe: 09:10:45.129 [192.168.1.214:47530]: dimcon____geheugen-van-nederland QUERY(3) for $i in /record[@version = 0] order by $i/system/index return $i OK 0.06 ms 09:10:45.129 [192.168.1.214:47530]: dimcon____geheugen-van-nederland QUERY(3) OK 0.03 ms 09:13:23.155 [192.168.1.214:47530]: dimcon____geheugen-van-nederland ITER(3) Error: Connection reset 09:13:23.155 [192.168.1.214:47530]: dimcon____geheugen-van-nederland LOGOUT admin OK I looked up the code, and it looks as though the whole (?) query is cached in memory upon retrieval. Given the DB is over 1.2 GB in size, our client server has a hard time (it only has 1,5 GB of Xmx). Is there any preferred way of dealing with this? What I am going to do for the moment, I think, is to override or intercept the creation of the ClientQuery and make an implementation that has a different caching strategy. Another approach may probably be to limit the query output and implement some custom iteration behavior - but if it that can be handled directly at the query level, I think it would make things easier. Manuel

2 6

db:add and ADD logic
by Fabrice Etanchaud 22 May '12

22 May '12

Dear all at BaseX, I don't understand the ADD/db:add behaviour when adding two documents with the same name : previous document is not replaced, but copied, and I did not find a way to correct, and delete only one of the documents, because they all have the same name... So, thanks to the db:replace function, I wrote a updating function which emulates desired behaviour (that adds or replaces the document if it already exists in the database). But if I try to insert the same document twice with this function (in the same updating query), I still get two documents in the database, because function is 'updating' and because of the pending update list... Are db:XXX functions really 'updating' ? Is there a way to delete only one of the duplicated documents ? Thank you again for your fantastic job. Regards, Fabrice ETANCHAUD Senior Software Engineer edital Berkenlaan 1 B-1831 Brussels, Belgium <skype:fabrice.etanchaud?call> +32 2 716 32 32 general +32 2 716 32 20 fax fetanchaud(a)edital.com corsearch.com <http://www.corsearch.com/> | edital.com <http://www.edital.com/> <outbind://899-00000000E439C83B50BF17409D8C27B7AE0425AF07005AA868536216F 543BAB9422EAEA5338B00000011020900005AA868536216F543BAB9422EAEA5338B00000 13AA36B0000/cid:image001.gif@01CC0F02.2C91B6C0> Confidentiality Notice: This email and its attachments (if any) contain confidential information of the sender. The information is intended only for the use by the direct addressees of the original sender of this email. If you are not an intended recipient of the original sender (or responsible for delivering the message to such person), you are hereby notified that any review, disclosure, copying, distribution or the taking of any action in reliance of the contents of and attachments to this email is strictly prohibited. If you have received this email in error, please immediately notify the sender at the address shown herein and permanently delete any copies of this email (digital or paper) in your possession.

2 1

Returned mail: see transcript for details
by sandeep＠marlabs.com 21 May '12

21 May '12

The original message was received at Mon, 21 May 2012 11:27:17 -0500 from [65.1.203.108] ----- The following addresses had permanent fatal errors ----- basex-talk(a)mailman.uni-konstanz.de

1 0

http server and client mode
by Gilles CARRY 21 May '12

21 May '12

Hello, I cannot set the http server to use a distant DB. In the restxq.xqm file, I always get Using this: let $count := (count(db:open('logs')/top/session)) in restxq.xqm, I always get the error: Stopped at line 15, column 38 in /usr/share/tomcat/webapps/BaseX72/restxq.xqm: [BASX0003] Database 'logs' not found. On the web browser. Using a CLI from a distant server, I can access the 'logs' DB. The db:list() instruction returns nothing. In web.xml, I set org.basex.mode to "client" and "host" to the IP of my DB server. Something I don't understand is that in the below output (db:system()) I can see that "host" is present whereas "mode" is not. Credential are OK. Any clue? When in "client" mode, is "dbpath" useful? Thanks, Gilles. <?xml version="1.0" encoding="UTF-8"?> <!DOCTYPE top PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd"> <top> <system> <generalinformation> <version>7.2</version> <usedmemory>21887 KB</usedmemory> </generalinformation> <mainoptions> <dbpath>/home/basex/BaseXData</dbpath> <debug>true</debug> <eventport>1985</eventport> <host>10.129.63.129</host> <httppath>/usr/share/tomcat/webapps/BaseX72/</httppath> <httpport>8984</httpport> <keepalive>0</keepalive> <lang>English</lang> <langkeys>false</langkeys> <nonproxyhosts></nonproxyhosts> <parallel>8</parallel> <port>1984</port> <proxyhost></proxyhost> <proxyport>80</proxyport> <repopath>/usr/share/tomcat/webapps/BaseX72//repo</repopath> <serverhost></serverhost> <serverport>1984</serverport> <stopport>8985</stopport> <timeout>0</timeout> </mainoptions> <options> <addarchives>true</addarchives> <addraw>false</addraw> <attrindex>true</attrindex> <autoflush>true</autoflush> <bindings></bindings> <cachequery>false</cachequery> <casesens>false</casesens> <catfile></catfile> <chop>true</chop> <compplan>true</compplan> <createfilter>*.xml</createfilter> <diacritics>false</diacritics> <dotcompact>false</dotcompact> <dotdisplay>true</dotdisplay> <dotplan>false</dotplan> <dotty>dotty</dotty> <dtd>false</dtd> <exporter></exporter> <forcecreate>false</forcecreate> <ftindex>false</ftindex> <htmlopt>html=false,omit-xml-declaration=false,method=xml,nons=false,nobogons=false,nodefaults=false,nocolons=false,norestart=false,ignorable=false,emptybogons=false,any=false,norootbogons=false,nocdata=false,lexical=false,encoding=utf-8</htmlopt> <intparse>true</intparse> <language>en</language> <lserror>0</lserror> <mainmem>false</mainmem> <maxcats>100</maxcats> <maxhits>-1</maxhits> <maxlen>96</maxlen> <maxstat>30</maxstat> <parser>xml</parser> <parseropt>encoding=UTF-8,flat=false,format=verbose,header=false,jsonml=false,lines=true,separator=comma</parseropt> <pathindex>true</pathindex> <queryinfo>false</queryinfo> <querypath></querypath> <runs>1</runs> <scoring>0</scoring> <serialize>true</serialize> <serializer></serializer> <skipcorrupt>false</skipcorrupt> <stemming>false</stemming> <stopwords></stopwords> <tailcalls>42</tailcalls> <textindex>true</textindex> <updindex>false</updindex> <wildcards>false</wildcards> <writeback>false</writeback> <xmlplan>false</xmlplan> <xquery3>true</xquery3> </options> </system> </top>

4 9

Use of XML Schema validation with BaseX [SEC=UNCLASSIFIED]
by Steve Cameron 21 May '12

21 May '12

Hi Just wondering if anyone has gone down this path, maybe via a Servlet filter? Thanks Stephen Cameron Antarctic Fisheries Computing and Database Support Provider Australian Antarctic Division Kingston, Tasmania [cid:image001.jpg@01CD374E.353A91A0] Phone: 03 62323571 ___________________________________________________________________________ Australian Antarctic Division - Commonwealth of Australia IMPORTANT: This transmission is intended for the addressee only. If you are not the intended recipient, you are notified that use or dissemination of this communication is strictly prohibited by Commonwealth law. If you have received this transmission in error, please notify the sender immediately by e-mail or by telephoning +61 3 6232 3209 and DELETE the message. Visit our web site at http://www.antarctica.gov.au/ ___________________________________________________________________________

2 1

Best way to insert large amounts of records
by Manuel Bernhardt 21 May '12

21 May '12

Hi, we're using BaseX to store multiple collections of documents (we call them records). These record are produced programmatically, by parsing an incoming stream on a server application and turning it into a document of the kind <record id="123" version="1"> ... </record> So far I took the following approach: - each collection of records is its own database in BaseX, for easier management - on insertion - set the session's autoflush to false - iterate over record - add them via add(id, document) - each 10000 records, flush - finally, flush once more - create the attributes index So for example now we have: name Resources Size Input Path ------------------------------------------------------------------------------------ col1 14141 19815190 col2 14750 16697081 col3 84450 253593687 col4 1012477 2107593252 col5 126058 186315175 col6 13767 14640701 col7 815991 730536864 col8 31189 39598405 col9 24733 91277637 col10 171906 202392553 ... and there'll be quite a bit more coming in. This kind of bulk insertion can also happen concurrently (I've set-up an actor pool at five for the moment). My questions are: - is this the most performant approach, or would it make sense to e.g. build one stream on the fly and somehow turn it into an inputstream to be sent via add? - is there a performance cost in adding with an ID? We don't really need them since we retrieve records via a query - and those resources aren't really files on the file-system - is there a performance penalty in doing this kind of parsing concurrently? - are there any JVM parameters that would help speed this up? I haven't quite found how to pass in JVM parameters when starting basexserver via the command line. Looks like BaseX gave itself an Xmx of 1866006528 (but that machine has 8GB so it could in theory get more. Thanks! Manuel

2 2

RESTXQ: If you ever need to set an http status code, or overwrite the annotations
by Philippe Rathé 20 May '12

20 May '12

Hello, this is an example of setting a 404 by overwriting what the REST annotations would produce. I had to gathered pieces of specs together in order to have it working. Please tell me if I'm missing the documentation that cover this. Feedback are welcome. declare %rest:path("/categories/{$category}") %rest:GET %rest:produces("application/atomcat+xml") %output:omit-xml-declaration("no") %output:media-type("application/atomcat+xml") function page:category($category as xs:string) { if (db:exists('categories', $category || '.xml')) then doc('categories/' || $category || '.xml') else let $reason := "Category scheme not found" return <rest:response> <http:response status="404" reason="{$reason}">  </http:response> <output:serialization-parameters> <output:omit-xml-declaration value="yes"/> <output:media-type value="text/plain"/> </output:serialization-parameters> </rest:response> }; Would produce the following on a not found document: HTTP/1.1 404 Not Found Content-Type: text/plain; charset=utf-8 Content-Length: 25 Server: Jetty(6.1.26) Category scheme not found -- Philippe

1 0

Scala client library for BaseX
by Manuel Bernhardt 20 May '12

20 May '12

Hi, I'd like to announce the first release of a scala client library for BaseX, which simplifies the idiomatic usage of BaseX within Scala applications. It's likely going to evolve quite a bit over the next weeks since we're in the process of learning how to best use BaseX. The source is available here: https://github.com/delving/basex-scala-client And I'll try to find some time to write some documentation this week-end. Comments, feedback etc. are of course very welcome! Cheers, Manuel

3 2

List all documents in a database
by Anupam Bakshi 20 May '12

20 May '12

Hello, I'm new to BaseX and XQuery, I wonder if someone can help with a simple question. I have added several documents to a BaseX database. Now I want to list and delete them. I didn't see commands to do that in the BaseX client 7.2.1. Any pointer or help will help me in understanding. Thanks, Anupam

3 4

RESTXQ: separate server into multiple files
by Philippe Rathé 18 May '12

18 May '12

Hello, In the Hello World example (restxq.xqm), would it be possible to split the file in two with half of the requests mappings be in one file in the other half in the other file. The reason I would like to do this is for separation of concern. You could see it as a controller file and in my use case I have two controllers. I have tried to add another file name abc.xqm aside restxq.xqm but I got the following error. The doc says it loads every xqm file in the root directory so I thought nothing would be necessary to configure. HTTP/1.1 500 Could not initialize class org.basex.http.restxq.RestXqResponse Could not initialize class org.basex.http.restxq.RestXqResponse</pre></p><h3>Caused by:</h3><pre>java.lang.NoClassDefFoundError: Could not initialize class org.basex.http.restxq.RestXqResponse at org.basex.http.restxq.RestXqModule.process(RestXqModule.java:97) at org.basex.http.restxq.RestXqServlet.run(RestXqServlet.java:28) at org.basex.http.BaseXServlet.service(BaseXServlet.java:34) at javax.servlet.http.HttpServlet.service(HttpServlet.java:820) at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511) at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:401) at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216) at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182) at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766) at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450) at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230) at org.mortbay.jetty.handler.HandlerCollection.handle(HandlerCollection.java:114) at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152) at org.mortbay.jetty.Server.handle(Server.java:326) at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542) at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928) at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549) at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212) at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404) at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410) at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582) Thanks, -- Philippe

2 2

Jump to page:

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

BaseX-Talk