Hi Christian,
I downloaded the latest SNAPSHOT this morning (20120903) and the problem is indeed fixed now. Using both the Oracle JDK and OpenJDK everything's running smoothly.
Thanks a lot for your excellent support!
Huib Verweij.
-----Original Message----- From: Christian Grün [mailto:christian.gruen@gmail.com] Sent: Sunday, 02 September, 2012 19:13 To: VERWEIJ Huib - Contractor Cc: basex-talk@mailman.uni-konstanz.de Subject: Re: [basex-talk] Concurreny troubles (?) and some remarks.
Hi Huib,
your test case seems to run fine on my machine. I'll send you a note in private, containing more details.
Best, Christian ___________________________
On Tue, Aug 28, 2012 at 3:15 PM, Christian Grün christian.gruen@gmail.com wrote:
Hi Huib,
thanks for sending me your case, I'll have a look at soon.
A note for everyone: all previous PDF versions of the documentation are now listed in our Wiki [1]. Next, we've added some words on the new HTTP Options, such as RESTXQPATH [2]. Some more general notes on the jetty.xml and web.xml configuration files will follow; those files have got more important for creating web pages, both in standalone mode and as web applications.
Christian
[1] http://docs.basex.org/wiki/Documentation [2] http://docs.basex.org/wiki/Options#HTTP_Options ___________________________
On Mon, Aug 27, 2012 at 3:09 PM, VERWEIJ Huib huib.verweij@nl.thalesgroup.com wrote:
Hi,
I seem to have a concurrency problem and I have a hard time fixing it. Our BaseX db is a very modest database, for testing we load a 138Kb file. We use JMeter to do performance testing on our Web Application deployment of BaseX 7.3. Randomly GET and PUT commands are sent to BaseX. When we use only one thread (simulated user) everything is fine, when we increase the number of threads we quickly end up with a corrupted database - two threads is generally slower to crash than three threads, with three threads it crashes fast.
Here are some stacktraces I get:
[INFO] Started Jetty Server BaseX 7.3 [Server] Server was started. Improper use? Potential bug? Your feedback is welcome: Contact: basex-talk@mailman.uni-konstanz.de Version: BaseX 7.3 Java: Sun Microsystems Inc., 1.6.0_22 OS: Linux, amd64 Stack Trace: java.lang.ArrayIndexOutOfBoundsException: 27 org.basex.util.list.ByteList.add(ByteList.java:39) org.basex.util.Compress.push(Compress.java:72) org.basex.util.Compress.pack(Compress.java:52) org.basex.data.DiskData.index(DiskData.java:387) org.basex.data.Data.attr(Data.java:1051) org.basex.data.Data.insert(Data.java:835) org.basex.query.up.primitives.InsertInto.apply(InsertInto.java:32)
org.basex.query.up.NodeUpdates.makePrimitivesEffective(NodeUpdates.java:216) org.basex.query.up.DatabaseUpdates.apply(DatabaseUpdates.java:152) org.basex.query.up.ContextModifier.apply(ContextModifier.java:66) org.basex.query.up.Updates.apply(Updates.java:183) org.basex.query.QueryContext.update(QueryContext.java:276) org.basex.http.restxq.RestXqResponse.create(RestXqResponse.java:76) org.basex.http.restxq.RestXqModule.process(RestXqModule.java:97) org.basex.http.restxq.RestXqServlet.run(RestXqServlet.java:28) org.basex.http.BaseXServlet.service(BaseXServlet.java:38) javax.servlet.http.HttpServlet.service(HttpServlet.java:820) org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:565)
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java :479)
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.j ava:119)
org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.jav a:524)
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandl er.java:225)
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1031)
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java: 406)
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandle r.java:186)
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandle r.java:965)
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.j ava:117)
org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(Cont extHandlerCollection.java:250)
org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerColl ection.java:149)
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:111) org.eclipse.jetty.server.Server.handle(Server.java:348)
org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(Abstrac tHttpConnection.java:452)
org.eclipse.jetty.server.AbstractHttpConnection.content(AbstractHttpC onnection.java:894)
org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(AbstractHttpConnection.java:948) org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:851)
org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235)
org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnecti on.java:77)
org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEn dPoint.java:620)
org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEnd Point.java:46)
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPoo l.java:603)
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:538) java.lang.Thread.run(Thread.java:679)
I am also getting:
[INFO] Started Jetty Server BaseX 7.3 [Server] Server was started. Improper use? Potential bug? Your feedback is welcome: Contact: basex-talk@mailman.uni-konstanz.de Version: BaseX 7.3 Java: Sun Microsystems Inc., 1.6.0_22 OS: Linux, amd64 Stack Trace: java.lang.ArrayIndexOutOfBoundsException: 22 org.basex.util.Compress.pull(Compress.java:132) org.basex.util.Compress.unpack(Compress.java:109) org.basex.data.DiskData.txt(DiskData.java:262) org.basex.data.DiskData.text(DiskData.java:229) org.basex.data.Data.atom(Data.java:208) org.basex.query.value.node.DBNode.string(DBNode.java:115) org.basex.query.value.node.ANode.string(ANode.java:50) org.basex.query.value.item.Str.eq(Str.java:80) org.basex.query.value.node.ANode.eq(ANode.java:61) org.basex.query.expr.CmpV$OpV$5.eval(CmpV.java:84) org.basex.query.expr.CmpG.eval(CmpG.java:258) org.basex.query.expr.CmpG.item(CmpG.java:213) org.basex.query.expr.CmpG.item(CmpG.java:25) org.basex.query.expr.ParseExpr.ebv(ParseExpr.java:81) org.basex.query.expr.ParseExpr.test(ParseExpr.java:94) org.basex.query.expr.Preds.preds(Preds.java:129) org.basex.query.path.IterStep$1.next(IterStep.java:48) org.basex.query.path.IterStep$1.next(IterStep.java:32) org.basex.query.path.IterPath$1.next(IterPath.java:67) org.basex.query.path.IterPath$1.next(IterPath.java:39) org.basex.query.iter.Iter.value(Iter.java:64) org.basex.query.expr.ParseExpr.value(ParseExpr.java:72) org.basex.query.QueryContext.value(QueryContext.java:303) org.basex.query.flwor.Let$1.next(Let.java:85) org.basex.query.flwor.FLWR$1.next(FLWR.java:63) org.basex.query.iter.Iter.value(Iter.java:64) org.basex.query.expr.ParseExpr.value(ParseExpr.java:72) org.basex.query.QueryContext.value(QueryContext.java:303) org.basex.query.expr.If.value(If.java:89) org.basex.query.QueryContext.value(QueryContext.java:303) org.basex.query.value.item.FuncItem.invValue(FuncItem.java:106) org.basex.query.func.FNFunc.foldLeft(FNFunc.java:196) org.basex.query.func.FNFunc.iter(FNFunc.java:39) org.basex.query.QueryContext.iter(QueryContext.java:292) org.basex.query.expr.ParseExpr.value(ParseExpr.java:72) org.basex.query.QueryContext.value(QueryContext.java:303) org.basex.query.func.UserFunc.value(UserFunc.java:154) org.basex.query.QueryContext.value(QueryContext.java:303) org.basex.query.func.BaseFuncCall.value(BaseFuncCall.java:56) org.basex.query.func.BaseFuncCall.iter(BaseFuncCall.java:69) org.basex.query.QueryContext.iter(QueryContext.java:292) org.basex.query.flwor.FLWR$1.next(FLWR.java:67) org.basex.query.iter.Iter.value(Iter.java:64) org.basex.query.expr.ParseExpr.value(ParseExpr.java:72) org.basex.query.QueryContext.value(QueryContext.java:303) org.basex.query.expr.If.value(If.java:89) org.basex.query.QueryContext.value(QueryContext.java:303) org.basex.query.value.item.FuncItem.invValue(FuncItem.java:106) org.basex.query.func.FNFunc.foldLeft(FNFunc.java:196) org.basex.query.func.FNFunc.iter(FNFunc.java:39) org.basex.query.QueryContext.iter(QueryContext.java:292) org.basex.query.expr.ParseExpr.value(ParseExpr.java:72) org.basex.query.QueryContext.value(QueryContext.java:303) org.basex.query.func.UserFunc.value(UserFunc.java:154) org.basex.query.QueryContext.value(QueryContext.java:303) org.basex.query.func.BaseFuncCall.value(BaseFuncCall.java:56) org.basex.query.func.BaseFuncCall.iter(BaseFuncCall.java:69) org.basex.query.QueryContext.iter(QueryContext.java:292) org.basex.query.flwor.FLWR$1.next(FLWR.java:67) org.basex.query.iter.Iter.value(Iter.java:64) org.basex.query.expr.ParseExpr.value(ParseExpr.java:72) org.basex.query.QueryContext.value(QueryContext.java:303) org.basex.query.expr.If.value(If.java:89) org.basex.query.QueryContext.value(QueryContext.java:303) org.basex.query.value.item.FuncItem.invValue(FuncItem.java:106) org.basex.query.func.FNFunc.foldLeft(FNFunc.java:196) org.basex.query.func.FNFunc.iter(FNFunc.java:39) org.basex.query.QueryContext.iter(QueryContext.java:292) org.basex.query.expr.ParseExpr.value(ParseExpr.java:72) org.basex.query.QueryContext.value(QueryContext.java:303) org.basex.query.func.UserFunc.value(UserFunc.java:154) org.basex.query.QueryContext.value(QueryContext.java:303) org.basex.query.func.BaseFuncCall.value(BaseFuncCall.java:56) org.basex.query.QueryContext.value(QueryContext.java:303) org.basex.query.flwor.Let$1.next(Let.java:85) org.basex.query.flwor.FLWR$1.next(FLWR.java:63) org.basex.query.iter.Iter.value(Iter.java:64) org.basex.query.expr.ParseExpr.value(ParseExpr.java:72) org.basex.query.QueryContext.value(QueryContext.java:303) org.basex.query.func.UserFunc.value(UserFunc.java:154) org.basex.query.QueryContext.value(QueryContext.java:303) org.basex.query.func.BaseFuncCall.value(BaseFuncCall.java:56) org.basex.query.QueryContext.value(QueryContext.java:303) org.basex.query.flwor.Let$1.next(Let.java:85) org.basex.query.flwor.GFLWOR.iter(GFLWOR.java:329) org.basex.query.flwor.GFLWOR.iter(GFLWOR.java:304) org.basex.query.QueryContext.iter(QueryContext.java:292) org.basex.query.expr.Constr.add(Constr.java:65) org.basex.query.expr.CElem.item(CElem.java:84) org.basex.query.expr.CElem.item(CElem.java:21) org.basex.query.expr.ParseExpr.value(ParseExpr.java:69) org.basex.query.QueryContext.value(QueryContext.java:303) org.basex.query.func.UserFunc.value(UserFunc.java:154) org.basex.query.QueryContext.value(QueryContext.java:303) org.basex.query.func.TailFuncCall.value(TailFuncCall.java:47) org.basex.query.QueryContext.value(QueryContext.java:303) org.basex.query.expr.If.value(If.java:89) org.basex.query.QueryContext.value(QueryContext.java:303) org.basex.query.func.UserFunc.value(UserFunc.java:154) org.basex.query.QueryContext.value(QueryContext.java:303) org.basex.query.func.TailFuncCall.value(TailFuncCall.java:47) org.basex.query.func.TailFuncCall.iter(TailFuncCall.java:55) org.basex.query.QueryContext.iter(QueryContext.java:292) org.basex.query.flwor.FLWR$1.next(FLWR.java:67) org.basex.query.flwor.FLWR$1.next(FLWR.java:59) org.basex.query.iter.Iter.value(Iter.java:64) org.basex.query.expr.ParseExpr.value(ParseExpr.java:72) org.basex.query.QueryContext.value(QueryContext.java:303) org.basex.query.func.UserFunc.value(UserFunc.java:154) org.basex.query.QueryContext.value(QueryContext.java:303) org.basex.query.func.BaseFuncCall.value(BaseFuncCall.java:56) org.basex.query.QueryContext.value(QueryContext.java:303) org.basex.http.restxq.RestXqResponse.create(RestXqResponse.java:75) org.basex.http.restxq.RestXqModule.process(RestXqModule.java:97) org.basex.http.restxq.RestXqServlet.run(RestXqServlet.java:28) org.basex.http.BaseXServlet.service(BaseXServlet.java:38) javax.servlet.http.HttpServlet.service(HttpServlet.java:820) org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:565)
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java :479)
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.j ava:119)
org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.jav a:524)
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandl er.java:225)
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1031)
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java: 406)
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandle r.java:186)
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandle r.java:965)
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.j ava:117)
org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(Cont extHandlerCollection.java:250)
org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerColl ection.java:149)
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:111) org.eclipse.jetty.server.Server.handle(Server.java:348)
org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(Abstrac tHttpConnection.java:452)
org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(Abstra ctHttpConnection.java:884)
org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:938) org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:630)
org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:230)
org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnecti on.java:77)
org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEn dPoint.java:620)
org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEnd Point.java:46)
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPoo l.java:603)
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:538) java.lang.Thread.run(Thread.java:679)
and also this:
14:52:02,452 - WARN - qtp384018721-20 - oup.nl.mms.plib.server.rest.AbstractEntityResource # HTTP response is not OK: code 500, ETag: empty, content: org.apache.http.conn.EofSensorInputStream@4acfed14 Improper use? Potential bug? Your feedback is welcome: Contact: basex-talk@mailman.uni-konstanz.de Version: BaseX 7.3 Java: Sun Microsystems Inc., 1.6.0_22 OS: Linux, amd64 Stack Trace: java.lang.ArrayIndexOutOfBoundsException 2012-08-27 14:52:02.959:WARN:oejs.ServletHandler:/basex/restxq/plib/platforms/ java.lang.ArrayIndexOutOfBoundsException: 0 at org.basex.util.Util.message(Util.java:166) at org.basex.util.TokenBuilder.addExt(TokenBuilder.java:327) at org.basex.util.TokenBuilder.addExt(TokenBuilder.java:345) at org.basex.util.Util.inf(Util.java:218) at org.basex.util.Util.info(Util.java:207) at org.basex.http.BaseXServlet.service(BaseXServlet.java:53) at javax.servlet.http.HttpServlet.service(HttpServlet.java:820) at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:565) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:479) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:119) at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:524) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:225) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1031) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:406) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:186) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:965) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:117) at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:250) at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:149) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:111) at org.eclipse.jetty.server.Server.handle(Server.java:348) at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:452) at org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:884) at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:938) at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:630) at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:230) at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:77) at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:620) at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:46) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:603) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:538) at java.lang.Thread.run(Thread.java:679)
What I did to try to fix it:
1 ---
Set org.basex.mode to "server". I set it in the "override.xml" file and later also set it in the .war file's web.xml. It seems to be running in "server" mode now. The INFO command output looks like:
============================== General Information Version: 7.3 Used Memory: 27080 KB
Main Options DBPATH: /home/dm218/NetBeansProjects/trunk/plib/plib-server/target/basex/data DEBUG: true EVENTPORT: 1985 HOST: localhost HTTPPATH: /home/dm218/NetBeansProjects/trunk/plib/plib-server/target/basex/http HTTPPORT: 8984 KEEPALIVE: 600 LANG: English LANGKEYS: false NONPROXYHOSTS: PARALLEL: 8 PORT: 1984 PROXYHOST: PROXYPORT: 80 REPOPATH: /home/dm218/NetBeansProjects/trunk/plib/plib-server/target/basex/repo SERVERHOST: SERVERPORT: 1984 STOPPORT: 8985 TIMEOUT: 30
Options ADDARCHIVES: true ADDRAW: false ATTRINDEX: true AUTOFLUSH: true BINDINGS: CACHEQUERY: false CASESENS: false CATFILE: CHOP: true COMPPLAN: true CREATEFILTER: *.xml DIACRITICS: false DOTCOMPACT: false DOTDISPLAY: true DOTPLAN: false DOTTY: dotty DTD: false EXPORTER: FORCECREATE: false FTINDEX: false HTMLOPT: html=false,omit-xml-declaration=false,method=xml,nons=false,nobogons= false,nodefaults=false,nocolons=false,norestart=false,ignorable=false ,emptybogons=false,any=false,norootbogons=false,nocdata=false,lexical =false,encoding=utf-8 INTPARSE: true LANGUAGE: en LSERROR: 0 MAINMEM: false MAXCATS: 100 MAXHITS: -1 MAXLEN: 96 MAXSTAT: 30 PARSER: xml PARSEROPT: encoding=UTF-8,flat=false,format=verbose,header=false,jsonml=false,li nes=true,separator=comma QUERYINFO: false QUERYPATH: RUNS: 1 SERIALIZE: true SERIALIZER: SKIPCORRUPT: false STEMMING: false STOPWORDS: TAILCALLS: 42 TEXTINDEX: true UPDINDEX: false WRITEBACK: false XMLPLAN: false XQUERY3: true
==============================
2 ---
I also tried using 7.4-SNAPSHOT (downloaded this morning) but it shows the same behaviour. It reports (I'm copying this by hand from another PC):
==========================
Unexpected error: org.basex.util.Util: java.lang.RunTimeException: Data Access out of bounds:
Pre value: 7290#used blocks: 31#total blocks: 31Access 30 (31 > 30)==========================
Some more remarks:
I would much appreciate a wiki-per-version, e.g. a 7.3 wiki, a7.4 wiki, etc. The adaptations for a new version are slightly confusing - BaseX changes a lot (new options, renames, what option to put in what config file, etc.). Also for people using an older version this can be handy.
the mailing list search service on the basex.org website (linksfrom the "Open Source" page) crashes.
(7.4) I am confused about the org.basex.restxqpath vs.org.basex.webpath settings, I put them both in and then it seemed to work. (Perhaps db:system() is also inconsistent wrt these settings.) RESTXQPATH is not mentioned in the OPTIONS list.
(7.4) perhaps I missed it and the math:uuid() function wasdeprecated already, but I found that I needed to adapt my .xqm script for 7.4 because math:uuid() was renamed to random:uuid(). It seemed a little abrupt to me, but as I said, perhaps it missed the deprecation. I also used db:system() and had to adapt that too because org.basex.httppath no longer exists.
I suspect I missed some configuration option or something like that and will appreciate any help!
Thanks,
Huib Verweij.
Disclaimer:
If you are not the intended recipient of this email, please notify the sender and delete it. Any unauthorized copying, disclosure or distribution of this email or its attachment(s) is forbidden. Thales Nederland BV will not accept liability for any damage caused by this email or its attachment(s). Thales Nederland BV is seated in Hengelo and is registered at the Chamber of Commerce under number 06061578.
BaseX-Talk mailing list BaseX-Talk@mailman.uni-konstanz.de https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk
------------------------------------------------------------------------------------------------------------ Disclaimer:
If you are not the intended recipient of this email, please notify the sender and delete it. Any unauthorized copying, disclosure or distribution of this email or its attachment(s) is forbidden. Thales Nederland BV will not accept liability for any damage caused by this email or its attachment(s). Thales Nederland BV is seated in Hengelo and is registered at the Chamber of Commerce under number 06061578. ------------------------------------------------------------------------------------------------------------