Hi,
I seem to have a concurrency problem and I have a hard time fixing it. Our BaseX db is a very modest database, for testing we load a 138Kb file. We use JMeter to do performance testing on our Web Application deployment of BaseX 7.3. Randomly GET and PUT commands are sent to BaseX. When we use only one thread (simulated user) everything is fine, when we increase the number of threads we quickly end up with a corrupted database - two threads is generally slower to crash than three threads, with three threads it crashes fast.
Here are some stacktraces I get:
[INFO] Started Jetty Server BaseX 7.3 [Server] Server was started. Improper use? Potential bug? Your feedback is welcome: Contact: basex-talk@mailman.uni-konstanz.demailto:basex-talk@mailman.uni-konstanz.de Version: BaseX 7.3 Java: Sun Microsystems Inc., 1.6.0_22 OS: Linux, amd64 Stack Trace: java.lang.ArrayIndexOutOfBoundsException: 27 org.basex.util.list.ByteList.add(ByteList.java:39) org.basex.util.Compress.push(Compress.java:72) org.basex.util.Compress.pack(Compress.java:52) org.basex.data.DiskData.index(DiskData.java:387) org.basex.data.Data.attr(Data.java:1051) org.basex.data.Data.insert(Data.java:835) org.basex.query.up.primitives.InsertInto.apply(InsertInto.java:32) org.basex.query.up.NodeUpdates.makePrimitivesEffective(NodeUpdates.java:216) org.basex.query.up.DatabaseUpdates.apply(DatabaseUpdates.java:152) org.basex.query.up.ContextModifier.apply(ContextModifier.java:66) org.basex.query.up.Updates.apply(Updates.java:183) org.basex.query.QueryContext.update(QueryContext.java:276) org.basex.http.restxq.RestXqResponse.create(RestXqResponse.java:76) org.basex.http.restxq.RestXqModule.process(RestXqModule.java:97) org.basex.http.restxq.RestXqServlet.run(RestXqServlet.java:28) org.basex.http.BaseXServlet.service(BaseXServlet.java:38) javax.servlet.http.HttpServlet.service(HttpServlet.java:820) org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:565) org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:479) org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:119) org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:524) org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:225) org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1031) org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:406) org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:186) org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:965) org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:117) org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:250) org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:149) org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:111) org.eclipse.jetty.server.Server.handle(Server.java:348) org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:452) org.eclipse.jetty.server.AbstractHttpConnection.content(AbstractHttpConnection.java:894) org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(AbstractHttpConnection.java:948) org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:851) org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235) org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:77) org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:620) org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:46) org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:603) org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:538) java.lang.Thread.run(Thread.java:679)
I am also getting:
[INFO] Started Jetty Server BaseX 7.3 [Server] Server was started. Improper use? Potential bug? Your feedback is welcome: Contact: basex-talk@mailman.uni-konstanz.demailto:basex-talk@mailman.uni-konstanz.de Version: BaseX 7.3 Java: Sun Microsystems Inc., 1.6.0_22 OS: Linux, amd64 Stack Trace: java.lang.ArrayIndexOutOfBoundsException: 22 org.basex.util.Compress.pull(Compress.java:132) org.basex.util.Compress.unpack(Compress.java:109) org.basex.data.DiskData.txt(DiskData.java:262) org.basex.data.DiskData.text(DiskData.java:229) org.basex.data.Data.atom(Data.java:208) org.basex.query.value.node.DBNode.string(DBNode.java:115) org.basex.query.value.node.ANode.string(ANode.java:50) org.basex.query.value.item.Str.eq(Str.java:80) org.basex.query.value.node.ANode.eq(ANode.java:61) org.basex.query.expr.CmpV$OpV$5.eval(CmpV.java:84) org.basex.query.expr.CmpG.eval(CmpG.java:258) org.basex.query.expr.CmpG.item(CmpG.java:213) org.basex.query.expr.CmpG.item(CmpG.java:25) org.basex.query.expr.ParseExpr.ebv(ParseExpr.java:81) org.basex.query.expr.ParseExpr.test(ParseExpr.java:94) org.basex.query.expr.Preds.preds(Preds.java:129) org.basex.query.path.IterStep$1.next(IterStep.java:48) org.basex.query.path.IterStep$1.next(IterStep.java:32) org.basex.query.path.IterPath$1.next(IterPath.java:67) org.basex.query.path.IterPath$1.next(IterPath.java:39) org.basex.query.iter.Iter.value(Iter.java:64) org.basex.query.expr.ParseExpr.value(ParseExpr.java:72) org.basex.query.QueryContext.value(QueryContext.java:303) org.basex.query.flwor.Let$1.next(Let.java:85) org.basex.query.flwor.FLWR$1.next(FLWR.java:63) org.basex.query.iter.Iter.value(Iter.java:64) org.basex.query.expr.ParseExpr.value(ParseExpr.java:72) org.basex.query.QueryContext.value(QueryContext.java:303) org.basex.query.expr.If.value(If.java:89) org.basex.query.QueryContext.value(QueryContext.java:303) org.basex.query.value.item.FuncItem.invValue(FuncItem.java:106) org.basex.query.func.FNFunc.foldLeft(FNFunc.java:196) org.basex.query.func.FNFunc.iter(FNFunc.java:39) org.basex.query.QueryContext.iter(QueryContext.java:292) org.basex.query.expr.ParseExpr.value(ParseExpr.java:72) org.basex.query.QueryContext.value(QueryContext.java:303) org.basex.query.func.UserFunc.value(UserFunc.java:154) org.basex.query.QueryContext.value(QueryContext.java:303) org.basex.query.func.BaseFuncCall.value(BaseFuncCall.java:56) org.basex.query.func.BaseFuncCall.iter(BaseFuncCall.java:69) org.basex.query.QueryContext.iter(QueryContext.java:292) org.basex.query.flwor.FLWR$1.next(FLWR.java:67) org.basex.query.iter.Iter.value(Iter.java:64) org.basex.query.expr.ParseExpr.value(ParseExpr.java:72) org.basex.query.QueryContext.value(QueryContext.java:303) org.basex.query.expr.If.value(If.java:89) org.basex.query.QueryContext.value(QueryContext.java:303) org.basex.query.value.item.FuncItem.invValue(FuncItem.java:106) org.basex.query.func.FNFunc.foldLeft(FNFunc.java:196) org.basex.query.func.FNFunc.iter(FNFunc.java:39) org.basex.query.QueryContext.iter(QueryContext.java:292) org.basex.query.expr.ParseExpr.value(ParseExpr.java:72) org.basex.query.QueryContext.value(QueryContext.java:303) org.basex.query.func.UserFunc.value(UserFunc.java:154) org.basex.query.QueryContext.value(QueryContext.java:303) org.basex.query.func.BaseFuncCall.value(BaseFuncCall.java:56) org.basex.query.func.BaseFuncCall.iter(BaseFuncCall.java:69) org.basex.query.QueryContext.iter(QueryContext.java:292) org.basex.query.flwor.FLWR$1.next(FLWR.java:67) org.basex.query.iter.Iter.value(Iter.java:64) org.basex.query.expr.ParseExpr.value(ParseExpr.java:72) org.basex.query.QueryContext.value(QueryContext.java:303) org.basex.query.expr.If.value(If.java:89) org.basex.query.QueryContext.value(QueryContext.java:303) org.basex.query.value.item.FuncItem.invValue(FuncItem.java:106) org.basex.query.func.FNFunc.foldLeft(FNFunc.java:196) org.basex.query.func.FNFunc.iter(FNFunc.java:39) org.basex.query.QueryContext.iter(QueryContext.java:292) org.basex.query.expr.ParseExpr.value(ParseExpr.java:72) org.basex.query.QueryContext.value(QueryContext.java:303) org.basex.query.func.UserFunc.value(UserFunc.java:154) org.basex.query.QueryContext.value(QueryContext.java:303) org.basex.query.func.BaseFuncCall.value(BaseFuncCall.java:56) org.basex.query.QueryContext.value(QueryContext.java:303) org.basex.query.flwor.Let$1.next(Let.java:85) org.basex.query.flwor.FLWR$1.next(FLWR.java:63) org.basex.query.iter.Iter.value(Iter.java:64) org.basex.query.expr.ParseExpr.value(ParseExpr.java:72) org.basex.query.QueryContext.value(QueryContext.java:303) org.basex.query.func.UserFunc.value(UserFunc.java:154) org.basex.query.QueryContext.value(QueryContext.java:303) org.basex.query.func.BaseFuncCall.value(BaseFuncCall.java:56) org.basex.query.QueryContext.value(QueryContext.java:303) org.basex.query.flwor.Let$1.next(Let.java:85) org.basex.query.flwor.GFLWOR.iter(GFLWOR.java:329) org.basex.query.flwor.GFLWOR.iter(GFLWOR.java:304) org.basex.query.QueryContext.iter(QueryContext.java:292) org.basex.query.expr.Constr.add(Constr.java:65) org.basex.query.expr.CElem.item(CElem.java:84) org.basex.query.expr.CElem.item(CElem.java:21) org.basex.query.expr.ParseExpr.value(ParseExpr.java:69) org.basex.query.QueryContext.value(QueryContext.java:303) org.basex.query.func.UserFunc.value(UserFunc.java:154) org.basex.query.QueryContext.value(QueryContext.java:303) org.basex.query.func.TailFuncCall.value(TailFuncCall.java:47) org.basex.query.QueryContext.value(QueryContext.java:303) org.basex.query.expr.If.value(If.java:89) org.basex.query.QueryContext.value(QueryContext.java:303) org.basex.query.func.UserFunc.value(UserFunc.java:154) org.basex.query.QueryContext.value(QueryContext.java:303) org.basex.query.func.TailFuncCall.value(TailFuncCall.java:47) org.basex.query.func.TailFuncCall.iter(TailFuncCall.java:55) org.basex.query.QueryContext.iter(QueryContext.java:292) org.basex.query.flwor.FLWR$1.next(FLWR.java:67) org.basex.query.flwor.FLWR$1.next(FLWR.java:59) org.basex.query.iter.Iter.value(Iter.java:64) org.basex.query.expr.ParseExpr.value(ParseExpr.java:72) org.basex.query.QueryContext.value(QueryContext.java:303) org.basex.query.func.UserFunc.value(UserFunc.java:154) org.basex.query.QueryContext.value(QueryContext.java:303) org.basex.query.func.BaseFuncCall.value(BaseFuncCall.java:56) org.basex.query.QueryContext.value(QueryContext.java:303) org.basex.http.restxq.RestXqResponse.create(RestXqResponse.java:75) org.basex.http.restxq.RestXqModule.process(RestXqModule.java:97) org.basex.http.restxq.RestXqServlet.run(RestXqServlet.java:28) org.basex.http.BaseXServlet.service(BaseXServlet.java:38) javax.servlet.http.HttpServlet.service(HttpServlet.java:820) org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:565) org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:479) org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:119) org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:524) org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:225) org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1031) org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:406) org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:186) org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:965) org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:117) org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:250) org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:149) org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:111) org.eclipse.jetty.server.Server.handle(Server.java:348) org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:452) org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:884) org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:938) org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:630) org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:230) org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:77) org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:620) org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:46) org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:603) org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:538) java.lang.Thread.run(Thread.java:679)
and also this:
14:52:02,452 - WARN - qtp384018721-20 - oup.nl.mms.plib.server.rest.AbstractEntityResource # HTTP response is not OK: code 500, ETag: empty, content: org.apache.http.conn.EofSensorInputStream@4acfed14mailto:org.apache.http.conn.EofSensorInputStream@4acfed14 Improper use? Potential bug? Your feedback is welcome: Contact: basex-talk@mailman.uni-konstanz.demailto:basex-talk@mailman.uni-konstanz.de Version: BaseX 7.3 Java: Sun Microsystems Inc., 1.6.0_22 OS: Linux, amd64 Stack Trace: java.lang.ArrayIndexOutOfBoundsException 2012-08-27 14:52:02.959:WARN:oejs.ServletHandler:/basex/restxq/plib/platforms/ java.lang.ArrayIndexOutOfBoundsException: 0 at org.basex.util.Util.message(Util.java:166) at org.basex.util.TokenBuilder.addExt(TokenBuilder.java:327) at org.basex.util.TokenBuilder.addExt(TokenBuilder.java:345) at org.basex.util.Util.inf(Util.java:218) at org.basex.util.Util.infohttp://org.basex.util.Util.info(Util.java:207) at org.basex.http.BaseXServlet.service(BaseXServlet.java:53) at javax.servlet.http.HttpServlet.service(HttpServlet.java:820) at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:565) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:479) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:119) at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:524) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:225) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1031) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:406) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:186) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:965) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:117) at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:250) at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:149) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:111) at org.eclipse.jetty.server.Server.handle(Server.java:348) at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:452) at org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:884) at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:938) at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:630) at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:230) at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:77) at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:620) at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:46) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:603) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:538) at java.lang.Thread.run(Thread.java:679)
What I did to try to fix it:
1 --- Set org.basex.mode to "server". I set it in the "override.xml" file and later also set it in the .war file's web.xml. It seems to be running in "server" mode now. The INFO command output looks like:
============================== General Information Version: 7.3 Used Memory: 27080 KB
Main Options DBPATH: /home/dm218/NetBeansProjects/trunk/plib/plib-server/target/basex/data DEBUG: true EVENTPORT: 1985 HOST: localhost HTTPPATH: /home/dm218/NetBeansProjects/trunk/plib/plib-server/target/basex/http HTTPPORT: 8984 KEEPALIVE: 600 LANG: English LANGKEYS: false NONPROXYHOSTS: PARALLEL: 8 PORT: 1984 PROXYHOST: PROXYPORT: 80 REPOPATH: /home/dm218/NetBeansProjects/trunk/plib/plib-server/target/basex/repo SERVERHOST: SERVERPORT: 1984 STOPPORT: 8985 TIMEOUT: 30
Options ADDARCHIVES: true ADDRAW: false ATTRINDEX: true AUTOFLUSH: true BINDINGS: CACHEQUERY: false CASESENS: false CATFILE: CHOP: true COMPPLAN: true CREATEFILTER: *.xml DIACRITICS: false DOTCOMPACT: false DOTDISPLAY: true DOTPLAN: false DOTTY: dotty DTD: false EXPORTER: FORCECREATE: false FTINDEX: false HTMLOPT: html=false,omit-xml-declaration=false,method=xml,nons=false,nobogons=false,nodefaults=false,nocolons=false,norestart=false,ignorable=false,emptybogons=false,any=false,norootbogons=false,nocdata=false,lexical=false,encoding=utf-8 INTPARSE: true LANGUAGE: en LSERROR: 0 MAINMEM: false MAXCATS: 100 MAXHITS: -1 MAXLEN: 96 MAXSTAT: 30 PARSER: xml PARSEROPT: encoding=UTF-8,flat=false,format=verbose,header=false,jsonml=false,lines=true,separator=comma QUERYINFO: false QUERYPATH: RUNS: 1 SERIALIZE: true SERIALIZER: SKIPCORRUPT: false STEMMING: false STOPWORDS: TAILCALLS: 42 TEXTINDEX: true UPDINDEX: false WRITEBACK: false XMLPLAN: false XQUERY3: true ==============================
2 --- I also tried using 7.4-SNAPSHOT (downloaded this morning) but it shows the same behaviour. It reports (I'm copying this by hand from another PC):
========================== Unexpected error: org.basex.util.Util: java.lang.RunTimeException: Data Access out of bounds:
- Pre value: 7290
- #used blocks: 31
- #total blocks: 31
- Access 30 (31 > 30) ==========================
Some more remarks:
- I would much appreciate a wiki-per-version, e.g. a 7.3 wiki, a 7.4 wiki, etc. The adaptations for a new version are slightly confusing - BaseX changes a lot (new options, renames, what option to put in what config file, etc.). Also for people using an older version this can be handy.
- the mailing list search service on the basex.org website (links from the "Open Source" page) crashes.
- (7.4) I am confused about the org.basex.restxqpath vs. org.basex.webpath settings, I put them both in and then it seemed to work. (Perhaps db:system() is also inconsistent wrt these settings.) RESTXQPATH is not mentioned in the OPTIONS list.
- (7.4) perhaps I missed it and the math:uuid() function was deprecated already, but I found that I needed to adapt my .xqm script for 7.4 because math:uuid() was renamed to random:uuid(). It seemed a little abrupt to me, but as I said, perhaps it missed the deprecation. I also used db:system() and had to adapt that too because org.basex.httppath no longer exists.
I suspect I missed some configuration option or something like that and will appreciate any help!
Thanks,
Huib Verweij.
------------------------------------------------------------------------------------------------------------ Disclaimer:
If you are not the intended recipient of this email, please notify the sender and delete it. Any unauthorized copying, disclosure or distribution of this email or its attachment(s) is forbidden. Thales Nederland BV will not accept liability for any damage caused by this email or its attachment(s). Thales Nederland BV is seated in Hengelo and is registered at the Chamber of Commerce under number 06061578. ------------------------------------------------------------------------------------------------------------