BaseX-Talk

basex-talk@mailman.uni-konstanz.de

1 participants
5252 discussions

Re: [basex-talk] Database corruption
by Antoni Mylka 14 Nov '12

14 Nov '12

Hi, Two further questions: 1. What's the cleanest way to get a log containing the actual query strings? AFAIU BaseX doesn't use any logging framework and the BaseXServlet.service has a finally block that prints the request headers (if Prop.debug is enabled) but not the request body. I thought about writing my own Filter for that purpose. 2. I've been trying to update BaseX to the latest snapshot, but the queries generated by my application, that used to work with 7.2.1 and 7.3 don't seem to work now. The body of a POST looks like this (\n means a newline character): <?xml version="1.0" encoding="UTF-8"?>\n <command xmlns="http://basex.org/rest"><text><![CDATA[\n ADD TO 5d03ebad-dc6a-492c-9527-bd2ce3830080\n <tx id="5d03ebad-dc6a-492c-9527-bd2ce3830080"\n dbId=""\n xmlns="http://our.namespace.com/xmlns/"\n xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"\n xmlns:xs="http://www.w3.org/2001/XMLSchema"\n <commit />\n <skip />\n </tx>\n \n ]]></text></command> Now the question is what's wrong with this query. I get a 400 Bad Request response with the following error message. Stopped at line 1, column 43:\n Syntax: ADD (TO [path]) [input]\n Add resource to database.\n \n Adds [input] as [path] to the database.\n As I said this used to work with 7.3. I tried to diff the StringParser class between 7.3 and the current trunk. It seems to use the InputParser on a single line and not on the entire query. I can't spot anything that would explain this behaviour though. All kinds of comments welcome. -- Antoni Myłka Software Engineer basis06 AG, Birkenweg 61, CH-3013 Bern - Fon +41 31 311 32 22 http://www.basis06.ch - source of smart business -- Antoni Myłka Software Engineer basis06 AG, Birkenweg 61, CH-3013 Bern - Fon +41 31 311 32 22 http://www.basis06.ch - source of smart business ----- Ursprüngliche Mail ----- Von: "Antoni Mylka" <amy(a)basis06.ch> An: "Christian Grün" <christian.gruen(a)gmail.com> Gesendet: Montag, 12. November 2012 15:42:42 Betreff: Re: [basex-talk] Database corruption Dear Christian, Thanks for the tip. Will fix the logging settings and retest with the latest snapshot. Best regards, -- Antoni Myłka Software Engineer basis06 AG, Birkenweg 61, CH-3013 Bern - Fon +41 31 311 32 22 http://www.basis06.ch - source of smart business ----- Ursprüngliche Mail ----- Von: "Christian Grün" <christian.gruen(a)gmail.com> An: "Antoni Mylka" <amy(a)basis06.ch> CC: "basex-talk" <basex-talk(a)mailman.uni-konstanz.de> Gesendet: Montag, 12. November 2012 15:34:23 Betreff: Re: [basex-talk] Database corruption Dear Antoni, without having read all the elaborate details in your mail (thanks), I would strongly advise to try the latest stable 7.5 snapshot [1], which contains some important improvements reg. concurrent updates. I'm well aware that it feels inconvenient to use beta releases in productive environments, but the current version of 7.5 is very stable, and will be released soon without any major enhancements or legacy issues. Hope this helps, Christian [1] http://files.basex.org/releases/latest/ ___________________________ On Mon, Nov 12, 2012 at 3:30 PM, Antoni Mylka <amy(a)basis06.ch> wrote: > Hi, > > We're developing an application that uses BaseX. In our tests, the application works properly for a couple of hours, makes queries to BaseX adds and removes data. Then suddenly some BaseX request will return the HTTP 500 response. From that point on all queries that make modifications in the database end in an error and the entire system grinds to a screeching halt. One of the experiments ran for four days, then we examined the access log to see that only the first 6 hours were OK, after that all BaseX queries would end in error. > > I'm going to investigate and will try to come up with as detailed a description of the issue as possible. For now the things I know are: > > 1. In our experiment, the application would make about 80000 successful queries over a period of 6 hours. Then it would get the first 500 error. > 2. Afterwards it "seems" that the queries which don't modify anything end with 200, while all the modification queries end with a 500 error. That's to be confirmed. > 3. BaseX 7.2.1 runs in a war deployed on Tomcat 7.0.32 on a linux system. > 4. The access logs don't show anything interesting. > 5. The catalina.out that shows the Tomcat standard output contains hundreds of exceptions of the following kind: > > Exception in thread "Thread-78197" java.lang.RuntimeException: Data Access out of bounds [pre:1422612, indexSize:5540, access:5540 > 5539] > at org.basex.util.Util.notexpected(Util.java:55) > at org.basex.io.random.TableDiskAccess.cursor(TableDiskAccess.java:478) > at org.basex.io.random.TableDiskAccess.read5(TableDiskAccess.java:211) > at org.basex.data.Data.textOff(Data.java:432) > at org.basex.data.DiskData.text(DiskData.java:228) > at org.basex.index.value.DiskValues.get(DiskValues.java:295) > at org.basex.index.value.UpdatableDiskValues.index(UpdatableDiskValues.java:133) > at org.basex.data.DiskData$1.run(DiskData.java:461) > > The thread name is different every time and the numbers in square brackets are also different. I don't see any patterns (yet). > 6. BaseX is accessed from a single thread in a single application. AFAIK there are no concurrent requests, though that is to be confirmed. > 7. The GC log shows that basex stays below 100M of used heap, which is well below the allotted Xmx setting of 2G. > 8. We did three experiments and the same effect has been observed in all three, albeit after different periods of time: (59K queries and 4:25 hours, 37K queries and 2:56 hours, 76.8K queries and 6 hours). > > Things I intend to do: > 1. Set up proper logging for BaseX, so that these errors don't end up in catalina.out, but in the proper log file. > 2. Catch the first offending query, see if there is anything unusual about it. It certainly doesn't differ much from the successful queries. It simply updates the BaseX state to keep it in sync with an external data source. All the update queries are generated by the same template and there have been hundreds of successful ones before the first error occurs. > 3. Manipulate the log settings, to see if it's possible to get something more meaningful out of them. > 4. Try newer versions: 7.3 and the current trunk. > > I've seen emails similar about "Data Access out of bounds" but they don't seem relevant. > > As I said, I'll try to come up with more information, but if the error or the stack trace looks familiar - I'd be grateful for pointers. > > Thanks in advance, > > -- > Antoni Myłka > Software Engineer > > basis06 AG, Birkenweg 61, CH-3013 Bern - Fon +41 31 311 32 22 > http://www.basis06.ch - source of smart business > > _______________________________________________ > BaseX-Talk mailing list > BaseX-Talk(a)mailman.uni-konstanz.de > https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk

2 7

xquery sql module init driver
by Jarosław Kowalewski 14 Nov '12

14 Nov '12

Hi, I try to connect with my database directly by sql module. sql:init("com.mysql.jdbc.Driver"), let $conn := sql:connect("jdbc:mysql://dbhost:port/dbname") return sql:execute($conn, "SELECT * FROM coffees WHERE price < 10") but I've problem with and I get error "[BXSQ0007] Could not initialize specified driver: 'com.mysql.jdbc.Driver'" java -cp pathto/mysql-connector-java-5.1.21-bin.jar -jar BaseX73.jar Thanks for a help Jarek

2 1

Re: [basex-talk] (no subject)
by Christian Grün 14 Nov '12

14 Nov '12

Hi John, there is no complete 7.5 available yet, but you may want to check out our documentation (search for 7.5) and the GitHub issue list (limit to Milestone 7.5) in order to get a first impression. Best, Christian Am 14.11.2012 05:18 schrieb "John Best" <johnbest5673(a)gmail.com>:

1 0

(no subject)
by John Best 13 Nov '12

13 Nov '12

Dear BaseX Team What are the NEW features in the coming version and where can I find the list of the same..?? -- Regards -- Have a nice day JBest

1 0

what is the best way to add/replace a lot of documents ?
by Fabrice ETANCHAUD 12 Nov '12

12 Nov '12

Dear all at basex, I am wondering what could be the most efficient way to add/replace a lot of documents (several thousands) at the same time. I tried db:add/db:replace but this gives heap memory overflow because of the pending list... Will I have better results with a .bxs with add/replace commands (and autoflush deactivated) ? Did any other user find his way in this use case ? Best regards, Fabrice

3 2

Function will return node(), but db:output(node()) returns 'Attribute xmlns redefined'
by France Baril 12 Nov '12

12 Nov '12

Hi, My search function used to return $html correctly *let* *$html* := <html *xmlns*=*"http://www.w3.org/1999/xhtml"*> <head><link *href*=*"*{*$cssfile*}*"* *rel*=*"stylesheet"* *type*=* "text/css"*/><title>Search help</title> <script *type*=*"text/javascript"* *xmlns*=*" http://www.w3.org/1999/xhtml"*>{*app:get-google-analytics-script*()/*text* ()}</script> </head> <body> <div> (…) <div>{*$result-list*}</div> </div></body></html> *return $html* I am now trying to keep records of users most popular queries in my DB, so I made the function an updating function and adapted the return value: *let* *$html* := <html *xmlns*=*"http://www.w3.org/1999/xhtml"*> <head><link *href*=*"*{*$cssfile*}*"* *rel*=*"stylesheet"* *type*=* "text/css"*/><title>Search help</title> <script *type*=*"text/javascript"* *xmlns*=*" http://www.w3.org/1999/xhtml"*>{*app:get-google-analytics-script*()/*text* ()}</script> </head> <body> <div> (…) <div>{*$result-list*}</div> </div></body></html> *return *(*feedback:analyze-user-search-and-result*(*$keywords*, * $result-ids*, *$this-menu*, *$application*, *$carrier*, *$country*, *$lang*, *$model*, *$os*), *db:output*(*$html*)) I got this error: error on line 30 at column 25: Attribute xmlns redefined After lots of research, I figured that when using db:output, the top element of $result-list is interpreted as coming from a different namespace. Adding xmlns="http://www.w3.org/1999/xhtml" to that top element resolved my issue, but I still don't get why this is happening, can you explain? More specifically, here are definitions of $result-lists and what they produce: let $result-list := <div/> would work when used in the 1st code sample, but not the second (updating function). let $result-list := <div xmlns:="http://www.w3.org/1999/xhtml"/> resolves the issue for the second (updating function). Thanks, France -- France Baril Architecte documentaire / Documentation architect france.baril(a)architextus.com (514) 572-0341

2 4

Database corruption
by Antoni Mylka 12 Nov '12

12 Nov '12

Hi, We're developing an application that uses BaseX. In our tests, the application works properly for a couple of hours, makes queries to BaseX adds and removes data. Then suddenly some BaseX request will return the HTTP 500 response. From that point on all queries that make modifications in the database end in an error and the entire system grinds to a screeching halt. One of the experiments ran for four days, then we examined the access log to see that only the first 6 hours were OK, after that all BaseX queries would end in error. I'm going to investigate and will try to come up with as detailed a description of the issue as possible. For now the things I know are: 1. In our experiment, the application would make about 80000 successful queries over a period of 6 hours. Then it would get the first 500 error. 2. Afterwards it "seems" that the queries which don't modify anything end with 200, while all the modification queries end with a 500 error. That's to be confirmed. 3. BaseX 7.2.1 runs in a war deployed on Tomcat 7.0.32 on a linux system. 4. The access logs don't show anything interesting. 5. The catalina.out that shows the Tomcat standard output contains hundreds of exceptions of the following kind: Exception in thread "Thread-78197" java.lang.RuntimeException: Data Access out of bounds [pre:1422612, indexSize:5540, access:5540 > 5539] at org.basex.util.Util.notexpected(Util.java:55) at org.basex.io.random.TableDiskAccess.cursor(TableDiskAccess.java:478) at org.basex.io.random.TableDiskAccess.read5(TableDiskAccess.java:211) at org.basex.data.Data.textOff(Data.java:432) at org.basex.data.DiskData.text(DiskData.java:228) at org.basex.index.value.DiskValues.get(DiskValues.java:295) at org.basex.index.value.UpdatableDiskValues.index(UpdatableDiskValues.java:133) at org.basex.data.DiskData$1.run(DiskData.java:461) The thread name is different every time and the numbers in square brackets are also different. I don't see any patterns (yet). 6. BaseX is accessed from a single thread in a single application. AFAIK there are no concurrent requests, though that is to be confirmed. 7. The GC log shows that basex stays below 100M of used heap, which is well below the allotted Xmx setting of 2G. 8. We did three experiments and the same effect has been observed in all three, albeit after different periods of time: (59K queries and 4:25 hours, 37K queries and 2:56 hours, 76.8K queries and 6 hours). Things I intend to do: 1. Set up proper logging for BaseX, so that these errors don't end up in catalina.out, but in the proper log file. 2. Catch the first offending query, see if there is anything unusual about it. It certainly doesn't differ much from the successful queries. It simply updates the BaseX state to keep it in sync with an external data source. All the update queries are generated by the same template and there have been hundreds of successful ones before the first error occurs. 3. Manipulate the log settings, to see if it's possible to get something more meaningful out of them. 4. Try newer versions: 7.3 and the current trunk. I've seen emails similar about "Data Access out of bounds" but they don't seem relevant. As I said, I'll try to come up with more information, but if the error or the stack trace looks familiar - I'd be grateful for pointers. Thanks in advance, -- Antoni Myłka Software Engineer basis06 AG, Birkenweg 61, CH-3013 Bern - Fon +41 31 311 32 22 http://www.basis06.ch - source of smart business

2 1

Compilation issue
by Antoni Mylka 12 Nov '12

12 Nov '12

Hi, I tried to clone and build the basex maven module today and I stumbled upon two issues. 1. When JavaCC is called, it modifies the RegExParser.java. The modified version contains the following lines: } catch(final ParseException pe) { throw Err.REGERR.thrw(ii, pe.getMessage()); } catch(final TokenMgrError err) { throw Err.REGERR.thrw(ii, err.getMessage()); } this causes a compilation error because the Err enum does not seem to contain the REGERR constant. 2. In German.lang the h_revert_file key had to be renamed to h_reopen_file. Otherwise the LangTest would fail. Am I doing something wrong? Are these issues real? -- Antoni Myłka Software Engineer basis06 AG, Birkenweg 61, CH-3013 Bern - Fon +41 31 311 32 22 http://www.basis06.ch - source of smart business

2 1

running xquery file with -c parameter returns unknown command: for
by bryan rasmussen 12 Nov '12

12 Nov '12

When I use -c to run a file and the file is empty_titles.bsx with the following XQUERY for $doc in db:open("dkcontent") where $doc//TITEL[. =""] return <doc>{$doc/*/@ID}</doc> returns fine. when the file is empty_titles.xq with the following: for $doc in db:open("dkcontent") where $doc//TITEL[. =""] return <doc>{$doc/*/@ID}</doc> I get Stopped at line 1, column 3: Unknown command: for. Try HELP. This is on Windows using basexclient. Thanks, Bryan Rasmussen

2 1

Re: [basex-talk] basex RUN local script when talking to Remote server
by Christian Grün 12 Nov '12

12 Nov '12

Hi Bryan (cc @ basex-talk), thanks for the confirmation. Yes, certain combinations of command-line arguments along with "-o" caused a buggy behavior. As usual, I've updated the latest snapshot containing the fix [1]. By the way, please note that all arguments will be evaluated from left to right in BaseX. This way, you can use a single BaseX call to write your results to several files, as e.g. shown here: basex -ofile1.txt -q1 -ofile2.txt -c"xquery 1 to 10" While this behavior may seem surprising at first glance, it has proven to be very helpful in our everyday's use. I tried to make this a little bit more explicit in our documentation [2]. Hope this helps, Christian [1] http://files.basex.org/releases/latest/ [2] http://docs.basex.org/wiki/Startup_Options

1 0

Jump to page:

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

BaseX-Talk