Extremely serious issue with magically disappearing databases
I have an embedded (soon to be unembedded) Base-X database supporting a website written in Scala. I have a very basic caching mechanism. The structure of the DB is such: <data> <cache> <page>...</page> <page>...</page> </cache> <somethings> <something>...</something> </somethings> <otherthings> <otherthing>...</otherthing> </otherthings> </data> I have a huge (1000+ lines) XQuery that takes all the somethings and otherthings recursively and constructs a <page> item that includes the html content of the page thus: <page uri="/" expires=""> <name>home</name> <content> <html> <head>...</head> <body>...</body> </html> </content> </page> This page is then saved into /data/cache. The way this works is simple. The main XQuery checks /data/cache/page[@uri=$uri] If it finds an unexpired page, it returns it. If it doesn't, it looks to see if such a page exists (in pieces) in the DB. If it does, it constructs the html page, inserts it into a <page> element along with some metadata (expires, etc.) and returns that. If no such page exists, it returns a 404 error page. Beauty, eh? And it works extremely well. Until I have an error in my XQuery, which, naturally, happens all the time during development. Just to make clear, here is the Scala that runs the query (from a Circumflex web app): get("(.)*".r) = { response.contentType("text/html") val url = uri.get(0).getOrElse("/") val out = executeQuery(Rx.pageQry(url)) match { case "" => "<p>Houston, we have a problem.</p>" case page if (page.substring(0,2) == "<p") => executeQuery("delete node /data/cache/page[@uri='" + url + "']") executeQuery("insert node " + page + " into /data/cache") executeQuery(Rx.pageQry(url)) case out => out } "<!DOCTYPE html>\n" + out } What this does is very simple. executeQuery simply runs the supplied query. Rx.pageQry just loads the huge query from a file and adds the call "local:getPage('/')" with the url inserted as a parameter. Right? Then the match looks to see if nothing at all comes back (case ""), or if a page that begins "<p" comes back (meaning a newly constructed query). The default is to spit the thing out. So in case one, we're screwed. In the default case, we serve the right page *or* a 404 error page. And in case two, we get the <page> for caching. In that instance, we delete any old page that might be in the cache with that URI, then insert the new page. Then we pull the html from the newly inserted page and serve that. I hope this is clear even if you don't write Scala. HERE IS THE PROBLEM: When I make an error in the big XQuery (the one loaded by Rx.pageQry), I get an error message from Base-X with the line number and the type of error. Beauty! So I fix the error and rerun the query AND MY DATABASE IS EMPTY. That's right: the entire database has been wiped. Running a / query gives me back <data> </data>. OUCH! If this happens in production (and this app is running in production right now), I might as well step in front of a bus. What, please, could be causing the database to dump itself? This has happened dozens of times! I'm willing to send the huge query off-list if anyone thinks that would help, but I don't see how the query *failing* and destroying the database could have anything to do with the query itself. I've looked at the Scala a million times and maybe I'm missing something, but the delete XQuery is /data/cache/page[etc.]. How can this destroy anything outside of /data/cache ? As for the insert query, how can inserting *anything* into /data/cache destroy things outside of /data/cache ? Am I missing something really basic, or is there a nasty bug in Base-X? Hopefully the former, but if it's the latter, I thought you should know. Could this be related to the embedded nature of the database? Thanks! Chas.
Hi Chas, the best would be if you could send me/us a working example (like the xml and the query).
and this app is running in production right now thats nice to hear, we will help you getting it to work.
Kind regards, Andreas Am 14.01.11 00:01, schrieb Charles F. Munat:
I have an embedded (soon to be unembedded) Base-X database supporting a website written in Scala. I have a very basic caching mechanism.
The structure of the DB is such:
<data> <cache> <page>...</page> <page>...</page> </cache> <somethings> <something>...</something> </somethings> <otherthings> <otherthing>...</otherthing> </otherthings> </data>
I have a huge (1000+ lines) XQuery that takes all the somethings and otherthings recursively and constructs a <page> item that includes the html content of the page thus:
<page uri="/" expires=""> <name>home</name> <content> <html> <head>...</head> <body>...</body> </html> </content> </page>
This page is then saved into /data/cache.
The way this works is simple. The main XQuery checks
/data/cache/page[@uri=$uri]
If it finds an unexpired page, it returns it. If it doesn't, it looks to see if such a page exists (in pieces) in the DB. If it does, it constructs the html page, inserts it into a <page> element along with some metadata (expires, etc.) and returns that. If no such page exists, it returns a 404 error page.
Beauty, eh?
And it works extremely well. Until I have an error in my XQuery, which, naturally, happens all the time during development.
Just to make clear, here is the Scala that runs the query (from a Circumflex web app):
get("(.)*".r) = { response.contentType("text/html")
val url = uri.get(0).getOrElse("/")
val out = executeQuery(Rx.pageQry(url)) match { case "" => "<p>Houston, we have a problem.</p>" case page if (page.substring(0,2) == "<p") => executeQuery("delete node /data/cache/page[@uri='" + url + "']") executeQuery("insert node " + page + " into /data/cache") executeQuery(Rx.pageQry(url)) case out => out }
"<!DOCTYPE html>\n" + out }
What this does is very simple. executeQuery simply runs the supplied query. Rx.pageQry just loads the huge query from a file and adds the call "local:getPage('/')" with the url inserted as a parameter. Right?
Then the match looks to see if nothing at all comes back (case ""), or if a page that begins "<p" comes back (meaning a newly constructed query). The default is to spit the thing out. So in case one, we're screwed. In the default case, we serve the right page *or* a 404 error page. And in case two, we get the <page> for caching. In that instance, we delete any old page that might be in the cache with that URI, then insert the new page. Then we pull the html from the newly inserted page and serve that. I hope this is clear even if you don't write Scala.
HERE IS THE PROBLEM:
When I make an error in the big XQuery (the one loaded by Rx.pageQry), I get an error message from Base-X with the line number and the type of error. Beauty! So I fix the error and rerun the query AND MY DATABASE IS EMPTY.
That's right: the entire database has been wiped. Running a / query gives me back <data> </data>. OUCH!
If this happens in production (and this app is running in production right now), I might as well step in front of a bus.
What, please, could be causing the database to dump itself? This has happened dozens of times! I'm willing to send the huge query off-list if anyone thinks that would help, but I don't see how the query *failing* and destroying the database could have anything to do with the query itself.
I've looked at the Scala a million times and maybe I'm missing something, but the delete XQuery is /data/cache/page[etc.]. How can this destroy anything outside of /data/cache ? As for the insert query, how can inserting *anything* into /data/cache destroy things outside of /data/cache ?
Am I missing something really basic, or is there a nasty bug in Base-X? Hopefully the former, but if it's the latter, I thought you should know. Could this be related to the embedded nature of the database?
Thanks!
Chas. _______________________________________________ BaseX-Talk mailing list BaseX-Talk@mailman.uni-konstanz.de https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk
This issue has arised by not closing the embedded database and running out of file handles. It has been solved by rewriting the code by the author and works now brilliant. Am 14.01.11 09:15, schrieb Andreas Weiler:
Hi Chas,
the best would be if you could send me/us a working example (like the xml and the query).
and this app is running in production right now thats nice to hear, we will help you getting it to work.
Kind regards, Andreas
Am 14.01.11 00:01, schrieb Charles F. Munat:
I have an embedded (soon to be unembedded) Base-X database supporting a website written in Scala. I have a very basic caching mechanism.
The structure of the DB is such:
<data> <cache> <page>...</page> <page>...</page> </cache> <somethings> <something>...</something> </somethings> <otherthings> <otherthing>...</otherthing> </otherthings> </data>
I have a huge (1000+ lines) XQuery that takes all the somethings and otherthings recursively and constructs a <page> item that includes the html content of the page thus:
<page uri="/" expires=""> <name>home</name> <content> <html> <head>...</head> <body>...</body> </html> </content> </page>
This page is then saved into /data/cache.
The way this works is simple. The main XQuery checks
/data/cache/page[@uri=$uri]
If it finds an unexpired page, it returns it. If it doesn't, it looks to see if such a page exists (in pieces) in the DB. If it does, it constructs the html page, inserts it into a <page> element along with some metadata (expires, etc.) and returns that. If no such page exists, it returns a 404 error page.
Beauty, eh?
And it works extremely well. Until I have an error in my XQuery, which, naturally, happens all the time during development.
Just to make clear, here is the Scala that runs the query (from a Circumflex web app):
get("(.)*".r) = { response.contentType("text/html")
val url = uri.get(0).getOrElse("/")
val out = executeQuery(Rx.pageQry(url)) match { case "" => "<p>Houston, we have a problem.</p>" case page if (page.substring(0,2) == "<p") => executeQuery("delete node /data/cache/page[@uri='" + url + "']") executeQuery("insert node " + page + " into /data/cache") executeQuery(Rx.pageQry(url)) case out => out }
"<!DOCTYPE html>\n" + out }
What this does is very simple. executeQuery simply runs the supplied query. Rx.pageQry just loads the huge query from a file and adds the call "local:getPage('/')" with the url inserted as a parameter. Right?
Then the match looks to see if nothing at all comes back (case ""), or if a page that begins "<p" comes back (meaning a newly constructed query). The default is to spit the thing out. So in case one, we're screwed. In the default case, we serve the right page *or* a 404 error page. And in case two, we get the <page> for caching. In that instance, we delete any old page that might be in the cache with that URI, then insert the new page. Then we pull the html from the newly inserted page and serve that. I hope this is clear even if you don't write Scala.
HERE IS THE PROBLEM:
When I make an error in the big XQuery (the one loaded by Rx.pageQry), I get an error message from Base-X with the line number and the type of error. Beauty! So I fix the error and rerun the query AND MY DATABASE IS EMPTY.
That's right: the entire database has been wiped. Running a / query gives me back <data> </data>. OUCH!
If this happens in production (and this app is running in production right now), I might as well step in front of a bus.
What, please, could be causing the database to dump itself? This has happened dozens of times! I'm willing to send the huge query off-list if anyone thinks that would help, but I don't see how the query *failing* and destroying the database could have anything to do with the query itself.
I've looked at the Scala a million times and maybe I'm missing something, but the delete XQuery is /data/cache/page[etc.]. How can this destroy anything outside of /data/cache ? As for the insert query, how can inserting *anything* into /data/cache destroy things outside of /data/cache ?
Am I missing something really basic, or is there a nasty bug in Base-X? Hopefully the former, but if it's the latter, I thought you should know. Could this be related to the embedded nature of the database?
Thanks!
Chas. _______________________________________________ BaseX-Talk mailing list BaseX-Talk@mailman.uni-konstanz.de https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk
_______________________________________________ BaseX-Talk mailing list BaseX-Talk@mailman.uni-konstanz.de https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk
participants (2)
-
Andreas Weiler -
Charles F. Munat