Here's a reproducible test case:

http://theta.cs.vt.edu/~gback/basex73bugs/basex73bug.zip

Download, extract, run 'reproduce.sh' and see the ill-formed XML.

- Godmar

On Wed, Jul 11, 2012 at 1:43 PM, Christian Grün christian.gruen@gmail.comwrote:

...

let me know if you can fix this with the information provided, or if you need a complete test case.

…always helpful.

...

Is this issue fixed in the last snapshot?

…just try ;)

thanks, christian

Thanks! I'll remove the String.replace() work-around I had.

Any insights on the issue with BaseX randomly locking databases and then failing CREATE DB/ALTER DB operations?

- Godmar

On Tue, Jul 24, 2012 at 6:16 AM, Christian Grün christian.gruen@gmail.comwrote:

Hi Godmar,

the ft:extract() namespace issue is now fixed. Feel free to check out the latest snapshot [1].

Christian

[1] https://github.com/BaseXdb/basex/issues/539 ____________________________________

...

BaseX returns this (ill-formed) XML:

<result-extract>
  <libx:body xmlns:libx="http://libx.org/xml/libx2"

xmlns="http://www.w3.org/2005/Atom%22%3E... that searches by <mark xmlns="http://libx.org/xml/libx2" xmlns="http://www.w3.org/2005/Atom">ISBN</mark> */ var cue = new libx.cues.CatalogCue('i', tuple.<mark xmlns="http://libx.org/xml/libx2" xmlns="http://www.w3.org/2005/Atom">isbn</mark>); <mark xmlns="http://libx.org/xml/libx2" xmlns="

http://www.w3.org/2005/Atom%22%3E

...

</mark>cue.insertBefore(tuple.position); cue.image.setAttribute('width', '16'...</libx:body> </result-extract>

resulting in:

[Fatal Error] :61:237: Attribute "xmlns" bound to namespace "http://www.w3.org/2005/Atom/" was already specified for element "mark".

The wrong XML appears numerous times inside <mark> tag. A <mark> element should not have more than one xmlns= attribute (why it has any, btw, I

don't

...

understand).

The ill-formed output appears to be produced by the 'ft:extract()'

function;

...

here's an excerpt from the XQuery:

for $extract in $entry/descendant::*[text() contains text "ISBN"] return <result-extract>{ft:extract($extract[text() contains text "ISBN"])}</result-extract>

This is a show stopper that prevents us from upgrading to 7.3, obviously

...

let me know if you can fix this with the information provided, or if you need a complete test case.

Is this issue fixed in the last snapshot?

• Godmar

---

BaseX-Talk mailing list BaseX-Talk@mailman.uni-konstanz.de https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk

On Tue, Jul 24, 2012 at 9:40 AM, Christian Grün christian.gruen@gmail.comwrote:

...

Any insights on the issue with BaseX randomly locking databases and then failing CREATE DB/ALTER DB operations?

I noticed your mail, but it's difficult to give a general assumption on what might be the problem. I'm aware of many existing use cases in which databases are created and dropped, and queries are sent.

As databass are implicitly locked if they are accessed by a client [1], you may need to explicitly close sessions in order to avoid that opened databases will not be properly closed.

I am very confused. I never explicitly "open" a database. I use sessions that issue a series of queries & commands, such as 'XQUERY', 'CREATE DB', and the like. I don't issue any 'OPEN' or 'CLOSE' commands.

I note that we are using the client/server architecture. A single server, with multiple client sessions. Each client session issues 'XQUERY', etc. commands via the org.basex. Closing a session simply disconnects - why should it have any effect on the db state?

Let me look at the code to illustrate the failure...

The message we're talking about is org.basex.core.BaseXException: Database 'PUB_feed_libx_dot_editions_at_gmail_dot_com_core' is currently opened by another client.

That's 'db_pinned_%' in your code... DB_PINNED_X in Text.java...

Happens during createDB:

./src/main/java/org/basex/core/cmd/CreateDB.java:92: if(context.pinned(name)) return error(DB_PINNED_X, name); ./src/main/java/org/basex/core/cmd/CreateDB.java:152: if(ctx.pinned(name)) throw new BaseXException(DB_PINNED_X, name); another client.

I'm guessing it's line 92.... Looking at context.pinned:

public boolean pinned(final String db) { return datas.pinned(db) || TableDiskAccess.locked(db, this); }

Leads to Datas.java.... and TableDiskAccess.java Datas.java simply records in memory which dbs are 'pinned' and how often. TableDiskAccess.java does physical file locking.

Looking for calls to 'unpin' ... finds one in Close.java:

/** * Evaluates the 'close' command and closes the current database. *

So it appears that BaseX expects database accesses to be wrapped in a OPEN...CLOSE brace. We don't issue neither OPEN nor CLOSE... so I'm guessing it's done internally as part of XQUERY?

Suppose one client run an XQUERY against a db, which involves (internally) one or more 'OPEN' calls, then accesses, then 'CLOSE' commands --- how do you synchronize this client with others? Do you have a master lock so that the server processes only one query/command at a time?

If so, that would be bad for performance... if not, your assumption that you can throw an error in CreateDB.run if the db is pinned is wrong. Instead, you'll need a 'wait_until_unpinned' method that CreateDB calls until the db it's trying to create is no longer pinned. It would need to wait on a signaling device that's signaled in unpin(). (Note that we are recreating a database with the same name over and over, even while some clients perform XQUERY on the then-current version of that database.) This mode of failure would, in essence, be a race condition that would manifest itself if there happened to be a read access to the db by one client while another is trying to create a db with the same name.

I actually don't think that's the failure we're seeing, based on the fact that restarting the server allows the operation to succeed. The latter would point to not calling 'close()' on some path in a previous db operation, leading to a stale entry in the datas. list.

- Godmar

If you should manage to create a test case, that would be great.

Christian

[1] http://docs.basex.org/wiki/Transaction_Management#Locking

On Tue, Jul 24, 2012 at 11:20 AM, Christian Grün christian.gruen@gmail.comwrote:

...

I am very confused. I never explicitly "open" a database. I use sessions that issue a series of queries & commands, such as 'XQUERY', 'CREATE DB', and the like. I don't issue any 'OPEN' or 'CLOSE' commands.

As you already guessed, the XQUERY commands is also capable of opening databases (otherwise, there would be no chance to access data inside those instances).

...

Suppose one client run an XQUERY against a db, which involves

(internally)

...

one or more 'OPEN' calls, then accesses, then 'CLOSE' commands --- how do you synchronize this client with others? Do you have a master lock so

that

...

the server processes only one query/command at a time?

The pin counter is used to find out how often a particular database is opened. Our architecture supports multiple read and single write operations at the same time [1].

...

if not, your assumption that you can throw an error in CreateDB.run if

the db is pinned is wrong. Instead, you'll need a 'wait_until_unpinned' method [...]

Well, it's tedious to guess what might be the problem as long as we can't reproduce your particular use case. If you believe you have some concrete ideas what might be the problem, though, feel free to check out the sources and give it a try.

I cannot make a reproducible test case, at least not yet, because of the nature of the failure, so I'm trying to help you make sure your design is correct.

If, indeed, you have a multiple-reader, single-writer design, *AND* if you correctly treat 'CREATE DB' as a write operation requiring exclusive access, then your design would be correct, and the failure I'm seeing shouldn't occur in client/server mode. (I can't easily find where you implement this, though, considering that 'pin' doesn't have a flag read-only or read/write - so I'm assuming the multiple-reader, single-writer lock is not done on a per-db basis.) Like I said, given the way we're accessing our db, this is a less likely mode of failure.

My guess is that you permanently fail to close a db after an XQUERY operation, leaving it in the pinned state. I'm not able to quickly find your code where you ensure this - presumably, you'd need some thread-local list of open dbs you'll have to close once a transaction completes or aborts.

- Godmar

ps: on that topic; on your wiki, you write:

"To avoid starvation of any transaction and wrong execution orders the waiting queue works with the FIFO principle ('First-In First-Out'), which states that the first process that arrives at the server will be the first one that will be executed. The FIFO principle cannot be adhered in a group of reading transactions, as they run in different threads and thus can overtake each other."

That claim is wrong - if you allow unrestricted overlapping read transactions, you don't guarantee the absence of starvation. You need to limit overlap of read transactions if you want to be starvation free, else pending writers may never get access.