Hello Colin,
this really sounds like a bug on our side to me. I opened a bug report on https://github.com/BaseXdb/basex/issues/1434 and I am sure Christian will quickly fix it when he gets back from a short vacation.
Cheers
Dirk
On 03/08/2017 06:20 PM, Colin Delacroix wrote:
Hi everyone,
(warning this is a long read)
I'm using server version 8.6.1 and writing a Java client to move lots of documents from a SQL database to a BaseX database. The basic version works just fine. Tested up to 1M documents.
Now I'm trying to do exactly the same but with AUTOFLUSH disabled + manually FLUSH every N documents and it fails after just a few inserts.
Pseudo code: (set up client session, create and open new database) session.execute(new Set(MainOptions.AUTOFLUSH, false)); int i = 0; while(doc = nextDocs()) { session.add(doc.getKey(), doc.getInputStream()); if(i++ % 10 == 0) { /* value reduced to 10 for testing purposes */ session.execute(new Flush()); } } (tear down session)
Remember tha
After the first FLUSH has been issued, the ADD directive starts randomly throwing a org.basex.core.BaseXException (when using the BaseXClient implementation it was its inner ok() method that started returning false).
I tracked the root cause in the server: the SAXParser complaining about the documents not being well-formed (the exact same documents are all imported fine with AUTOFLUSH on). In the server standard logs I can see why: parts of those documents are suddenly missing (usually the beginning + command part) at parse time!
That is the symptom, now for the cause I'm also observing that the ADD and FLUSH directive in the logs are out of order, exactly like what would happen in the FLUSH commands were not properly queued.
I was expecting something like : ADD ADD ADD (N times) FLUSH ADD ADD ADD (N times) FLUSH etc.
Instead I'm observing ADD FLUSH ADD FLUSH ADD FLUSH (etc) then ADD ADD ADD (lots of them) ERROR
So for now I'm keeping autoflush on :) but is there anything obvious I could try there? (server runs on linux openjdk 1.8.0_91, client runs in Tomcat on Windows Oracle JDK 1.8.0_121)
Thanks, Colin