Hello Colin,

this really sounds like a bug on our side to me. I opened a bug report
on https://github.com/BaseXdb/basex/issues/1434 and I am sure Christian
will quickly fix it when he gets back from a short vacation.

Cheers

Dirk

On 03/08/2017 06:20 PM, Colin Delacroix wrote:
> Hi everyone,
>
> (warning this is a long read)
>
> I'm using server version 8.6.1 and writing a Java client to move lots
> of documents from a SQL database to a BaseX database.
> The basic version works just fine. Tested up to 1M documents.
>
> Now I'm trying to do exactly the same but with AUTOFLUSH disabled +
> manually FLUSH every N documents and it fails after just a few inserts.
>
> Pseudo code:
> (set up client session, create and open new database)
> session.execute(new Set(MainOptions.AUTOFLUSH, false));
> int i = 0;
> while(doc = nextDocs()) {
>   session.add(doc.getKey(), doc.getInputStream());
>   if(i++ % 10 == 0) { /* value reduced to 10 for testing purposes */
>     session.execute(new Flush());
>   }
> }
> (tear down session)
>
> Remember tha
>
> After the first FLUSH has been issued, the ADD  directive starts
> randomly throwing a org.basex.core.BaseXException (when using the
> BaseXClient implementation it was its inner ok() method that started
> returning false).
>
> I tracked the root cause in the server: the SAXParser complaining
> about the documents not being well-formed (the exact same documents
> are all imported fine with AUTOFLUSH on). In the server standard logs
> I can see why: parts of those documents are suddenly missing (usually
> the beginning + command part) at parse time!
>
> That is the symptom, now for the cause I'm also observing that the ADD
> and FLUSH directive in the logs are out of order, exactly like what
> would happen in the FLUSH commands were not properly queued.
>
> I was expecting something like :
> ADD
> ADD
> ADD
> (N times)
> FLUSH
> ADD
> ADD
> ADD
> (N times)
> FLUSH
> etc.
>
> Instead I'm observing
> ADD
> FLUSH
> ADD
> FLUSH
> ADD
> FLUSH
> (etc)
> then
> ADD
> ADD
> ADD
> (lots of them)
> ERROR
>
>
> So for now I'm keeping autoflush on :) but is there anything obvious I
> could try there?
> (server runs on linux openjdk 1.8.0_91, client runs in Tomcat on
> Windows Oracle JDK 1.8.0_121)
>
> Thanks,
> Colin
>

--
Dirk Kirsten, BaseX GmbH, http://basexgmbh.de
|-- Firmensitz: Blarerstrasse 56, 78462 Konstanz
|-- Registergericht Freiburg, HRB: 708285, Geschäftsführer:
|   Dr. Christian Grün, Dr. Alexander Holupirek, Michael Seiferle
`-- Phone: 0049 7531 91 68 276, Fax: 0049 7531 20 05 22