Hi everyone,
(warning this is a long read)
I'm using server version 8.6.1 and writing a Java client to move lots of documents from a SQL database to a BaseX database. The basic version works just fine. Tested up to 1M documents.
Now I'm trying to do exactly the same but with AUTOFLUSH disabled + manually FLUSH every N documents and it fails after just a few inserts.
Pseudo code: (set up client session, create and open new database) session.execute(new Set(MainOptions.AUTOFLUSH, false)); int i = 0; while(doc = nextDocs()) { session.add(doc.getKey(), doc.getInputStream()); if(i++ % 10 == 0) { /* value reduced to 10 for testing purposes */ session.execute(new Flush()); } } (tear down session)
Remember tha
After the first FLUSH has been issued, the ADD directive starts randomly throwing a org.basex.core.BaseXException (when using the BaseXClient implementation it was its inner ok() method that started returning false).
I tracked the root cause in the server: the SAXParser complaining about the documents not being well-formed (the exact same documents are all imported fine with AUTOFLUSH on). In the server standard logs I can see why: parts of those documents are suddenly missing (usually the beginning + command part) at parse time!
That is the symptom, now for the cause I'm also observing that the ADD and FLUSH directive in the logs are out of order, exactly like what would happen in the FLUSH commands were not properly queued.
I was expecting something like : ADD ADD ADD (N times) FLUSH ADD ADD ADD (N times) FLUSH etc.
Instead I'm observing ADD FLUSH ADD FLUSH ADD FLUSH (etc) then ADD ADD ADD (lots of them) ERROR
So for now I'm keeping autoflush on :) but is there anything obvious I could try there? (server runs on linux openjdk 1.8.0_91, client runs in Tomcat on Windows Oracle JDK 1.8.0_121)
Thanks, Colin
Hello Colin,
this really sounds like a bug on our side to me. I opened a bug report on https://github.com/BaseXdb/basex/issues/1434 and I am sure Christian will quickly fix it when he gets back from a short vacation.
Cheers
Dirk
On 03/08/2017 06:20 PM, Colin Delacroix wrote:
Hi everyone,
(warning this is a long read)
I'm using server version 8.6.1 and writing a Java client to move lots of documents from a SQL database to a BaseX database. The basic version works just fine. Tested up to 1M documents.
Now I'm trying to do exactly the same but with AUTOFLUSH disabled + manually FLUSH every N documents and it fails after just a few inserts.
Pseudo code: (set up client session, create and open new database) session.execute(new Set(MainOptions.AUTOFLUSH, false)); int i = 0; while(doc = nextDocs()) { session.add(doc.getKey(), doc.getInputStream()); if(i++ % 10 == 0) { /* value reduced to 10 for testing purposes */ session.execute(new Flush()); } } (tear down session)
Remember tha
After the first FLUSH has been issued, the ADD directive starts randomly throwing a org.basex.core.BaseXException (when using the BaseXClient implementation it was its inner ok() method that started returning false).
I tracked the root cause in the server: the SAXParser complaining about the documents not being well-formed (the exact same documents are all imported fine with AUTOFLUSH on). In the server standard logs I can see why: parts of those documents are suddenly missing (usually the beginning + command part) at parse time!
That is the symptom, now for the cause I'm also observing that the ADD and FLUSH directive in the logs are out of order, exactly like what would happen in the FLUSH commands were not properly queued.
I was expecting something like : ADD ADD ADD (N times) FLUSH ADD ADD ADD (N times) FLUSH etc.
Instead I'm observing ADD FLUSH ADD FLUSH ADD FLUSH (etc) then ADD ADD ADD (lots of them) ERROR
So for now I'm keeping autoflush on :) but is there anything obvious I could try there? (server runs on linux openjdk 1.8.0_91, client runs in Tomcat on Windows Oracle JDK 1.8.0_121)
Thanks, Colin
Thanks for your quick answer Dirk.
I worked on a test case in the basex-tests module and could not reproduce the issue.
Then back to my code I spotted a concurrency issue that would access the ClientSession object from 2 different threads at the same time; since the class isn't thread-safe that explains the data corruption entirely.
You can close the issue and sorry for the noise.
Cheers, Colin
On Thu, Mar 9, 2017 at 2:03 PM, Dirk Kirsten dk@basex.org wrote:
Hello Colin,
this really sounds like a bug on our side to me. I opened a bug report on https://github.com/BaseXdb/basex/issues/1434 and I am sure Christian will quickly fix it when he gets back from a short vacation.
Cheers
Dirk
On 03/08/2017 06:20 PM, Colin Delacroix wrote:
Hi everyone,
(warning this is a long read)
I'm using server version 8.6.1 and writing a Java client to move lots of documents from a SQL database to a BaseX database. The basic version works just fine. Tested up to 1M documents.
Now I'm trying to do exactly the same but with AUTOFLUSH disabled + manually FLUSH every N documents and it fails after just a few inserts.
Pseudo code: (set up client session, create and open new database) session.execute(new Set(MainOptions.AUTOFLUSH, false)); int i = 0; while(doc = nextDocs()) { session.add(doc.getKey(), doc.getInputStream()); if(i++ % 10 == 0) { /* value reduced to 10 for testing purposes */ session.execute(new Flush()); } } (tear down session)
Remember tha
After the first FLUSH has been issued, the ADD directive starts randomly throwing a org.basex.core.BaseXException (when using the BaseXClient implementation it was its inner ok() method that started returning false).
I tracked the root cause in the server: the SAXParser complaining about the documents not being well-formed (the exact same documents are all imported fine with AUTOFLUSH on). In the server standard logs I can see why: parts of those documents are suddenly missing (usually the beginning + command part) at parse time!
That is the symptom, now for the cause I'm also observing that the ADD and FLUSH directive in the logs are out of order, exactly like what would happen in the FLUSH commands were not properly queued.
I was expecting something like : ADD ADD ADD (N times) FLUSH ADD ADD ADD (N times) FLUSH etc.
Instead I'm observing ADD FLUSH ADD FLUSH ADD FLUSH (etc) then ADD ADD ADD (lots of them) ERROR
So for now I'm keeping autoflush on :) but is there anything obvious I could try there? (server runs on linux openjdk 1.8.0_91, client runs in Tomcat on Windows Oracle JDK 1.8.0_121)
Thanks, Colin
-- Dirk Kirsten, BaseX GmbH, http://basexgmbh.de |-- Firmensitz: Blarerstrasse 56, 78462 Konstanz |-- Registergericht Freiburg, HRB: 708285, Geschäftsführer: | Dr. Christian Grün, Dr. Alexander Holupirek, Michael Seiferle `-- Phone: 0049 7531 91 68 276, Fax: 0049 7531 20 05 22
basex-talk@mailman.uni-konstanz.de
-
Colin Delacroix -
Dirk Kirsten