Hi Sven,

which filesystem do you use?

Some databases perform slow in not optimized XFS systens.

http://ondrejcertik.blogspot.com/2008/02/xfs-is-20x-slower-than-ext3-with.ht...

May also be hardware related, try to test your disk performance with hdparm and compare it to your windows machine.

Greetings,

Am 28.10.2011 12:06, schrieb Sven Rega:

Hy Christian,

I currently don't plan to change the database to a new version, because "never change a running system" :-) But on weekend i will update my process to test the new database version 7.1.

Some current Profiling results: I test the process with the profiling software VisualVM from sun/oracle and i think, i find the bottleneck: org.basex.io.in.BufferInput.next() On the windows machine i see, that this method is called 11150 times and executes this method in sum in 1890ms. (11150/1890= 5,89 calls per ms) On the linux machine i see, that the method is called 255635 times and needs in sum 834981ms (255635/834981 = 0,306 calls per ms). So the factor between both executions is around 20.

Do you have any idea, which reason could be in this method?

king regards Sven

On 27.10.2011 22:40, Christian Grün wrote:

...

Hi Sven,

thanks for your mail. Usually, there shouldn't be any notable differences between Windows and Linux (only the Mac JVM is known to be slower). Before giving any further advices, it might be interesting to hear if the problems persist with the latest version of BaseX (Version 7.01) ?

Best, Christian ___________________________

On Thu, Oct 27, 2011 at 10:36 PM, Sven Regasven.rega@gmx.de wrote:

...

Hy,

i'm using the basexserver 6.7.1 on a windows (i7; 8GB; 2,8GHz) and on different linux machines (i7; 16GB, 3,4GHz). the same xml file (~2,5GByte) and one simple query has i big difference in speed between both machines and i don't know, where the bottleneck is coming from. on the windows machine i get througput of ~600 to 1000 xml entities per second. On the linux machine i only get max. 110 xml entities per second.

I use the same jvm setting on both machines: -server -Xmx3g I also use the index (text, path, attribtue) on both machines.

The query:

"declare function local:ProductValues() { for $n in //Product[(@ID='A' or @ID='D' or @ID='G')]/Values return <pv><utid>{data($n/../@ID)}</utid><id>{data($n/../@SecondID)}</id>{$n}</pv>

}; local:ProductValues()"

I read from the basex server over lan with 1GBit.

Does anybody have a idea, what could be the reason for such speed difference between windows and linux?

king regards Sven _______________________________________________ BaseX-Talk mailing list BaseX-Talk@mailman.uni-konstanz.de https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk

---

BaseX-Talk mailing list BaseX-Talk@mailman.uni-konstanz.de https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk

Hy Christian,

Currently i try the basex version 7.0.1. Now, the speed/performance isn't the problem (is very fast now :-) ) but i get an problem with my xml data and i think, it is an heisenbug.

In my xml file i have a structure with content (including entities) like this (in real with multiple articles):

<root> <article ID="xys"> <values> .... <value ..>.....</value> <value ..>.....</value> <value ID="abcdefg" refcontent="other article">&t;b&gt;System requirements: &lt;/b&gt;</value> </values> </article> </root>

The problem: i import the same file multiple times and execute a query like for $n in //article[(@ID='Article' or @ID='Other')]/values return <pv><utid>{data($n/../@ID)}</utid>{$n}</pv>

Sometime, all is ok without any problem, but after the next import (same file) i get the error from java from my next processing step with sax/xpath: Error on line 115 of document : The entity "t" was referenced, but not declared. Nested exception: The entity "t" was referenced, but not declared. If looked into <pv>..</pv> part, which throws the exception i see, that the data is corrupt: basexserver returns: ..... <value ID="abcdefg" refcontent="other article">&t;b&gt;System requirements: &lt;/b&gt;</value>.... (look at ">&t;" , there is a missing l from &lt;) After the exception i looked into my basexserver with basexclient to check this data and see the correct data in the database. Also if i only query this specific article, i get the correct data.

I also started the basexserver with the apache commons xml resolver.jar in the classpath, but no effort.

What i'm doing wrong? Or is this a bug? Why this problem was never seen in the basex 6.7.1 version?

king regards Sven

On 27.10.2011 22:40, Christian Grün wrote:

Hi Sven,

thanks for your mail. Usually, there shouldn't be any notable differences between Windows and Linux (only the Mac JVM is known to be slower). Before giving any further advices, it might be interesting to hear if the problems persist with the latest version of BaseX (Version 7.01) ?

Best, Christian ___________________________

On Thu, Oct 27, 2011 at 10:36 PM, Sven Regasven.rega@gmx.de wrote:

...

Hy,

i'm using the basexserver 6.7.1 on a windows (i7; 8GB; 2,8GHz) and on different linux machines (i7; 16GB, 3,4GHz). the same xml file (~2,5GByte) and one simple query has i big difference in speed between both machines and i don't know, where the bottleneck is coming from. on the windows machine i get througput of ~600 to 1000 xml entities per second. On the linux machine i only get max. 110 xml entities per second.

I use the same jvm setting on both machines: -server -Xmx3g I also use the index (text, path, attribtue) on both machines.

The query:

"declare function local:ProductValues() { for $n in //Product[(@ID='A' or @ID='D' or @ID='G')]/Values return <pv><utid>{data($n/../@ID)}</utid><id>{data($n/../@SecondID)}</id>{$n}</pv> }; local:ProductValues()"

I read from the basex server over lan with 1GBit.

Does anybody have a idea, what could be the reason for such speed difference between windows and linux?

king regards Sven _______________________________________________ BaseX-Talk mailing list BaseX-Talk@mailman.uni-konstanz.de https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk

Hi Christian,

Since Version 7.0, we use our internal XML parser as default parser. It would be interesting to hear what happens if you switch back to the Java XML parser (see [1] for details).

[1] http://docs.basex.org/wiki/Parser#XML_Parsers

That site still says that the SAX parser is the default. Which information is correct?

Kind regards, Ralf

Hy Christian,

i tested the import with the sax and with the internal parser and results are also corrupt. Process: import data from one large xml file (2,5GB) into one empty db

my data: <root> <article ID="xys"> <values> .... <value ..>.....</value> <value ..>.....</value> <value ID="abcdefg" refcontent="other article">&t;b&gt;System requirements: &lt;/b&gt;</value> </values> </article> </root>

QUERY_1: for $n in //article[(@ID='Article' or @ID='Other')]/values return <pv><utid>{data($n/../@ID)}</utid>{$n}</pv>

QUERY_2: for $n in //article[(@ID='Article' or @ID='Other') and ID="abc"]/values return <pv><utid>{data($n/../@ID)}</utid>{$n}</pv>

If i query with QUERY_1, i get one corrupt results which looks like: <pv> <utid>abc</utid> .. <value ...> ...... <value <----- here the closing ">" is missing .. </pv>

but if i use query QUERY_2, which only grab the one specific article with the corrupt data, i get a correct result, without the missing closing tag.

So the DB contains all correct information, but the query (QUERY_1) over all article returns corrupt data.

Do you have some other ideas for the reason of this problem/bug? Do you know, which code parts are changed from version 6.7.1 to 7.0.1, which are involved in this problem?

regards Sven

On 02.11.2011 18:20, Christian Grün wrote:

Hi Sven,

...

Currently i try the basex version 7.0.1. Now, the speed/performance isn't the problem (is very fast now :-) )

..always nice to hear..

...

but i get an problem with my xml data and i think, it is an heisenbug.

..not that nice to hear ;)..

...

What i'm doing wrong? Or is this a bug? Why this problem was never seen in the basex 6.7.1 version?

Since Version 7.0, we use our internal XML parser as default parser. It would be interesting to hear what happens if you switch back to the Java XML parser (see [1] for details).

Best, Christian

[1] http://docs.basex.org/wiki/Parser#XML_Parsers

Hy Christian,

Some idea from me: could it be a buffer problem (overlapping issue) ? The Query_1 produce data with around 2,1 GB. With the same query, the angle bracket is always missing on the same entity on the same (byte) offset in the result. May be some output buffer from the server side or some input buffer from the client side has a bug?

What do you mean with serialization api? for connecting and querying i use the java api: - org.basex.server.ClientSession - org.basex.server.ClientQuery - while(query.more()) - query.next()

regards Sven

On 03.11.2011 13:54, Christian Grün wrote:

Hi Sven,

I'm not sure what to think about this issue. This could also be a serialization problem, as angle brackets (<,>) are not stored in the database at all. Which API do you use to serialize the results/have you tried another API as well?

Christian ___________________________

On Thu, Nov 3, 2011 at 1:38 PM, Sven Regasven.rega@gmx.de wrote:

...

Hy Christian,

i tested the import with the sax and with the internal parser and results are also corrupt. Process: import data from one large xml file (2,5GB) into one empty db

my data:

<root> <article ID="xys"> <values> .... <value ..>.....</value> <value ..>.....</value> <value ID="abcdefg" refcontent="other article">&t;b&gt;System requirements: &lt;/b&gt;</value> </values> </article> </root>

QUERY_1: for $n in //article[(@ID='Article' or @ID='Other')]/values return<pv><utid>{data($n/../@ID)}</utid>{$n}</pv>

QUERY_2: for $n in //article[(@ID='Article' or @ID='Other') and ID="abc"]/values return<pv><utid>{data($n/../@ID)}</utid>{$n}</pv>

If i query with QUERY_1, i get one corrupt results which looks like:

<pv> <utid>abc</utid> .. <value ...> ......<value<----- here the closing ">" is missing .. </pv>

but if i use query QUERY_2, which only grab the one specific article with the corrupt data, i get a correct result, without the missing closing tag.

So the DB contains all correct information, but the query (QUERY_1) over all article returns corrupt data.

Do you have some other ideas for the reason of this problem/bug? Do you know, which code parts are changed from version 6.7.1 to 7.0.1, which are involved in this problem?

regards Sven

On 02.11.2011 18:20, Christian Grün wrote:

...

Hi Sven,

...

Currently i try the basex version 7.0.1. Now, the speed/performance isn't the problem (is very fast now :-) )

..always nice to hear..

...

but i get an problem with my xml data and i think, it is an heisenbug.

..not that nice to hear ;)..

...

What i'm doing wrong? Or is this a bug? Why this problem was never seen in the basex 6.7.1 version?

Since Version 7.0, we use our internal XML parser as default parser. It would be interesting to hear what happens if you switch back to the Java XML parser (see [1] for details).

Best, Christian

[1] http://docs.basex.org/wiki/Parser#XML_Parsers

Hy Dimitar,

With the standalone mode, the described problem is not present. So it could be really a problem in the client/server communication.

regards, Sven

On 03.11.2011 14:34, Dimitar Popov wrote:

Am Donnerstag, 3. November 2011, 14:15:31 schrieb Sven Rega:

...

Hy Christian,

Some idea from me: could it be a buffer problem (overlapping issue) ? The Query_1 produce data with around 2,1 GB. With the same query, the angle bracket is always missing on the same entity on the same (byte) offset in the result. May be some output buffer from the server side or some input buffer from the client side has a bug?

Hi Sven,

can you try running basex as stand-alone application (i.e. not client-server). Thus, we'll be able to see if the problem is in the client-server communication...

Greetz, Dimitar

Hy Christian,

I think also, that it's a client problem. With the direct execution of ClientQuery.execute i get an exception:

Exception in thread "main" java.lang.NegativeArraySizeException at java.util.Arrays.copyOf(Arrays.java:2786) at org.basex.io.out.ArrayOutput.write(ArrayOutput.java:37) at org.basex.server.ClientSession.receive(ClientSession.java:305) at org.basex.server.ClientSession.receive(ClientSession.java:268) at org.basex.server.ClientSession.execute(ClientSession.java:174) at org.basex.server.Session.execute(Session.java:51) at test1.main(test1.java:66)

The data, which i receive from the basexDB, has a size around of 2,16 GByte. The max. int value could be 2,147,483,647 and so its clear, that the receive-Method get an negative value.

void receive(final BufferInput bi, final OutputStream os) throws IOException { final DecodingInput di = new DecodingInput(bi); for(int b; (b = di.read()) != -1;) os.write(b); }

May be the size of my result with session.query() and the one missing character depends also on the max integer value?!?

regars Sven

On 03.11.2011 18:56, Christian Grün wrote:

Hi Sven,

sounds indeed like a client issue. I have no idea, however, what the problem could be, as our client code is pretty light-weight. Did you manage to encounter the same problem by calling ClientQuery.execute (..or is the result too large to be generated by this command)? It would be great if you could provide us with a reproducible example - but I know it's sometimes difficult to generate one.

Christian ___________________________

On Thu, Nov 3, 2011 at 5:08 PM, Sven Regasven.rega@gmx.de wrote:

...

Hy Dimitar,

With the standalone mode, the described problem is not present. So it could be really a problem in the client/server communication.

regards, Sven

On 03.11.2011 14:34, Dimitar Popov wrote:

...

Am Donnerstag, 3. November 2011, 14:15:31 schrieb Sven Rega:

...

Hy Christian,

Some idea from me: could it be a buffer problem (overlapping issue) ? The Query_1 produce data with around 2,1 GB. With the same query, the angle bracket is always missing on the same entity on the same (byte) offset in the result. May be some output buffer from the server side or some input buffer from the client side has a bug?

Hi Sven,

can you try running basex as stand-alone application (i.e. not client-server). Thus, we'll be able to see if the problem is in the client-server communication...

Greetz, Dimitar

if i looked into the code of org.basex.server.Query is see the object value "pos" which is from the java type int. In the basex version 6.7.1 this "pos" variable did not exists. Also in the new 7.0.1 version some caching mechanism seems to be implemented, which was not in the old version. I think, in these methods the problem could be raised.

Since Version 7.0 of BaseX, to avoid deadlocks and increased memory usage, iterative query results are completely sent to the client before the actual iterator is triggered; this is why the "pos" variable disappeared in the Query class.

Christian