Hi Oliver, I tried both GUI mode and terminal client. I get the same error there. And also Map Visualization worked fine for the same collection before I created indexes. Don't you have the same problem in terminal mode? How big is your collection then (number of files and size in MB)? Best regards, Ivan 20.10.2010 18:21, Oliver Egli wrote:
Hi Ivan,
I used to have the same problem. If you are working with the GUI mode, try to close the Map Visualization. (Open another smaller DB, close the Visualization, and try to open your Database again). That worked fine for me.
Regards, Oli
-----Ursprüngliche Nachricht----- Von: basex-talk-bounces@mailman.uni-konstanz.de [mailto:basex-talk-bounces@mailman.uni-konstanz.de] Im Auftrag von Ivan Lagunov Gesendet: Mittwoch, 20. Oktober 2010 16:00 An: Christian Grün Cc: BaseX Talk Betreff: [basex-talk] Index issue for a large collection
Hi,
I have a database collection with over 17000 documents (about 190MB in total). Everything worked fine until I created text and attribute indexes to improve performance. Now I cannot open database and perform any operation with it because I get 'Error: Out of Main Memory'. Setting "intparse on" doesn't help. I've increased Xmx value up to 2560m (maximum value for my system), it doesn't help too. Is there any way to fix it? Could you provide an approximate size of memory that is required for text and attribute indexes depending on the size of the database? Is there any possibility for me to use indexes on current system? Thanks in advance!
Best regards, Ivan
_______________________________________________ BaseX-Talk mailing list BaseX-Talk@mailman.uni-konstanz.de https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk
Ivan, hm, 190 MB sounds like a pretty small collection; 2.5 GB RAM should suffice to process 10 times the data and more. Have you tried version 6.2.9 of BaseX? Could you please check out the debugging flag on command line? Try sth. like: java -cp basex.jar org.basex.BaseX -d -c "create db ...." Thanks, Christian On Wed, Oct 20, 2010 at 10:42 PM, Ivan Lagunov <ilagunov@evelopers.com> wrote:
Hi Oliver,
I tried both GUI mode and terminal client. I get the same error there. And also Map Visualization worked fine for the same collection before I created indexes. Don't you have the same problem in terminal mode? How big is your collection then (number of files and size in MB)?
Best regards, Ivan
20.10.2010 18:21, Oliver Egli wrote:
Hi Ivan,
I used to have the same problem. If you are working with the GUI mode, try to close the Map Visualization. (Open another smaller DB, close the Visualization, and try to open your Database again). That worked fine for me.
Regards, Oli
-----Ursprüngliche Nachricht----- Von: basex-talk-bounces@mailman.uni-konstanz.de [mailto:basex-talk-bounces@mailman.uni-konstanz.de] Im Auftrag von Ivan Lagunov Gesendet: Mittwoch, 20. Oktober 2010 16:00 An: Christian Grün Cc: BaseX Talk Betreff: [basex-talk] Index issue for a large collection
Hi,
I have a database collection with over 17000 documents (about 190MB in total). Everything worked fine until I created text and attribute indexes to improve performance. Now I cannot open database and perform any operation with it because I get 'Error: Out of Main Memory'. Setting "intparse on" doesn't help. I've increased Xmx value up to 2560m (maximum value for my system), it doesn't help too. Is there any way to fix it? Could you provide an approximate size of memory that is required for text and attribute indexes depending on the size of the database? Is there any possibility for me to use indexes on current system? Thanks in advance!
Best regards, Ivan
_______________________________________________ BaseX-Talk mailing list BaseX-Talk@mailman.uni-konstanz.de https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk
Dear Ivan, there are good news...Christian has fixed the index problem. Get the newest version here... http://basex.org/maven/org/basex/basex/6.3 Kind regards, Andreas Am 21.10.2010 16:09, schrieb Christian Grün:
Ivan,
hm, 190 MB sounds like a pretty small collection; 2.5 GB RAM should suffice to process 10 times the data and more. Have you tried version 6.2.9 of BaseX? Could you please check out the debugging flag on command line? Try sth. like:
java -cp basex.jar org.basex.BaseX -d -c "create db ...."
Thanks, Christian
On Wed, Oct 20, 2010 at 10:42 PM, Ivan Lagunov<ilagunov@evelopers.com> wrote:
Hi Oliver,
I tried both GUI mode and terminal client. I get the same error there. And also Map Visualization worked fine for the same collection before I created indexes. Don't you have the same problem in terminal mode? How big is your collection then (number of files and size in MB)?
Best regards, Ivan
20.10.2010 18:21, Oliver Egli wrote:
Hi Ivan,
I used to have the same problem. If you are working with the GUI mode, try to close the Map Visualization. (Open another smaller DB, close the Visualization, and try to open your Database again). That worked fine for me.
Regards, Oli
-----Ursprüngliche Nachricht----- Von: basex-talk-bounces@mailman.uni-konstanz.de [mailto:basex-talk-bounces@mailman.uni-konstanz.de] Im Auftrag von Ivan Lagunov Gesendet: Mittwoch, 20. Oktober 2010 16:00 An: Christian Grün Cc: BaseX Talk Betreff: [basex-talk] Index issue for a large collection
Hi,
I have a database collection with over 17000 documents (about 190MB in total). Everything worked fine until I created text and attribute indexes to improve performance. Now I cannot open database and perform any operation with it because I get 'Error: Out of Main Memory'. Setting "intparse on" doesn't help. I've increased Xmx value up to 2560m (maximum value for my system), it doesn't help too. Is there any way to fix it? Could you provide an approximate size of memory that is required for text and attribute indexes depending on the size of the database? Is there any possibility for me to use indexes on current system? Thanks in advance!
Best regards, Ivan
_______________________________________________ BaseX-Talk mailing list BaseX-Talk@mailman.uni-konstanz.de https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk
_______________________________________________ BaseX-Talk mailing list BaseX-Talk@mailman.uni-konstanz.de https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk
Dear Andreas, I've started the server with the following command: java -Xms256m -Xmx2560m -XX:MaxPermSize=256m -cp basex.jar org.basex.BaseXServer -d Then I've tried to 'open database' in basex client. Finally I've got the following exception: BaseX 6.3 [Server] Server was started. java.util.Arrays.copyOf(Arrays.java:2834) org.basex.util.IntList.add(IntList.java:53) org.basex.data.Data.doc(Data.java:195) org.basex.core.Context.update(Context.java:126) org.basex.core.Context.openDB(Context.java:108) org.basex.core.cmd.Open.run(Open.java:35) org.basex.core.Command.run(Command.java:236) org.basex.core.Command.exec(Command.java:218) org.basex.core.Command.execute(Command.java:66) org.basex.server.ServerProcess.run(ServerProcess.java:161) Memory usage of the server java process reaches 1.6GB (I was monitoring it visually using 'top' linux command so real usage might have been even higher). Best regards, Ivan On 10/25/2010 02:25 PM, Andreas Weiler wrote:
Dear Ivan,
there are good news...Christian has fixed the index problem. Get the newest version here...
http://basex.org/maven/org/basex/basex/6.3
Kind regards, Andreas
Am 21.10.2010 16:09, schrieb Christian Grün:
Ivan,
hm, 190 MB sounds like a pretty small collection; 2.5 GB RAM should suffice to process 10 times the data and more. Have you tried version 6.2.9 of BaseX? Could you please check out the debugging flag on command line? Try sth. like:
java -cp basex.jar org.basex.BaseX -d -c "create db ...."
Thanks, Christian
On Wed, Oct 20, 2010 at 10:42 PM, Ivan Lagunov<ilagunov@evelopers.com> wrote:
Hi Oliver,
I tried both GUI mode and terminal client. I get the same error there. And also Map Visualization worked fine for the same collection before I created indexes. Don't you have the same problem in terminal mode? How big is your collection then (number of files and size in MB)?
Best regards, Ivan
20.10.2010 18:21, Oliver Egli wrote:
Hi Ivan,
I used to have the same problem. If you are working with the GUI mode, try to close the Map Visualization. (Open another smaller DB, close the Visualization, and try to open your Database again). That worked fine for me.
Regards, Oli
-----Ursprüngliche Nachricht----- Von: basex-talk-bounces@mailman.uni-konstanz.de [mailto:basex-talk-bounces@mailman.uni-konstanz.de] Im Auftrag von Ivan Lagunov Gesendet: Mittwoch, 20. Oktober 2010 16:00 An: Christian Grün Cc: BaseX Talk Betreff: [basex-talk] Index issue for a large collection
Hi,
I have a database collection with over 17000 documents (about 190MB in total). Everything worked fine until I created text and attribute indexes to improve performance. Now I cannot open database and perform any operation with it because I get 'Error: Out of Main Memory'. Setting "intparse on" doesn't help. I've increased Xmx value up to 2560m (maximum value for my system), it doesn't help too. Is there any way to fix it? Could you provide an approximate size of memory that is required for text and attribute indexes depending on the size of the database? Is there any possibility for me to use indexes on current system? Thanks in advance!
Best regards, Ivan
_______________________________________________ BaseX-Talk mailing list BaseX-Talk@mailman.uni-konstanz.de https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk
_______________________________________________ BaseX-Talk mailing list BaseX-Talk@mailman.uni-konstanz.de https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk
Dear Ivan, did u create the database with the newest version? Kind regards, Andreas Am 25.10.2010 12:51, schrieb Ivan Lagunov:
Dear Andreas,
I've started the server with the following command: java -Xms256m -Xmx2560m -XX:MaxPermSize=256m -cp basex.jar org.basex.BaseXServer -d
Then I've tried to 'open database' in basex client.
Finally I've got the following exception: BaseX 6.3 [Server] Server was started. java.util.Arrays.copyOf(Arrays.java:2834) org.basex.util.IntList.add(IntList.java:53) org.basex.data.Data.doc(Data.java:195) org.basex.core.Context.update(Context.java:126) org.basex.core.Context.openDB(Context.java:108) org.basex.core.cmd.Open.run(Open.java:35) org.basex.core.Command.run(Command.java:236) org.basex.core.Command.exec(Command.java:218) org.basex.core.Command.execute(Command.java:66) org.basex.server.ServerProcess.run(ServerProcess.java:161)
Memory usage of the server java process reaches 1.6GB (I was monitoring it visually using 'top' linux command so real usage might have been even higher).
Best regards, Ivan
On 10/25/2010 02:25 PM, Andreas Weiler wrote:
Dear Ivan,
there are good news...Christian has fixed the index problem. Get the newest version here...
http://basex.org/maven/org/basex/basex/6.3
Kind regards, Andreas
Am 21.10.2010 16:09, schrieb Christian Grün:
Ivan,
hm, 190 MB sounds like a pretty small collection; 2.5 GB RAM should suffice to process 10 times the data and more. Have you tried version 6.2.9 of BaseX? Could you please check out the debugging flag on command line? Try sth. like:
java -cp basex.jar org.basex.BaseX -d -c "create db ...."
Thanks, Christian
On Wed, Oct 20, 2010 at 10:42 PM, Ivan Lagunov<ilagunov@evelopers.com> wrote:
Hi Oliver,
I tried both GUI mode and terminal client. I get the same error there. And also Map Visualization worked fine for the same collection before I created indexes. Don't you have the same problem in terminal mode? How big is your collection then (number of files and size in MB)?
Best regards, Ivan
20.10.2010 18:21, Oliver Egli wrote:
Hi Ivan,
I used to have the same problem. If you are working with the GUI mode, try to close the Map Visualization. (Open another smaller DB, close the Visualization, and try to open your Database again). That worked fine for me.
Regards, Oli
-----Ursprüngliche Nachricht----- Von: basex-talk-bounces@mailman.uni-konstanz.de [mailto:basex-talk-bounces@mailman.uni-konstanz.de] Im Auftrag von Ivan Lagunov Gesendet: Mittwoch, 20. Oktober 2010 16:00 An: Christian Grün Cc: BaseX Talk Betreff: [basex-talk] Index issue for a large collection
Hi,
I have a database collection with over 17000 documents (about 190MB in total). Everything worked fine until I created text and attribute indexes to improve performance. Now I cannot open database and perform any operation with it because I get 'Error: Out of Main Memory'. Setting "intparse on" doesn't help. I've increased Xmx value up to 2560m (maximum value for my system), it doesn't help too. Is there any way to fix it? Could you provide an approximate size of memory that is required for text and attribute indexes depending on the size of the database? Is there any possibility for me to use indexes on current system? Thanks in advance!
Best regards, Ivan
_______________________________________________ BaseX-Talk mailing list BaseX-Talk@mailman.uni-konstanz.de https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk
_______________________________________________ BaseX-Talk mailing list BaseX-Talk@mailman.uni-konstanz.de https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk
Dear Andreas, It worked fine as soon as I created the database with the newest version. But I'd like to be confident of BaseX that I won't lose my data as it happened for the version 6.1. Best regards, Ivan On 10/25/2010 03:21 PM, Andreas Weiler wrote:
Dear Ivan,
did u create the database with the newest version?
Kind regards, Andreas
Am 25.10.2010 12:51, schrieb Ivan Lagunov:
Dear Andreas,
I've started the server with the following command: java -Xms256m -Xmx2560m -XX:MaxPermSize=256m -cp basex.jar org.basex.BaseXServer -d
Then I've tried to 'open database' in basex client.
Finally I've got the following exception: BaseX 6.3 [Server] Server was started. java.util.Arrays.copyOf(Arrays.java:2834) org.basex.util.IntList.add(IntList.java:53) org.basex.data.Data.doc(Data.java:195) org.basex.core.Context.update(Context.java:126) org.basex.core.Context.openDB(Context.java:108) org.basex.core.cmd.Open.run(Open.java:35) org.basex.core.Command.run(Command.java:236) org.basex.core.Command.exec(Command.java:218) org.basex.core.Command.execute(Command.java:66) org.basex.server.ServerProcess.run(ServerProcess.java:161)
Memory usage of the server java process reaches 1.6GB (I was monitoring it visually using 'top' linux command so real usage might have been even higher).
Best regards, Ivan
On 10/25/2010 02:25 PM, Andreas Weiler wrote:
Dear Ivan,
there are good news...Christian has fixed the index problem. Get the newest version here...
http://basex.org/maven/org/basex/basex/6.3
Kind regards, Andreas
Am 21.10.2010 16:09, schrieb Christian Grün:
Ivan,
hm, 190 MB sounds like a pretty small collection; 2.5 GB RAM should suffice to process 10 times the data and more. Have you tried version 6.2.9 of BaseX? Could you please check out the debugging flag on command line? Try sth. like:
java -cp basex.jar org.basex.BaseX -d -c "create db ...."
Thanks, Christian
On Wed, Oct 20, 2010 at 10:42 PM, Ivan Lagunov<ilagunov@evelopers.com> wrote:
Hi Oliver,
I tried both GUI mode and terminal client. I get the same error there. And also Map Visualization worked fine for the same collection before I created indexes. Don't you have the same problem in terminal mode? How big is your collection then (number of files and size in MB)?
Best regards, Ivan
20.10.2010 18:21, Oliver Egli wrote:
Hi Ivan,
I used to have the same problem. If you are working with the GUI mode, try to close the Map Visualization. (Open another smaller DB, close the Visualization, and try to open your Database again). That worked fine for me.
Regards, Oli
-----Ursprüngliche Nachricht----- Von: basex-talk-bounces@mailman.uni-konstanz.de [mailto:basex-talk-bounces@mailman.uni-konstanz.de] Im Auftrag von Ivan Lagunov Gesendet: Mittwoch, 20. Oktober 2010 16:00 An: Christian Grün Cc: BaseX Talk Betreff: [basex-talk] Index issue for a large collection
Hi,
I have a database collection with over 17000 documents (about 190MB in total). Everything worked fine until I created text and attribute indexes to improve performance. Now I cannot open database and perform any operation with it because I get 'Error: Out of Main Memory'. Setting "intparse on" doesn't help. I've increased Xmx value up to 2560m (maximum value for my system), it doesn't help too. Is there any way to fix it? Could you provide an approximate size of memory that is required for text and attribute indexes depending on the size of the database? Is there any possibility for me to use indexes on current system? Thanks in advance!
Best regards, Ivan
_______________________________________________ BaseX-Talk mailing list BaseX-Talk@mailman.uni-konstanz.de https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk
_______________________________________________ BaseX-Talk mailing list BaseX-Talk@mailman.uni-konstanz.de https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk
Hi Ivan, Am 25.10.2010 14:47, schrieb Ivan Lagunov:
It worked fine as soon as I created the database with the newest version. But I'd like to be confident of BaseX that I won't lose my data as it happened for the version 6.1.
the database corruption you reported was most probably caused by an evil bug in the XQuery Update code that we were able to fix in version 6.2.8. It destroyed parts of the index structure, leading to reads on empty DB pages. The stack trace you provided is a result of that. So I think the problem shouldn't happen again :-). Cheers Leo
participants (4)
-
Andreas Weiler -
Christian Grün -
Ivan Lagunov -
Leonard Wörteler