Hello all.
I've tried to execute query
for $i in collection('stars')/csv/record return insert node (attribute {'id'} {generate-id()} ) into $i
with this http://www.onlinedisk.ru/file/889304/ file (1000000 strings with 3 integer in any in csv format). I had a hang of basex and onece got the following:
Exception in thread "Thread-526" java.lang.OutOfMemoryError: Java heap space Exception in thread "AWT-EventQueue-0" java.lang.OutOfMemoryError: GC overhead limit exceeded at java.awt.EventQueue.postEvent(EventQueue.java:261) at java.awt.EventQueue.postEventPrivate(EventQueue.java:215) Exception in thread "AWT-XAWT" java.lang.OutOfMemoryError: GC overhead limit exceeded Exception in thread "SIGTERM handler" java.lang.OutOfMemoryError: GC overhead limit exceeded
and again - the folowing:
Exception in thread "AWT-XAWT" java.lang.OutOfMemoryError: Java heap space at java.util.logging.LogRecord.<init>(LogRecord.java:151) at java.util.logging.Logger.log(Logger.java:607) at sun.awt.X11.XToolkit.processException(XToolkit.java:535) at sun.awt.X11.XToolkit.run(XToolkit.java:614) at sun.awt.X11.XToolkit.run(XToolkit.java:543) at java.lang.Thread.run(Thread.java:679)
Hang was even when I changed generate-id() to "1". Seems like a bug.
Markin Alex.
Hi Alex,
for $i in collection('stars')/csv/record return insert node (attribute {'id'} {generate-id()} ) into $i
before any update is performed, all operations first need to be put on the pending update list [1]. It looks as if this list requires too much memory. How much memory did you assign to the JRE?
Christian
[1] http://docs.basex.org/wiki/XQuery_Update#Pending_Update_List __________________________________________________
with this file (1000000 strings with 3 integer in any in csv format). I had a hang of basex and onece got the following:
Exception in thread "Thread-526" java.lang.OutOfMemoryError: Java heap space Exception in thread "AWT-EventQueue-0" java.lang.OutOfMemoryError: GC overhead limit exceeded at java.awt.EventQueue.postEvent(EventQueue.java:261) at java.awt.EventQueue.postEventPrivate(EventQueue.java:215) Exception in thread "AWT-XAWT" java.lang.OutOfMemoryError: GC overhead limit exceeded Exception in thread "SIGTERM handler" java.lang.OutOfMemoryError: GC overhead limit exceeded
and again - the folowing:
Exception in thread "AWT-XAWT" java.lang.OutOfMemoryError: Java heap space at java.util.logging.LogRecord.<init>(LogRecord.java:151) at java.util.logging.Logger.log(Logger.java:607) at sun.awt.X11.XToolkit.processException(XToolkit.java:535) at sun.awt.X11.XToolkit.run(XToolkit.java:614) at sun.awt.X11.XToolkit.run(XToolkit.java:543) at java.lang.Thread.run(Thread.java:679)
Hang was even when I changed generate-id() to "1". Seems like a bug. Markin Alex.
Hi Alex,
I've tried to further reduce memory consumption of XQuery Update operations (however, this won't change the general behavior, as the XQUF specification will always expect update operations to be cached before they can be evaluated). Feel free to try the latest snapshot [1].
Christian
[1] http://files.basex.org/releases/latest/ ___________________________
for $i in collection('stars')/csv/record return insert node (attribute {'id'} {generate-id()} ) into $i
before any update is performed, all operations first need to be put on the pending update list [1]. It looks as if this list requires too much memory. How much memory did you assign to the JRE?
Christian
[1] http://docs.basex.org/wiki/XQuery_Update#Pending_Update_List __________________________________________________
with this file (1000000 strings with 3 integer in any in csv format). I had a hang of basex and onece got the following:
Exception in thread "Thread-526" java.lang.OutOfMemoryError: Java heap space Exception in thread "AWT-EventQueue-0" java.lang.OutOfMemoryError: GC overhead limit exceeded at java.awt.EventQueue.postEvent(EventQueue.java:261) at java.awt.EventQueue.postEventPrivate(EventQueue.java:215) Exception in thread "AWT-XAWT" java.lang.OutOfMemoryError: GC overhead limit exceeded Exception in thread "SIGTERM handler" java.lang.OutOfMemoryError: GC overhead limit exceeded
and again - the folowing:
Exception in thread "AWT-XAWT" java.lang.OutOfMemoryError: Java heap space at java.util.logging.LogRecord.<init>(LogRecord.java:151) at java.util.logging.Logger.log(Logger.java:607) at sun.awt.X11.XToolkit.processException(XToolkit.java:535) at sun.awt.X11.XToolkit.run(XToolkit.java:614) at sun.awt.X11.XToolkit.run(XToolkit.java:543) at java.lang.Thread.run(Thread.java:679)
Hang was even when I changed generate-id() to "1". Seems like a bug. Markin Alex.
I didn't find where the default value of jre memory allocation is set, but tried to call basex (BaseX73-20120612.205120) with the following command:
java -cp "$CP" $VM "${vm_args[@]}" -Xms1024m -Xmx3072m org.basex.BaseXGUI "${general_args[@]}"
and it still hanging. Even GUI does not response.
Markin Alex.
2012/6/12 Christian Grün christian.gruen@gmail.com
Hi Alex,
I've tried to further reduce memory consumption of XQuery Update operations (however, this won't change the general behavior, as the XQUF specification will always expect update operations to be cached before they can be evaluated). Feel free to try the latest snapshot [1].
Christian
[1] http://files.basex.org/releases/latest/ ___________________________
for $i in collection('stars')/csv/record return insert node (attribute {'id'} {generate-id()} ) into $i
before any update is performed, all operations first need to be put on the pending update list [1]. It looks as if this list requires too much memory. How much memory did you assign to the JRE?
Christian
[1] http://docs.basex.org/wiki/XQuery_Update#Pending_Update_List __________________________________________________
with this file (1000000 strings with 3 integer in any in csv format). I
had
a hang of basex and onece got the following:
Exception in thread "Thread-526" java.lang.OutOfMemoryError: Java heap
space
Exception in thread "AWT-EventQueue-0" java.lang.OutOfMemoryError: GC overhead limit exceeded at java.awt.EventQueue.postEvent(EventQueue.java:261) at java.awt.EventQueue.postEventPrivate(EventQueue.java:215) Exception in thread "AWT-XAWT" java.lang.OutOfMemoryError: GC overhead
limit
exceeded Exception in thread "SIGTERM handler" java.lang.OutOfMemoryError: GC overhead limit exceeded
and again - the folowing:
Exception in thread "AWT-XAWT" java.lang.OutOfMemoryError: Java heap
space
at java.util.logging.LogRecord.<init>(LogRecord.java:151) at java.util.logging.Logger.log(Logger.java:607) at sun.awt.X11.XToolkit.processException(XToolkit.java:535) at sun.awt.X11.XToolkit.run(XToolkit.java:614) at sun.awt.X11.XToolkit.run(XToolkit.java:543) at java.lang.Thread.run(Thread.java:679)Hang was even when I changed generate-id() to "1". Seems like a bug. Markin Alex.
Hi Alex,
it looks as if the sheer number of updates to be performed is simply too large. This may look surprising, regarding the general simplicity of your use case (..add 1 mio. attributes to a document..), but the general problem of the update standard is that all update operations (and the accompanying meta-data) have to be cached first.
I have a number of optimizations in mind that could further reduce memory consumption (such as introducing hashes for identical QNames and node instances), but this will take a little bit longer.
Christian ___________________________
I didn't find where the default value of jre memory allocation is set, but tried to call basex (BaseX73-20120612.205120) with the following command:
java -cp "$CP" $VM "${vm_args[@]}" -Xms1024m -Xmx3072m org.basex.BaseXGUI "${general_args[@]}"
and it still hanging. Even GUI does not response.
Markin Alex.
2012/6/12 Christian Grün christian.gruen@gmail.com
Hi Alex,
I've tried to further reduce memory consumption of XQuery Update operations (however, this won't change the general behavior, as the XQUF specification will always expect update operations to be cached before they can be evaluated). Feel free to try the latest snapshot [1].
Christian
[1] http://files.basex.org/releases/latest/ ___________________________
for $i in collection('stars')/csv/record return insert node (attribute {'id'} {generate-id()} ) into $i
before any update is performed, all operations first need to be put on the pending update list [1]. It looks as if this list requires too much memory. How much memory did you assign to the JRE?
Christian
[1] http://docs.basex.org/wiki/XQuery_Update#Pending_Update_List __________________________________________________
with this file (1000000 strings with 3 integer in any in csv format). I had a hang of basex and onece got the following:
Exception in thread "Thread-526" java.lang.OutOfMemoryError: Java heap space Exception in thread "AWT-EventQueue-0" java.lang.OutOfMemoryError: GC overhead limit exceeded at java.awt.EventQueue.postEvent(EventQueue.java:261) at java.awt.EventQueue.postEventPrivate(EventQueue.java:215) Exception in thread "AWT-XAWT" java.lang.OutOfMemoryError: GC overhead limit exceeded Exception in thread "SIGTERM handler" java.lang.OutOfMemoryError: GC overhead limit exceeded
and again - the folowing:
Exception in thread "AWT-XAWT" java.lang.OutOfMemoryError: Java heap space at java.util.logging.LogRecord.<init>(LogRecord.java:151) at java.util.logging.Logger.log(Logger.java:607) at sun.awt.X11.XToolkit.processException(XToolkit.java:535) at sun.awt.X11.XToolkit.run(XToolkit.java:614) at sun.awt.X11.XToolkit.run(XToolkit.java:543) at java.lang.Thread.run(Thread.java:679)
Hang was even when I changed generate-id() to "1". Seems like a bug. Markin Alex.
basex-talk@mailman.uni-konstanz.de
-
Alex Markin -
Christian Grün