Hi,
I have a database about 200 MB in size made up of approximately 150 000 documents of similar size and structure as children of the root node.
When I run the following script in basexgui a significant amount of memory is consumed (over 1 GB if I'm reading the display correctly), but I do get a result:
for $doc in db:get('docs') return $doc update { delete node .//*[local-name() = ('A', 'B', 'C', 'D')] }
When I run it over basexhttp I get a java.lang.OutOfMemoryError: Java heap space. I have increased the memory available to the JVM to 4 GB but this has not affected the failure of the script.
How can I resolve this? Is there some rewriting of the script that would help, or is it more specific to basexhttp?
Many thanks,
Jack
Sorry, the first line of the script should end with
/*/*
Could the problem be related to UPDINDEX, AUTOOPTIMIZE and/or FTINDEX all being set to true on basexhttp but false on basexgui? I thought it wouldn't make a difference because the updates occur in main memory, but perhaps it does.
Best,
Jack
On Wed, 7 Feb 2024, 4:03 pm Jack Steyn, steynjack@gmail.com wrote:
Hi,
I have a database about 200 MB in size made up of approximately 150 000 documents of similar size and structure as children of the root node.
When I run the following script in basexgui a significant amount of memory is consumed (over 1 GB if I'm reading the display correctly), but I do get a result:
for $doc in db:get('docs') return $doc update { delete node .//*[local-name() = ('A', 'B', 'C', 'D')] }
When I run it over basexhttp I get a java.lang.OutOfMemoryError: Java heap space. I have increased the memory available to the JVM to 4 GB but this has not affected the failure of the script.
How can I resolve this? Is there some rewriting of the script that would help, or is it more specific to basexhttp?
Many thanks,
Jack
Hi Jack,
If you run the query via basexhttp, how do you retrieve the results, i.e., which client do you use?
Can you possibly provide us with a self-contained example, something like…
for $i in 1 to 500000 return <xml/> update { insert node <a/> into . }
…and some steps to reproduce the behavior?
Thanks in advance, Christian
On Wed, Feb 7, 2024 at 6:04 AM Jack Steyn steynjack@gmail.com wrote:
Hi,
I have a database about 200 MB in size made up of approximately 150 000 documents of similar size and structure as children of the root node.
When I run the following script in basexgui a significant amount of memory is consumed (over 1 GB if I'm reading the display correctly), but I do get a result:
for $doc in db:get('docs') return $doc update { delete node .//*[local-name() = ('A', 'B', 'C', 'D')] }
When I run it over basexhttp I get a java.lang.OutOfMemoryError: Java heap space. I have increased the memory available to the JVM to 4 GB but this has not affected the failure of the script.
How can I resolve this? Is there some rewriting of the script that would help, or is it more specific to basexhttp?
Many thanks,
Jack
Hi Christian,
Sorry, I should have provided a self-contained example to begin with.
In any case, I was running BaseX 10.0; after noticing that 10.6 boasts 'Much more memory-efficient representation of XML fragments', I upgraded to 10.7 and the problem appears to be resolved (and wow, there is a big difference in performance – kudos!).
Many thanks,
Jack
On Wed, 7 Feb 2024, 6:09 pm Christian Grün, christian.gruen@gmail.com wrote:
Hi Jack,
If you run the query via basexhttp, how do you retrieve the results, i.e., which client do you use?
Can you possibly provide us with a self-contained example, something like…
for $i in 1 to 500000 return <xml/> update { insert node <a/> into . }
…and some steps to reproduce the behavior?
Thanks in advance, Christian
On Wed, Feb 7, 2024 at 6:04 AM Jack Steyn steynjack@gmail.com wrote:
Hi,
I have a database about 200 MB in size made up of approximately 150 000 documents of similar size and structure as children of the root node.
When I run the following script in basexgui a significant amount of memory is consumed (over 1 GB if I'm reading the display correctly), but I do get a result:
for $doc in db:get('docs') return $doc update { delete node .//*[local-name() = ('A', 'B', 'C', 'D')] }
When I run it over basexhttp I get a java.lang.OutOfMemoryError: Java heap space. I have increased the memory available to the JVM to 4 GB but this has not affected the failure of the script.
How can I resolve this? Is there some rewriting of the script that would help, or is it more specific to basexhttp?
Many thanks,
Jack
…great to hear, thanks.
On Thu, Feb 8, 2024 at 7:30 AM Jack Steyn steynjack@gmail.com wrote:
Hi Christian,
Sorry, I should have provided a self-contained example to begin with.
In any case, I was running BaseX 10.0; after noticing that 10.6 boasts 'Much more memory-efficient representation of XML fragments', I upgraded to 10.7 and the problem appears to be resolved (and wow, there is a big difference in performance – kudos!).
Many thanks,
Jack
On Wed, 7 Feb 2024, 6:09 pm Christian Grün, christian.gruen@gmail.com wrote:
Hi Jack,
If you run the query via basexhttp, how do you retrieve the results, i.e., which client do you use?
Can you possibly provide us with a self-contained example, something like…
for $i in 1 to 500000 return <xml/> update { insert node <a/> into . }
…and some steps to reproduce the behavior?
Thanks in advance, Christian
On Wed, Feb 7, 2024 at 6:04 AM Jack Steyn steynjack@gmail.com wrote:
Hi,
I have a database about 200 MB in size made up of approximately 150 000 documents of similar size and structure as children of the root node.
When I run the following script in basexgui a significant amount of memory is consumed (over 1 GB if I'm reading the display correctly), but I do get a result:
for $doc in db:get('docs') return $doc update { delete node .//*[local-name() = ('A', 'B', 'C', 'D')] }
When I run it over basexhttp I get a java.lang.OutOfMemoryError: Java heap space. I have increased the memory available to the JVM to 4 GB but this has not affected the failure of the script.
How can I resolve this? Is there some rewriting of the script that would help, or is it more specific to basexhttp?
Many thanks,
Jack
basex-talk@mailman.uni-konstanz.de
-
Christian Grün -
Jack Steyn