Bonjour Simon,
I would send a query for each document, externalizing the loop in java.
A question : could you process be written in xquery ? That way you might not face memory overflow.
Best regards, Fabrice Etanchaud CERFrance Poitou-Charentes
De : basex-talk-bounces@mailman.uni-konstanz.de [mailto:basex-talk-bounces@mailman.uni-konstanz.de] De la part de Simon Chatelain Envoyé : vendredi 22 septembre 2017 09:34 À : BaseX Objet : [basex-talk] OutOfMemoryError at Query#more()
Hello, I am facing an issue while retrieving some big amount of XML documents from a BaseX collection. Each document (as an XML file) is around 10 KB, and in the problematic case I must retrieve around 70000 of them. I am using Session#query(String query) then Query#more() and Query#next() to iterate through the result of my query.
try (final Query query = l_Session.query(“query”)) { while (query.more()) { String xml = query.next(); } } If there is more than a certain amount of XML document in the result of my query I get a OutOfMemoryError (full stack trace in attached file) when executing query.more().
I did the test with BaseX 8.6.6 and 8.6.7, Java 8, VM arguments –Xmx1024m
Increasing the Xmx value is not a solution as I don’t know what the maximum amount of data I will have to retrieve in the future. So what I need is a reliable way of executing such queries and iterate through the result without exploding the heap size. I also try to use QueryProcessor and QueryProcessor#iter() instead of Session#query(String query). But is it safe to use it knowing that my application is multithreaded and that each thread has its own session to query or add elements from/to multiple collections? Moreover, for now all access to BaseX are done through a session, so my application can run with an embedded BaseX or with a BaseX server. If I start using QueryProcessor, then it will be embedded BaseX only, right?
I also attached a simple example showing the problem.
Any advice would be much appreciated
Thanks Simon