Re: [basex-talk] memory usage when writing files

Hi Lars,

Here is some background information for the reported behavior (sorry in advance if this is known to you anyway): The functional semantics of XQuery requires that repeated calls to fn:doc and fn:collection return the same documents. This can e.g. be shown by the following query:

doc('x.xml') is doc('x.xml')

As it's difficult to guess in advance which of the opened documents will possibly be requested again in the same query, they are all kept in main-memory until query evaluation is completed.

However, things are different with functions like fetch:xml [1]. You may need to tweak your query a little bit, because the function will always give you single XML documents.

Does this help? Christian

[1] http://docs.basex.org/wiki/Fetch_Module#fetch:xml

On Fri, Mar 27, 2015 at 10:41 AM, Lars Johnsen yoonsen@gmail.com wrote:

Hi all

Here is code that gradually eats up memory, whether run in GUI or as command. All it does is creating temporary collections out of folders, and writing them to file.

Is there a simple way to avoid this code to eat up memory? It runs out of memory (set at 12GB for command, 18GB in GUI) after 300 folders or so, and it has to process 20 000 of them.

Best Lars G Johnsen Norwegian National Library

Here is the actual code

(: process list of folders :)

for $collections in file:list($digibooks) let $html := $htmlfiles || substring-before($collections, "_ocr") || ".html"

 return

    (: code is rerun so check if files exist :)

    if (not(file:exists($html))) then
    try {

       (: create a temporary collection of the files and write result to

disk :)

        file:write(
          $html,
          db:digibok-to-html(
            collection($digibooks || $collections))
        )

    } catch * {
        $err:code
    }
    else
      ()