When adding documents from a directory I get things working using a few small documents and
Add(docPath).execute(CONTEXT)
but when I want to add hundreds of documents (average size 1Mb) I get a memory error. I am using Jython and have -J-Xms1200M and -J-Xmx1200m
should I use an iterator and only add one document at a time? [this doesn't seem to me to be the way to use Add() ]
TIA
Peter
Dear Peter,
thanks for your feedback. Indeed the Add command is supposed to be used for smaller-sized inputs. To minimize the risk of runtime failures, the command caches the parsed input in main-memory before it's inserted in the database. In your scenario, I'd advise you to either…
1. Create a new database and add all documents via CreateDB 2. Adding all documents using your own iterator.
Option 1 shoud be the preferred choice as long as you don't plan to perform regular updates.
Hope this helps, Christian ___________________________
Christian Grün University of Konstanz Department of Computer & Information Science D-78457 Konstanz, Germany Tel: +49-7531-88-4449, Fax: +49-7531-88-3577 http://www.inf.uni-konstanz.de/~gruen
On Sun, Aug 22, 2010 at 3:22 PM, pw@themail.co.uk wrote:
When adding documents from a directory I get things working using a few small documents and
Add(docPath).execute(CONTEXT)
but when I want to add hundreds of documents (average size 1Mb) I get a memory error. I am using Jython and have -J-Xms1200M and -J-Xmx1200m
should I use an iterator and only add one document at a time? [this doesn't seem to me to be the way to use Add() ]
TIA
Peter
BaseX-Talk mailing list BaseX-Talk@mailman.uni-konstanz.de https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk
basex-talk@mailman.uni-konstanz.de
-
Christian Grün -
pw＠themail.co.uk