Hi Fabrice,<div><br></div><div>indeed, the problem with large updates is the size of the pending update list.</div><div><br></div><div>1) If you want to stick with XQuery you could spread the complete add/replace</div><div>

over several queries and consequently reduce the size of the pending update</div><div>list. I&#39;m aware that this may not be a completely self-maintaining solution if</div><div>the size of your update varies considerably.</div>

<div><br></div><div>At some point in the future (I hope within the next few months) we plan to</div><div>integrate another XQUF optimization that caches the expensive parts of the</div><div>pending update list on disk. Most likely this will solve your problem, but until</div>

<div>then ...</div><div><br></div><div>2) ... switching to a sequence of BaseX commands is an option. Each BaseX</div><div>command is executed separately (at least update-wise), which keeps the</div><div>the PUL at bay. On the other hand, dividing the bulk update into its atomic</div>

<div>parts comes with a price tag ... Performance should be ok if you limit yourself</div><div>to db:add. Documents are added to the end of the BaseX table. In contrast,</div><div>the table access pattern for db:replace is more random.</div>

<div><br></div><div><b>In Conclusion - </b>if you are able to spread out your update over several XQueries,</div><div>that&#39;s the way to go. The more adds/replaces you combine in a query, the faster</div><div>it will be. If dividing the update into smaller parts is a problem, a sequence of</div>

<div>commands may be your only choice. You could also consider a mixed solution:</div><div>commands for all &#39;adds&#39; (as they&#39;re fast anyway) and xqueries for all replaces ...</div><div><br></div><div>Hope this helps!</div>

<div><br></div><div>Cheers,</div><div>Lukas</div><div><br></div><div><br></div><div><br><div class="gmail_quote">On Sun, Nov 11, 2012 at 8:27 PM, Fabrice ETANCHAUD <span dir="ltr">&lt;<a href="mailto:fabrice.etanchaud@orange.fr" target="_blank">fabrice.etanchaud@orange.fr</a>&gt;</span> wrote:<br>


<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><p>Dear all at basex,</p>
<p> </p>
<p>I am wondering what could be the most efficient way to add/replace a lot of documents (several thousands) at the same time.</p>
<p> </p>
<p>I tried db:add/db:replace but this gives heap memory overflow because of the pending list...</p>
<p> </p>
<p>Will I have better results with a .bxs with add/replace commands (and autoflush deactivated) ?</p>
<p> </p>
<p>Did any other user find his way in this use case ?</p>
<p> </p>
<p>Best regards,</p>
<p> </p>
<p>Fabrice</p><br>_______________________________________________<br>
BaseX-Talk mailing list<br>
<a href="mailto:BaseX-Talk@mailman.uni-konstanz.de" target="_blank">BaseX-Talk@mailman.uni-konstanz.de</a><br>
<a href="https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk" target="_blank">https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk</a><br>
<br></blockquote></div><br></div>