Hi all, I have downloaded/started the latest dev snapshot of 7.7. My database consists of an extract of the OpenStreetMap data (only Germany) which is 29GB in size (xml/OSM format). I have managed to load the file into the database but the indexes creation procedure fails. I have tried -Xmx4g, but if the progress bar of the GUI is right, I would need the 10x of heap space to create the attribute indexes.
Any ideas/workarounds to create the attribute indexes for such a big database?
Thanks!
Hi Erdal,
the index split heuristic seems to fail in your case, so you could try to set some fixed index split sizes [1]. I agree this requires some manual fine tuning.
Best, Christian
[1] http://docs.basex.org/wiki/Indexes#Index_Construction ___________________________
On Tue, Apr 23, 2013 at 7:29 PM, Erdal Karaca erdal.karaca.de@gmail.com wrote:
Hi all, I have downloaded/started the latest dev snapshot of 7.7. My database consists of an extract of the OpenStreetMap data (only Germany) which is 29GB in size (xml/OSM format). I have managed to load the file into the database but the indexes creation procedure fails. I have tried -Xmx4g, but if the progress bar of the GUI is right, I would need the 10x of heap space to create the attribute indexes.
Any ideas/workarounds to create the attribute indexes for such a big database?
Thanks!
BaseX-Talk mailing list BaseX-Talk@mailman.uni-konstanz.de https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk
Hi Christian, Thanks for the quick reply!
What value should I set the option INDEXSPLITSIZE to?
Thanks!
2013/4/23 Christian Grün christian.gruen@gmail.com
Hi Erdal,
the index split heuristic seems to fail in your case, so you could try to set some fixed index split sizes [1]. I agree this requires some manual fine tuning.
Best, Christian
[1] http://docs.basex.org/wiki/Indexes#Index_Construction ___________________________
On Tue, Apr 23, 2013 at 7:29 PM, Erdal Karaca erdal.karaca.de@gmail.com wrote:
Hi all, I have downloaded/started the latest dev snapshot of 7.7. My database consists of an extract of the OpenStreetMap data (only
Germany)
which is 29GB in size (xml/OSM format). I have managed to load the file into the database but the indexes
creation
procedure fails. I have tried -Xmx4g, but if the progress bar of the GUI
is
right, I would need the 10x of heap space to create the attribute
indexes.
Any ideas/workarounds to create the attribute indexes for such a big database?
Thanks!
BaseX-Talk mailing list BaseX-Talk@mailman.uni-konstanz.de https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk
Please first check out my link; you’ll have to try a little. ___________________________
On Tue, Apr 23, 2013 at 7:58 PM, Erdal Karaca erdal.karaca.de@gmail.com wrote:
Hi Christian, Thanks for the quick reply!
What value should I set the option INDEXSPLITSIZE to?
Thanks!
2013/4/23 Christian Grün christian.gruen@gmail.com
Hi Erdal,
the index split heuristic seems to fail in your case, so you could try to set some fixed index split sizes [1]. I agree this requires some manual fine tuning.
Best, Christian
[1] http://docs.basex.org/wiki/Indexes#Index_Construction ___________________________
On Tue, Apr 23, 2013 at 7:29 PM, Erdal Karaca erdal.karaca.de@gmail.com wrote:
Hi all, I have downloaded/started the latest dev snapshot of 7.7. My database consists of an extract of the OpenStreetMap data (only Germany) which is 29GB in size (xml/OSM format). I have managed to load the file into the database but the indexes creation procedure fails. I have tried -Xmx4g, but if the progress bar of the GUI is right, I would need the 10x of heap space to create the attribute indexes.
Any ideas/workarounds to create the attribute indexes for such a big database?
Thanks!
BaseX-Talk mailing list BaseX-Talk@mailman.uni-konstanz.de https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk
basex-talk@mailman.uni-konstanz.de
-
Christian Grün -
Erdal Karaca