Hi Bram
How did you set the java heap space? Usual way would be by using the command line and setting e.g.
java -Xmx2g
However, the object which throws the error is already nearly 1,5GB in size, so if other stuff has to be allocates as well it could simply be that it is still not enough. Try increasing it to at least 4GB.
Indicating the memory leak: Well, it could certainly be, but I personally doubt it. First of all, having millions of databases is a lot, so some performance impact is expected. And given the scale of your databases I wouldn't call 600ms "terribly slow", you just seem to have a lot of data. When you create a new database there is also stuff which has to be checked (I would guess for example that BaseX checks that the database not already exists, which runtime obviously depends on the number of databases you already have), so I would guess the performance is somewhat expected. Of course, it might be possible to optimize this and try to make BaseX more performant for your given scenario.
Cheers Dirk
Senacor Technologies Aktiengesellschaft - Sitz: Eschborn - Amtsgericht Frankfurt am Main - Reg.-Nr.: HRB 105546 Vorstand: Matthias Tomann, Marcus Purzer - Aufsichtsratsvorsitzender: Daniel Grözinger
-----Ursprüngliche Nachricht----- Von: basex-talk-bounces@mailman.uni-konstanz.de [mailto:basex-talk-bounces@mailman.uni-konstanz.de] Im Auftrag von Bram Vanroy Gesendet: Dienstag, 15. August 2017 22:22 An: basex-talk@mailman.uni-konstanz.de Betreff: [basex-talk] Could not reserve enough space for object heap
Hi all
I'm running into an issue with many databases. I.e. one server instance with millions of databases. When creating all of these, I found that the more databases are included on the instance, the slower further database generation got. For instance, I could see in the logs that in the first < 10.000 databases the creation happened smoothly with around 50ms per file of 1-4kB. However, when having more and more databases for this server instance, things got very slow: for an XML file of 1-4kB the logs show ~600ms. This is terribly slow, as you can imagine.
At first I thought something was wrong with my hardware, but I checked on another system and the same issues arises. Then I thought maybe Java is doing something strange, so I figured I'd reboot and see if that cleared some stuff up. But now when I try to launch 'basex' or 'basexserver', I get the following message:
Could not reserve enough space for 1433600KB object heap
I googled the issue, and it was suggested that I added a JAVA option to my system's variable (I'm on Windows 10 64 bit, BaseX 8.6.4) indicating the memory it could use. I set that to 2048MB. But still the same issue persists.
I have contacted the list before, with issues of generating millions of database with the same server instance, and this seems another one related to the problem. I am no expert AT ALL, but isn't it possible there is some sort of micro memory leak that only becomes apparent when creating an amount of databases of this magnitude? If not, other ideas are welcome as well. At least on how to get rid of the Java error mentioned above.
Kind regards
Bram Vanroy