I can confirm that explicitly using a 64 bit JVM does not solve the issue I mentioned in my first email.
The problem persists; when working with a single server instance the creation of very small databases (~1-5kb) goes smoothly at the beginning but turns into a nightmare when reaching a certain threshold. Let's say around ~1M databases, the time to create subsequent databases increases dramatically. We're talking about a 10x duration increase.
At first I thought I was doing something wrong, but for the life of me I can't figure out what! The java process doesn't utilize any more or less resources than it does when just starting, so that's not it either. Do you have any information that might help here? Does a server do anything to the databases when creating new ones? E.g. when one creates a new database, is a list of all databases re-indexed or something like that? This could explain the issue. But other inspiration is welcome as well, because I don't know where to look.
Thanks in advance
-----Oorspronkelijk bericht----- Van: Christian Grün [mailto:christian.gruen@gmail.com] Verzonden: woensdag 16 augustus 2017 11:31 Aan: Bram Vanroy Bram.Vanroy@ugent.be CC: BaseX basex-talk@mailman.uni-konstanz.de Onderwerp: Re: [basex-talk] Could not reserve enough space for object heap
I am running 64 bit, a x86-x64 JVM so I'd thought I could assign a larger amount of memory to it.
Hm. You could try to specify -d64 to ensure that the 64bit JVM is used.
Why does it need that much memory to just launch? Is it trying to load some thing of all the existing databases in memory?
As the error is returned by Java, not by BaseX, it’s most probably returnend before any databases are touched. Maybe it helps to consult some other Java forum pages and compare the configuration with yours?
Hope this helps, Christian
Citeren Christian Grün christian.gruen@gmail.com:
Hi Bram,
As was indicated before, the JVM is restricted to appr. 1.4 -1.5, sometimes 1.6 GB on 32 bit Windows systems.
Generally (and as you have impressively proven), it is indeed possible to have millions of databases in a single BaseX instance. In practice, it is recommendable to find a tradeoff between number of databases and number of documents/XML nodes per database. If there is no obvious way how to distribute your documents, an additional mapping database might be reasonable in order to look up where a document has gone.
Hope this helps, Christian
On Tue, Aug 15, 2017 at 10:22 PM, Bram Vanroy Bram.Vanroy@ugent.be wrote:
Hi all
I'm running into an issue with many databases. I.e. one server instance with millions of databases. When creating all of these, I found that the more databases are included on the instance, the slower further database generation got. For instance, I could see in the logs that in the first < 10.000 databases the creation happened smoothly with around 50ms per file of 1-4kB. However, when having more and more databases for this server instance, things got very slow: for an XML file of 1-4kB the logs show ~600ms. This is terribly slow, as you can imagine.
At first I thought something was wrong with my hardware, but I checked on another system and the same issues arises. Then I thought maybe Java is doing something strange, so I figured I'd reboot and see if that cleared some stuff up. But now when I try to launch 'basex' or 'basexserver', I get the following message:
Could not reserve enough space for 1433600KB object heapI googled the issue, and it was suggested that I added a JAVA option to my system's variable (I'm on Windows 10 64 bit, BaseX 8.6.4) indicating the memory it could use. I set that to 2048MB. But still the same issue persists.
I have contacted the list before, with issues of generating millions of database with the same server instance, and this seems another one related to the problem. I am no expert AT ALL, but isn't it possible there is some sort of micro memory leak that only becomes apparent when creating an amount of databases of this magnitude? If not, other ideas are welcome as well. At least on how to get rid of the Java error mentioned above.
Kind regards
Bram Vanroy