Hi Tim,
Thanks for testing the new Store Module.
The fluctuations in memory consumption and the long lags mainly indicate that the garbage collector is pretty busy. Some data structures will be much smaller if serialized to disk, so I assume your amount of data is simply too large to be efficiently organized in main memory. If you need all data in practce, you could think about only preserving meta data and index structures in memory.
Feel free to provide us with test cases if you feel that the behavior is suspicious enough.
Cheers, Christian
On Tue, Nov 8, 2022 at 5:53 AM Tim Thompson timathom@gmail.com wrote:
Hello,
I've been experimenting with the new Store module, which seems very handy. I have a custom store that is about 2GB in size. In the GUI (BaseX 10.3 beta), when I try to load it into memory with store:read(), the reported memory usage appears to max out (with 32GB of available memory):
Total Memory: 32 GB Reserved Memory: 32 GB Used Memory: 32 GB
However, monitoring memory usage with htop shows a smaller increase in memory usage: from 5GB at baseline, it peaks around 15GB, then goes down to 9GB, all while the BaseX GUI appears to be frozen.
After a couple of minutes, the store does load, but GUI performance seems to be degraded, and it continues to report max memory usage.
Any thoughts?
Tim
-- Tim A. Thompson (he, him) Librarian for Applied Metadata Research Yale University Library www.linkedin.com/in/timathompson