Hello,
I'm trying to load a large dataset (Medline, 16.7 million records, 66Gb data on disk) into BaseX. The input data is divided into 563 files of 30000 records each. When I try to load the data into a database with the following command: create db path/dir medline_full I get the following error message: Error: "medline08n0507.xml" (Line 2620386): Document is too large for being processed.
The document itself can be loaded into a seperate database without problems and I also managed to load a merged file containing 1 million records, so the file size itself seems not to be the problem.
I used the gui implementation running on a quadcore server with SuSe 10.2 64bit and 8Gb of memory, although the java version is 32bit 1.6.0_18
Therefore my question, does BaseX have a datalimit, either on file-, record- or nodelevel? Or are there hardware limitiations that could result in such an error message?
With kind regards Judith