Hi Max - you have given me food for thought.
I was not aware of the selective indexing and will investigate.
Partitioning might well be the answer - I'll ask around and see if the people close to the data have some ideas about how data might be logically partitioned. Maybe having a master database - without the elements - "pointing" at various database instance partitions is the answer?
Thanks for the ideas.
From: Date: Fri, 8 Jul 2016 15:24:16 +0200 Subject: Re: [basex-talk] Improving performance in a 40GB database To: james.hn.sears@outlook.com CC: basex-talk@mailman.uni-konstanz.de
Hi James,
did you see: http://docs.basex.org/wiki/Index#Selective_Indexing
Otherwise for large amounts of data I found it helpful to create another database with only contents that need to be full text searchable or otherwise help to limit the scope of the data. Then, with more specific information (e.g. ttle or category names etc.) fetch the data from the main database.
As well, databases in BaseX are really lightweight, so you could create multiple dbs and spread the 40 GB over several dbs. Then, query multiple dbs at once. Could be faster than querying one large db (and updating is much faster as well).
Regards,
Max