Hi James,
did you see: http://docs.basex.org/wiki/Index#Selective_Indexing
Otherwise for large amounts of data I found it helpful to create another database with only contents that need to be full text searchable or otherwise help to limit the scope of the data. Then, with more specific information (e.g. ttle or category names etc.) fetch the data from the main database.
As well, databases in BaseX are really lightweight, so you could create multiple dbs and spread the 40 GB over several dbs. Then, query multiple dbs at once. Could be faster than querying one large db (and updating is much faster as well).
Regards,
Max
2016-07-07 10:05 GMT+02:00 James Sears james.hn.sears@outlook.com:
Hi Liam - at the moment, I'm expecting our most common user case will be to search across all the books. It's not possible to be more specific until later on in the query.
In fact, it's likely that specific filtering would be via an export of the xml that gets imported into excel - where standard filters would be applied by the (non technical) user.
Subject: Re: [basex-talk] Improving performance in a 40GB database From: liam@w3.org To: james.hn.sears@outlook.com CC: basex-talk@mailman.uni-konstanz.de Date: Wed, 6 Jul 2016 14:56:43 -0400
On Wed, 2016-07-06 at 16:02 +0100, James Sears wrote:
let $void := prof:void(for $book in //book
Are you able to make that //book more specific? or can book elements occur at any level? BaseX is possibly fetching every element node in the database to see if it's an element of type book or not.
or can you look at the generated query plan in more detail?
Liam
-- Liam R. E. Quin liam@w3.org The World Wide Web Consortium (W3C)