Hi Aravanis,
thanks for the details. Yes, it’s obvious that the index is used in your query. Out of interest, I’ve created a full-text index from 10 medline files (~800mb). I’ve listed the results below. Do you get similar performance for 10% of the data? I may try a larger medline portion in a second test.
Next, I would be interested what results you get when using Java’s default memory allocation (i.e., if you start BaseX without -Xmx, or if you assign a small value such as 64 or 256m). As BaseX relies on the caching patterns of your operating system, it could be that the memory assigned to BaseX cannot be utilized by the OS.
Best, Christian
Compiling: - simplifying descendant-or-self step(s) - applying full-text index
Query: //AbstractText[text() contains text 'KINASES']
Optimized Query: db:fulltext("medline", "KINASES")/parent::*:AbstractText
Result: - Hit(s): 84 Items - Updated: 0 Items - Printed: 104 KB - Read Locking: local [medline] - Write Locking: none
Timing: - Parsing: 0.0 ms - Compiling: 0.16 ms - Evaluating: 9.27 ms - Printing: 11.53 ms - Total Time: 20.98 ms ___________________________
2013/9/30 Aravanis, G. g.aravanis@student.tue.nl:
Hi Christian,
I have tried many queries and a typical example of what I get is the following:
Total Time: 21492.91 ms
Compiling:
- simplifying descendant-or-self step(s)
- applying full-text index
Query: //AbstractText[text() contains text 'KINASES']
Optimized Query: db:fulltext("1-100un", "KINASES")/parent::*:AbstractText
Result:
- Hit(s): 167 Items
- Updated: 0 Items
- Printed: 191 KB
- Read Locking: local [1-100un]
- Write Locking: none
Timing:
- Parsing: 0.0 ms
- Compiling: 0.27 ms
- Evaluating: 21324.7 ms
- Printing: 167.93 ms
- Total Time: 21492.91 ms
Query plan:
<QueryPlan> <CachedPath> <FTIndexAccess data="1-100un"> <FTWords> <Str value="KINASES" type="xs:string"/> </FTWords> </FTIndexAccess> <IterStep axis="parent" test="*:AbstractText"/> </CachedPath> </QueryPlan>
Regards, George ________________________________________ Από: Christian Grün [christian.gruen@gmail.com] Στάλθηκε: Δευτέρα, 30 Σεπτεμβρίου 2013 12:20 μμ Προς: Aravanis, G. Κοιν.: basex-talk@mailman.uni-konstanz.de Θέμα: Re: [basex-talk] Keep index in main memory-performance
Hi George,
I’m not sure if your queries are using the full-text at all, as a minimum of 8 seconds doesn’t sound that thrilling. Could you please check the output in the InfoView panel in the GUI (or the command-line output created via -V) and see what’s written there? Next, feel free to pass us on one of the queries you have tried.
Best, Christian ___________________________
2013/9/29 Aravanis, G. g.aravanis@student.tue.nl:
Hi community,
I have created a DB of around 7 GB documents in BaseX (100 first records of Medline) with full text index option enabled (index size 1 GB).
I test it locally on my laptop. The problem is that I observe great variations in the free text query times from 8s up to couple of minutes. Main memory is underused (allocated 2 GB) and the processor has quite low utilization. I observe high usage of the hard disk. I have read that the whole db could be loaded in the main memory, but this is not an option for an 100 GB db collection that I would like to have.
My question is if there is a possibility to load only the index in the main memory in order to accelerate the process of searching. Any other performance tips are very welcome.
Thank you, George Aravanis _______________________________________________ BaseX-Talk mailing list BaseX-Talk@mailman.uni-konstanz.de https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk
BaseX-Talk mailing list BaseX-Talk@mailman.uni-konstanz.de https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk