Hi Dirk and Fabrice!, thank you so much for your help with these subjects!, it has been of great help to read this answers, the link that Fabrice suggested and reading about  indexes also. Now is much clear for me how BaseX works!, I have to recognize some part of my brain still fights and sticks to RDBMS model after more than 12 years working with it, but at the same time I'm feeling pretty excited about working with BaseX. I'll keep you guys posted on how everything goes on this project!, and will come back to you also if I get into some other question ;)

Thank you so much!

Oscar H


2014-09-25 5:54 GMT-05:00 Dirk Kirsten <dk@basex.org>:
Hello Oscar,

As Fabrice already suggested, maintaining a separate collection with
node-id mappings might be a viable solution. Another option could be to
split your documents up in a way that the relevant information is stored
in one collection (which is indexed) and all the other supplemental
information is stored in another collection. This way, the first
collection should be rather small and the text index should work fine.

>
> So, from all the information we receive, at this moment I estimate we only
> need around 25%, I though about having different databases with full and
> partial information but the thing is that somehow the requirements are not
> entirely defined on one hand, and on the other, there's information that we
> use on the queries and some other that we still need to display to its
> owner and that we're displaying using XSLT.

If you need to display additional information, it is no problem to
access multiple collections in a single XQuery. So splitting up the data
should not be a show-stopper.

>
> == Question 1: Indexes are only required for some fields ==
> We usually need to locate the records by some id, or query over some of the
> elements available on the XML files, but those are pretty much always the
> same, so those are the elements that I'd like to have indexed. That's why I
> don't see a reason for having indexes over the contents of all the elements
> since is unlikely (at least right now) we'll make use of those and instead
> they consume a lot of hard drive.


You currently can't define an index to just select certain elements. It
would certainly be very nice to have super-flexible indexes, but as you
can guess this is a non-trivial task. Maintaining separate collections
is currently the way to go.
>
> == Question 2: to store files on the filesystem or as raw on BaseX? ==
> Right now, we're storing the information we receive as XML files on the
> file system on a RAID 10, anyway what's your advice?, to keep the files
> stored on the filesystem directly or to let BaseX handle those (I think
> this is the difference between add/replace and store commands right?), is
> there any article you could point me I could use for reference?, as I see
> BaseX right now it is handling the queries and the index information right
> now but depends on the filesystem to retrieve the entire document, am I
> right?

If you have a non-small collection of documents, simply storing them in
the file system is certainly not very performant. Using XQuery, you can
read from the file system, but that means parsing has to be executed
each time.

As Fabrice pointed out (thanks!), the concept is different than what you
described here. Using add/replace parses an XML file and adds it to the
database. During parsing, the XML file will be stored in a binary
format, to be able to optimize queries and to access relevant data much
faster. You can not add/replace any binary file to BaseX, as it would
not be parseable. Store, on the other hand, simply copies the file and
can therefore handle any binary file. This is useful if you e.g. want to
store media files within your DB, but you most likely do not want to
store XML files in a binary way, as it is similar in performance as
reading from the plain filesystem.

In short: You most likely want to add your documents to a collection.

>
> == Question 3: dynamic optimize and index updates? ==
> As you can imagine, I'll need to have the indexes updated
> since"data-mining" will be done with the information from the people
> registered on it. I've seen is not possible to run the "optimize" command
> while the app is up, I'm not sure about the indexes getting updated on real
> time either, but this somehow is troubling me since the idea is to have the
> app running 24x7, and if we get to have a lot of registered users, to
> update the indexes or to optimize the db will take a long time, isn't it?.
> So any strategies on this?

I don't quite get what you mean by "optimize can not be run when the app
is up". Optimize can not be run if the database is opened by another
context (as it is updating and we maintain ACID), but your app shouldn't
hold open the database all the time.

One option you might want to look into is updating indexes (see
http://docs.basex.org/wiki/Options#UPDINDEX), it might be beneficial for
your use case. You still have to trigger the indexing by using optimize.

One common strategy for such scenarios is also to maintain separate
collections. One with the most current data, which is not indexed and
can be updated quite fast. And then another collection with the bulk of
the data, which is indexed and can be access fast. A cron job would than
schedule to current data to be transferred to the other collection
during times of low load on the server. This way, your updates will be
performed on a rather small collection without the need to optimize the
indexes all the time, while read operations can be fast as the majority
of the data is nicely indexed. Again, accessing both collections is no
hassle with XQuery.

>
> == Question 4: connection pooling ==
> I have only found XQJ-Pool to be used with BaseX, does anybody know about
> any other pooling mechanism available for BaseX?

I am not aware of any.

Cheers,
Dirk


--
Dirk Kirsten, BaseX GmbH, http://basex.org
|-- Firmensitz: Blarerstrasse 56, 78462 Konstanz
|-- Registergericht Freiburg, HRB: 708285, Geschäftsführer:
|   Dr. Christian Grün, Dr. Alexander Holupirek, Michael Seiferle
`-- Phone: 0049 7531 28 28 676, Fax: 0049 7531 20 05 22