Hello,
We have a data transformation pipeline that works with XML files of different sizes, sometimes big (up to several gigabytes). We are using BaseX to do the transformations. For smaller files we use the MAINMEM option, because the whole database can fit in memory. But for some files we can't do that, but all databases we create are simply disposable, and we'd like to control where we put them, and destroy them after the processing. Is there a special option or any other way to specify where particular database's files will reside? How to specify that when we call CreateDB command?
thank you, Vladimir
Hi Vladimir,
The DBPATH option is the one you’ll need to assign. As it’s a global option, it should be assigned at startup time [1].
Best, Christian
[1] https://docs.basex.org/wiki/Options
On Thu, Jul 16, 2020 at 6:11 AM Vladimir Churyukin vladich@gmail.com wrote:
Hello,
We have a data transformation pipeline that works with XML files of different sizes, sometimes big (up to several gigabytes). We are using BaseX to do the transformations. For smaller files we use the MAINMEM option, because the whole database can fit in memory. But for some files we can't do that, but all databases we create are simply disposable, and we'd like to control where we put them, and destroy them after the processing. Is there a special option or any other way to specify where particular database's files will reside? How to specify that when we call CreateDB command?
thank you, Vladimir
Yes, I've seen that option. But there is no way to set it per database, correct? I'm asking because by nature our operations are ad-hoc, we don't really "startup" the instances, we create a database, process the data, then destroy the database. Is there some internal limitation why this option needs to be global?
thank you, -Vladimir
On Thu, Jul 16, 2020 at 4:36 AM Christian Grün christian.gruen@gmail.com wrote:
Hi Vladimir,
The DBPATH option is the one you’ll need to assign. As it’s a global option, it should be assigned at startup time [1].
Best, Christian
[1] https://docs.basex.org/wiki/Options
On Thu, Jul 16, 2020 at 6:11 AM Vladimir Churyukin vladich@gmail.com wrote:
Hello,
We have a data transformation pipeline that works with XML files of
different sizes, sometimes big (up to several gigabytes).
We are using BaseX to do the transformations. For smaller files we use the MAINMEM option, because the whole database
can fit in memory. But for some files we can't do that, but all databases we create are simply disposable, and we'd like to control where we put them, and destroy them after the processing.
Is there a special option or any other way to specify where particular
database's files will reside?
How to specify that when we call CreateDB command?
thank you, Vladimir
Right, the option is global. As BaseX has been designed to serve concurrent requests, it would introduce unexpected side effects of the path was changed at runtime.
If you are careful, you can try to change the path by assigning a new value to Context.soptions.
Vladimir Churyukin vladich@gmail.com schrieb am Do., 16. Juli 2020, 17:33:
Yes, I've seen that option. But there is no way to set it per database, correct? I'm asking because by nature our operations are ad-hoc, we don't really "startup" the instances, we create a database, process the data, then destroy the database. Is there some internal limitation why this option needs to be global?
thank you, -Vladimir
On Thu, Jul 16, 2020 at 4:36 AM Christian Grün christian.gruen@gmail.com wrote:
Hi Vladimir,
The DBPATH option is the one you’ll need to assign. As it’s a global option, it should be assigned at startup time [1].
Best, Christian
[1] https://docs.basex.org/wiki/Options
On Thu, Jul 16, 2020 at 6:11 AM Vladimir Churyukin vladich@gmail.com wrote:
Hello,
We have a data transformation pipeline that works with XML files of
different sizes, sometimes big (up to several gigabytes).
We are using BaseX to do the transformations. For smaller files we use the MAINMEM option, because the whole database
can fit in memory. But for some files we can't do that, but all databases we create are simply disposable, and we'd like to control where we put them, and destroy them after the processing.
Is there a special option or any other way to specify where particular
database's files will reside?
How to specify that when we call CreateDB command?
thank you, Vladimir
Ah no, I'm not talking about changing it in runtime, I'm talking about specifying the path on database creation, for example when CreateDB command is executed. There shouldn't be concurrency problems at the moment of database creation, correct?
-Vladimir
On Thu, Jul 16, 2020 at 8:41 AM Christian Grün christian.gruen@gmail.com wrote:
Right, the option is global. As BaseX has been designed to serve concurrent requests, it would introduce unexpected side effects of the path was changed at runtime.
If you are careful, you can try to change the path by assigning a new value to Context.soptions.
Vladimir Churyukin vladich@gmail.com schrieb am Do., 16. Juli 2020, 17:33:
Yes, I've seen that option. But there is no way to set it per database, correct? I'm asking because by nature our operations are ad-hoc, we don't really "startup" the instances, we create a database, process the data, then destroy the database. Is there some internal limitation why this option needs to be global?
thank you, -Vladimir
On Thu, Jul 16, 2020 at 4:36 AM Christian Grün christian.gruen@gmail.com wrote:
Hi Vladimir,
The DBPATH option is the one you’ll need to assign. As it’s a global option, it should be assigned at startup time [1].
Best, Christian
[1] https://docs.basex.org/wiki/Options
On Thu, Jul 16, 2020 at 6:11 AM Vladimir Churyukin vladich@gmail.com wrote:
Hello,
We have a data transformation pipeline that works with XML files of
different sizes, sometimes big (up to several gigabytes).
We are using BaseX to do the transformations. For smaller files we use the MAINMEM option, because the whole
database can fit in memory. But for some files we can't do that, but all databases we create are simply disposable, and we'd like to control where we put them, and destroy them after the processing.
Is there a special option or any other way to specify where particular
database's files will reside?
How to specify that when we call CreateDB command?
thank you, Vladimir
basex-talk@mailman.uni-konstanz.de
-
Christian Grün -
Vladimir Churyukin