A RESTXQ definition could look something like this:

|declare| |    %rest:path||(||"/a/{$client}/{$path}"||)| |function| |yourmodule:test(||$client as xs:string||, ||$||path ||as xs:string||||) {| |let $db := switch ($client)| |    case "clientA" return db:open("dbA")| ||| | |    case "clientA" return db:open("dbA")||| |        default return db:open("dbX")| |    return yourmodule:do-something-with($db, $path) | |};|

And the called function |do-something-with| could also depend on the client

Am 21.02.2020 um 03:48 schrieb DYER Chris:

Hi Markus,

Our intent is to use REST and RESTXQ interfaces to interact with data on a /single/ database using some server side XQueries.

However, we may have 2 (or more) databases provided by some server, and associated xqueries are specific to the contents of each database (as the contents of each of these databases may be structured differently).

We would like for a client to be able to open a single database and be able to run associated XQueries, without requiring client side knowledge of which server side xqueries are associated with the database it has opened.

Does this help describe what we are trying to achieve?

Cheers, Chris

*From:*BaseX-Talk basex-talk-bounces@mailman.uni-konstanz.de *On Behalf Of *Markus Wittenberg *Sent:* Friday, 21 February 2020 12:08 PM *To:* basex-talk@mailman.uni-konstanz.de *Subject:* Re: [basex-talk] Database specific xquery modules

Hi Chris,

as you might already know, BaseX has built-in REST support [1] which you can use to query and update data via HTTP. You can also use the RESTXQ [2] functionality to define arbitrary server side XQueries which can also be requested by HTTP and are also suitable for web applications. Both methods are not restricted to a single database.

Best regards,

Markus

[1] http://docs.basex.org/wiki/REST

[2] http://docs.basex.org/wiki/RESTXQ

Am 21.02.2020 um 02:05 schrieb DYER Chris:

I am looking for some ideas on some architecture surrounding
xquery modules.

Our situation is that we have multiple client applications in
various languages that will connect to a database.

We will have some reasonably complex xqueries to extract data in
certain structures that will be used by each of these clients.

So to avoid having to duplicate the query logic we would like the
xqueries to reside "in" the database.

Additionally, we would like to use the same BaseX installation for
multiple databases, corresponding to multiple projects.

The data in each of these databases may be against a different
schema and our xqueries are essentially tied to the schema of the
data. Thus, each database may need slightly different xqueries.

The xquery library modules looks to be global for the BaseX
instance, as does the loading of main modules off of disk.

Is there any way to tie the query module or file to a specific
database?

We are trying to avoid being limited to 1 database per BaseX instance.

An alternative would be to embed the xqueries within our client
applications, but that involves duplication. Also, we are not sure
if there are any performance difference with having the queries
stored in the repo vs client side?

For background, we are looking at migrating from oracle to BaseX.
So, we are effectively looking for an equivalent of multiple
clients connecting to a userspace and using stored procedures
which are specific to that userspace.

Any ideas on how to achieve this currently? Or is this something
that BaseX could support?

Cheers,

Chris

-- Markus Wittenberg Tel +49 (0)341 248 475 36 Mailwittenberg@axxepta.de mailto:wittenberg@axxepta.de

---

axxepta solutions GmbH Lehmgrubenweg 17, 88131 Lindau Amtsgericht Berlin HRB 97544B Geschäftsführer: Karsten Becke, Maximilian Gärber

If you want to use RESTXQ you would have to put the modules, or least the functions defining the endpoints, in the webapp directory and not in the repo directory. And far as I know, BaseX doesn't support installation for modules in webapp as for in repo. So you have to explicitly change/add files in webapp directory.

You could write a module for each database, defining its own endpoints, and maybe still reuse some code by importing a module with common functions in every specific module.

Am 21.02.2020 um 07:11 schrieb DYER Chris:

Hi Markus,

Not quite what I am aiming for.

We /do/ want a single BaseX Instance to provide access to multiple databases.

Further to this though, a set of client applications will only connect to one database. We have multiple sets of client applications, with each set each connecting to one database.

We do want to modularise the functions, however as an example we want to be able to have 2 versions of the same function, where 1 version works with the schema of the data in database 1, and the second version with a different schema in database 2.

If I could explicitly link the function modules with a database, I could call the same function from both application sets, and use the context of the open database to call the correct version of the function.

**

An example situation may help

Say we have Database for “Black Books”, which has book data in xml structures which conform to some version (1.1) of a schema.

We want a query stored on the server that returns books on art

Get-Books-About-Art

We have a client application specifically for this database that calls this function.

It will only need to query the data in this database.

Now we add a second Database for "Goliath Books"

The book data in this database uses a modified xml structure (version 2 of a schema)

We have a client application for this database that also wants to get books about art.

The query for this database is slightly different because the xml schema has changed.

I would like to store both versions of Get-Books-About-Art in the database.

Then for each client to connect to their associated database, and be able to execute the correct version of the xquery function.

The idea is that each time I add a new database that has data in some format, I can install associated query modules that are restricted to the context of that database to avoid conflicts with existing databases.

Maybe what I am looking for is effectively a database specific repo to install modules to, rather than a basex install repo.

*From:*Markus Wittenberg wittenberg@axxepta.de *Sent:* Friday, 21 February 2020 2:15 PM *To:* DYER Chris Chris.Dyer@sydac.com; basex-talk@mailman.uni-konstanz.de *Subject:* Re: [basex-talk] Database specific xquery modules

A RESTXQ definition could look something like this:

|declare|

|%rest:path("/a/{$client}/{$path}")|

|function| |yourmodule:test($client as xs:string, $path as xs:string) {|

|let $db := switch ($client)|

|    case "clientA" return db:open("dbA")|

|    case "clientA" return db:open("dbA")|

|        default return db:open("dbX")|

|    return yourmodule:do-something-with($db, $path)|

|};|

And the called function |do-something-with| could also depend on the client

Am 21.02.2020 um 03:48 schrieb DYER Chris:

Hi Markus,

Our intent is to use REST and RESTXQ interfaces to interact with
data on a /single/ database using some server side XQueries.

However, we may have 2 (or more) databases provided by some
server, and associated xqueries are specific to the contents of
each database (as the contents of each of these databases may be
structured differently).

We would like for a client to be able to open a single database
and be able to run associated XQueries, without requiring client
side knowledge of which server side xqueries are associated with
the database it has opened.

Does this help describe what we are trying to achieve?

Cheers,
Chris

*From:*BaseX-Talk <basex-talk-bounces@mailman.uni-konstanz.de>
<mailto:basex-talk-bounces@mailman.uni-konstanz.de> *On Behalf Of
*Markus Wittenberg
*Sent:* Friday, 21 February 2020 12:08 PM
*To:* basex-talk@mailman.uni-konstanz.de
<mailto:basex-talk@mailman.uni-konstanz.de>
*Subject:* Re: [basex-talk] Database specific xquery modules

 Hi Chris,

as you might already know, BaseX has built-in REST support [1]
which you can use to query and update data via HTTP. You can also
use the RESTXQ [2] functionality to define arbitrary server side
XQueries which can also be requested by HTTP and are also suitable
for web applications. Both methods are not restricted to a single
database.

Best regards,

  Markus

[1] http://docs.basex.org/wiki/REST

[2] http://docs.basex.org/wiki/RESTXQ

Am 21.02.2020 um 02:05 schrieb DYER Chris:

    I am looking for some ideas on some architecture surrounding
    xquery modules.

    Our situation is that we have multiple client applications in
    various languages that will connect to a database.

    We will have some reasonably complex xqueries to extract data
    in certain structures that will be used by each of these clients.

    So to avoid having to duplicate the query logic we would like
    the xqueries to reside "in" the database.

    Additionally, we would like to use the same BaseX installation
    for multiple databases, corresponding to multiple projects.

    The data in each of these databases may be against a different
    schema and our xqueries are essentially tied to the schema of
    the data. Thus, each database may need slightly different
    xqueries.

    The xquery library modules looks to be global for the BaseX
    instance, as does the loading of main modules off of disk.

    Is there any way to tie the query module or file to a specific
    database?

    We are trying to avoid being limited to 1 database per BaseX
    instance.

    An alternative would be to embed the xqueries within our
    client applications, but that involves duplication. Also, we
    are not sure if there are any performance difference with
    having the queries stored in the repo vs client side?

    For background, we are looking at migrating from oracle to
    BaseX. So, we are effectively looking for an equivalent of
    multiple clients connecting to a userspace and using stored
    procedures which are specific to that userspace.

    Any ideas on how to achieve this currently? Or is this
    something that BaseX could support?

    Cheers,

    Chris

-- 

Markus Wittenberg



Tel +49 (0)341 248 475 36

Mailwittenberg@axxepta.de  <mailto:wittenberg@axxepta.de>



----



axxepta solutions GmbH

Lehmgrubenweg 17, 88131 Lindau



Amtsgericht Berlin HRB 97544B

Geschäftsführer: Karsten Becke, Maximilian Gärber

-- Markus Wittenberg Tel +49 (0)341 248 475 36 Mailwittenberg@axxepta.de mailto:wittenberg@axxepta.de

---

axxepta solutions GmbH Lehmgrubenweg 17, 88131 Lindau Amtsgericht Berlin HRB 97544B Geschäftsführer: Karsten Becke, Maximilian Gärber

Hi Steve, I do not think what we are trying to achieve is necessarily OO related (although I can see why it may look that way). It may be more that we would be looking at another layer of encapsulation of some basex features.

Your suggestion on storing the queries within the database and then evaluating them does sound like a possible option for us. It would be nice to achieve this scoping of the queries using explicitly registered function modules.

This is all based on what we are currently achieving with our oracle database solution. We have a single oracle database server, and we have developers create new user spaces on that server which encapsulates a set of tables and associated stored procedures (functions) for working with those tables. Those functions are only visible within the scope of the user space

It sounds like this sort of use case would generally be handled with separate basex installs/servers. So I am possibly looking at a feature request.

Cheers, Chris

From: Majewski, Steven Dennis (sdm7g) sdm7g@virginia.edu Sent: Saturday, 22 February 2020 12:37 AM To: DYER Chris Chris.Dyer@sydac.com; basex-talk@mailman.uni-konstanz.de Cc: Markus Wittenberg wittenberg@axxepta.de Subject: Re: [basex-talk] Database specific xquery modules

I’m not sure this is the best approach, but you seem determined to try to figure out how to turn XQuery into an Object-Oriented language, so, if you want to bind query functions to a particular database, you can put the functions in the database, and use xquery:eval() http://docs.basex.org/wiki/XQuery_Module to call the function as a node() argument to eval. You would be using the context database as the “self” variable and xquery:eval() as dispatcher.

If you don’t literally want to bind functions to DB, but instead dispatch by the schema of the data, you could use a typeswitch if the schema have different namespaces or root elements,. ( I’ve tried this to switch between processing TEIP4 and TEIP5, for example. )

— Steve M.

On Feb 21, 2020, at 1:41 AM, Markus Wittenberg <wittenberg@axxepta.demailto:wittenberg@axxepta.de> wrote:

If you want to use RESTXQ you would have to put the modules, or least the functions defining the endpoints, in the webapp directory and not in the repo directory. And far as I know, BaseX doesn't support installation for modules in webapp as for in repo. So you have to explicitly change/add files in webapp directory. You could write a module for each database, defining its own endpoints, and maybe still reuse some code by importing a module with common functions in every specific module. Am 21.02.2020 um 07:11 schrieb DYER Chris: Hi Markus, Not quite what I am aiming for.

We do want a single BaseX Instance to provide access to multiple databases.

Further to this though, a set of client applications will only connect to one database. We have multiple sets of client applications, with each set each connecting to one database.

We do want to modularise the functions, however as an example we want to be able to have 2 versions of the same function, where 1 version works with the schema of the data in database 1, and the second version with a different schema in database 2.

If I could explicitly link the function modules with a database, I could call the same function from both application sets, and use the context of the open database to call the correct version of the function.

An example situation may help Say we have Database for “Black Books”, which has book data in xml structures which conform to some version (1.1) of a schema. We want a query stored on the server that returns books on art Get-Books-About-Art We have a client application specifically for this database that calls this function. It will only need to query the data in this database.

Now we add a second Database for "Goliath Books" The book data in this database uses a modified xml structure (version 2 of a schema) We have a client application for this database that also wants to get books about art. The query for this database is slightly different because the xml schema has changed.

I would like to store both versions of Get-Books-About-Art in the database. Then for each client to connect to their associated database, and be able to execute the correct version of the xquery function.

The idea is that each time I add a new database that has data in some format, I can install associated query modules that are restricted to the context of that database to avoid conflicts with existing databases.

Maybe what I am looking for is effectively a database specific repo to install modules to, rather than a basex install repo.

From: Markus Wittenberg wittenberg@axxepta.demailto:wittenberg@axxepta.de Sent: Friday, 21 February 2020 2:15 PM To: DYER Chris Chris.Dyer@sydac.commailto:Chris.Dyer@sydac.com; basex-talk@mailman.uni-konstanz.demailto:basex-talk@mailman.uni-konstanz.de Subject: Re: [basex-talk] Database specific xquery modules

A RESTXQ definition could look something like this: declare %rest:path("/a/{$client}/{$path}") function yourmodule:test($client as xs:string, $path as xs:string) { let $db := switch ($client) case "clientA" return db:open("dbA") case "clientA" return db:open("dbA") default return db:open("dbX") return yourmodule:do-something-with($db, $path) }; And the called function do-something-with could also depend on the client

Am 21.02.2020 um 03:48 schrieb DYER Chris: Hi Markus, Our intent is to use REST and RESTXQ interfaces to interact with data on a single database using some server side XQueries.

However, we may have 2 (or more) databases provided by some server, and associated xqueries are specific to the contents of each database (as the contents of each of these databases may be structured differently).

We would like for a client to be able to open a single database and be able to run associated XQueries, without requiring client side knowledge of which server side xqueries are associated with the database it has opened.

Does this help describe what we are trying to achieve?

Cheers, Chris

From: BaseX-Talk basex-talk-bounces@mailman.uni-konstanz.demailto:basex-talk-bounces@mailman.uni-konstanz.de On Behalf Of Markus Wittenberg Sent: Friday, 21 February 2020 12:08 PM To: basex-talk@mailman.uni-konstanz.demailto:basex-talk@mailman.uni-konstanz.de Subject: Re: [basex-talk] Database specific xquery modules

Hi Chris, as you might already know, BaseX has built-in REST support [1] which you can use to query and update data via HTTP. You can also use the RESTXQ [2] functionality to define arbitrary server side XQueries which can also be requested by HTTP and are also suitable for web applications. Both methods are not restricted to a single database. Best regards, Markus

[1] http://docs.basex.org/wiki/REST [2] http://docs.basex.org/wiki/RESTXQ

Am 21.02.2020 um 02:05 schrieb DYER Chris: I am looking for some ideas on some architecture surrounding xquery modules.

Our situation is that we have multiple client applications in various languages that will connect to a database. We will have some reasonably complex xqueries to extract data in certain structures that will be used by each of these clients. So to avoid having to duplicate the query logic we would like the xqueries to reside "in" the database.

Additionally, we would like to use the same BaseX installation for multiple databases, corresponding to multiple projects. The data in each of these databases may be against a different schema and our xqueries are essentially tied to the schema of the data. Thus, each database may need slightly different xqueries.

The xquery library modules looks to be global for the BaseX instance, as does the loading of main modules off of disk. Is there any way to tie the query module or file to a specific database?

We are trying to avoid being limited to 1 database per BaseX instance.

An alternative would be to embed the xqueries within our client applications, but that involves duplication. Also, we are not sure if there are any performance difference with having the queries stored in the repo vs client side?

For background, we are looking at migrating from oracle to BaseX. So, we are effectively looking for an equivalent of multiple clients connecting to a userspace and using stored procedures which are specific to that userspace.

Any ideas on how to achieve this currently? Or is this something that BaseX could support?

Cheers, Chris

--

Markus Wittenberg

Tel +49 (0)341 248 475 36

Mail wittenberg@axxepta.demailto:wittenberg@axxepta.de

----

axxepta solutions GmbH

Lehmgrubenweg 17, 88131 Lindau

Amtsgericht Berlin HRB 97544B

Geschäftsführer: Karsten Becke, Maximilian Gärber

--

Markus Wittenberg

Tel +49 (0)341 248 475 36

Mail wittenberg@axxepta.demailto:wittenberg@axxepta.de

----

axxepta solutions GmbH

Lehmgrubenweg 17, 88131 Lindau

Amtsgericht Berlin HRB 97544B

Geschäftsführer: Karsten Becke, Maximilian Gärber

--

Markus Wittenberg

Tel +49 (0)341 248 475 36

Mail wittenberg@axxepta.demailto:wittenberg@axxepta.de

----

axxepta solutions GmbH

Lehmgrubenweg 17, 88131 Lindau

Amtsgericht Berlin HRB 97544B

Geschäftsführer: Karsten Becke, Maximilian Gärber

Hi Christian,

As I understand Chris' issue, it is that he wants to be able to (for example) call the following two REST GETs ...

localhost:8984/rest/BookstoreA?run=list_books.xq localhost:8984/rest/BookstoreB?run=list_books.xq

... and have it call two different XQueries - list_books.xq may need to be different for each database (perhaps due to a change in underlying data representation). Consequently, the desire is that XQueries can be associated/bound to a particular database, and not be global.

This way the client does not have to worry about the databases' internal data representation or query implementation. Similarly, when creating the database it would only be necessary to import/add the desired XQueries, and not have to worry about what other databases and global XQueries already sit in the BaseX instance.

It isn't clear to me how RESTXQ helps with this. There would still have to be a single RESTXQuery module (not one per database), just bound to a particular endpoint pattern. It would need to know how to work for all different databases, based on the database name. The database name could be obtained from the endpoint and used, but to have XQueries know specific database names seems undesirable, as they can't be deployed unmodified for different databases.

The storing of XQueries within a database and using xquery:eval could work (where the XQueries are loaded from an XPath in the context), but it seems that this would change the syntax ... localhost:8984/rest/BookstoreA?run=execute_query.xq&query=list_books localhost:8984/rest/BookstoreB?run=execute_query.xq&query=list_books ... but this requires trampolining and forwarding of variables, options, etc ... and be more complex. Possible for simple cases, I guess, but it doesn't feel natural.

The compartmentalisation of XQueries within the webapp folder could allow for this ... localhost:8984/rest/BookstoreA?run=BookstoreA/list_books.xq localhost:8984/rest/BookstoreB?run=BookstoreB/list_books.xq ... which is a bit unnatural and open to error for the client, but seems possible. It would be nice not to have to worry about directory structures on the server. I don't think that this would be an option if using RESTXQ though, because the path to the query is not relevant, just the bound endpoint.

Or, we can just run different servers, and configure them accordingly ... localhost:8984/rest/BookstoreA?run=list_books.xq localhost:8985/rest/BookstoreB?run=list_books.xq

However, I imagine a BaseX instance on a server that multiple projects all use as their development database. Each project would create its own database within that instance. It is much easier to CREATE DB than to have to get access to the server, set up another instance, configure ports etc. If every database was its own instance of BaseX I can imagine far more than 12 instances, or 12 concurrent HTTP servers running - our company's development database instance currently has 108 different databases in it.

So, is it a lot of work to achieve something like the initial syntax above? localhost:8984/rest/BookstoreA?run=list_books.xq localhost:8984/rest/BookstoreB?run=list_books.xq

Hopefully that makes sense. Thanks, Luke

-----Original Message----- From: BaseX-Talk basex-talk-bounces@mailman.uni-konstanz.de On Behalf Of Christian Grün Sent: Tuesday, 25 February 2020 3:20 AM To: DYER Chris Chris.Dyer@sydac.com Cc: basex-talk@mailman.uni-konstanz.de Subject: Re: [basex-talk] Database specific xquery modules

Hi Chris,

Our situation is that we have multiple client applications in various languages that will connect to a database.

Are you already using some of the clients that are available for communication with the BaseX client/server architecture [1], or are (planning to) communicating with the HTTP server?

In the first case, or if you use HTTP with the basic REST API, it’s recommendable to give your users access to this specific database [2]. You could then store all queries on the server, and launch them via the RUN command [3] or the equivalent REST 'run' operation [4].

In the second case, and if you decide to write a RESTXQ application, you will have full flexibility, as you control in your server-side entry point which database is to be opened and accessed [5]. As indicated by Steve, you can use xquery:eval [6] to address and evaluate queries that are either organized as static strings, stored in the file system or stored in a database (possibly a separate instance that contains only queries).

Compared to most relational DBMS, BaseX is much more light-weight, and startup time is minimal. If you have a limited number of database that you want to provide, it’s perfectly ok to launch a number of BaseX instances. I remember a use case in which we had deployed 12 BaseX instances on a single server, mostly because we wanted to simulate horizontal scalability.

Personally, I would recommend the RESTXQ approach, but it also depends on how much time you want to invest in writing more complex XQuery applications.

Cheers, Christian

[1] http://docs.basex.org/wiki/Clients [2] http://docs.basex.org/wiki/User_Management [3] http://docs.basex.org/wiki/Commands#RUN [4] http://docs.basex.org/wiki/REST#GET_Method [5] http://docs.basex.org/wiki/RESTXQ [6] http://docs.basex.org/wiki/XQuery_Module#xquery:eval

On Mon, Feb 24, 2020 at 6:22 AM DYER Chris Chris.Dyer@sydac.com wrote:

Hi Steve,

I do not think what we are trying to achieve is necessarily OO related (although I can see why it may look that way).

It may be more that we would be looking at another layer of encapsulation of some basex features.

Your suggestion on storing the queries within the database and then evaluating them does sound like a possible option for us. It would be nice to achieve this scoping of the queries using explicitly registered function modules.

This is all based on what we are currently achieving with our oracle database solution.

We have a single oracle database server, and we have developers create new user spaces on that server which encapsulates a set of tables and associated stored procedures (functions) for working with those tables. Those functions are only visible within the scope of the user space

It sounds like this sort of use case would generally be handled with separate basex installs/servers.

So I am possibly looking at a feature request.

Cheers, Chris

From: Majewski, Steven Dennis (sdm7g) sdm7g@virginia.edu Sent: Saturday, 22 February 2020 12:37 AM To: DYER Chris Chris.Dyer@sydac.com; basex-talk@mailman.uni-konstanz.de Cc: Markus Wittenberg wittenberg@axxepta.de Subject: Re: [basex-talk] Database specific xquery modules

I’m not sure this is the best approach, but you seem determined to try to figure out how to turn XQuery into an Object-Oriented language, so, if you want to bind query functions to a particular database, you can put the functions in the database, and use xquery:eval()

http://docs.basex.org/wiki/XQuery_Module to call the function as a node() argument to eval.

You would be using the context database as the “self” variable and xquery:eval() as dispatcher.

If you don’t literally want to bind functions to DB, but instead dispatch by the schema of the data, you could use a typeswitch if the schema have different namespaces or root elements,.

( I’ve tried this to switch between processing TEIP4 and TEIP5, for example. )

— Steve M.

On Feb 21, 2020, at 1:41 AM, Markus Wittenberg wittenberg@axxepta.de wrote:

If you want to use RESTXQ you would have to put the modules, or least the functions defining the endpoints, in the webapp directory and not in the repo directory. And far as I know, BaseX doesn't support installation for modules in webapp as for in repo. So you have to explicitly change/add files in webapp directory.

You could write a module for each database, defining its own endpoints, and maybe still reuse some code by importing a module with common functions in every specific module.

Am 21.02.2020 um 07:11 schrieb DYER Chris:

Hi Markus,

Not quite what I am aiming for.

We do want a single BaseX Instance to provide access to multiple databases.

Further to this though, a set of client applications will only connect to one database. We have multiple sets of client applications, with each set each connecting to one database.

We do want to modularise the functions, however as an example we want to be able to have 2 versions of the same function, where 1 version works with the schema of the data in database 1, and the second version with a different schema in database 2.

If I could explicitly link the function modules with a database, I could call the same function from both application sets, and use the context of the open database to call the correct version of the function.

An example situation may help

Say we have Database for “Black Books”, which has book data in xml structures which conform to some version (1.1) of a schema.

We want a query stored on the server that returns books on art

Get-Books-About-Art

We have a client application specifically for this database that calls this function.

It will only need to query the data in this database.

Now we add a second Database for "Goliath Books"

The book data in this database uses a modified xml structure (version 2 of a schema)

We have a client application for this database that also wants to get books about art.

The query for this database is slightly different because the xml schema has changed.

I would like to store both versions of Get-Books-About-Art in the database.

Then for each client to connect to their associated database, and be able to execute the correct version of the xquery function.

The idea is that each time I add a new database that has data in some format, I can install associated query modules that are restricted to the context of that database to avoid conflicts with existing databases.

Maybe what I am looking for is effectively a database specific repo to install modules to, rather than a basex install repo.

From: Markus Wittenberg wittenberg@axxepta.de Sent: Friday, 21 February 2020 2:15 PM To: DYER Chris Chris.Dyer@sydac.com; basex-talk@mailman.uni-konstanz.de Subject: Re: [basex-talk] Database specific xquery modules

A RESTXQ definition could look something like this:

declare

%rest:path("/a/{$client}/{$path}")

function yourmodule:test($client as xs:string, $path as xs:string) {

let $db := switch ($client)

    case "clientA" return db:open("dbA")

    case "clientA" return db:open("dbA")

    default return db:open("dbX")

return yourmodule:do-something-with($db, $path)

};

And the called function do-something-with could also depend on the client

Am 21.02.2020 um 03:48 schrieb DYER Chris:

Hi Markus,

Our intent is to use REST and RESTXQ interfaces to interact with data on a single database using some server side XQueries.

However, we may have 2 (or more) databases provided by some server, and associated xqueries are specific to the contents of each database (as the contents of each of these databases may be structured differently).

We would like for a client to be able to open a single database and be able to run associated XQueries, without requiring client side knowledge of which server side xqueries are associated with the database it has opened.

Does this help describe what we are trying to achieve?

Cheers, Chris

From: BaseX-Talk basex-talk-bounces@mailman.uni-konstanz.de On Behalf Of Markus Wittenberg Sent: Friday, 21 February 2020 12:08 PM To: basex-talk@mailman.uni-konstanz.de Subject: Re: [basex-talk] Database specific xquery modules

Hi Chris,

as you might already know, BaseX has built-in REST support [1] which you can use to query and update data via HTTP. You can also use the RESTXQ [2] functionality to define arbitrary server side XQueries which can also be requested by HTTP and are also suitable for web applications. Both methods are not restricted to a single database.

Best regards,

Markus

[1] http://docs.basex.org/wiki/REST

[2] http://docs.basex.org/wiki/RESTXQ

Am 21.02.2020 um 02:05 schrieb DYER Chris:

I am looking for some ideas on some architecture surrounding xquery modules.

Our situation is that we have multiple client applications in various languages that will connect to a database.

We will have some reasonably complex xqueries to extract data in certain structures that will be used by each of these clients.

So to avoid having to duplicate the query logic we would like the xqueries to reside "in" the database.

Additionally, we would like to use the same BaseX installation for multiple databases, corresponding to multiple projects.

The data in each of these databases may be against a different schema and our xqueries are essentially tied to the schema of the data. Thus, each database may need slightly different xqueries.

The xquery library modules looks to be global for the BaseX instance, as does the loading of main modules off of disk.

Is there any way to tie the query module or file to a specific database?

We are trying to avoid being limited to 1 database per BaseX instance.

An alternative would be to embed the xqueries within our client applications, but that involves duplication. Also, we are not sure if there are any performance difference with having the queries stored in the repo vs client side?

For background, we are looking at migrating from oracle to BaseX. So, we are effectively looking for an equivalent of multiple clients connecting to a userspace and using stored procedures which are specific to that userspace.

Any ideas on how to achieve this currently? Or is this something that BaseX could support?

Cheers,

Chris

--

Markus Wittenberg

Tel +49 (0)341 248 475 36

Mail wittenberg@axxepta.de

---

axxepta solutions GmbH

Lehmgrubenweg 17, 88131 Lindau

Amtsgericht Berlin HRB 97544B

Geschäftsführer: Karsten Becke, Maximilian Gärber

--

Markus Wittenberg

Tel +49 (0)341 248 475 36

Mail wittenberg@axxepta.de

---

axxepta solutions GmbH

Lehmgrubenweg 17, 88131 Lindau

Amtsgericht Berlin HRB 97544B

Geschäftsführer: Karsten Becke, Maximilian Gärber

--

Markus Wittenberg

Tel +49 (0)341 248 475 36

Mail wittenberg@axxepta.de

---

axxepta solutions GmbH

Lehmgrubenweg 17, 88131 Lindau

Amtsgericht Berlin HRB 97544B

Geschäftsführer: Karsten Becke, Maximilian Gärber

I was thinking that there would be a simpler way to bind function namespaces and databases, but that runs into the asymmetry that string in import and namespace declarations in the prolog must be literal strings and can’t be variables. ( I still tend to think from XSLT mental models instead of XQuery sometimes. ) And it seems that even the extended Q{} syntax requires a literal, so it’s not simple to make the function namespace variable.

I can do something like this ( with modules for English, French, Italian added to my repo ):

declare variable $it := 'http://example.org/modules/Italian' ; declare variable $fr := 'http://example.org/modules/French' ; declare variable $head := 'import module namespace lang = "' ; declare variable $tail := '"; ' ;

for $lang in ( $it, $fr ) let $prolog := $head || $lang || $tail return xquery:eval( $prolog || 'lang:hello("Everybody!")' )

Returns =>

Ciao Everybody! Bonjour Everybody!

In the XSLT 1 days, we used to get around the lack of dynamic eval by the functional programming technique of having one stylesheet produce another with the variables resolved. So another way of packaging this might be to create a RESTXQ endpoint that generates an XQuery module from a template and use that endpoint URL in an “import module … at” expression. But that location must also be a literal, so it’s just another variation of the above.

I can also think of some tricks using higher level functions, but so far, nothing that makes it all simple.

— Steve M.

On Feb 25, 2020, at 5:50 AM, Christian Grün christian.gruen@gmail.com wrote:

Hi Luke,

...

It isn't clear to me how RESTXQ helps with this.

Here is a (very minimal) example how an XQuery endpoint could look like:

declare %rest:path('{$db}/{$query}') function rest:execute( $db as xs:string, $query as xs:string ) { xquery:eval( xs:anyURI($db || '/' || $query), map { '': db:open($db) } ) };

One endpoint for all databases would suffice indeed.

Extending the semantics of our REST API would be another option, but we may need to clarify various questions (will there be an extra server directory for each database? What happens if people want to access both global and database-specific queries? etc).

Hope this helps, Christian

I will try to describe our situation a little bit more clearly. This is focused on our development environment, but similar situations occur in a production environment.

We have multiple developers, each working on multiple projects. Each project will require a database. Each database will need to be accessed by multiple applications. Applications may be developed and executed by multiple developers on a netwok. We want to have a central server so that all developers are connecting to a single location.

The content in each of these databases may be similar but with some structural (schema) differences. As such, any queries for interacting with the data may be implemented slightly differently.

When a new project is started, we would have a developer create an associated database on the central server. We would like to create associated queries/functions to access the data for this new project. These would only be applicable to the database we just created, so we would not want them executed on another database. They also may conflict with similar functions for other project databases, so keeping them isolated is ideal.

From the suggestions, it sounds like storing the queries as data within the database is the most likely solution in the current software. It does mean we lose some of the benefits of them being actual modules.

As per Christian’s response, the software would need modifications to allow these isolated modules. I would expect some isolated directory structure per database would be necessary (if utilised). I would also expect this to be used in conjunction to the current global modules (probably less concerned about how the implementation would end up).

Cheers, Chris

From: Majewski, Steven Dennis (sdm7g) sdm7g@virginia.edu Sent: Wednesday, 26 February 2020 4:21 AM To: DYER Chris Chris.Dyer@sydac.com Cc: basex-talk@mailman.uni-konstanz.de Subject: Re: [basex-talk] Database specific xquery modules

Also, question for the OP (Chris):

It’s not completely clear what exactly you want to hide from whom.

That is, if you only want to hide the implementation details from API clients, then RESTXQ and some conditional code is probably sufficient, however your comparison with local stored procedures in SQL make me think you mean that database clients my provide their own implementations of lower level functions, and you want higher level procedures to be able to call those functions without being concerned with the implementations. ( Which is what made me think of comparison with Object Oriented programming. )

— Steve M.

On Feb 25, 2020, at 11:17 AM, Majewski, Steven Dennis (sdm7g) <sdm7g@virginia.edumailto:sdm7g@virginia.edu> wrote:

I was thinking that there would be a simpler way to bind function namespaces and databases, but that runs into the asymmetry that string in import and namespace declarations in the prolog must be literal strings and can’t be variables. ( I still tend to think from XSLT mental models instead of XQuery sometimes. ) And it seems that even the extended Q{} syntax requires a literal, so it’s not simple to make the function namespace variable.

I can do something like this ( with modules for English, French, Italian added to my repo ):

declare variable $it := 'http://example.org/modules/Italian' ; declare variable $fr := 'http://example.org/modules/French' ; declare variable $head := 'import module namespace lang = "' ; declare variable $tail := '"; ' ;

for $lang in ( $it, $fr ) let $prolog := $head || $lang || $tail return xquery:eval( $prolog || 'lang:hello("Everybody!")' )

Returns =>

Ciao Everybody! Bonjour Everybody!

In the XSLT 1 days, we used to get around the lack of dynamic eval by the functional programming technique of having one stylesheet produce another with the variables resolved. So another way of packaging this might be to create a RESTXQ endpoint that generates an XQuery module from a template and use that endpoint URL in an “import module … at” expression. But that location must also be a literal, so it’s just another variation of the above.

I can also think of some tricks using higher level functions, but so far, nothing that makes it all simple.

— Steve M.

On Feb 25, 2020, at 5:50 AM, Christian Grün <christian.gruen@gmail.commailto:christian.gruen@gmail.com> wrote:

Hi Luke,

It isn't clear to me how RESTXQ helps with this.

Here is a (very minimal) example how an XQuery endpoint could look like:

declare %rest:path('{$db}/{$query}') function rest:execute( $db as xs:string, $query as xs:string ) { xquery:eval( xs:anyURI($db || '/' || $query), map { '': db:open($db) } ) };

One endpoint for all databases would suffice indeed.

Extending the semantics of our REST API would be another option, but we may need to clarify various questions (will there be an extra server directory for each database? What happens if people want to access both global and database-specific queries? etc).

Hope this helps, Christian

Just to be sure: Have you read my response to Luke, and have you thought about using RESTXQ and providing a single entry point (for the beginning) for queries that calls xquery:eval?

Yes, I have seen the response. Using the single restxq entry point and xquery:eval is one of the more likely solutions for us.

You can also store your queries in the filesystem of your server. And you could add other endpoints to store and update your custom database queries.

Not entirely sure what you are suggesting here.

I understand that there are multiple methods we could employ to direct our applications to execute the correct queries, with varying levels of effort and varying returns on cleanliness and ease of use. I would suggest that extending the current module repository system would be a more natural approach, and would bring in some concepts that have parallels in other database applications. Granted this might be a large amount of effort, so I would be expecting this to be a feature request.

For now, we will likely go with whichever is the simplest for us to achieve

Chris

So, would the convention of using the database name as the function namespace for queries to that database be sufficient ? Or is convention not a sufficient encapsulation ? ( i.e. do you need stronger enforcement, so that db1 users can’t possibly call db2:func() ? )

— Steve M.

On Feb 26, 2020, at 11:07 PM, DYER Chris Chris.Dyer@sydac.com wrote:

I will try to describe our situation a little bit more clearly. This is focused on our development environment, but similar situations occur in a production environment.

We have multiple developers, each working on multiple projects. Each project will require a database. Each database will need to be accessed by multiple applications. Applications may be developed and executed by multiple developers on a netwok. We want to have a central server so that all developers are connecting to a single location.

The content in each of these databases may be similar but with some structural (schema) differences. As such, any queries for interacting with the data may be implemented slightly differently.

When a new project is started, we would have a developer create an associated database on the central server. We would like to create associated queries/functions to access the data for this new project. These would only be applicable to the database we just created, so we would not want them executed on another database. They also may conflict with similar functions for other project databases, so keeping them isolated is ideal.

From the suggestions, it sounds like storing the queries as data within the database is the most likely solution in the current software. It does mean we lose some of the benefits of them being actual modules.

As per Christian’s response, the software would need modifications to allow these isolated modules. I would expect some isolated directory structure per database would be necessary (if utilised). I would also expect this to be used in conjunction to the current global modules (probably less concerned about how the implementation would end up).

Cheers, Chris

From: Majewski, Steven Dennis (sdm7g) sdm7g@virginia.edu Sent: Wednesday, 26 February 2020 4:21 AM To: DYER Chris Chris.Dyer@sydac.com Cc: basex-talk@mailman.uni-konstanz.de Subject: Re: [basex-talk] Database specific xquery modules

Also, question for the OP (Chris):

It’s not completely clear what exactly you want to hide from whom.

That is, if you only want to hide the implementation details from API clients, then RESTXQ and some conditional code is probably sufficient, however your comparison with local stored procedures in SQL make me think you mean that database clients my provide their own implementations of lower level functions, and you want higher level procedures to be able to call those functions without being concerned with the implementations. ( Which is what made me think of comparison with Object Oriented programming. )

— Steve M.

On Feb 25, 2020, at 11:17 AM, Majewski, Steven Dennis (sdm7g) <sdm7g@virginia.edu mailto:sdm7g@virginia.edu> wrote:

I was thinking that there would be a simpler way to bind function namespaces and databases, but that runs into the asymmetry that string in import and namespace declarations in the prolog must be literal strings and can’t be variables. ( I still tend to think from XSLT mental models instead of XQuery sometimes. ) And it seems that even the extended Q{} syntax requires a literal, so it’s not simple to make the function namespace variable.

I can do something like this ( with modules for English, French, Italian added to my repo ):

declare variable $it := 'http://example.org/modules/Italian http://example.org/modules/Italian' ; declare variable $fr := 'http://example.org/modules/French http://example.org/modules/French' ; declare variable $head := 'import module namespace lang = "' ; declare variable $tail := '"; ' ;

for $lang in ( $it, $fr ) let $prolog := $head || $lang || $tail return xquery:eval( $prolog || 'lang:hello("Everybody!")' )

Returns =>

Ciao Everybody! Bonjour Everybody!

In the XSLT 1 days, we used to get around the lack of dynamic eval by the functional programming technique of having one stylesheet produce another with the variables resolved. So another way of packaging this might be to create a RESTXQ endpoint that generates an XQuery module from a template and use that endpoint URL in an “import module … at” expression. But that location must also be a literal, so it’s just another variation of the above.

I can also think of some tricks using higher level functions, but so far, nothing that makes it all simple.

— Steve M.

On Feb 25, 2020, at 5:50 AM, Christian Grün <christian.gruen@gmail.com mailto:christian.gruen@gmail.com> wrote:

Hi Luke,

It isn't clear to me how RESTXQ helps with this.

Here is a (very minimal) example how an XQuery endpoint could look like:

declare %rest:path('{$db}/{$query}') function rest:execute( $db as xs:string, $query as xs:string ) { xquery:eval( xs:anyURI($db || '/' || $query), map { '': db:open($db) } ) };

One endpoint for all databases would suffice indeed.

Extending the semantics of our REST API would be another option, but we may need to clarify various questions (will there be an extra server directory for each database? What happens if people want to access both global and database-specific queries? etc).

Hope this helps, Christian

So, would the convention of using the database name as the function namespace for queries to that database be sufficient ? Or is convention not a sufficient encapsulation ? ( i.e. do you need stronger enforcement, so that db1 users can’t possibly call db2:func() ? )

A stronger enforcement on the encapsulation would be ideal. I would suggest a built in concept of database specific modules should give you this.

Using Convention and the database name in the function namespace could potentially be sufficient in terms of referencing. It would likely be more convenient if this was not through namespacing though. The database our applications connect to is configurable at application start up time, so we would need to take some care in making sure the function namespaces are generated as well. We would also need to avoid conflicts in their installation location on disk.

if you DO go the route of storing procedures in the database, you might consider whether you could express those procedures as XSLT stylesheets that you could call from XQuery using xslt:transform() function.

For sure, something to consider here. We would likely start with raw/text to see how we go and potentially look to formalise using xslt at a later stage.

Chris