Dear all,
I'm wondering why the code in [1] is not working as expected. Expected means getting back a map with 100 keys.
Instead I get a random number of keys which denotes the fact (proved by tracing) that the map in store is updated concurrently without respecting the lock annotations.
Is there a way to obtain synchronized access to key/values of the default store?
Thank you.
Kind regards,
Marco.
[1]
declare %updating %basex:lock('CONFIG') function local:write($k as xs:string) { store:put("config", map:put(store:get("config"), $k, random:integer(10))) };
let $s1 := store:put("config", map{}) let $s2 := xquery:fork-join(for $i in (1 to 100) return function(){ local:write("key" || $i) }) return count(map:keys(store:get("config")))
Hi Marco,
even if %basex:lock can be added to specific function calls, it will only be considered before the full query is evaluated. As a consequence, your current query will only be evaluated if no other updating query with CONFIG locks is running at the same time.
Apart from that, the store functions are not handled by our transaction management. Instead, they will be immediately executed, and we do our best to ensure that parallel store operations do not result in inconsistent states (basically by treating each store update as an atomic operation). We may need to clarify this in our documentation.
What has been your motivation for adding entries to the store in parallel?
Best, Christian
Marco Lettere m.lettere@gmail.com schrieb am Mi., 22. Jan. 2025, 17:21:
Dear all,
I'm wondering why the code in [1] is not working as expected. Expected means getting back a map with 100 keys.
Instead I get a random number of keys which denotes the fact (proved by tracing) that the map in store is updated concurrently without respecting the lock annotations.
Is there a way to obtain synchronized access to key/values of the default store?
Thank you.
Kind regards,
Marco.
[1]
declare %updating %basex:lock('CONFIG') function local:write($k as xs:string) { store:put("config", map:put(store:get("config"), $k, random:integer(10))) };
let $s1 := store:put("config", map{}) let $s2 := xquery:fork-join(for $i in (1 to 100) return function(){ local:write("key" || $i) }) return count(map:keys(store:get("config")))
Thanks for checking into this Christian.
I'm experimenting with some sort of workflow oriented programming because our projects are getting more and more complex and we'd like to have a way of "standardizing" some patterns and getting a bit of introspection and observability of what happens to a complex workflow.
We'd love to use the store as a shared memory among functions annotated as tasks for many reasons such as keeping a global log of the workflow, reducing dependency on function signatures or chaining together tasks executed in subsequent continuations (aka subworkflows).
Those are kind of things that you can benefit from when using workflow engines. But we don't want to rely on those because they have a huge impact in terms of provisioning and maintenance. So we wanted to check whether a light weight solution in code is viable.
I'm currently rewriting the code to use a DB instead of the store. But it would have been so nice to have it without requiring an external db server... But for example, as soon as I have two tasks running in parallel (fork-join) and they want to add a log line to the sequence of logs that introduces the risk of loosing data.
Hope it clears up the intent even if a bit convoluted explanation. :-)
Ciao, M.
On 27/01/25 17:15, Christian Grün wrote:
Hi Marco,
even if %basex:lock can be added to specific function calls, it will only be considered before the full query is evaluated. As a consequence, your current query will only be evaluated if no other updating query with CONFIG locks is running at the same time.
Apart from that, the store functions are not handled by our transaction management. Instead, they will be immediately executed, and we do our best to ensure that parallel store operations do not result in inconsistent states (basically by treating each store update as an atomic operation). We may need to clarify this in our documentation.
What has been your motivation for adding entries to the store in parallel?
Best, Christian
Marco Lettere m.lettere@gmail.com schrieb am Mi., 22. Jan. 2025, 17:21:
Dear all, I'm wondering why the code in [1] is not working as expected. Expected means getting back a map with 100 keys. Instead I get a random number of keys which denotes the fact (proved by tracing) that the map in store is updated concurrently without respecting the lock annotations. Is there a way to obtain synchronized access to key/values of the default store? Thank you. Kind regards, Marco. [1] declare %updating %basex:lock('CONFIG') function local:write($k as xs:string) { store:put("config", map:put(store:get("config"), $k, random:integer(10))) }; let $s1 := store:put("config", map{}) let $s2 := xquery:fork-join(for $i in (1 to 100) return function(){ local:write("key" || $i) }) return count(map:keys(store:get("config")))
Hi Marco,
When I try your code (with 10.7 or 11.6 ) I get
[XPTY0004] Function is updating: %updating fn() as empty-sequence()...
Which is what I would expect from the hint in the documentation [1] Do you not get this error message?
/Andy
[1] https://docs.basex.org/main/XQuery_Functions#xquery:fork-join
On Mon, 27 Jan 2025 at 16:41, Marco Lettere m.lettere@gmail.com wrote:
Thanks for checking into this Christian.
I'm experimenting with some sort of workflow oriented programming because our projects are getting more and more complex and we'd like to have a way of "standardizing" some patterns and getting a bit of introspection and observability of what happens to a complex workflow.
We'd love to use the store as a shared memory among functions annotated as tasks for many reasons such as keeping a global log of the workflow, reducing dependency on function signatures or chaining together tasks executed in subsequent continuations (aka subworkflows).
Those are kind of things that you can benefit from when using workflow engines. But we don't want to rely on those because they have a huge impact in terms of provisioning and maintenance. So we wanted to check whether a light weight solution in code is viable.
I'm currently rewriting the code to use a DB instead of the store. But it would have been so nice to have it without requiring an external db server... But for example, as soon as I have two tasks running in parallel (fork-join) and they want to add a log line to the sequence of logs that introduces the risk of loosing data.
Hope it clears up the intent even if a bit convoluted explanation. :-) Ciao, M.
On 27/01/25 17:15, Christian Grün wrote:
Hi Marco,
even if %basex:lock can be added to specific function calls, it will only be considered before the full query is evaluated. As a consequence, your current query will only be evaluated if no other updating query with CONFIG locks is running at the same time.
Apart from that, the store functions are not handled by our transaction management. Instead, they will be immediately executed, and we do our best to ensure that parallel store operations do not result in inconsistent states (basically by treating each store update as an atomic operation). We may need to clarify this in our documentation.
What has been your motivation for adding entries to the store in parallel?
Best, Christian
Marco Lettere m.lettere@gmail.com schrieb am Mi., 22. Jan. 2025, 17:21:
Dear all,
I'm wondering why the code in [1] is not working as expected. Expected means getting back a map with 100 keys.
Instead I get a random number of keys which denotes the fact (proved by tracing) that the map in store is updated concurrently without respecting the lock annotations.
Is there a way to obtain synchronized access to key/values of the default store?
Thank you.
Kind regards,
Marco.
[1]
declare %updating %basex:lock('CONFIG') function local:write($k as xs:string) { store:put("config", map:put(store:get("config"), $k, random:integer(10))) };
let $s1 := store:put("config", map{}) let $s2 := xquery:fork-join(for $i in (1 to 100) return function(){ local:write("key" || $i) }) return count(map:keys(store:get("config")))
Hi Andy,
sorry for not specifying but we usually set the following property in the .basex file in order to being able to blend updating and non updating functions.
# Local Options MIXUPDATES = true
M.
On 27/01/25 18:30, Andy Bunce wrote:
Hi Marco,
When I try your code (with 10.7 or 11.6 ) I get
[XPTY0004] Function is updating: %updating fn() as empty-sequence()...
Which is what I would expect from the hint in the documentation [1] Do you not get this error message?
/Andy
[1] https://docs.basex.org/main/XQuery_Functions#xquery:fork-join
On Mon, 27 Jan 2025 at 16:41, Marco Lettere m.lettere@gmail.com wrote:
Thanks for checking into this Christian. I'm experimenting with some sort of workflow oriented programming because our projects are getting more and more complex and we'd like to have a way of "standardizing" some patterns and getting a bit of introspection and observability of what happens to a complex workflow. We'd love to use the store as a shared memory among functions annotated as tasks for many reasons such as keeping a global log of the workflow, reducing dependency on function signatures or chaining together tasks executed in subsequent continuations (aka subworkflows). Those are kind of things that you can benefit from when using workflow engines. But we don't want to rely on those because they have a huge impact in terms of provisioning and maintenance. So we wanted to check whether a light weight solution in code is viable. I'm currently rewriting the code to use a DB instead of the store. But it would have been so nice to have it without requiring an external db server... But for example, as soon as I have two tasks running in parallel (fork-join) and they want to add a log line to the sequence of logs that introduces the risk of loosing data. Hope it clears up the intent even if a bit convoluted explanation. :-) Ciao, M. On 27/01/25 17:15, Christian Grün wrote:Hi Marco, even if %basex:lock can be added to specific function calls, it will only be considered before the full query is evaluated. As a consequence, your current query will only be evaluated if no other updating query with CONFIG locks is running at the same time. Apart from that, the store functions are not handled by our transaction management. Instead, they will be immediately executed, and we do our best to ensure that parallel store operations do not result in inconsistent states (basically by treating each store update as an atomic operation). We may need to clarify this in our documentation. What has been your motivation for adding entries to the store in parallel? Best, Christian Marco Lettere <m.lettere@gmail.com> schrieb am Mi., 22. Jan. 2025, 17:21: Dear all, I'm wondering why the code in [1] is not working as expected. Expected means getting back a map with 100 keys. Instead I get a random number of keys which denotes the fact (proved by tracing) that the map in store is updated concurrently without respecting the lock annotations. Is there a way to obtain synchronized access to key/values of the default store? Thank you. Kind regards, Marco. [1] declare %updating %basex:lock('CONFIG') function local:write($k as xs:string) { store:put("config", map:put(store:get("config"), $k, random:integer(10))) }; let $s1 := store:put("config", map{}) let $s2 := xquery:fork-join(for $i in (1 to 100) return function(){ local:write("key" || $i) }) return count(map:keys(store:get("config")))
Ahh, I see. I can reproduce your result by omitting the %updating. I guess the store functions, like the file functions, are updating (but they just don't like to admit it 😀.)
Thanks /Andy
On Mon, 27 Jan 2025 at 17:37, Marco Lettere m.lettere@gmail.com wrote:
Hi Andy,
sorry for not specifying but we usually set the following property in the .basex file in order to being able to blend updating and non updating functions.
# Local Options MIXUPDATES = true
M. On 27/01/25 18:30, Andy Bunce wrote:
Hi Marco,
When I try your code (with 10.7 or 11.6 ) I get
[XPTY0004] Function is updating: %updating fn() as empty-sequence()...
Which is what I would expect from the hint in the documentation [1] Do you not get this error message?
/Andy
[1] https://docs.basex.org/main/XQuery_Functions#xquery:fork-join
On Mon, 27 Jan 2025 at 16:41, Marco Lettere m.lettere@gmail.com wrote:
Thanks for checking into this Christian.
I'm experimenting with some sort of workflow oriented programming because our projects are getting more and more complex and we'd like to have a way of "standardizing" some patterns and getting a bit of introspection and observability of what happens to a complex workflow.
We'd love to use the store as a shared memory among functions annotated as tasks for many reasons such as keeping a global log of the workflow, reducing dependency on function signatures or chaining together tasks executed in subsequent continuations (aka subworkflows).
Those are kind of things that you can benefit from when using workflow engines. But we don't want to rely on those because they have a huge impact in terms of provisioning and maintenance. So we wanted to check whether a light weight solution in code is viable.
I'm currently rewriting the code to use a DB instead of the store. But it would have been so nice to have it without requiring an external db server... But for example, as soon as I have two tasks running in parallel (fork-join) and they want to add a log line to the sequence of logs that introduces the risk of loosing data.
Hope it clears up the intent even if a bit convoluted explanation. :-) Ciao, M.
On 27/01/25 17:15, Christian Grün wrote:
Hi Marco,
even if %basex:lock can be added to specific function calls, it will only be considered before the full query is evaluated. As a consequence, your current query will only be evaluated if no other updating query with CONFIG locks is running at the same time.
Apart from that, the store functions are not handled by our transaction management. Instead, they will be immediately executed, and we do our best to ensure that parallel store operations do not result in inconsistent states (basically by treating each store update as an atomic operation). We may need to clarify this in our documentation.
What has been your motivation for adding entries to the store in parallel?
Best, Christian
Marco Lettere m.lettere@gmail.com schrieb am Mi., 22. Jan. 2025, 17:21:
Dear all,
I'm wondering why the code in [1] is not working as expected. Expected means getting back a map with 100 keys.
Instead I get a random number of keys which denotes the fact (proved by tracing) that the map in store is updated concurrently without respecting the lock annotations.
Is there a way to obtain synchronized access to key/values of the default store?
Thank you.
Kind regards,
Marco.
[1]
declare %updating %basex:lock('CONFIG') function local:write($k as xs:string) { store:put("config", map:put(store:get("config"), $k, random:integer(10))) };
let $s1 := store:put("config", map{}) let $s2 := xquery:fork-join(for $i in (1 to 100) return function(){ local:write("key" || $i) }) return count(map:keys(store:get("config")))
basex-talk@mailman.uni-konstanz.de
-
Andy Bunce -
Christian Grün -
Marco Lettere