Re: [basex-talk] unexpected behavior from prof:track()

Dear Michael,

You provided us with an excellent analysis and summary of what BaseX currently does. An addendum:

If a built-in function is directly called, its arguments will be evaluated by the function. If a static or dynamic function is called, however, the arguments will be evaluated before the actual function is invoked. While it may probably be possible to defer the execution, there are various aspects complicating that (the body of a dynamically evaluated function may be recursive; arguments reside in a different dynamic context than the function body; etc.).

If the dynamic function is statically known, and if the arguments are deterministic, the arguments can be inlined at compile time:

prof:track#1( (1 to 1000000) ! <a/> )

…will be rewritten to…

let $value_1 := (1 to 1000000) ! <a/> return prof:track($value_1)

…will be rewritten to…

prof:track( (1 to 1000000) ! <a/> )

That doesn’t work if the argument is internally tagged as non-deterministic (which is the case for prof:sleep).

It doesn’t work, either, if fn:function-lookup is used, as the function will only be invoked at runtime (i.e., after compiling the query) … or it didn’t, as I had another look at our implementation: fn:function-lookup will now be pre-evaluated if its arguments are static and if it returns a value at compile time. As a result, the following expression …

function-lookup(xs:QName('prof:track'), 1)( (1 to 1000000) ! <a/> )

… will be simplified to:

prof:track( (1 to 1000000) ! <a/> )

Due to the non-deterministic nature of its argument, as before, the following function …

function-lookup(xs:QName('prof:track'), 1)( prof:sleep(1000) )

… will only be rewritten to …

let $value_1 := prof:sleep(1000) return prof:track($value_1)

A new snapshot is available [1] (BaseX 9.7.3 will be released until end of this month).

Hope this helps, your feedback is welcome, Christian

[1] https://files.basex.org/releases/latest/

On Mon, Jun 20, 2022 at 1:23 AM C. M. Sperberg-McQueen cmsmcq@blackmesatech.com wrote:

Short version:

When I call prof:track(...) with an expression as argument, I get the results I would expect. When I assign prof:track#1 to a variable and then call it with an expression by referring to that variable, the time shown is always 0. Is this the expected behavior?

Details:

Consider the following XQuery module:

let $f := function($s as xs:string) {
              trace('hi'),
              prof:sleep(2000),
              trace($s)
          },
    $pt := (function-lookup(
              QName('http://basex.org/modules/prof',
                    'track'),
              1),
              function($item) {
                  map {'time':'?', 'value': $item}
              }
           )[1]

return (prof:track($f('world')), $pt($f('friends!')) )

It defines two variables:

• $f, a function that produces output (so I can see that it ran) and calls prof:sleep (so it takes more than a couple of ms), and

• $pt, which will be the prof:track() function if it exists (as it will in BaseX) or a substitute function which just evaluates the expression but does not provide any timing information.

It then calls function $f twice, once wrapped in prof:track() and once wrapped in $pt(). The output follows:

map {
  "memory": 0,
  "time": 2.00069e3,
  "value": ("hi", "world")
}
map {
  "memory": 0,
  "time": 0.0e0,
  "value": ("hi", "friends!")
}

As may be seen, the 'time' value in the map is 0.

Conjecture:

I suspect that this has to do with the fact that prof:track() is not a normal function -- in Lisp or Scheme, it would have to be implemented as a macro or 'special form', meaning that the arguments to the call are not evaluated before calling the function. I wonder whether what is happening here is that when the call to $pt($f('friends')) is handled, the argument is being evaluated (producing the sequence ('hi', 'friends!')) and then prof:track() is being called. Since it has no work to do, prof:track() then just returns the result, accurately showing a time cost of 0.

Background:

In a test harness I use to run tests for an XQuery program I am writing, I initially used prof:track() to keep track of how long each test takes to run. Recently, as part of an effort to ensure that the program is implementation-neutral, I rewrote the test harness so that it could run under other XQuery engines as well -- hence the indirection and the definition of $pt.

I realize that I could get the BaseX version to work by saying something like

let $prof-track := function-lookup(...) let $fall-back := function(...)

if (exists($prof-track)) then prof:track(...) else $fall-back(...)

but my recollection is that I tried this first and it provoked fatal errors in other engines, since prof:track() is not available in the static context in those engines.

Is there any way to write a single module so that prof:track() is used (and produces useful information) if available, and the fallback function is used otherwise? I would really like to avoid generating different versions of the test harness for different XQuery engines.

Many thanks for any thoughts or advice anyone on this list can provide.

Michael Sperberg-McQueen

-- C. M. Sperberg-McQueen Black Mesa Technologies LLC http://blackmesatech.com