Programmatically extracting function signatures
Hi all - I was curious if there was a way to extract function signatures from BaseX. I'm currently reading through the (excellent) wiki but was curious if there was a more automated process I might be able to use. I've been trying to think of something clever with function-lookup, but I can't seem to make it dynamic based on module namespace; e.g. ``` for $fun in function-lookup(xs:QName("archive:delete"), 1) return $fun ``` I may just grab the wiki and parse it :), but I'm also interested in changes between versions of the processor. Any thoughts or suggestions would be greatly appreciated! Best, Bridger
If you look in lib/ folder the file modules.zip contains the BaseX modules as .xqm stub files with functions marked as external. The entry for archive:delete is shown below. At one point I am sure these *.xqm could be parsed by functions like inspect:module and inspect:xqdoc Currently they raise errors like [XQST0045] Function '...' is in reserved namespace. I believe were generated by scraping the wiki :) However: * some of these are a little out of date (at least in 9.02) * modules.zip is not present in recent 9.1 betas This is a shame as I hoped to use to one day use these in some kind of IDE for content assist. (although I have hoped that for a long time without actually doing it.) Regards /Andy (:~ : Deletes entries from an <code>$archive</code> . : The format of <code>$entries</code> is the same as for <a href="#archive:create">archive:create</a> . : : @error bxerr:ARCH0005 the entries of the given archive cannot be modified. : @error bxerr:ARCH9999 archive creation failed for some other reason. :) declare function archive:delete($archive as xs:base64Binary, $entries as item()*) as xs:base64Binary external; On 17 September 2018 at 16:43, Bridger Dyson-Smith <bdysonsmith@gmail.com> wrote:
Hi all -
I was curious if there was a way to extract function signatures from BaseX. I'm currently reading through the (excellent) wiki but was curious if there was a more automated process I might be able to use. I've been trying to think of something clever with function-lookup, but I can't seem to make it dynamic based on module namespace; e.g.
``` for $fun in function-lookup(xs:QName("archive:delete"), 1) return $fun ```
I may just grab the wiki and parse it :), but I'm also interested in changes between versions of the processor.
Any thoughts or suggestions would be greatly appreciated! Best, Bridger
Andy - thank you for that suggestion! On Mon, Sep 17, 2018 at 12:13 PM Andy Bunce <bunce.andy@gmail.com> wrote:
If you look in lib/ folder the file modules.zip contains the BaseX modules as .xqm stub files with functions marked as external. The entry for archive:delete is shown below. At one point I am sure these *.xqm could be parsed by functions like inspect:module and inspect:xqdoc Currently they raise errors like [XQST0045] Function '...' is in reserved namespace.
I believe were generated by scraping the wiki :)
However: * some of these are a little out of date (at least in 9.02) * modules.zip is not present in recent 9.1 betas
This is a shame as I hoped to use to one day use these in some kind of IDE for content assist. (although I have hoped that for a long time without actually doing it.)
This is exactly the use case I have. I'm attempting to help work on an IntelliJ plugin that will leverage the signatures (for things I don't quite understand :).
Regards
/Andy
(:~ : Deletes entries from an <code>$archive</code> . : The format of <code>$entries</code> is the same as for <a href="#archive:create">archive:create</a> . : : @error bxerr:ARCH0005 the entries of the given archive cannot be modified. : @error bxerr:ARCH9999 archive creation failed for some other reason. :) declare function archive:delete($archive as xs:base64Binary, $entries as item()*) as xs:base64Binary external;
Thanks again for the suggestion. Best, Bridger
On 17 September 2018 at 16:43, Bridger Dyson-Smith <bdysonsmith@gmail.com> wrote:
Hi all -
I was curious if there was a way to extract function signatures from BaseX. I'm currently reading through the (excellent) wiki but was curious if there was a more automated process I might be able to use. I've been trying to think of something clever with function-lookup, but I can't seem to make it dynamic based on module namespace; e.g.
``` for $fun in function-lookup(xs:QName("archive:delete"), 1) return $fun ```
I may just grab the wiki and parse it :), but I'm also interested in changes between versions of the processor.
Any thoughts or suggestions would be greatly appreciated! Best, Bridger
Sounds good. Of course I meant the etc/ folder not lib/. /Andy On 17 September 2018 at 17:28, Bridger Dyson-Smith <bdysonsmith@gmail.com> wrote:
Andy - thank you for that suggestion!
On Mon, Sep 17, 2018 at 12:13 PM Andy Bunce <bunce.andy@gmail.com> wrote:
If you look in lib/ folder the file modules.zip contains the BaseX modules as .xqm stub files with functions marked as external. The entry for archive:delete is shown below. At one point I am sure these *.xqm could be parsed by functions like inspect:module and inspect:xqdoc Currently they raise errors like [XQST0045] Function '...' is in reserved namespace.
I believe were generated by scraping the wiki :)
However: * some of these are a little out of date (at least in 9.02) * modules.zip is not present in recent 9.1 betas
This is a shame as I hoped to use to one day use these in some kind of IDE for content assist. (although I have hoped that for a long time without actually doing it.)
This is exactly the use case I have. I'm attempting to help work on an IntelliJ plugin that will leverage the signatures (for things I don't quite understand :).
Regards
/Andy
(:~ : Deletes entries from an <code>$archive</code> . : The format of <code>$entries</code> is the same as for <a href="#archive:create">archive:create</a> . : : @error bxerr:ARCH0005 the entries of the given archive cannot be modified. : @error bxerr:ARCH9999 archive creation failed for some other reason. :) declare function archive:delete($archive as xs:base64Binary, $entries as item()*) as xs:base64Binary external;
Thanks again for the suggestion. Best, Bridger
On 17 September 2018 at 16:43, Bridger Dyson-Smith <bdysonsmith@gmail.com
wrote:
Hi all -
I was curious if there was a way to extract function signatures from BaseX. I'm currently reading through the (excellent) wiki but was curious if there was a more automated process I might be able to use. I've been trying to think of something clever with function-lookup, but I can't seem to make it dynamic based on module namespace; e.g.
``` for $fun in function-lookup(xs:QName("archive:delete"), 1) return $fun ```
I may just grab the wiki and parse it :), but I'm also interested in changes between versions of the processor.
Any thoughts or suggestions would be greatly appreciated! Best, Bridger
Hi Bridger, I’m excited to hear about your contributions to the IntelliJ XQuery plugin. This will be a good reason indeed for updating the stub files, which (as Andy has correctly guessed) are based on our Wiki pages. It’s been a while ago since I looked at this, but I’ll be glad to create updated versions of the files. I’ve created a new issue for this step [1]. For a start, the files of the 9.0.2 release might do the job. Cheers, Christian [1] https://github.com/BaseXdb/basex/issues/1623 On Mon, Sep 17, 2018 at 6:23 PM Bridger Dyson-Smith <bdysonsmith@gmail.com> wrote:
Andy - thank you for that suggestion!
On Mon, Sep 17, 2018 at 12:13 PM Andy Bunce <bunce.andy@gmail.com> wrote:
If you look in lib/ folder the file modules.zip contains the BaseX modules as .xqm stub files with functions marked as external. The entry for archive:delete is shown below. At one point I am sure these *.xqm could be parsed by functions like inspect:module and inspect:xqdoc Currently they raise errors like [XQST0045] Function '...' is in reserved namespace.
I believe were generated by scraping the wiki :)
However: * some of these are a little out of date (at least in 9.02) * modules.zip is not present in recent 9.1 betas
This is a shame as I hoped to use to one day use these in some kind of IDE for content assist. (although I have hoped that for a long time without actually doing it.)
This is exactly the use case I have. I'm attempting to help work on an IntelliJ plugin that will leverage the signatures (for things I don't quite understand :).
Regards /Andy
(:~ : Deletes entries from an <code>$archive</code> . : The format of <code>$entries</code> is the same as for <a href="#archive:create">archive:create</a> . : : @error bxerr:ARCH0005 the entries of the given archive cannot be modified. : @error bxerr:ARCH9999 archive creation failed for some other reason. :) declare function archive:delete($archive as xs:base64Binary, $entries as item()*) as xs:base64Binary external;
Thanks again for the suggestion. Best, Bridger
On 17 September 2018 at 16:43, Bridger Dyson-Smith <bdysonsmith@gmail.com> wrote:
Hi all -
I was curious if there was a way to extract function signatures from BaseX. I'm currently reading through the (excellent) wiki but was curious if there was a more automated process I might be able to use. I've been trying to think of something clever with function-lookup, but I can't seem to make it dynamic based on module namespace; e.g.
``` for $fun in function-lookup(xs:QName("archive:delete"), 1) return $fun ```
I may just grab the wiki and parse it :), but I'm also interested in changes between versions of the processor.
Any thoughts or suggestions would be greatly appreciated! Best, Bridger
Hi Bridger, I am glad to report I have created new stub files for the BaseX XQuery Modules [1]. They’ll now be included in the official releases again [2]. I have also uploaded the script that I wrote for generating the xqdoc output [3]. It’s far from perfect, but definitely more complete than the old version. If you encounter any errors, please don’t hesitate to tell me. Could you give us a little update on your contribution to the IntelliJ XQuery plugin? Cheers, Christian [1] https://github.com/BaseXdb/basex/issues/1623 [2] http://files.basex.org/releases/latest/ [3] https://github.com/BaseXdb/basex-dist/blob/master/wiki2xqdoc.xq On Mon, Sep 17, 2018 at 5:38 PM Bridger Dyson-Smith <bdysonsmith@gmail.com> wrote:
Hi all -
I was curious if there was a way to extract function signatures from BaseX. I'm currently reading through the (excellent) wiki but was curious if there was a more automated process I might be able to use. I've been trying to think of something clever with function-lookup, but I can't seem to make it dynamic based on module namespace; e.g.
``` for $fun in function-lookup(xs:QName("archive:delete"), 1) return $fun ```
I may just grab the wiki and parse it :), but I'm also interested in changes between versions of the processor.
Any thoughts or suggestions would be greatly appreciated! Best, Bridger
Hi Christian - thank you for your work on the xqdoc files (and the link to the xquery used for generating them)! The current state of XQuery plugins for the JetBrains/IntelliJ IDEs has changed a bit since I added the IDEA documentation to the wiki. Grzegorz Ligas' XQuery Support plugin has been forked by the talented developers at OverStory, with lots of MarkLogic-specific features added. Reece Dunn also has been working on an XQuery plugin that takes a slightly different approach: rather than focusing on one implementation, Reece is working on a plugin that provides broader language support. The very minor work that I've done has been for Reece - I'm (slowly) adding implementation builtin function signatures that will be used by his plugin to provide improved static analysis. I hope that provides a bit of information regarding news with XQuery/JetBrains plugins. Best, Bridger On Tue, Oct 30, 2018 at 9:13 AM Christian Grün <christian.gruen@gmail.com> wrote:
Hi Bridger,
I am glad to report I have created new stub files for the BaseX XQuery Modules [1]. They’ll now be included in the official releases again [2]. I have also uploaded the script that I wrote for generating the xqdoc output [3]. It’s far from perfect, but definitely more complete than the old version. If you encounter any errors, please don’t hesitate to tell me.
Could you give us a little update on your contribution to the IntelliJ XQuery plugin?
Cheers, Christian
[1] https://github.com/BaseXdb/basex/issues/1623 [2] http://files.basex.org/releases/latest/ [3] https://github.com/BaseXdb/basex-dist/blob/master/wiki2xqdoc.xq
On Mon, Sep 17, 2018 at 5:38 PM Bridger Dyson-Smith <bdysonsmith@gmail.com> wrote:
Hi all -
I was curious if there was a way to extract function signatures from
BaseX. I'm currently reading through the (excellent) wiki but was curious if there was a more automated process I might be able to use. I've been trying to think of something clever with function-lookup, but I can't seem to make it dynamic based on module namespace; e.g.
``` for $fun in function-lookup(xs:QName("archive:delete"), 1) return $fun ```
I may just grab the wiki and parse it :), but I'm also interested in
changes between versions of the processor.
Any thoughts or suggestions would be greatly appreciated! Best, Bridger
On Wed, 7 Nov 2018 at 01:27, Bridger Dyson-Smith <bdysonsmith@gmail.com> wrote:
Hi Christian -
thank you for your work on the xqdoc files (and the link to the xquery used for generating them)!
The current state of XQuery plugins for the JetBrains/IntelliJ IDEs has changed a bit since I added the IDEA documentation to the wiki. Grzegorz Ligas' XQuery Support plugin has been forked by the talented developers at OverStory, with lots of MarkLogic-specific features added. Reece Dunn also has been working on an XQuery plugin that takes a slightly different approach: rather than focusing on one implementation, Reece is working on a plugin that provides broader language support.
To add some background... My plugin started as an attempt to address various issues in the XQuery parser in Grzegorz' plugin. [1] was the point where I decided to attempt creating my own plugin, after reporting various issues to Grzegorz. Specifically, my plugin aimed to: 1. Address the keyword vs identifier issue in various places [2]. 2. Provide a robust lexer and parser with error recovery. 3. Provide full support for the MarkLogic syntax in addition to XQuery 3 support -- this later evolved into adding support for other XQuery extensions (updating, full text, scripting), XQuery 3.1 support, and BaseX and Saxon vendor extension support. Version 1.2 of my plugin has support for the BaseX update, fuzzy, and non-deterministic extensions. Version 1.3 (in development) adds support for the BaseX 9.1 ternary if, elvis operator, and if without else syntax extensions. My plan is to publish 1.3 when the IntelliJ 2018.3 release candidate is released, so the plugin is available for the full release. My plugin does not currently support running queries, debugging, code reformatting, auto-completion, and various other features that Grzegorz' plugin supports. These are planned (I am currently working on running queries), but these things take time. I've also been improving function lookup in 1.3 to be standards conformant -- this is complex, and is not 100% complete. The technical challenges in implementing XQuery support in an IDE are different to those implementing it in a query processor as it needs to support the functionality and capabiilties of both the IDE and XQuery. My long-term plans are to implement as many of the XPath and XQuery static errors from the error condition list as possible to spot all statically determinable errors in the IDE, as well as various other inspections. I'm also considering adding specific support for the XPath subset of XQuery and integrating that with XSLT for better XPath support in IntelliJ. The very minor work that I've done has been for Reece - I'm (slowly) adding
implementation builtin function signatures that will be used by his plugin to provide improved static analysis.
My plans have always been to describe the built-in functions and static context as XQuery files. I completed this for the XQuery and MarkLogic functions, and Bridger has added BaseX, EXPath/EXQuery, and Saxon built-in function definitions. In the future, I want to add complete API descriptions using xqdoc comments so I can integrate them into the IDE when my plugin supports displaying the xqdoc information in place. I also want to use the information in the function annotations to report functions that require a different version of the XQuery processor. [1] https://github.com/ligasgr/intellij-xquery/issues/199 [2] I have solved this by making the NCName and keyword tokens implement a common interface, then check for that interface when looking for NCNames. I then remove the keyword styling on keywords in NCName positions after the code has been parsed. I also use this to support the reserved function name functionality. Kind regards, Reece I hope that provides a bit of information regarding news with
XQuery/JetBrains plugins. Best, Bridger
On Tue, Oct 30, 2018 at 9:13 AM Christian Grün <christian.gruen@gmail.com> wrote:
Hi Bridger,
I am glad to report I have created new stub files for the BaseX XQuery Modules [1]. They’ll now be included in the official releases again [2]. I have also uploaded the script that I wrote for generating the xqdoc output [3]. It’s far from perfect, but definitely more complete than the old version. If you encounter any errors, please don’t hesitate to tell me.
Could you give us a little update on your contribution to the IntelliJ XQuery plugin?
Cheers, Christian
[1] https://github.com/BaseXdb/basex/issues/1623 [2] http://files.basex.org/releases/latest/ [3] https://github.com/BaseXdb/basex-dist/blob/master/wiki2xqdoc.xq
On Mon, Sep 17, 2018 at 5:38 PM Bridger Dyson-Smith <bdysonsmith@gmail.com> wrote:
Hi all -
I was curious if there was a way to extract function signatures from
BaseX. I'm currently reading through the (excellent) wiki but was curious if there was a more automated process I might be able to use. I've been trying to think of something clever with function-lookup, but I can't seem to make it dynamic based on module namespace; e.g.
``` for $fun in function-lookup(xs:QName("archive:delete"), 1) return $fun ```
I may just grab the wiki and parse it :), but I'm also interested in
changes between versions of the processor.
Any thoughts or suggestions would be greatly appreciated! Best, Bridger
participants (4)
-
Andy Bunce -
Bridger Dyson-Smith -
Christian Grün -
Reece Dunn