I'm using BaseX 8.4 but have recreated the issue with 8.5.4 beta 8d90b66
I'm POSTing xml files to a RESTXQ function using the following script
let $fpath := 'C:\incoming\SAQD\mo' let $targetBasex := 'http://localhost:8984/saqd/mo/' let $list := sort(file:list($fpath), function($fname) {fn:analyze-string($fname,'.*_(.*?).xml$')/fn:match/fn:group[@nr="1"]/string()}) for $file in $list let $abspath := $fpath||''||$file let $body := try {fetch:xml($abspath, map { 'chop': true() }) } catch * {()}
return if ( not(empty($body)) ) then (http:send-request(http:request href="{$targetBasex||$file}" method='POST' <http:body media-type="text/xml">{$body}</http:body> </http:request>), file:move($abspath,$fpath||'\processed'||$file) ) else ()
The XML file has no namespace declarations. The RESTXQ function does a db:update using the body received and the http-client namespace has been added to the root node.
If I POST to the same function using curl the stored document is unchanged. The function is as follows.
declare %rest:path("/saqd/mo/{$fname}") (: %rest:consumes("application/xml", "text/xml") :) %rest:POST("{$body}") %updating function saqd:mo($fname, $body as document-node()) { try {(db:replace('SAQD', '/MO/'||$fname, $body ), db:output(http:send-request(<http:request href="{' http://example.org:8082/home/VALIDSCHEMA/xml/current/Metoffice_Data/%27%7C%7... }" method='PUT' username='validuser' password='validpassword' send-authorization='true'> <http:body media-type='text/xml'>{$body}</http:body> </http:request>)) ) } catch * {db:output( rest:response <http:response status="500" message="{$err:code||' : '||$err:description}"> </http:response> </rest:response>)} };
The file PUT inside the db:output statement also gets the namespace added i.e. it happens even when the file was POSTed with curl so the recieved document still had no namespaces.
Is this expected behaviour, a bug, or am I using http:send-request incorrectly?
Regards,
chris sweeney
Hi Chris,
I appreciate your accurate comments. As a lot is going on in your query example, however, it made it difficult for me to strip it down to the basic problem(s) you’ve been encountering. If I get it right; for example, I didn’t understand why the request to example.org is wrapped with db:output, what it is doing, etc.
Do you think it’s possible to simplify the example a little? The shorter, the better ;)
Thanks in advance Christian
On Wed, Sep 14, 2016 at 1:35 PM, chrisis chrisisanon@gmail.com wrote:
I'm using BaseX 8.4 but have recreated the issue with 8.5.4 beta 8d90b66
I'm POSTing xml files to a RESTXQ function using the following script
let $fpath := 'C:\incoming\SAQD\mo' let $targetBasex := 'http://localhost:8984/saqd/mo/' let $list := sort(file:list($fpath), function($fname) {fn:analyze-string($fname,'.*_(.*?).xml$')/fn:match/fn:group[@nr="1"]/string()}) for $file in $list let $abspath := $fpath||''||$file let $body := try {fetch:xml($abspath, map { 'chop': true() }) } catch * {()}
return if ( not(empty($body)) ) then (http:send-request(http:request href="{$targetBasex||$file}" method='POST' <http:body media-type="text/xml">{$body}</http:body> </http:request>), file:move($abspath,$fpath||'\processed'||$file) ) else ()
The XML file has no namespace declarations. The RESTXQ function does a db:update using the body received and the http-client namespace has been added to the root node.
If I POST to the same function using curl the stored document is unchanged. The function is as follows.
declare %rest:path("/saqd/mo/{$fname}") (: %rest:consumes("application/xml", "text/xml") :) %rest:POST("{$body}") %updating function saqd:mo($fname, $body as document-node()) { try {(db:replace('SAQD', '/MO/'||$fname, $body ), db:output(http:send-request(<http:request href="{'http://example.org:8082/home/VALIDSCHEMA/xml/current/Metoffice_Data/%27%7C%7..." method='PUT' username='validuser' password='validpassword' send-authorization='true'> <http:body media-type='text/xml'>{$body}</http:body> </http:request>)) ) } catch * {db:output( rest:response <http:response status="500" message="{$err:code||' : '||$err:description}"> </http:response> </rest:response>)} };
The file PUT inside the db:output statement also gets the namespace added i.e. it happens even when the file was POSTed with curl so the recieved document still had no namespaces.
Is this expected behaviour, a bug, or am I using http:send-request incorrectly?
Regards,
chris sweeney
Hi Christian
Sorry, chopped that out of the real thing in a bit of a hurry and made a bit of a mess of it. Here's something slightly closer to a SSCE.
Module under WEBPATH
xquery version "3.1" encoding "utf-8"; module namespace example = 'http://localhost/example'; declare %rest:path("/example/post/{$fname}") %rest:POST("{$body}") %updating function example:post($fname, $body as document-node()) {db:replace('exampledb', '/post/'||$fname, $body) };
Script run in BaseX GUI
let $file := 'C:\temp\xq\example.xml' let $fname := 'example.xml' let $targetrestxqproc := 'http://localhost:8984/example/post/' let $body := fetch:xml($file, map { 'chop': true() }) return http:send-request(<http:request href="{$targetrestxqproc||$fname}" method='POST'> <http:body media-type='text/xml'>{$body}</http:body> </http:request>)
Result in exampledb is <example xmlns:http="http://expath.org/ns/http-client"/>
Using curl from a DOS prompt
c:\apps\curl\curl -F"file=@example.xml;type=application/xml" http://localhost:8984/example/post/examplecurl.xml
Result in exampledb is <example/>
The addition of the namespace in the first case came as a bit of a surprise to me.
Thanks, chris
Hm. I can’t reproduce it..
1. I stored your module as 'example.xqm' in the WEBPATH directory 2. I started basexhttp 3. I created an empty 'exampledb' database in the GUI and closed it again 4. I ran your script in the GUI (after rewriting fetch:xml(...) with <example/>) 5. I ran db:open('exambledb')
Did I do something wrong?
By the way, if possible, remove everything from your examples that required manual post-processing. Could you please check the following code and tell me how the returned XML looks like?
_ example.xqm ___________
module namespace example = 'http://localhost/example'; declare %rest:path("/example") %rest:POST("{$body}") function example:post($body) { $body };
_ test.xqm ___________
http:send-request( <http:request href="http://localhost:8984/example" method='POST'> <http:body media-type='text/xml'> <example/> </http:body> </http:request> )[2]
On my system, I get "<example/>".
On Thu, Sep 15, 2016 at 2:34 PM, chrisis chrisisanon@gmail.com wrote:
Hi Christian
Sorry, chopped that out of the real thing in a bit of a hurry and made a bit of a mess of it. Here's something slightly closer to a SSCE.
Module under WEBPATH
xquery version "3.1" encoding "utf-8"; module namespace example = 'http://localhost/example'; declare %rest:path("/example/post/{$fname}") %rest:POST("{$body}") %updating function example:post($fname, $body as document-node()) {db:replace('exampledb', '/post/'||$fname, $body) };
Script run in BaseX GUI
let $file := 'C:\temp\xq\example.xml' let $fname := 'example.xml' let $targetrestxqproc := 'http://localhost:8984/example/post/' let $body := fetch:xml($file, map { 'chop': true() }) return http:send-request(<http:request href="{$targetrestxqproc||$fname}" method='POST'> <http:body media-type='text/xml'>{$body}</http:body> </http:request>)
Result in exampledb is
<example xmlns:http="http://expath.org/ns/http-client"/>
Thanks for the hints. I can finally agree it’s a bug ;) I have added a GitHub issue for it [1]. The problem only occurs with "database nodes" (which are the ones created via "fn:doc", "fetch:xml", or by applying transformations via "copy $c := ..." or "... update {}".
This looks to me like a generall weakness of the EXPath spec in general: A namespace declaration will always be valid for all descendant nodes of an element. As it doesn’t make sense to keep the http namespace when supplying bodies via http:send-request, we are explicitly removing the namespace from the body node before sending it out. As there is no such functionality in XQuery like “removing namespaces”, this operation is not well-defined, and it’s probably the reason for this bug.
I hope I’ll find time to fix this soon.
Cheers, Christian
[1] https://github.com/BaseXdb/basex/issues/1353
On Fri, Sep 16, 2016 at 4:06 PM, Christian Grün christian.gruen@gmail.com wrote:
Hm. I can’t reproduce it..
- I stored your module as 'example.xqm' in the WEBPATH directory
- I started basexhttp
- I created an empty 'exampledb' database in the GUI and closed it again
- I ran your script in the GUI (after rewriting fetch:xml(...) with <example/>)
- I ran db:open('exambledb')
Did I do something wrong?
By the way, if possible, remove everything from your examples that required manual post-processing. Could you please check the following code and tell me how the returned XML looks like?
_ example.xqm ___________
module namespace example = 'http://localhost/example'; declare %rest:path("/example") %rest:POST("{$body}") function example:post($body) { $body };
_ test.xqm ___________
http:send-request( <http:request href="http://localhost:8984/example" method='POST'> <http:body media-type='text/xml'> <example/> </http:body> </http:request> )[2]
On my system, I get "<example/>".
On Thu, Sep 15, 2016 at 2:34 PM, chrisis chrisisanon@gmail.com wrote:
Hi Christian
Sorry, chopped that out of the real thing in a bit of a hurry and made a bit of a mess of it. Here's something slightly closer to a SSCE.
Module under WEBPATH
xquery version "3.1" encoding "utf-8"; module namespace example = 'http://localhost/example'; declare %rest:path("/example/post/{$fname}") %rest:POST("{$body}") %updating function example:post($fname, $body as document-node()) {db:replace('exampledb', '/post/'||$fname, $body) };
Script run in BaseX GUI
let $file := 'C:\temp\xq\example.xml' let $fname := 'example.xml' let $targetrestxqproc := 'http://localhost:8984/example/post/' let $body := fetch:xml($file, map { 'chop': true() }) return http:send-request(<http:request href="{$targetrestxqproc||$fname}" method='POST'> <http:body media-type='text/xml'>{$body}</http:body> </http:request>)
Result in exampledb is
<example xmlns:http="http://expath.org/ns/http-client"/>
Thanks for the explanation. I’ll keep an eye out for the fix but I can easily work round this issue for now.
cheers,
chris
On 16 Sep 2016, at 17:15, Christian Grün christian.gruen@gmail.com wrote:
Thanks for the hints. I can finally agree it’s a bug ;) I have added a GitHub issue for it [1]. The problem only occurs with "database nodes" (which are the ones created via "fn:doc", "fetch:xml", or by applying transformations via "copy $c := ..." or "... update {}".
This looks to me like a generall weakness of the EXPath spec in general: A namespace declaration will always be valid for all descendant nodes of an element. As it doesn’t make sense to keep the http namespace when supplying bodies via http:send-request, we are explicitly removing the namespace from the body node before sending it out. As there is no such functionality in XQuery like “removing namespaces”, this operation is not well-defined, and it’s probably the reason for this bug.
I hope I’ll find time to fix this soon.
Cheers, Christian
[1] https://github.com/BaseXdb/basex/issues/1353
On Fri, Sep 16, 2016 at 4:06 PM, Christian Grün christian.gruen@gmail.com wrote:
Hm. I can’t reproduce it..
- I stored your module as 'example.xqm' in the WEBPATH directory
- I started basexhttp
- I created an empty 'exampledb' database in the GUI and closed it again
- I ran your script in the GUI (after rewriting fetch:xml(...) with <example/>)
- I ran db:open('exambledb')
Did I do something wrong?
By the way, if possible, remove everything from your examples that required manual post-processing. Could you please check the following code and tell me how the returned XML looks like?
_ example.xqm ___________
module namespace example = 'http://localhost/example'; declare %rest:path("/example") %rest:POST("{$body}") function example:post($body) { $body };
_ test.xqm ___________
http:send-request( <http:request href="http://localhost:8984/example" method='POST'> <http:body media-type='text/xml'> <example/> </http:body> </http:request> )[2]
On my system, I get "<example/>".
On Thu, Sep 15, 2016 at 2:34 PM, chrisis chrisisanon@gmail.com wrote:
Hi Christian
Sorry, chopped that out of the real thing in a bit of a hurry and made a bit of a mess of it. Here's something slightly closer to a SSCE.
Module under WEBPATH
xquery version "3.1" encoding "utf-8"; module namespace example = 'http://localhost/example'; declare %rest:path("/example/post/{$fname}") %rest:POST("{$body}") %updating function example:post($fname, $body as document-node()) {db:replace('exampledb', '/post/'||$fname, $body) };
Script run in BaseX GUI
let $file := 'C:\temp\xq\example.xml' let $fname := 'example.xml' let $targetrestxqproc := 'http://localhost:8984/example/post/' let $body := fetch:xml($file, map { 'chop': true() }) return http:send-request(<http:request href="{$targetrestxqproc||$fname}" method='POST'> <http:body media-type='text/xml'>{$body}</http:body> </http:request>)
Result in exampledb is
<example xmlns:http="http://expath.org/ns/http-client"/>
Dear Chris,
I had another look at the issue that was triggered by our discussion [1]. I believe the problem is indeed one that would first need to be clarified in the EXPath HTTP Specification, so I decided to close it.
The best workaround is indeed to pass the payload via the third argument of the function..
Best, Christian
[1] https://github.com/BaseXdb/basex/issues/1353
On Fri, Sep 16, 2016 at 6:28 PM, chris sweeney chrisisanon@gmail.com wrote:
Thanks for the explanation. I’ll keep an eye out for the fix but I can easily work round this issue for now.
cheers,
chris
On 16 Sep 2016, at 17:15, Christian Grün christian.gruen@gmail.com wrote:
Thanks for the hints. I can finally agree it’s a bug ;) I have added a GitHub issue for it [1]. The problem only occurs with "database nodes" (which are the ones created via "fn:doc", "fetch:xml", or by applying transformations via "copy $c := ..." or "... update {}".
This looks to me like a generall weakness of the EXPath spec in general: A namespace declaration will always be valid for all descendant nodes of an element. As it doesn’t make sense to keep the http namespace when supplying bodies via http:send-request, we are explicitly removing the namespace from the body node before sending it out. As there is no such functionality in XQuery like “removing namespaces”, this operation is not well-defined, and it’s probably the reason for this bug.
I hope I’ll find time to fix this soon.
Cheers, Christian
[1] https://github.com/BaseXdb/basex/issues/1353
On Fri, Sep 16, 2016 at 4:06 PM, Christian Grün christian.gruen@gmail.com wrote:
Hm. I can’t reproduce it..
- I stored your module as 'example.xqm' in the WEBPATH directory
- I started basexhttp
- I created an empty 'exampledb' database in the GUI and closed it again
- I ran your script in the GUI (after rewriting fetch:xml(...) with <example/>)
- I ran db:open('exambledb')
Did I do something wrong?
By the way, if possible, remove everything from your examples that required manual post-processing. Could you please check the following code and tell me how the returned XML looks like?
_ example.xqm ___________
module namespace example = 'http://localhost/example'; declare %rest:path("/example") %rest:POST("{$body}") function example:post($body) { $body };
_ test.xqm ___________
http:send-request( <http:request href="http://localhost:8984/example" method='POST'> <http:body media-type='text/xml'> <example/> </http:body> </http:request> )[2]
On my system, I get "<example/>".
On Thu, Sep 15, 2016 at 2:34 PM, chrisis chrisisanon@gmail.com wrote:
Hi Christian
Sorry, chopped that out of the real thing in a bit of a hurry and made a bit of a mess of it. Here's something slightly closer to a SSCE.
Module under WEBPATH
xquery version "3.1" encoding "utf-8"; module namespace example = 'http://localhost/example'; declare %rest:path("/example/post/{$fname}") %rest:POST("{$body}") %updating function example:post($fname, $body as document-node()) {db:replace('exampledb', '/post/'||$fname, $body) };
Script run in BaseX GUI
let $file := 'C:\temp\xq\example.xml' let $fname := 'example.xml' let $targetrestxqproc := 'http://localhost:8984/example/post/' let $body := fetch:xml($file, map { 'chop': true() }) return http:send-request(<http:request href="{$targetrestxqproc||$fname}" method='POST'> <http:body media-type='text/xml'>{$body}</http:body> </http:request>)
Result in exampledb is
<example xmlns:http="http://expath.org/ns/http-client"/>
basex-talk@mailman.uni-konstanz.de
-
chris sweeney -
chrisis
-
Christian Grün