Hello,
I have problem with http:send-request. I want to send post method request with form data but I encountered problem with characters encoding.
Example:
Request:
for $x in http:send-request(
<http:request method='post' href='http://localhost:3333%27%3E
<http:header name="User-Agent" value="Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.99 Safari/537.36"/>
<http:body media-type='application/x-www-form-urlencoded; charset=UTF-8'>
<![CDATA[start=0&length=100&sort=date_type&page=0&size=100]]>
</http:body>
</http:request>
)[2]/json/data/results/content
return $x
Optimized Query:
util:item-at(http:send-request(element Q{http://expath.org/ns/http-client%7Drequest { (attribute method { ("post") }, attribute href { ("http://localhost:3333") }, element Q{http://expath.org/ns/http-client%7Dheader { (attribute name { ("User-Agent") }, attribute value { ("Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.99 Safari/537.36") }) }, element Q{http://expath.org/ns/http-client%7Dbody { (attribute media-type { ("application/x-www-form-urlencoded; charset=UTF-8") }, "
 start=0&length=100&sort=date_type&page=0&size=100
 ") }) }), 2)/json/data/results/content
Request send:
POST / HTTP/1.1
User-Agent: Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.99 Safari/537.36
Content-Type: application/x-www-form-urlencoded; charset=UTF-8
Host: copart.com:3333
Accept: text/html, image/gif, image/jpeg, *; q=.2, */*; q=.2
Connection: keep-alive
Content-Length: 75
start=0&length=100&sort=date_type&page=0&size=100
Payload is not correctly encoded it should be:
start=0&length=100&sort=date_type&page=0&size=100
How can I force function http:send-request to serialize CDATA without escaping ?
Best Regards
Bogdan Bogucki
Hi Bogdan,
If you want your arguments parsed as HTTP query parameters, it should suffice to attach them to the URL:
let $url := 'http://localhost:3333' let $params := ``[start=0&length=100&sort=date_type&page=0&size=100]`` return http:send-request( <http:request method='post' href='{ $url || '?' || $params }'> ...
If this is not what you need, you might need to give us more information on how the request will be handled server-side (for example, do you use RESTXQ?).
Best, Christian
On Mon, Jul 2, 2018 at 11:11 PM Bogdan Bogucki bbogucki@hussar.pl wrote:
Hello,
I have problem with http:send-request. I want to send post method request with form data but I encountered problem with characters encoding.
Example:
Request:
for $x in http:send-request(
<http:request method='post' href=
<http:header name="User-Agent" value="Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.99 Safari/537.36"/> <http:body media-type='application/x-www-form-urlencoded; charset=UTF-8'> <![CDATA[start=0&length=100&sort=date_type&page=0&size=100]]> </http:body></http:request>
)[2]/json/data/results/content
return $x
Optimized Query:
util:item-at(http:send-request(element Q{http://expath.org/ns/http-client%7Drequest { (attribute method { ("post") }, attribute href { ("http://localhost:3333") }, element Q{http://expath.org/ns/http-client%7Dheader { (attribute name { ("User-Agent") }, attribute value { ("Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.99 Safari/537.36") }) }, element Q{http://expath.org/ns/http-client%7Dbody { (attribute media-type { ("application/x-www-form-urlencoded; charset=UTF-8") }, "
 start=0&length=100&sort=date_type&page=0&size=100
 ") }) }), 2)/json/data/results/content
Request send:
POST / HTTP/1.1
User-Agent: Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.99 Safari/537.36
Content-Type: application/x-www-form-urlencoded; charset=UTF-8
Host: copart.com:3333
Accept: text/html, image/gif, image/jpeg, *; q=.2, */*; q=.2
Connection: keep-alive
Content-Length: 75
start=0&length=100&sort=date_type&page=0&size=100Payload is not correctly encoded it should be:
start=0&length=100&sort=date_type&page=0&size=100
How can I force function http:send-request to serialize CDATA without escaping ?
Best Regards
Bogdan Bogucki
Hello,
I don't want pass values in query string. Server side expect parameters in payload. HTML Form parameters in POST method are passed in payload not in URI. This is the reason why I put parameters in body. Problem is escaping & character with & How can I force BaseX to not escape characters in CDATA element ?
This is my code: <http:request method='post' href='http://localhost:3333%27%3E <http:header name="User-Agent" value="Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.99 Safari/537.36"/> <http:body media-type='application/x-www-form-urlencoded; charset=UTF-8'> <![CDATA[start=0&length=100&sort=date_type&page=0&size=100]]> </http:body> </http:request>
On server side I received following request:
POST / HTTP/1.1 User-Agent: Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.99 Safari/537.36 Content-Type: application/x-www-form-urlencoded; charset=UTF-8
Host: copart.com:3333
Accept: text/html, image/gif, image/jpeg, *; q=.2, */*; q=.2
Connection: keep-alive Content-Length: 75
start=0&length=100&sort=date_type&page=0&size=100
I expect that CDATA will be not escaped, or I am wrong ?
Pozdrawiam Bogdan Bogucki
HUSSAR SYSTEMS
NIP: 532-170-12-00 Regon: 142624070
e-mail: bbogucki@hussar.pl tel. kom.: +48 607 409 301 www: www.hussar.pl
-----Wiadomość oryginalna----- Od: Christian Grün [mailto:christian.gruen@gmail.com] Wysłano: 3 lipca 2018 12:11 Do: bbogucki@hussar.pl DW: BaseX Temat: Re: [basex-talk] http:send-request - problem with form data
Hi Bogdan,
If you want your arguments parsed as HTTP query parameters, it should suffice to attach them to the URL:
let $url := 'http://localhost:3333' let $params := ``[start=0&length=100&sort=date_type&page=0&size=100]`` return http:send-request( <http:request method='post' href='{ $url || '?' || $params }'> ...
If this is not what you need, you might need to give us more information on how the request will be handled server-side (for example, do you use RESTXQ?).
Best, Christian
On Mon, Jul 2, 2018 at 11:11 PM Bogdan Bogucki bbogucki@hussar.pl wrote:
Hello,
I have problem with http:send-request. I want to send post method request with form data but I encountered problem with characters encoding.
Example:
Request:
for $x in http:send-request(
<http:request method='post' href=
<http:header name="User-Agent" value="Mozilla/5.0 (Windows NT 6.1;Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.99 Safari/537.36"/>
<http:body media-type='application/x-www-form-urlencoded;charset=UTF-8'>
<![CDATA[start=0&length=100&sort=date_type&page=0&size=100]]> </http:body></http:request>
)[2]/json/data/results/content
return $x
Optimized Query:
util:item-at(http:send-request(element Q{http://expath.org/ns/http-client%7Drequest { (attribute method { ("post") }, attribute href { ("http://localhost:3333") }, element Q{http://expath.org/ns/http-client%7Dheader { (attribute name { ("User-Agent") }, attribute value { ("Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.99 Safari/537.36") }) }, element Q{http://expath.org/ns/http-client%7Dbody { (attribute media-type { ("application/x-www-form-urlencoded; charset=UTF-8") }, "
 start=0&length=100&sort=date_type&page=0&size=100
 ") }) }), 2)/json/data/results/content
Request send:
POST / HTTP/1.1
User-Agent: Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.99 Safari/537.36
Content-Type: application/x-www-form-urlencoded; charset=UTF-8
Host: copart.com:3333
Accept: text/html, image/gif, image/jpeg, *; q=.2, */*; q=.2
Connection: keep-alive
Content-Length: 75
start=0&length=100&sort=date_type&page=0&size=100Payload is not correctly encoded it should be:
start=0&length=100&sort=date_type&page=0&size=100
How can I force function http:send-request to serialize CDATA without escaping ?
Best Regards
Bogdan Bogucki
Hi Bogdan,
I didn’t try it, but what happens if you specify the body as third argument?
Cheers, Christian
On Tue, Jul 3, 2018 at 10:14 PM Bogdan Bogucki bbogucki@hussar.pl wrote:
Hello,
I don't want pass values in query string. Server side expect parameters in payload. HTML Form parameters in POST method are passed in payload not in URI. This is the reason why I put parameters in body. Problem is escaping & character with & How can I force BaseX to not escape characters in CDATA element ?
This is my code: <http:request method='post' href='http://localhost:3333%27%3E <http:header name="User-Agent" value="Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.99 Safari/537.36"/> <http:body media-type='application/x-www-form-urlencoded; charset=UTF-8'> <![CDATA[start=0&length=100&sort=date_type&page=0&size=100]]> </http:body> </http:request>
On server side I received following request:
POST / HTTP/1.1 User-Agent: Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.99 Safari/537.36 Content-Type: application/x-www-form-urlencoded; charset=UTF-8
Host: copart.com:3333
Accept: text/html, image/gif, image/jpeg, *; q=.2, */*; q=.2
Connection: keep-alive Content-Length: 75
start=0&length=100&sort=date_type&page=0&size=100
I expect that CDATA will be not escaped, or I am wrong ?
Pozdrawiam Bogdan Bogucki
HUSSAR SYSTEMS
NIP: 532-170-12-00 Regon: 142624070
e-mail: bbogucki@hussar.pl tel. kom.: +48 607 409 301 www: www.hussar.pl
-----Wiadomość oryginalna----- Od: Christian Grün [mailto:christian.gruen@gmail.com] Wysłano: 3 lipca 2018 12:11 Do: bbogucki@hussar.pl DW: BaseX Temat: Re: [basex-talk] http:send-request - problem with form data
Hi Bogdan,
If you want your arguments parsed as HTTP query parameters, it should suffice to attach them to the URL:
let $url := 'http://localhost:3333' let $params := ``[start=0&length=100&sort=date_type&page=0&size=100]`` return http:send-request( <http:request method='post' href='{ $url || '?' || $params }'> ...
If this is not what you need, you might need to give us more information on how the request will be handled server-side (for example, do you use RESTXQ?).
Best, Christian
On Mon, Jul 2, 2018 at 11:11 PM Bogdan Bogucki bbogucki@hussar.pl wrote:
Hello,
I have problem with http:send-request. I want to send post method request with form data but I encountered problem with characters encoding.
Example:
Request:
for $x in http:send-request(
<http:request method='post' href=
<http:header name="User-Agent" value="Mozilla/5.0 (Windows NT 6.1;Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.99 Safari/537.36"/>
<http:body media-type='application/x-www-form-urlencoded;charset=UTF-8'>
<![CDATA[start=0&length=100&sort=date_type&page=0&size=100]]> </http:body></http:request>
)[2]/json/data/results/content
return $x
Optimized Query:
util:item-at(http:send-request(element Q{http://expath.org/ns/http-client%7Drequest { (attribute method { ("post") }, attribute href { ("http://localhost:3333") }, element Q{http://expath.org/ns/http-client%7Dheader { (attribute name { ("User-Agent") }, attribute value { ("Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.99 Safari/537.36") }) }, element Q{http://expath.org/ns/http-client%7Dbody { (attribute media-type { ("application/x-www-form-urlencoded; charset=UTF-8") }, "
 start=0&length=100&sort=date_type&page=0&size=100
 ") }) }), 2)/json/data/results/content
Request send:
POST / HTTP/1.1
User-Agent: Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.99 Safari/537.36
Content-Type: application/x-www-form-urlencoded; charset=UTF-8
Host: copart.com:3333
Accept: text/html, image/gif, image/jpeg, *; q=.2, */*; q=.2
Connection: keep-alive
Content-Length: 75
start=0&length=100&sort=date_type&page=0&size=100Payload is not correctly encoded it should be:
start=0&length=100&sort=date_type&page=0&size=100
How can I force function http:send-request to serialize CDATA without escaping ?
Best Regards
Bogdan Bogucki
Hello,
It works with body as a third parameter. Body has to be defined in http request because without that third parameter is ignored.
Working example:
for $x in http:send-request( <http:request method='post'> <http:header name="User-Agent" value="Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.99 Safari/537.36"/> <http:body media-type='application/x-www-form-urlencoded; charset=UTF-8'/> </http:request> ,'http:localhost:3333', 'start=0&length=100& ')[2]/data/results/content return $x
Thank you Christian for help.
Best Regards Bogdan Bogucki
-----Wiadomość oryginalna----- Od: Christian Grün [mailto:christian.gruen@gmail.com] Wysłano: 4 lipca 2018 18:31 Do: bbogucki@hussar.pl DW: BaseX Temat: Re: [basex-talk] http:send-request - problem with form data
Hi Bogdan,
I didn’t try it, but what happens if you specify the body as third argument?
Cheers, Christian
On Tue, Jul 3, 2018 at 10:14 PM Bogdan Bogucki bbogucki@hussar.pl wrote:
Hello,
I don't want pass values in query string. Server side expect parameters in payload. HTML Form parameters in POST method are passed in payload not in URI. This is the reason why I put parameters in body. Problem is escaping & character with & How can I force BaseX to not escape characters in CDATA element ?
This is my code: <http:request method='post' href='http://localhost:3333%27%3E <http:header name="User-Agent" value="Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.99 Safari/537.36"/> <http:body media-type='application/x-www-form-urlencoded; charset=UTF-8'> <![CDATA[start=0&length=100&sort=date_type&page=0&size=100]]> </http:body> </http:request>
On server side I received following request:
POST / HTTP/1.1 User-Agent: Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.99 Safari/537.36 Content-Type: application/x-www-form-urlencoded; charset=UTF-8
Host: copart.com:3333
Accept: text/html, image/gif, image/jpeg, *; q=.2, */*; q=.2
Connection: keep-alive Content-Length: 75
start=0&length=100&sort=date_type&page=0&size=100
I expect that CDATA will be not escaped, or I am wrong ?
Pozdrawiam Bogdan Bogucki
HUSSAR SYSTEMS
NIP: 532-170-12-00 Regon: 142624070
e-mail: bbogucki@hussar.pl tel. kom.: +48 607 409 301 www: www.hussar.pl
-----Wiadomość oryginalna----- Od: Christian Grün [mailto:christian.gruen@gmail.com] Wysłano: 3 lipca 2018 12:11 Do: bbogucki@hussar.pl DW: BaseX Temat: Re: [basex-talk] http:send-request - problem with form data
Hi Bogdan,
If you want your arguments parsed as HTTP query parameters, it should suffice to attach them to the URL:
let $url := 'http://localhost:3333' let $params := ``[start=0&length=100&sort=date_type&page=0&size=100]`` return http:send-request( <http:request method='post' href='{ $url || '?' || $params }'> ...
If this is not what you need, you might need to give us more information on how the request will be handled server-side (for example, do you use RESTXQ?).
Best, Christian
On Mon, Jul 2, 2018 at 11:11 PM Bogdan Bogucki bbogucki@hussar.pl wrote:
Hello,
I have problem with http:send-request. I want to send post method request with form data but I encountered problem with characters encoding.
Example:
Request:
for $x in http:send-request(
<http:request method='post' href=
<http:header name="User-Agent" value="Mozilla/5.0 (Windows NT6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.99 Safari/537.36"/>
<http:body media-type='application/x-www-form-urlencoded;charset=UTF-8'>
<![CDATA[start=0&length=100&sort=date_type&page=0&size=100]]> </http:body></http:request>
)[2]/json/data/results/content
return $x
Optimized Query:
util:item-at(http:send-request(element Q{http://expath.org/ns/http-client%7Drequest { (attribute method { ("post") }, attribute href { ("http://localhost:3333") }, element Q{http://expath.org/ns/http-client%7Dheader { (attribute name { ("User-Agent") }, attribute value { ("Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.99 Safari/537.36") }) }, element Q{http://expath.org/ns/http-client%7Dbody { (attribute media-type { ("application/x-www-form-urlencoded; charset=UTF-8") }, "
 start=0&length=100&sort=date_type&page=0&size=100
 ") }) }), 2)/json/data/results/content
Request send:
POST / HTTP/1.1
User-Agent: Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.99 Safari/537.36
Content-Type: application/x-www-form-urlencoded; charset=UTF-8
Host: copart.com:3333
Accept: text/html, image/gif, image/jpeg, *; q=.2, */*; q=.2
Connection: keep-alive
Content-Length: 75
start=0&length=100&sort=date_type&page=0&size=100
Payload is not correctly encoded it should be:
start=0&length=100&sort=date_type&page=0&size=100
How can I force function http:send-request to serialize CDATA without escaping ?
Best Regards
Bogdan Bogucki
And thanks for the confirmation. I will try to find out why the two alternatives make a difference.
Bogdan Bogucki bbogucki@hussar.pl schrieb am Mi., 4. Juli 2018, 21:56:
Hello,
It works with body as a third parameter. Body has to be defined in http request because without that third parameter is ignored.
Working example:
for $x in http:send-request( <http:request method='post'> <http:header name="User-Agent" value="Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.99 Safari/537.36"/> <http:body media-type='application/x-www-form-urlencoded; charset=UTF-8'/> </http:request> ,'http:localhost:3333', 'start=0&length=100& ')[2]/data/results/content return $x
Thank you Christian for help.
Best Regards Bogdan Bogucki
-----Wiadomość oryginalna----- Od: Christian Grün [mailto:christian.gruen@gmail.com] Wysłano: 4 lipca 2018 18:31 Do: bbogucki@hussar.pl DW: BaseX Temat: Re: [basex-talk] http:send-request - problem with form data
Hi Bogdan,
I didn’t try it, but what happens if you specify the body as third argument?
Cheers, Christian
On Tue, Jul 3, 2018 at 10:14 PM Bogdan Bogucki bbogucki@hussar.pl wrote:
Hello,
I don't want pass values in query string. Server side expect parameters
in payload.
HTML Form parameters in POST method are passed in payload not in URI. This is the reason why I put parameters in body. Problem is escaping & character with & How can I force BaseX to not escape characters in
CDATA element ?
This is my code: <http:request method='post' href='http://localhost:3333%27%3E <http:header name="User-Agent" value="Mozilla/5.0 (Windows NT 6.1;
Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.99 Safari/537.36"/>
<http:body media-type='application/x-www-form-urlencoded;charset=UTF-8'>
<![CDATA[start=0&length=100&sort=date_type&page=0&size=100]]> </http:body></http:request>
On server side I received following request:
POST / HTTP/1.1 User-Agent: Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.99 Safari/537.36 Content-Type: application/x-www-form-urlencoded; charset=UTF-8
Host: copart.com:3333
Accept: text/html, image/gif, image/jpeg, *; q=.2, */*; q=.2
Connection: keep-alive Content-Length: 75
start=0&length=100&sort=date_type&page=0&size=100
I expect that CDATA will be not escaped, or I am wrong ?
Pozdrawiam Bogdan Bogucki
HUSSAR SYSTEMS
NIP: 532-170-12-00 Regon: 142624070
e-mail: bbogucki@hussar.pl tel. kom.: +48 607 409 301 www: www.hussar.pl
-----Wiadomość oryginalna----- Od: Christian Grün [mailto:christian.gruen@gmail.com] Wysłano: 3 lipca 2018 12:11 Do: bbogucki@hussar.pl DW: BaseX Temat: Re: [basex-talk] http:send-request - problem with form data
Hi Bogdan,
If you want your arguments parsed as HTTP query parameters, it should
suffice to attach them to the URL:
let $url := 'http://localhost:3333' let $params := ``[start=0&length=100&sort=date_type&page=0&size=100]`` return http:send-request( <http:request method='post' href='{ $url || '?' || $params }'> ...
If this is not what you need, you might need to give us more information
on how the request will be handled server-side (for example, do you use RESTXQ?).
Best, Christian
On Mon, Jul 2, 2018 at 11:11 PM Bogdan Bogucki bbogucki@hussar.pl
wrote:
Hello,
I have problem with http:send-request. I want to send post method
request with form data but I encountered problem with characters encoding.
Example:
Request:
for $x in http:send-request(
<http:request method='post' href=
<http:header name="User-Agent" value="Mozilla/5.0 (Windows NT6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.99 Safari/537.36"/>
<http:body media-type='application/x-www-form-urlencoded;charset=UTF-8'>
<![CDATA[start=0&length=100&sort=date_type&page=0&size=100]]> </http:body></http:request>
)[2]/json/data/results/content
return $x
Optimized Query:
util:item-at(http:send-request(element Q{
http://expath.org/ns/http-client%7Drequest { (attribute method { ("post") }, attribute href { ("http://localhost:3333") }, element Q{ http://expath.org/ns/http-client%7Dheader { (attribute name { ("User-Agent") }, attribute value { ("Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.99 Safari/537.36") }) }, element Q{http://expath.org/ns/http-client%7Dbody { (attribute media-type { ("application/x-www-form-urlencoded; charset=UTF-8") }, "
 start=0&length=100&sort=date_type&page=0&size=100
 ") }) }), 2)/json/data/results/content
Request send:
POST / HTTP/1.1
User-Agent: Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.99 Safari/537.36
Content-Type: application/x-www-form-urlencoded; charset=UTF-8
Host: copart.com:3333
Accept: text/html, image/gif, image/jpeg, *; q=.2, */*; q=.2
Connection: keep-alive
Content-Length: 75
start=0&length=100&sort=date_type&page=0&size=100
Payload is not correctly encoded it should be:
start=0&length=100&sort=date_type&page=0&size=100
How can I force function http:send-request to serialize CDATA without
escaping ?
Best Regards
Bogdan Bogucki
Hi Bogdan,
I finally had some time to get to the bottom of the conversion inconsistencies:
If the value is supplied as child of the http:body node, it will be serialized as text node. If it’s supplied as extra parameter, it will be serialized as string. If text nodes are serialized, entities will be encoded. Strings will be returned without encoded entities:
<a><![CDATA[&]]></a> → <a>&</a> ``[&]`` → &
I agree this is not very intuitive, and I have now rewritten this as follows: If the supplied output media-type does not result in one of the default serialization methods of the spec (xml, xhtml, html, json, text, adaptive), the text node will be atomized (i.e., converted to an item of type xs:untypedAtomic), which will then be serialized as plain string. With the new snapshot [1], it should make no difference if you supply your value as child of http:body or as extra argument.
As a general guide, I would recommended everyone to use the third argument for body items. It will be both safer and faster (in particular if binary data can be streamed). As the rich syntax of http:send-request is regularly confusing users on the list, I have reopened an old Github issue, which is about providing additional and simpler functions for the most widely used HTTP methods [2].
Best, Christian
[1] http://files.basex.org/releases/latest/ [2] https://github.com/BaseXdb/basex/issues/914
On Wed, Jul 4, 2018 at 10:05 PM Christian Grün christian.gruen@gmail.com wrote:
And thanks for the confirmation. I will try to find out why the two alternatives make a difference.
Bogdan Bogucki bbogucki@hussar.pl schrieb am Mi., 4. Juli 2018, 21:56:
Hello,
It works with body as a third parameter. Body has to be defined in http request because without that third parameter is ignored.
Working example:
for $x in http:send-request( <http:request method='post'> <http:header name="User-Agent" value="Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.99 Safari/537.36"/> <http:body media-type='application/x-www-form-urlencoded; charset=UTF-8'/> </http:request> ,'http:localhost:3333', 'start=0&length=100& ')[2]/data/results/content return $x
Thank you Christian for help.
Best Regards Bogdan Bogucki
-----Wiadomość oryginalna----- Od: Christian Grün [mailto:christian.gruen@gmail.com] Wysłano: 4 lipca 2018 18:31 Do: bbogucki@hussar.pl DW: BaseX Temat: Re: [basex-talk] http:send-request - problem with form data
Hi Bogdan,
I didn’t try it, but what happens if you specify the body as third argument?
Cheers, Christian
On Tue, Jul 3, 2018 at 10:14 PM Bogdan Bogucki bbogucki@hussar.pl wrote:
Hello,
I don't want pass values in query string. Server side expect parameters in payload. HTML Form parameters in POST method are passed in payload not in URI. This is the reason why I put parameters in body. Problem is escaping & character with & How can I force BaseX to not escape characters in CDATA element ?
This is my code: <http:request method='post' href='http://localhost:3333%27%3E <http:header name="User-Agent" value="Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.99 Safari/537.36"/> <http:body media-type='application/x-www-form-urlencoded; charset=UTF-8'> <![CDATA[start=0&length=100&sort=date_type&page=0&size=100]]> </http:body> </http:request>
On server side I received following request:
POST / HTTP/1.1 User-Agent: Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.99 Safari/537.36 Content-Type: application/x-www-form-urlencoded; charset=UTF-8
Host: copart.com:3333
Accept: text/html, image/gif, image/jpeg, *; q=.2, */*; q=.2
Connection: keep-alive Content-Length: 75
start=0&length=100&sort=date_type&page=0&size=100
I expect that CDATA will be not escaped, or I am wrong ?
Pozdrawiam Bogdan Bogucki
HUSSAR SYSTEMS
NIP: 532-170-12-00 Regon: 142624070
e-mail: bbogucki@hussar.pl tel. kom.: +48 607 409 301 www: www.hussar.pl
-----Wiadomość oryginalna----- Od: Christian Grün [mailto:christian.gruen@gmail.com] Wysłano: 3 lipca 2018 12:11 Do: bbogucki@hussar.pl DW: BaseX Temat: Re: [basex-talk] http:send-request - problem with form data
Hi Bogdan,
If you want your arguments parsed as HTTP query parameters, it should suffice to attach them to the URL:
let $url := 'http://localhost:3333' let $params := ``[start=0&length=100&sort=date_type&page=0&size=100]`` return http:send-request( <http:request method='post' href='{ $url || '?' || $params }'> ...
If this is not what you need, you might need to give us more information on how the request will be handled server-side (for example, do you use RESTXQ?).
Best, Christian
On Mon, Jul 2, 2018 at 11:11 PM Bogdan Bogucki bbogucki@hussar.pl wrote:
Hello,
I have problem with http:send-request. I want to send post method request with form data but I encountered problem with characters encoding.
Example:
Request:
for $x in http:send-request(
<http:request method='post' href=
<http:header name="User-Agent" value="Mozilla/5.0 (Windows NT6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.99 Safari/537.36"/>
<http:body media-type='application/x-www-form-urlencoded;charset=UTF-8'>
<![CDATA[start=0&length=100&sort=date_type&page=0&size=100]]> </http:body></http:request>
)[2]/json/data/results/content
return $x
Optimized Query:
util:item-at(http:send-request(element Q{http://expath.org/ns/http-client%7Drequest { (attribute method { ("post") }, attribute href { ("http://localhost:3333") }, element Q{http://expath.org/ns/http-client%7Dheader { (attribute name { ("User-Agent") }, attribute value { ("Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.99 Safari/537.36") }) }, element Q{http://expath.org/ns/http-client%7Dbody { (attribute media-type { ("application/x-www-form-urlencoded; charset=UTF-8") }, "
 start=0&length=100&sort=date_type&page=0&size=100
 ") }) }), 2)/json/data/results/content
Request send:
POST / HTTP/1.1
User-Agent: Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.99 Safari/537.36
Content-Type: application/x-www-form-urlencoded; charset=UTF-8
Host: copart.com:3333
Accept: text/html, image/gif, image/jpeg, *; q=.2, */*; q=.2
Connection: keep-alive
Content-Length: 75
start=0&length=100&sort=date_type&page=0&size=100
Payload is not correctly encoded it should be:
start=0&length=100&sort=date_type&page=0&size=100
How can I force function http:send-request to serialize CDATA without escaping ?
Best Regards
Bogdan Bogucki
Hi Christian,
Great!
I have one doubt why when we don't define body in http request third parameter is ignored ?
Only when body element is provided parseBody with bodies is invoked. I have to check source code to figure out how to pass body payload via third argument.
if(body != null) { final QNm pl = body.qname(); // single part request if(pl.eq(Q_BODY)) { parseBody(body, bodies, hr.payloadAtts, hr.payload); hr.isMultipart = false; // multipart request } else if(pl.eq(Q_MULTIPART)) { parseMultipart(body, bodies.iter(), hr.payloadAtts, hr.parts); hr.isMultipart = true; } else { throw HC_REQ_X.get(info, "Unknown payload element: " + body.qname()); } }
Best Regards Bogdan Bogucki
-----Wiadomość oryginalna----- Od: Christian Grün [mailto:christian.gruen@gmail.com] Wysłano: 12 lipca 2018 15:36 Do: Bogdan Bogucki DW: BaseX Temat: Re: [basex-talk] http:send-request - problem with form data
Hi Bogdan,
I finally had some time to get to the bottom of the conversion inconsistencies:
If the value is supplied as child of the http:body node, it will be serialized as text node. If it’s supplied as extra parameter, it will be serialized as string. If text nodes are serialized, entities will be encoded. Strings will be returned without encoded entities:
<a><![CDATA[&]]></a> → <a>&</a> ``[&]`` → &
I agree this is not very intuitive, and I have now rewritten this as follows: If the supplied output media-type does not result in one of the default serialization methods of the spec (xml, xhtml, html, json, text, adaptive), the text node will be atomized (i.e., converted to an item of type xs:untypedAtomic), which will then be serialized as plain string. With the new snapshot [1], it should make no difference if you supply your value as child of http:body or as extra argument.
As a general guide, I would recommended everyone to use the third argument for body items. It will be both safer and faster (in particular if binary data can be streamed). As the rich syntax of http:send-request is regularly confusing users on the list, I have reopened an old Github issue, which is about providing additional and simpler functions for the most widely used HTTP methods [2].
Best, Christian
[1] http://files.basex.org/releases/latest/ [2] https://github.com/BaseXdb/basex/issues/914
On Wed, Jul 4, 2018 at 10:05 PM Christian Grün christian.gruen@gmail.com wrote:
And thanks for the confirmation. I will try to find out why the two alternatives make a difference.
Bogdan Bogucki bbogucki@hussar.pl schrieb am Mi., 4. Juli 2018, 21:56:
Hello,
It works with body as a third parameter. Body has to be defined in http request because without that third parameter is ignored.
Working example:
for $x in http:send-request( <http:request method='post'> <http:header name="User-Agent" value="Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.99 Safari/537.36"/> <http:body media-type='application/x-www-form-urlencoded; charset=UTF-8'/> </http:request> ,'http:localhost:3333', 'start=0&length=100& ')[2]/data/results/content return $x
Thank you Christian for help.
Best Regards Bogdan Bogucki
-----Wiadomość oryginalna----- Od: Christian Grün [mailto:christian.gruen@gmail.com] Wysłano: 4 lipca 2018 18:31 Do: bbogucki@hussar.pl DW: BaseX Temat: Re: [basex-talk] http:send-request - problem with form data
Hi Bogdan,
I didn’t try it, but what happens if you specify the body as third argument?
Cheers, Christian
On Tue, Jul 3, 2018 at 10:14 PM Bogdan Bogucki bbogucki@hussar.pl wrote:
Hello,
I don't want pass values in query string. Server side expect parameters in payload. HTML Form parameters in POST method are passed in payload not in URI. This is the reason why I put parameters in body. Problem is escaping & character with & How can I force BaseX to not escape characters in CDATA element ?
This is my code: <http:request method='post' href='http://localhost:3333%27%3E <http:header name="User-Agent" value="Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.99 Safari/537.36"/> <http:body media-type='application/x-www-form-urlencoded; charset=UTF-8'> <![CDATA[start=0&length=100&sort=date_type&page=0&size=100]]> </http:body> </http:request>
On server side I received following request:
POST / HTTP/1.1 User-Agent: Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.99 Safari/537.36 Content-Type: application/x-www-form-urlencoded; charset=UTF-8
Host: copart.com:3333
Accept: text/html, image/gif, image/jpeg, *; q=.2, */*; q=.2
Connection: keep-alive Content-Length: 75
start=0&length=100&sort=date_type&page=0&size=100
I expect that CDATA will be not escaped, or I am wrong ?
Pozdrawiam Bogdan Bogucki
HUSSAR SYSTEMS
NIP: 532-170-12-00 Regon: 142624070
e-mail: bbogucki@hussar.pl tel. kom.: +48 607 409 301 www: www.hussar.pl
-----Wiadomość oryginalna----- Od: Christian Grün [mailto:christian.gruen@gmail.com] Wysłano: 3 lipca 2018 12:11 Do: bbogucki@hussar.pl DW: BaseX Temat: Re: [basex-talk] http:send-request - problem with form data
Hi Bogdan,
If you want your arguments parsed as HTTP query parameters, it should suffice to attach them to the URL:
let $url := 'http://localhost:3333' let $params := ``[start=0&length=100&sort=date_type&page=0&size=100]`` return http:send-request( <http:request method='post' href='{ $url || '?' || $params }'> ...
If this is not what you need, you might need to give us more information on how the request will be handled server-side (for example, do you use RESTXQ?).
Best, Christian
On Mon, Jul 2, 2018 at 11:11 PM Bogdan Bogucki bbogucki@hussar.pl wrote:
Hello,
I have problem with http:send-request. I want to send post method request with form data but I encountered problem with characters encoding.
Example:
Request:
for $x in http:send-request(
<http:request method='post' href=
<http:header name="User-Agent" value="Mozilla/5.0 (Windows NT6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.99 Safari/537.36"/>
<http:body media-type='application/x-www-form-urlencoded;charset=UTF-8'>
<![CDATA[start=0&length=100&sort=date_type&page=0&size=100]]> </http:body></http:request>
)[2]/json/data/results/content
return $x
Optimized Query:
util:item-at(http:send-request(element Q{http://expath.org/ns/http-client%7Drequest { (attribute method { ("post") }, attribute href { ("http://localhost:3333") }, element Q{http://expath.org/ns/http-client%7Dheader { (attribute name { ("User-Agent") }, attribute value { ("Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.99 Safari/537.36") }) }, element Q{http://expath.org/ns/http-client%7Dbody { (attribute media-type { ("application/x-www-form-urlencoded; charset=UTF-8") }, "
 start=0&length=100&sort=date_type&page=0&size=100
 ") }) }), 2)/json/data/results/content
Request send:
POST / HTTP/1.1
User-Agent: Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.99 Safari/537.36
Content-Type: application/x-www-form-urlencoded; charset=UTF-8
Host: copart.com:3333
Accept: text/html, image/gif, image/jpeg, *; q=.2, */*; q=.2
Connection: keep-alive
Content-Length: 75
start=0&length=100&sort=date_type&page=0&size=100
Payload is not correctly encoded it should be:
start=0&length=100&sort=date_type&page=0&size=100
How can I force function http:send-request to serialize CDATA without escaping ?
Best Regards
Bogdan Bogucki
Hi Bogdan,
I have one doubt why when we don't define body in http request third parameter is ignored ?
It was defined like this in the EXPath spec [1]:
“When a body element has an empty content (i.e. it has no child node at all) its content is given by the parameter $bodies. In a single part request, this param must have at most one item. If the body is empty, the param cannot be the empty sequence. In a multipart request, $bodies must have as many items as there are empty body elements. If there are three empty body elements, the content of the first of them is $bodies[1], and so on. The number of empty body elements must be equal to the number of items in $bodies.”
I guess it would have been more user-friendly if an error was raised if these requirements are not met by the function call.
Anyway, as indicated, I’d be happy if we had functions with unambiguous function arguments in the future. The Zorba HTTP Client Module could serve as inspiration [2]. I have never tried it in practice, but it looks pretty well-designed (an additional options argument should probably added to the simple functions).
Best, Christian
[1] http://expath.org/spec/http-client#d2e362 [2] http://www.zorba.io/documentation/3.0/modules/zorba/io/http-client/
Only when body element is provided parseBody with bodies is invoked. I have to check source code to figure out how to pass body payload via third argument.
if(body != null) { final QNm pl = body.qname(); // single part request if(pl.eq(Q_BODY)) { parseBody(body, bodies, hr.payloadAtts, hr.payload); hr.isMultipart = false; // multipart request } else if(pl.eq(Q_MULTIPART)) { parseMultipart(body, bodies.iter(), hr.payloadAtts, hr.parts); hr.isMultipart = true; } else { throw HC_REQ_X.get(info, "Unknown payload element: " + body.qname()); } }
Best Regards Bogdan Bogucki
-----Wiadomość oryginalna----- Od: Christian Grün [mailto:christian.gruen@gmail.com] Wysłano: 12 lipca 2018 15:36 Do: Bogdan Bogucki DW: BaseX Temat: Re: [basex-talk] http:send-request - problem with form data
Hi Bogdan,
I finally had some time to get to the bottom of the conversion inconsistencies:
If the value is supplied as child of the http:body node, it will be serialized as text node. If it’s supplied as extra parameter, it will be serialized as string. If text nodes are serialized, entities will be encoded. Strings will be returned without encoded entities:
<a><![CDATA[&]]></a> → <a>&</a> ``[&]`` → &
I agree this is not very intuitive, and I have now rewritten this as follows: If the supplied output media-type does not result in one of the default serialization methods of the spec (xml, xhtml, html, json, text, adaptive), the text node will be atomized (i.e., converted to an item of type xs:untypedAtomic), which will then be serialized as plain string. With the new snapshot [1], it should make no difference if you supply your value as child of http:body or as extra argument.
As a general guide, I would recommended everyone to use the third argument for body items. It will be both safer and faster (in particular if binary data can be streamed). As the rich syntax of http:send-request is regularly confusing users on the list, I have reopened an old Github issue, which is about providing additional and simpler functions for the most widely used HTTP methods [2].
Best, Christian
[1] http://files.basex.org/releases/latest/ [2] https://github.com/BaseXdb/basex/issues/914
On Wed, Jul 4, 2018 at 10:05 PM Christian Grün christian.gruen@gmail.com wrote:
And thanks for the confirmation. I will try to find out why the two alternatives make a difference.
Bogdan Bogucki bbogucki@hussar.pl schrieb am Mi., 4. Juli 2018, 21:56:
Hello,
It works with body as a third parameter. Body has to be defined in http request because without that third parameter is ignored.
Working example:
for $x in http:send-request( <http:request method='post'> <http:header name="User-Agent" value="Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.99 Safari/537.36"/> <http:body media-type='application/x-www-form-urlencoded; charset=UTF-8'/> </http:request> ,'http:localhost:3333', 'start=0&length=100& ')[2]/data/results/content return $x
Thank you Christian for help.
Best Regards Bogdan Bogucki
-----Wiadomość oryginalna----- Od: Christian Grün [mailto:christian.gruen@gmail.com] Wysłano: 4 lipca 2018 18:31 Do: bbogucki@hussar.pl DW: BaseX Temat: Re: [basex-talk] http:send-request - problem with form data
Hi Bogdan,
I didn’t try it, but what happens if you specify the body as third argument?
Cheers, Christian
On Tue, Jul 3, 2018 at 10:14 PM Bogdan Bogucki bbogucki@hussar.pl wrote:
Hello,
I don't want pass values in query string. Server side expect parameters in payload. HTML Form parameters in POST method are passed in payload not in URI. This is the reason why I put parameters in body. Problem is escaping & character with & How can I force BaseX to not escape characters in CDATA element ?
This is my code: <http:request method='post' href='http://localhost:3333%27%3E <http:header name="User-Agent" value="Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.99 Safari/537.36"/> <http:body media-type='application/x-www-form-urlencoded; charset=UTF-8'> <![CDATA[start=0&length=100&sort=date_type&page=0&size=100]]> </http:body> </http:request>
On server side I received following request:
POST / HTTP/1.1 User-Agent: Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.99 Safari/537.36 Content-Type: application/x-www-form-urlencoded; charset=UTF-8
Host: copart.com:3333
Accept: text/html, image/gif, image/jpeg, *; q=.2, */*; q=.2
Connection: keep-alive Content-Length: 75
start=0&length=100&sort=date_type&page=0&size=100
I expect that CDATA will be not escaped, or I am wrong ?
Pozdrawiam Bogdan Bogucki
HUSSAR SYSTEMS
NIP: 532-170-12-00 Regon: 142624070
e-mail: bbogucki@hussar.pl tel. kom.: +48 607 409 301 www: www.hussar.pl
-----Wiadomość oryginalna----- Od: Christian Grün [mailto:christian.gruen@gmail.com] Wysłano: 3 lipca 2018 12:11 Do: bbogucki@hussar.pl DW: BaseX Temat: Re: [basex-talk] http:send-request - problem with form data
Hi Bogdan,
If you want your arguments parsed as HTTP query parameters, it should suffice to attach them to the URL:
let $url := 'http://localhost:3333' let $params := ``[start=0&length=100&sort=date_type&page=0&size=100]`` return http:send-request( <http:request method='post' href='{ $url || '?' || $params }'> ...
If this is not what you need, you might need to give us more information on how the request will be handled server-side (for example, do you use RESTXQ?).
Best, Christian
On Mon, Jul 2, 2018 at 11:11 PM Bogdan Bogucki bbogucki@hussar.pl wrote:
Hello,
I have problem with http:send-request. I want to send post method request with form data but I encountered problem with characters encoding.
Example:
Request:
for $x in http:send-request(
<http:request method='post' href=
<http:header name="User-Agent" value="Mozilla/5.0 (Windows NT6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.99 Safari/537.36"/>
<http:body media-type='application/x-www-form-urlencoded;charset=UTF-8'>
<![CDATA[start=0&length=100&sort=date_type&page=0&size=100]]> </http:body></http:request>
)[2]/json/data/results/content
return $x
Optimized Query:
util:item-at(http:send-request(element Q{http://expath.org/ns/http-client%7Drequest { (attribute method { ("post") }, attribute href { ("http://localhost:3333") }, element Q{http://expath.org/ns/http-client%7Dheader { (attribute name { ("User-Agent") }, attribute value { ("Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.99 Safari/537.36") }) }, element Q{http://expath.org/ns/http-client%7Dbody { (attribute media-type { ("application/x-www-form-urlencoded; charset=UTF-8") }, "
 start=0&length=100&sort=date_type&page=0&size=100
 ") }) }), 2)/json/data/results/content
Request send:
POST / HTTP/1.1
User-Agent: Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.99 Safari/537.36
Content-Type: application/x-www-form-urlencoded; charset=UTF-8
Host: copart.com:3333
Accept: text/html, image/gif, image/jpeg, *; q=.2, */*; q=.2
Connection: keep-alive
Content-Length: 75
start=0&length=100&sort=date_type&page=0&size=100
Payload is not correctly encoded it should be:
start=0&length=100&sort=date_type&page=0&size=100
How can I force function http:send-request to serialize CDATA without escaping ?
Best Regards
Bogdan Bogucki
The Zorba HTTP spec is very interesting. I think the original EXPath HTTP spec (and others) may have taken a more map-and-array form had those data structures been available in XQuery at the time. It would be interesting to reconsider some of these specs in a future major revision to them.
On Thu, Jul 12, 2018 at 4:33 PM Christian Grün christian.gruen@gmail.com wrote:
Hi Bogdan,
I have one doubt why when we don't define body in http request third
parameter is ignored ?
It was defined like this in the EXPath spec [1]:
“When a body element has an empty content (i.e. it has no child node at all) its content is given by the parameter $bodies. In a single part request, this param must have at most one item. If the body is empty, the param cannot be the empty sequence. In a multipart request, $bodies must have as many items as there are empty body elements. If there are three empty body elements, the content of the first of them is $bodies[1], and so on. The number of empty body elements must be equal to the number of items in $bodies.”
I guess it would have been more user-friendly if an error was raised if these requirements are not met by the function call.
Anyway, as indicated, I’d be happy if we had functions with unambiguous function arguments in the future. The Zorba HTTP Client Module could serve as inspiration [2]. I have never tried it in practice, but it looks pretty well-designed (an additional options argument should probably added to the simple functions).
Best, Christian
[1] http://expath.org/spec/http-client#d2e362 [2] http://www.zorba.io/documentation/3.0/modules/zorba/io/http-client/
Only when body element is provided parseBody with bodies is invoked. I have to check source code to figure out how to pass body payload via
third argument.
if(body != null) { final QNm pl = body.qname(); // single part request if(pl.eq(Q_BODY)) { parseBody(body, bodies, hr.payloadAtts, hr.payload); hr.isMultipart = false; // multipart request } else if(pl.eq(Q_MULTIPART)) { parseMultipart(body, bodies.iter(), hr.payloadAtts, hr.parts); hr.isMultipart = true; } else { throw HC_REQ_X.get(info, "Unknown payload element: " +
body.qname());
}}
Best Regards Bogdan Bogucki
-----Wiadomość oryginalna----- Od: Christian Grün [mailto:christian.gruen@gmail.com] Wysłano: 12 lipca 2018 15:36 Do: Bogdan Bogucki DW: BaseX Temat: Re: [basex-talk] http:send-request - problem with form data
Hi Bogdan,
I finally had some time to get to the bottom of the conversion
inconsistencies:
If the value is supplied as child of the http:body node, it will be
serialized as text node. If it’s supplied as extra parameter, it will be serialized as string. If text nodes are serialized, entities will be encoded. Strings will be returned without encoded entities:
<a><![CDATA[&]]></a> → <a>&</a> ``[&]`` → &
I agree this is not very intuitive, and I have now rewritten this as follows: If the supplied output media-type does not result in one of the
default serialization methods of the spec (xml, xhtml, html, json, text, adaptive), the text node will be atomized (i.e., converted to an item of type xs:untypedAtomic), which will then be serialized as plain string. With the new snapshot [1], it should make no difference if you supply your value as child of http:body or as extra argument.
As a general guide, I would recommended everyone to use the third
argument for body items. It will be both safer and faster (in particular if binary data can be streamed). As the rich syntax of http:send-request is regularly confusing users on the list, I have reopened an old Github issue, which is about providing additional and simpler functions for the most widely used HTTP methods [2].
Best, Christian
[1] http://files.basex.org/releases/latest/ [2] https://github.com/BaseXdb/basex/issues/914
On Wed, Jul 4, 2018 at 10:05 PM Christian Grün <
christian.gruen@gmail.com> wrote:
And thanks for the confirmation. I will try to find out why the two
alternatives make a difference.
Bogdan Bogucki bbogucki@hussar.pl schrieb am Mi., 4. Juli 2018,
21:56:
Hello,
It works with body as a third parameter. Body has to be defined in
http request because without that third parameter is ignored.
Working example:
for $x in http:send-request( <http:request method='post'> <http:header name="User-Agent" value="Mozilla/5.0 (Windows NT
6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.99 Safari/537.36"/>
<http:body media-type='application/x-www-form-urlencoded;charset=UTF-8'/>
</http:request> ,'http:localhost:3333', 'start=0&length=100& ')[2]/data/results/content return $x
Thank you Christian for help.
Best Regards Bogdan Bogucki
-----Wiadomość oryginalna----- Od: Christian Grün [mailto:christian.gruen@gmail.com] Wysłano: 4 lipca 2018 18:31 Do: bbogucki@hussar.pl DW: BaseX Temat: Re: [basex-talk] http:send-request - problem with form data
Hi Bogdan,
I didn’t try it, but what happens if you specify the body as third
argument?
Cheers, Christian
On Tue, Jul 3, 2018 at 10:14 PM Bogdan Bogucki bbogucki@hussar.pl
wrote:
Hello,
I don't want pass values in query string. Server side expect
parameters in payload.
HTML Form parameters in POST method are passed in payload not in
URI.
This is the reason why I put parameters in body. Problem is escaping & character with & How can I force BaseX to not escape
characters in CDATA element ?
This is my code: <http:request method='post' href='http://localhost:3333%27%3E <http:header name="User-Agent" value="Mozilla/5.0 (Windows NT
6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.99 Safari/537.36"/>
<http:body media-type='application/x-www-form-urlencoded;charset=UTF-8'>
<![CDATA[start=0&length=100&sort=date_type&page=0&size=100]]> </http:body></http:request>
On server side I received following request:
POST / HTTP/1.1 User-Agent: Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.99 Safari/537.36 Content-Type: application/x-www-form-urlencoded; charset=UTF-8
Host: copart.com:3333
Accept: text/html, image/gif, image/jpeg, *; q=.2, */*; q=.2
Connection: keep-alive Content-Length: 75
start=0&length=100&sort=date_type&page=0&size=100
I expect that CDATA will be not escaped, or I am wrong ?
Pozdrawiam Bogdan Bogucki
HUSSAR SYSTEMS
NIP: 532-170-12-00 Regon: 142624070
e-mail: bbogucki@hussar.pl tel. kom.: +48 607 409 301 www: www.hussar.pl
-----Wiadomość oryginalna----- Od: Christian Grün [mailto:christian.gruen@gmail.com] Wysłano: 3 lipca 2018 12:11 Do: bbogucki@hussar.pl DW: BaseX Temat: Re: [basex-talk] http:send-request - problem with form data
Hi Bogdan,
If you want your arguments parsed as HTTP query parameters, it
should suffice to attach them to the URL:
let $url := 'http://localhost:3333' let $params :=
``[start=0&length=100&sort=date_type&page=0&size=100]``
return http:send-request( <http:request method='post' href='{ $url || '?' || $params }'>
...
If this is not what you need, you might need to give us more
information on how the request will be handled server-side (for example, do you use RESTXQ?).
Best, Christian
On Mon, Jul 2, 2018 at 11:11 PM Bogdan Bogucki bbogucki@hussar.pl
wrote:
Hello,
I have problem with http:send-request. I want to send post method
request with form data but I encountered problem with characters encoding.
Example:
Request:
for $x in http:send-request(
<http:request method='post' href=
<http:header name="User-Agent" value="Mozilla/5.0 (Windows NT6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.99 Safari/537.36"/>
<http:body media-type='application/x-www-form-urlencoded;charset=UTF-8'>
<![CDATA[start=0&length=100&sort=date_type&page=0&size=100]]> </http:body></http:request>
)[2]/json/data/results/content
return $x
Optimized Query:
util:item-at(http:send-request(element Q{
http://expath.org/ns/http-client%7Drequest { (attribute method { ("post") }, attribute href { ("http://localhost:3333") }, element Q{ http://expath.org/ns/http-client%7Dheader { (attribute name { ("User-Agent") }, attribute value { ("Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.99 Safari/537.36") }) }, element Q{http://expath.org/ns/http-client%7Dbody { (attribute media-type { ("application/x-www-form-urlencoded; charset=UTF-8") }, "
 start=0&length=100&sort=date_type&page=0&size=100
 ") }) }), 2)/json/data/results/content
Request send:
POST / HTTP/1.1
User-Agent: Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.99 Safari/537.36
Content-Type: application/x-www-form-urlencoded; charset=UTF-8
Host: copart.com:3333
Accept: text/html, image/gif, image/jpeg, *; q=.2, */*; q=.2
Connection: keep-alive
Content-Length: 75
start=0&length=100&sort=date_type&page=0&size=100
Payload is not correctly encoded it should be:
start=0&length=100&sort=date_type&page=0&size=100
How can I force function http:send-request to serialize CDATA
without escaping ?
Best Regards
Bogdan Bogucki
Hi Joe,
Thanks for your assessment. I completely agree: If we manage to create a new revision of the existing EXPath spec, the extensions might be available in other XQuery implementations as well.
It could even be backward-compatible: We could keep http:send-request, and a new http:send function could be the one that has a single href argument, a single location for body items, takes advantage of maps, etc.
Do you know if eXist-db might benefit from such an extension as well? Or is the custom http://exist-db.org/xquery/httpclient module the one that is used by most developers?
Christian
Joe Wicentowski joewiz@gmail.com schrieb am Fr., 13. Juli 2018, 03:41:
The Zorba HTTP spec is very interesting. I think the original EXPath HTTP spec (and others) may have taken a more map-and-array form had those data structures been available in XQuery at the time. It would be interesting to reconsider some of these specs in a future major revision to them.
On Thu, Jul 12, 2018 at 4:33 PM Christian Grün christian.gruen@gmail.com wrote:
Hi Bogdan,
I have one doubt why when we don't define body in http request third
parameter is ignored ?
It was defined like this in the EXPath spec [1]:
“When a body element has an empty content (i.e. it has no child node at all) its content is given by the parameter $bodies. In a single part request, this param must have at most one item. If the body is empty, the param cannot be the empty sequence. In a multipart request, $bodies must have as many items as there are empty body elements. If there are three empty body elements, the content of the first of them is $bodies[1], and so on. The number of empty body elements must be equal to the number of items in $bodies.”
I guess it would have been more user-friendly if an error was raised if these requirements are not met by the function call.
Anyway, as indicated, I’d be happy if we had functions with unambiguous function arguments in the future. The Zorba HTTP Client Module could serve as inspiration [2]. I have never tried it in practice, but it looks pretty well-designed (an additional options argument should probably added to the simple functions).
Best, Christian
[1] http://expath.org/spec/http-client#d2e362 [2] http://www.zorba.io/documentation/3.0/modules/zorba/io/http-client/
Only when body element is provided parseBody with bodies is invoked. I have to check source code to figure out how to pass body payload via
third argument.
if(body != null) { final QNm pl = body.qname(); // single part request if(pl.eq(Q_BODY)) { parseBody(body, bodies, hr.payloadAtts, hr.payload); hr.isMultipart = false; // multipart request } else if(pl.eq(Q_MULTIPART)) { parseMultipart(body, bodies.iter(), hr.payloadAtts, hr.parts); hr.isMultipart = true; } else { throw HC_REQ_X.get(info, "Unknown payload element: " +
body.qname());
}}
Best Regards Bogdan Bogucki
-----Wiadomość oryginalna----- Od: Christian Grün [mailto:christian.gruen@gmail.com] Wysłano: 12 lipca 2018 15:36 Do: Bogdan Bogucki DW: BaseX Temat: Re: [basex-talk] http:send-request - problem with form data
Hi Bogdan,
I finally had some time to get to the bottom of the conversion
inconsistencies:
If the value is supplied as child of the http:body node, it will be
serialized as text node. If it’s supplied as extra parameter, it will be serialized as string. If text nodes are serialized, entities will be encoded. Strings will be returned without encoded entities:
<a><![CDATA[&]]></a> → <a>&</a> ``[&]`` → &
I agree this is not very intuitive, and I have now rewritten this as follows: If the supplied output media-type does not result in one of
the default serialization methods of the spec (xml, xhtml, html, json, text, adaptive), the text node will be atomized (i.e., converted to an item of type xs:untypedAtomic), which will then be serialized as plain string. With the new snapshot [1], it should make no difference if you supply your value as child of http:body or as extra argument.
As a general guide, I would recommended everyone to use the third
argument for body items. It will be both safer and faster (in particular if binary data can be streamed). As the rich syntax of http:send-request is regularly confusing users on the list, I have reopened an old Github issue, which is about providing additional and simpler functions for the most widely used HTTP methods [2].
Best, Christian
[1] http://files.basex.org/releases/latest/ [2] https://github.com/BaseXdb/basex/issues/914
On Wed, Jul 4, 2018 at 10:05 PM Christian Grün <
christian.gruen@gmail.com> wrote:
And thanks for the confirmation. I will try to find out why the two
alternatives make a difference.
Bogdan Bogucki bbogucki@hussar.pl schrieb am Mi., 4. Juli 2018,
21:56:
Hello,
It works with body as a third parameter. Body has to be defined in
http request because without that third parameter is ignored.
Working example:
for $x in http:send-request( <http:request method='post'> <http:header name="User-Agent" value="Mozilla/5.0 (Windows NT
6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.99 Safari/537.36"/>
<http:body media-type='application/x-www-form-urlencoded;charset=UTF-8'/>
</http:request> ,'http:localhost:3333', 'start=0&length=100& ')[2]/data/results/content return $x
Thank you Christian for help.
Best Regards Bogdan Bogucki
-----Wiadomość oryginalna----- Od: Christian Grün [mailto:christian.gruen@gmail.com] Wysłano: 4 lipca 2018 18:31 Do: bbogucki@hussar.pl DW: BaseX Temat: Re: [basex-talk] http:send-request - problem with form data
Hi Bogdan,
I didn’t try it, but what happens if you specify the body as third
argument?
Cheers, Christian
On Tue, Jul 3, 2018 at 10:14 PM Bogdan Bogucki bbogucki@hussar.pl
wrote:
Hello,
I don't want pass values in query string. Server side expect
parameters in payload.
HTML Form parameters in POST method are passed in payload not in
URI.
This is the reason why I put parameters in body. Problem is escaping & character with & How can I force BaseX to not
escape characters in CDATA element ?
This is my code: <http:request method='post' href='http://localhost:3333%27%3E <http:header name="User-Agent" value="Mozilla/5.0 (Windows NT
6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.99 Safari/537.36"/>
<http:body media-type='application/x-www-form-urlencoded;charset=UTF-8'>
<![CDATA[start=0&length=100&sort=date_type&page=0&size=100]]> </http:body></http:request>
On server side I received following request:
POST / HTTP/1.1 User-Agent: Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.99 Safari/537.36 Content-Type: application/x-www-form-urlencoded; charset=UTF-8
Host: copart.com:3333
Accept: text/html, image/gif, image/jpeg, *; q=.2, */*; q=.2
Connection: keep-alive Content-Length: 75
start=0&length=100&sort=date_type&page=0&size=100
I expect that CDATA will be not escaped, or I am wrong ?
Pozdrawiam Bogdan Bogucki
HUSSAR SYSTEMS
NIP: 532-170-12-00 Regon: 142624070
e-mail: bbogucki@hussar.pl tel. kom.: +48 607 409 301 www: www.hussar.pl
-----Wiadomość oryginalna----- Od: Christian Grün [mailto:christian.gruen@gmail.com] Wysłano: 3 lipca 2018 12:11 Do: bbogucki@hussar.pl DW: BaseX Temat: Re: [basex-talk] http:send-request - problem with form data
Hi Bogdan,
If you want your arguments parsed as HTTP query parameters, it
should suffice to attach them to the URL:
let $url := 'http://localhost:3333' let $params :=
``[start=0&length=100&sort=date_type&page=0&size=100]``
return http:send-request( <http:request method='post' href='{ $url || '?' || $params }'>
...
If this is not what you need, you might need to give us more
information on how the request will be handled server-side (for example, do you use RESTXQ?).
Best, Christian
On Mon, Jul 2, 2018 at 11:11 PM Bogdan Bogucki bbogucki@hussar.pl
wrote:
> > Hello, > > > > I have problem with http:send-request. I want to send post
method request with form data but I encountered problem with characters encoding.
> > Example: > > > > Request: > > for $x in http:send-request( > > <http:request method='post' href= > > 'http://localhost:3333%27%3E > > <http:header name="User-Agent" value="Mozilla/5.0 (Windows NT > 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) > Chrome/67.0.3396.99 Safari/537.36"/> > > <http:body media-type='application/x-www-form-urlencoded; > charset=UTF-8'> > > <![CDATA[start=0&length=100&sort=date_type&page=0&size=100]]> > > </http:body> > > </http:request> > > )[2]/json/data/results/content > > return $x > > > > Optimized Query: > > util:item-at(http:send-request(element Q{
http://expath.org/ns/http-client%7Drequest { (attribute method { ("post") }, attribute href { ("http://localhost:3333") }, element Q{ http://expath.org/ns/http-client%7Dheader { (attribute name { ("User-Agent") }, attribute value { ("Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.99 Safari/537.36") }) }, element Q{http://expath.org/ns/http-client%7Dbody { (attribute media-type { ("application/x-www-form-urlencoded; charset=UTF-8") }, "
 start=0&length=100&sort=date_type&page=0&size=100
 ") }) }), 2)/json/data/results/content
> > > > Request send: > > POST / HTTP/1.1 > > User-Agent: Mozilla/5.0 (Windows NT 6.1; Win64; x64) > AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.99 > Safari/537.36 > > Content-Type: application/x-www-form-urlencoded; charset=UTF-8 > > Host: copart.com:3333 > > Accept: text/html, image/gif, image/jpeg, *; q=.2, */*; q=.2 > > Connection: keep-alive > > Content-Length: 75 > > > > > > > start=0&length=100&sort=date_type&page=0&size=100 > > > > Payload is not correctly encoded it should be: > > start=0&length=100&sort=date_type&page=0&size=100 > > > > How can I force function http:send-request to serialize CDATA
without escaping ?
> > > > > > Best Regards > > Bogdan Bogucki > >
-- Sent from my iPhone
basex-talk@mailman.uni-konstanz.de
-
Bogdan Bogucki -
Christian Grün -
Joe Wicentowski