In case it helps: I had a similar problem in eXist some years ago. My fix was to put the attribute `http-version="1.0"` on the request element. The HTTP client defaults to HTTP v1.1, but for whatever reason, only HTTP 1.0 worked for me.
~Ash ________________________________ From: BaseX-Talk basex-talk-bounces@mailman.uni-konstanz.de on behalf of Tim Thompson timathom@gmail.com Sent: Wednesday, February 2, 2022 1:59 PM To: Christian Grün christian.gruen@gmail.com; BaseX basex-talk@mailman.uni-konstanz.de Subject: Re: [basex-talk] Issue with HTTP client and authentication
Ah! Thanks for clarifying that. So, now it is working for GET requests in BaseX, but still not for POST.
So, this works: http:send-request( <http:request method="GET" href="{$endpoint}?query={encode-for-uri('select * where {?s ?p ?o} limit 1')}" username="{$user}" password="{$pass}" auth-method="Digest"> </http:request> )
But when I try to POST a request body with Digest auth, it is still failing (httpbin doesn't seem to support testing Digest with POST, unfortunately). The request hangs for a while, then fails with 401 Unauthorized. Wireshark reveals that the first network response from the server is actually a 502 Bad Gateway.
Attached are two screenshots from Wireshark: first, the request/response sequence with curl, then with BaseX.
Here's what Wireshark shows for the first request via curl:
Hypertext Transfer Protocol POST /v1/graphs/sparql HTTP/1.1\r\n [Expert Info (Chat/Sequence): POST /v1/graphs/sparql HTTP/1.1\r\n] [POST /v1/graphs/sparql HTTP/1.1\r\n] [Severity level: Chat] [Group: Sequence] Request Method: POST Request URI: /v1/graphs/sparql Request Version: HTTP/1.1 Host: example.orghttps://nam12.safelinks.protection.outlook.com/?url=http%3A%2F%2Fexample.org%2F&data=04%7C01%7Cas.clark%40northeastern.edu%7Cdf41115443c24ad1a67a08d9e67e4565%7Ca8eec281aaa34daeac9b9a398b9215e7%7C0%7C0%7C637794253163659084%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=j3Wipf2PLgF3njCsb6BTN1J%2BwrOOfe1%2Fcm%2F%2BUiGTMQk%3D&reserved=0\r\n User-Agent: curl/7.71.1\r\n Accept: */*\r\n Content-type: application/sparql-query\r\n Content-Length: 0\r\n [Content length: 0] \r\n [Full request URI: example.org/v1/graphs/sparqlhttps://nam12.safelinks.protection.outlook.com/?url=http%3A%2F%2Fexample.org%2Fv1%2Fgraphs%2Fsparql&data=04%7C01%7Cas.clark%40northeastern.edu%7Cdf41115443c24ad1a67a08d9e67e4565%7Ca8eec281aaa34daeac9b9a398b9215e7%7C0%7C0%7C637794253163659084%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=PeCGztKkndWthFeKyUrYUljM2oq4ivkyP1ugAYL64QU%3D&reserved=0]
Then via BaseX:
Hypertext Transfer Protocol POST /v1/graphs/sparql HTTP/1.1\r\n [Expert Info (Chat/Sequence): POST /v1/graphs/sparql HTTP/1.1\r\n] [POST /v1/graphs/sparql HTTP/1.1\r\n] [Severity level: Chat] [Group: Sequence] Request Method: POST Request URI: /v1/graphs/sparql Request Version: HTTP/1.1 Content-type: application/sparql-query\r\n Authorization: Digest\r\n User-Agent: Java/1.8.0_121\r\n Host: example.orghttps://nam12.safelinks.protection.outlook.com/?url=http%3A%2F%2Fexample.org%2F&data=04%7C01%7Cas.clark%40northeastern.edu%7Cdf41115443c24ad1a67a08d9e67e4565%7Ca8eec281aaa34daeac9b9a398b9215e7%7C0%7C0%7C637794253163659084%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=j3Wipf2PLgF3njCsb6BTN1J%2BwrOOfe1%2Fcm%2F%2BUiGTMQk%3D&reserved=0\r\n Accept: text/html, image/gif, image/jpeg, *; q=.2, */*; q=.2\r\n Connection: keep-alive\r\n \r\n [Full request URI: example.org/v1/graphs/sparqlhttps://nam12.safelinks.protection.outlook.com/?url=http%3A%2F%2Fexample.org%2Fv1%2Fgraphs%2Fsparql&data=04%7C01%7Cas.clark%40northeastern.edu%7Cdf41115443c24ad1a67a08d9e67e4565%7Ca8eec281aaa34daeac9b9a398b9215e7%7C0%7C0%7C637794253163659084%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=PeCGztKkndWthFeKyUrYUljM2oq4ivkyP1ugAYL64QU%3D&reserved=0]
I notice that curl doesn't include the Authorization header on the first request, but I am out of my depth here.
Best, Tim
-- Tim A. Thompson Metadata Librarian Yale University Library
On Wed, Feb 2, 2022 at 12:40 PM Christian Grün <christian.gruen@gmail.commailto:christian.gruen@gmail.com> wrote: Hi Tim,
Your request should be successful if you omit the send-authorization attribute:
“If send-authorization is true (default value is false) and the authentication method supports generating the header Authorization without challenge, the request contains this header. The default value is to send a non-authenticated request, and if the response is an authentication challenge, then only send the credentials in a second message.” [1]
If you use Basic authentication, you can reduce the communication to a single request by sending the authorization with the first request. With Digest authentication, there will always be two requests.
With the (still to be finalized) HTTP Client module 2.0, send-authorization will disappear, and it will be up to the implementation to decide how many requests will be sent [2].
Hope this helps, Christian
[1] http://expath.org/spec/http-client#d2e430https://nam12.safelinks.protection.outlook.com/?url=http%3A%2F%2Fexpath.org%2Fspec%2Fhttp-client%23d2e430&data=04%7C01%7Cas.clark%40northeastern.edu%7Cdf41115443c24ad1a67a08d9e67e4565%7Ca8eec281aaa34daeac9b9a398b9215e7%7C0%7C0%7C637794253163659084%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=RREH6BeyWOV8juK%2Brf0UuofvBiC2JOv7B1%2BWXLYCjUQ%3D&reserved=0 [2] https://expath.github.io/expath-cg/specs/http-client-2/https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fexpath.github.io%2Fexpath-cg%2Fspecs%2Fhttp-client-2%2F&data=04%7C01%7Cas.clark%40northeastern.edu%7Cdf41115443c24ad1a67a08d9e67e4565%7Ca8eec281aaa34daeac9b9a398b9215e7%7C0%7C0%7C637794253163659084%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=owfYuuSXSpkkPTnQhBIEJ1wBcKPCs4oFnaaVEvndvMA%3D&reserved=0
On Wed, Feb 2, 2022 at 6:19 PM Tim Thompson <timathom@gmail.commailto:timathom@gmail.com> wrote:
Thanks, Andy! That's very handy. So, when I test against httpbin, the same thing happens.
curl request: curl --digest --user user:pw -X GET 'http://httpbin.org/digest-auth/auth/user/pwhttps://nam12.safelinks.protection.outlook.com/?url=http%3A%2F%2Fhttpbin.org%2Fdigest-auth%2Fauth%2Fuser%2Fpw&data=04%7C01%7Cas.clark%40northeastern.edu%7Cdf41115443c24ad1a67a08d9e67e4565%7Ca8eec281aaa34daeac9b9a398b9215e7%7C0%7C0%7C637794253163659084%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=SWqtnqsC98VlfYr7TeIUggFdoR7dJDZtkNJxz9051jw%3D&reserved=0'
response: { "authenticated": true, "user": "user" }
BaseX request: let $endpoint := "http://httpbin.org/digest-auth/auth/user/pwhttps://nam12.safelinks.protection.outlook.com/?url=http%3A%2F%2Fhttpbin.org%2Fdigest-auth%2Fauth%2Fuser%2Fpw&data=04%7C01%7Cas.clark%40northeastern.edu%7Cdf41115443c24ad1a67a08d9e67e4565%7Ca8eec281aaa34daeac9b9a398b9215e7%7C0%7C0%7C637794253163659084%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=SWqtnqsC98VlfYr7TeIUggFdoR7dJDZtkNJxz9051jw%3D&reserved=0"
let $response := ( http:send-request( <http:request method="GET" href="{$endpoint}" username="user" password="pw" auth-method="Digest" send-authorization="true"> <http:header name="Content-Type" value="{$type}; charset=utf-8"/> </http:request> ) ) return $response
response: <http:response xmlns:http="http://expath.org/ns/http-clienthttps://nam12.safelinks.protection.outlook.com/?url=http%3A%2F%2Fexpath.org%2Fns%2Fhttp-client&data=04%7C01%7Cas.clark%40northeastern.edu%7Cdf41115443c24ad1a67a08d9e67e4565%7Ca8eec281aaa34daeac9b9a398b9215e7%7C0%7C0%7C637794253163659084%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=%2FNS%2BIceZOx6yP%2Fp4ikxkNqAhYq44F49nUaE3u8iijr0%3D&reserved=0" status="401" message="UNAUTHORIZED"> <http:header name="Server" value="gunicorn/19.9.0"/> <http:header name="Access-Control-Allow-Origin" value="*"/> <http:header name="Access-Control-Allow-Credentials" value="true"/> <http:header name="WWW-Authenticate" value="Digest realm="me@kennethreitz.commailto:quot%3Bme@kennethreitz.com", nonce="36d59385e96e1689595f4f14fa2921ca", qop="auth", opaque="a21df3287b5cb9cbf8b6a68b2c15fa1d", algorithm=MD5, stale=FALSE"/> <http:header name="Connection" value="keep-alive"/> <http:header name="Set-Cookie" value="fake=fake_value; Path=/"/> <http:header name="Set-Cookie" value="stale_after=never; Path=/"/> <http:header name="Content-Length" value="0"/> <http:header name="Date" value="Wed, 02 Feb 2022 17:14:13 GMT"/> <http:header name="Content-Type" value="text/html; charset=utf-8"/> </http:response>
-- Tim A. Thompson Metadata Librarian Yale University Library
On Wed, Feb 2, 2022 at 12:06 PM Andy Bunce <bunce.andy@gmail.commailto:bunce.andy@gmail.com> wrote:
I find httpbin.orghttps://nam12.safelinks.protection.outlook.com/?url=http%3A%2F%2Fhttpbin.org%2F&data=04%7C01%7Cas.clark%40northeastern.edu%7Cdf41115443c24ad1a67a08d9e67e4565%7Ca8eec281aaa34daeac9b9a398b9215e7%7C0%7C0%7C637794253163659084%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=5ZA6A7rJL%2B2YD%2BPvtA9wtLUjJVf3EAZw3Fb5ifo4H%2BU%3D&reserved=0 a useful resource to test this kind of thing [1] and [2] /Andy
[1] http://httpbin.org/#/Auth/get_basic_auth__user___passwd_https://nam12.safelinks.protection.outlook.com/?url=http%3A%2F%2Fhttpbin.org%2F%23%2FAuth%2Fget_basic_auth__user___passwd_&data=04%7C01%7Cas.clark%40northeastern.edu%7Cdf41115443c24ad1a67a08d9e67e4565%7Ca8eec281aaa34daeac9b9a398b9215e7%7C0%7C0%7C637794253163659084%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=TU9Itw0Bs4Wbk51bnStj2oFqWMq%2BrdIOxgUwq5fFFrM%3D&reserved=0 [2] http://httpbin.org/basic-auth/user1/mypasshttps://nam12.safelinks.protection.outlook.com/?url=http%3A%2F%2Fhttpbin.org%2Fbasic-auth%2Fuser1%2Fmypass&data=04%7C01%7Cas.clark%40northeastern.edu%7Cdf41115443c24ad1a67a08d9e67e4565%7Ca8eec281aaa34daeac9b9a398b9215e7%7C0%7C0%7C637794253163659084%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=lH8HeRKkVu%2Fev4Er8lokTCZ0xqyS3HOYtBBqgTSw0PY%3D&reserved=0
On Wed, 2 Feb 2022 at 16:58, Christian Grün <christian.gruen@gmail.commailto:christian.gruen@gmail.com> wrote:
Good to hear; so at least Digest is detected at this stage ;)
Tim Thompson <timathom@gmail.commailto:timathom@gmail.com> schrieb am Mi., 2. Feb. 2022, 17:54:
With "digest" I get: "[experr:HC0004] Invalid authentication method: digest."
-- Tim A. Thompson Metadata Librarian Yale University Library
On Wed, Feb 2, 2022 at 11:52 AM Christian Grün <christian.gruen@gmail.commailto:christian.gruen@gmail.com> wrote:
Yes, it should still be supported. I assume it doesn't make a difference if you use "Digest" or "digest"?
Tim Thompson <timathom@gmail.commailto:timathom@gmail.com> schrieb am Mi., 2. Feb. 2022, 17:48:
Thanks, Christian. Right, that wasn't a working example. The server I'm querying is behind a firewall, so I can't really reproduce the issue unfortunately.
The BaseX documentation states that the HTTP client does support Digest authentication. However, I did some poking around using Wireshark and limited knowledge of network traffic. When I run the request in curl, the expected protocol is played out, and I see the evidence in Wireshark:
(1) The server responds with "401 Unauthorized" and provides the nonce value. (2) The client (curl) then does its business (supplies the username and password, resends the request, etc.)
But when I submit the request via BaseX, it never gets past step (1). It never seems to send the username and password at all. Is Digest authentication still supported?
Best, Tim
-- Tim A. Thompson Metadata Librarian Yale University Library
On Wed, Feb 2, 2022 at 10:13 AM Christian Grün <christian.gruen@gmail.commailto:christian.gruen@gmail.com> wrote: > > Hi Tim, > > Difficult to tell; both the curl and the XQuery variants give me 404. > Do you think you could provide us with an example that works out of > the box? > > Best, > Christian > > > > On Wed, Feb 2, 2022 at 12:54 AM Tim Thompson <timathom@gmail.commailto:timathom@gmail.com> wrote: > > > > Hello, > > > > I'm trying to post a SPARQL query to an endpoint using Digest authentication with the HTTP client. > > > > The query works fine using curl: > > > > curl --digest --user user:pass -X POST -d@'test.rq' \ > > -H "Content-type: application/sparql-query" \ > > 'http://example.org/sparqlhttps://nam12.safelinks.protection.outlook.com/?url=http%3A%2F%2Fexample.org%2Fsparql&data=04%7C01%7Cas.clark%40northeastern.edu%7Cdf41115443c24ad1a67a08d9e67e4565%7Ca8eec281aaa34daeac9b9a398b9215e7%7C0%7C0%7C637794253163659084%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=8uhboyxkhGWHk8HbZRKgDXOAbDRa2UwIigI1H5vWxt0%3D&reserved=0' > > > > But the equivalent request in BaseX fails with 401 Unauthorized: > > > > let $endpoint := "http://example.org/sparqlhttps://nam12.safelinks.protection.outlook.com/?url=http%3A%2F%2Fexample.org%2Fsparql&data=04%7C01%7Cas.clark%40northeastern.edu%7Cdf41115443c24ad1a67a08d9e67e4565%7Ca8eec281aaa34daeac9b9a398b9215e7%7C0%7C0%7C637794253163659084%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=8uhboyxkhGWHk8HbZRKgDXOAbDRa2UwIigI1H5vWxt0%3D&reserved=0" > > let $user := "user" > > let $pass := "pass" > > let $type := "application/sparql-query" > > > > let $response := ( > > http:send-request( > > http:request > method="POST" > > href="{$endpoint}" > > username="{$user}" > > password="{$pass}" > > auth-method="Digest" > > send-authorization="true"> > > http:header > name="Content-Type" > > value="{$type}; charset=utf-8"/> > > http:body > media-type="{$type}">{ > > ``[ > > select * where {?s ?p ?o} limit 1 > > ]`` > > }</http:body> > > </http:request> > > ) > > ) > > return > > $response > > > > Any ideas about what might be causing the BaseX HTTP client to be denied here? > > > > Thanks in advance, > > Tim > > > > > > -- > > Tim A. Thompson > > Metadata Librarian > > Yale University Library > >