Thanks, Ash! That attribute doesn't seem to be available in BaseX, as far as I can tell.

--
Tim A. Thompson
Metadata Librarian
Yale University Library


On Wed, Feb 2, 2022 at 2:21 PM Clark, Ash <as.clark@northeastern.edu> wrote:
In case it helps: I had a similar problem in eXist some years ago. My fix was to put the attribute `http-version="1.0"` on the request element. The HTTP client defaults to HTTP v1.1, but for whatever reason, only HTTP 1.0 worked for me.

~Ash
From: BaseX-Talk <basex-talk-bounces@mailman.uni-konstanz.de> on behalf of Tim Thompson <timathom@gmail.com>
Sent: Wednesday, February 2, 2022 1:59 PM
To: Christian Grün <christian.gruen@gmail.com>; BaseX <basex-talk@mailman.uni-konstanz.de>
Subject: Re: [basex-talk] Issue with HTTP client and authentication
 
Ah! Thanks for clarifying that. So, now it is working for GET requests in BaseX, but still not for POST.

So, this works:
http:send-request(
  <http:request
      method="GET"        
      href="{$endpoint}?query={encode-for-uri('select * where {?s ?p ?o} limit 1')}"
      username="{$user}"
      password="{$pass}"
      auth-method="Digest">          
  </http:request>
)

But when I try to POST a request body with Digest auth, it is still failing (httpbin doesn't seem to support testing Digest with POST, unfortunately). The request hangs for a while, then fails with 401 Unauthorized. Wireshark reveals that the first network response from the server is actually a 502 Bad Gateway.

Attached are two screenshots from Wireshark: first, the request/response sequence with curl, then with BaseX.

Here's what Wireshark shows for the first request via curl:

Hypertext Transfer Protocol
    POST /v1/graphs/sparql HTTP/1.1\r\n
        [Expert Info (Chat/Sequence): POST /v1/graphs/sparql HTTP/1.1\r\n]
            [POST /v1/graphs/sparql HTTP/1.1\r\n]
            [Severity level: Chat]
            [Group: Sequence]
        Request Method: POST
        Request URI: /v1/graphs/sparql
        Request Version: HTTP/1.1
    Host: example.org\r\n
    User-Agent: curl/7.71.1\r\n
    Accept: */*\r\n
    Content-type: application/sparql-query\r\n
    Content-Length: 0\r\n
        [Content length: 0]
    \r\n
    [Full request URI: example.org/v1/graphs/sparql]

Then via BaseX:

Hypertext Transfer Protocol
    POST /v1/graphs/sparql HTTP/1.1\r\n
        [Expert Info (Chat/Sequence): POST /v1/graphs/sparql HTTP/1.1\r\n]
            [POST /v1/graphs/sparql HTTP/1.1\r\n]
            [Severity level: Chat]
            [Group: Sequence]
        Request Method: POST
        Request URI: /v1/graphs/sparql
        Request Version: HTTP/1.1
    Content-type: application/sparql-query\r\n
    Authorization: Digest\r\n
    User-Agent: Java/1.8.0_121\r\n
    Host: example.org\r\n
    Accept: text/html, image/gif, image/jpeg, *; q=.2, */*; q=.2\r\n
    Connection: keep-alive\r\n
    \r\n
    [Full request URI: example.org/v1/graphs/sparql]

I notice that curl doesn't include the Authorization header on the first request, but I am out of my depth here.

Best,
Tim


--
Tim A. Thompson
Metadata Librarian
Yale University Library


On Wed, Feb 2, 2022 at 12:40 PM Christian Grün <christian.gruen@gmail.com> wrote:
Hi Tim,

Your request should be successful if you omit the send-authorization attribute:

“If send-authorization is true (default value is false) and the
authentication method supports generating the header Authorization
without challenge, the request contains this header. The default value
is to send a non-authenticated request, and if the response is an
authentication challenge, then only send the credentials in a second
message.” [1]

If you use Basic authentication, you can reduce the communication to a
single request by sending the authorization with the first request.
With Digest authentication, there will always be two requests.

With the (still to be finalized) HTTP Client module 2.0,
send-authorization will disappear, and it will be up to the
implementation to decide how many requests will be sent [2].

Hope this helps,
Christian

[1] http://expath.org/spec/http-client#d2e430
[2] https://expath.github.io/expath-cg/specs/http-client-2/



On Wed, Feb 2, 2022 at 6:19 PM Tim Thompson <timathom@gmail.com> wrote:
>
> Thanks, Andy! That's very handy. So, when I test against httpbin, the same thing happens.
>
> curl request:
> curl --digest --user user:pw -X GET 'http://httpbin.org/digest-auth/auth/user/pw'
>
> response:
> {
>   "authenticated": true,
>   "user": "user"
> }
>
> BaseX request:
> let $endpoint := "http://httpbin.org/digest-auth/auth/user/pw"
>
> let $response := (
>   http:send-request(
>     <http:request
>         method="GET"
>         href="{$endpoint}"
>         username="user"
>         password="pw"
>         auth-method="Digest"
>         send-authorization="true">
>         <http:header
>             name="Content-Type"
>             value="{$type}; charset=utf-8"/>
>     </http:request>
>   )
> )
> return
>   $response
>
> response:
> <http:response xmlns:http="http://expath.org/ns/http-client" status="401" message="UNAUTHORIZED">
>   <http:header name="Server" value="gunicorn/19.9.0"/>
>   <http:header name="Access-Control-Allow-Origin" value="*"/>
>   <http:header name="Access-Control-Allow-Credentials" value="true"/>
>   <http:header name="WWW-Authenticate" value="Digest realm=&quot;me@kennethreitz.com&quot;, nonce=&quot;36d59385e96e1689595f4f14fa2921ca&quot;, qop=&quot;auth&quot;, opaque=&quot;a21df3287b5cb9cbf8b6a68b2c15fa1d&quot;, algorithm=MD5, stale=FALSE"/>
>   <http:header name="Connection" value="keep-alive"/>
>   <http:header name="Set-Cookie" value="fake=fake_value; Path=/"/>
>   <http:header name="Set-Cookie" value="stale_after=never; Path=/"/>
>   <http:header name="Content-Length" value="0"/>
>   <http:header name="Date" value="Wed, 02 Feb 2022 17:14:13 GMT"/>
>   <http:header name="Content-Type" value="text/html; charset=utf-8"/>
> </http:response>
>
>
> --
> Tim A. Thompson
> Metadata Librarian
> Yale University Library
>
>
> On Wed, Feb 2, 2022 at 12:06 PM Andy Bunce <bunce.andy@gmail.com> wrote:
>>
>> I find  httpbin.org a useful resource to test this kind of thing [1] and [2]
>> /Andy
>>
>> [1] http://httpbin.org/#/Auth/get_basic_auth__user___passwd_
>> [2] http://httpbin.org/basic-auth/user1/mypass
>>
>> On Wed, 2 Feb 2022 at 16:58, Christian Grün <christian.gruen@gmail.com> wrote:
>>>
>>> Good to hear; so at least Digest is detected at this stage ;)
>>>
>>>
>>> Tim Thompson <timathom@gmail.com> schrieb am Mi., 2. Feb. 2022, 17:54:
>>>>
>>>> With "digest" I get: "[experr:HC0004] Invalid authentication method: digest."
>>>>
>>>> --
>>>> Tim A. Thompson
>>>> Metadata Librarian
>>>> Yale University Library
>>>>
>>>> On Wed, Feb 2, 2022 at 11:52 AM Christian Grün <christian.gruen@gmail.com> wrote:
>>>>>
>>>>> Yes, it should still be supported. I assume it doesn't make a difference if you use "Digest" or "digest"?
>>>>>
>>>>>
>>>>>
>>>>> Tim Thompson <timathom@gmail.com> schrieb am Mi., 2. Feb. 2022, 17:48:
>>>>>>
>>>>>> Thanks, Christian. Right, that wasn't a working example. The server I'm querying is behind a firewall, so I can't really reproduce the issue unfortunately.
>>>>>>
>>>>>> The BaseX documentation states that the HTTP client does support Digest authentication. However, I did some poking around using Wireshark and limited knowledge of network traffic. When I run the request in curl, the expected protocol is played out, and I see the evidence in Wireshark:
>>>>>>
>>>>>> (1) The server responds with "401 Unauthorized" and provides the nonce value.
>>>>>> (2) The client (curl) then does its business (supplies the username and password, resends the request, etc.)
>>>>>>
>>>>>> But when I submit the request via BaseX, it never gets past step (1). It never seems to send the username and password at all. Is Digest authentication still supported?
>>>>>>
>>>>>> Best,
>>>>>> Tim
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Tim A. Thompson
>>>>>> Metadata Librarian
>>>>>> Yale University Library
>>>>>>
>>>>>>
>>>>>> On Wed, Feb 2, 2022 at 10:13 AM Christian Grün <christian.gruen@gmail.com> wrote:
>>>>>>>
>>>>>>> Hi Tim,
>>>>>>>
>>>>>>> Difficult to tell; both the curl and the XQuery variants give me 404.
>>>>>>> Do you think you could provide us with an example that works out of
>>>>>>> the box?
>>>>>>>
>>>>>>> Best,
>>>>>>> Christian
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Wed, Feb 2, 2022 at 12:54 AM Tim Thompson <timathom@gmail.com> wrote:
>>>>>>> >
>>>>>>> > Hello,
>>>>>>> >
>>>>>>> > I'm trying to post a SPARQL query to an endpoint using Digest authentication with the HTTP client.
>>>>>>> >
>>>>>>> > The query works fine using curl:
>>>>>>> >
>>>>>>> > curl --digest --user user:pass -X POST -d@'test.rq' \
>>>>>>> >     -H "Content-type: application/sparql-query" \
>>>>>>> >     'http://example.org/sparql'
>>>>>>> >
>>>>>>> > But the equivalent request in BaseX fails with 401 Unauthorized:
>>>>>>> >
>>>>>>> > let $endpoint := "http://example.org/sparql"
>>>>>>> > let $user := "user"
>>>>>>> > let $pass := "pass"
>>>>>>> > let $type := "application/sparql-query"
>>>>>>> >
>>>>>>> > let $response := (
>>>>>>> >   http:send-request(
>>>>>>> >     <http:request
>>>>>>> >         method="POST"
>>>>>>> >         href="{$endpoint}"
>>>>>>> >         username="{$user}"
>>>>>>> >         password="{$pass}"
>>>>>>> >         auth-method="Digest"
>>>>>>> >         send-authorization="true">
>>>>>>> >         <http:header
>>>>>>> >             name="Content-Type"
>>>>>>> >             value="{$type}; charset=utf-8"/>
>>>>>>> >         <http:body
>>>>>>> >             media-type="{$type}">{
>>>>>>> >             ``[
>>>>>>> >               select * where {?s ?p ?o} limit 1
>>>>>>> >             ]``
>>>>>>> >           }</http:body>
>>>>>>> >     </http:request>
>>>>>>> >   )
>>>>>>> > )
>>>>>>> > return
>>>>>>> >   $response
>>>>>>> >
>>>>>>> > Any ideas about what might be causing the BaseX HTTP client to be denied here?
>>>>>>> >
>>>>>>> > Thanks in advance,
>>>>>>> > Tim
>>>>>>> >
>>>>>>> >
>>>>>>> > --
>>>>>>> > Tim A. Thompson
>>>>>>> > Metadata Librarian
>>>>>>> > Yale University Library
>>>>>>> >