Hi Bogdan,

The current http:send-request implementation is based on the default Java HttpURLConnection, which does not resolve redirects that use different protocols [1]. This is the reason why your request will not be fully processed (as it e.g. happens when you use the browser).

It seems that your initial request to the https protocol returns a 302 redirect to a (now unsafe) http URL, which returns another redirect to https. I don’t know who maintains the discussed web site, but it could be worth contacting the admins and asking them if they could update and simplify their redirect policy.

Hope this helps, Christian

[1] https://stackoverflow.com/a/1884427/1018790

On Fri, Oct 25, 2019 at 1:52 PM Bogdan Bogucki bbogucki@hussar.pl wrote:

Hello,

I encounter problem with http:send-request function. I need handle multiple forward requests (3) with cookies.

First request returns cookies information which are required with next requests. Output from http:send-request dosen't contains all fields which are returned by server.

Response from browser is flowing: cache-control private cf-cache-status DYNAMIC cf-ray 52b3cedc5b9bcc9f-WAW content-type text/html; charset=utf-8 date Fri, 25 Oct 2019 11:21:37 GMT expect-ct max-age=604800, report-uri="ht….com/cdn-cgi/beacon/expect-ct" location http://www.pracuj.pl/praca/jav%E2%80%A6eloper-warszawa,oferta,7171988 server cloudflare set-cookie __cfduid=daf54110b2a87d66c2a53…; domain=.pracuj.pl; HttpOnly set-cookie _yaic=13; expires=Fri, 31-Dec-9999 23:59:59 GMT; path=/ set-cookie _urnadiam=A; domain=.pracuj.pl…3:00:00 GMT; path=/; HttpOnly x-aspnet-version 4.0.30319 x-aspnetmvc-version 5.2 X-Firefox-Spdy h2 x-powered-by ASP.NET x-ua-compatible IE=edge

Request:

http:send-request( <http:request method='get' follow-redirect='false'> <http:header name="User-Agent" value="Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:69.0) Gecko/20100101 Firefox/69.0"/> <http:header name="Accept" value="text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8"/> <http:header name="Accept-Languag" value="pl,en-US;q=0.7,en;q=0.3"/> <http:header name="DNT" value="1"/> <http:header name="Connection" value="keep-alive"/> <http:header name="Upgrade-Insecure-Requests" value="1"/> <http:body media-type='application/x-www-form-urlencoded; charset=UTF-8'/> </http:request>, 'https://www.pracuj.pl/praca/java-developer-warszawa,oferta,7171988')

Response form http:send-request is:

<http:response xmlns:http="http://expath.org/ns/http-client" status="302" message="Found"> <http:header name="Transfer-Encoding" value="chunked"/> <http:header name="CF-RAY" value="52b3d0de097fb839-WAW"/> <http:header name="Server" value="cloudflare"/> <http:header name="Connection" value="keep-alive"/> <http:header name="Date" value="Fri, 25 Oct 2019 11:23:00 GMT"/> <http:header name="X-AspNetMvc-Version" value="5.2"/> <http:header name="X-UA-Compatible" value="IE=edge"/> <http:header name="CF-Cache-Status" value="DYNAMIC"/> <http:header name="Cache-Control" value="private"/> <http:header name="X-AspNet-Version" value="4.0.30319"/> <http:header name="Set-Cookie" value="_urnadiam=B; domain=.pracuj.pl; expires=Tue, 31-Dec-2019 23:00:00 GMT; path=/; HttpOnly"/> <http:header name="Expect-CT" value="max-age=604800, report-uri=&quot;https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct%22%22/%3E <http:header name="X-Powered-By" value="ASP.NET"/> <http:header name="Content-Type" value="text/html; charset=utf-8"/> <http:body media-type="text/html"/> </http:response>

Two set-cookie fields is missing.

How should I invoke http:send-request to receive proper result ?

Regards

Bogdan