J. Chang J. Chang - 2 months ago 9
Vb.net Question

Why can't I get this webclient to work for this URL?

Dim wc = New System.Net.WebClient

Dim apistring = wc.DownloadString("https://www.coinexchange.io/api/v1/getmarketsummaries")


The URL open just fine in browser. Somehow webclient can't get that. Hmmm....

What's the problem?

Update: I used a modified webclient with useragent and cookies and it works. I think it checks for things like user agent but I do not know.

I still do not know what the problem is and still curious. If anyone want to examine and check feel free.

Basically what exactly this site look for, and what software we can use to easily check what the problem is.

Answer Source

Some websites will not respond to a plain HTTP Request that contain only the Host header. They require additional common headers that typically would be set when being originated in a web browser.

Most commonly when a WebClient request fails the server is looking for the User-Agent or the Accept header. The server may rely on these headers to determine how to output the Response to client. A typical example is when an API looks at the Accept for text/html, application/xml or text/javascript or tapplication/json to determine if it should return HTML, XML, Javascript or JSON.

Depending on the site it might also look for Referer, 'Cookie', Accept-Language and/or Accept-Encoding headers.

Trying a combination of those values based on what your browser produces.

For this particular website, the browser sends:

enter image description here

The header this site is looking for is the User-Agent header. If it is not present it closes the connection and returns no response.