bawjensen bawjensen - 5 months ago 14
JSON Question

Go - HTTP Requests, JSON, reusing connections

I'm using Go to make many requests over HTTPS, and I'm having issues with not reusing connections, and running out of ports. The requests I'm making are to an API that returns data in the JSON format, which I then

json.Decode
into a Go value.

According to questions I've come across on this site (#1, #2), in order for Go to reuse the connection for another request, I must read the entirety of the response's body before closing (note this wasn't always the behavior, as stated here).

Previously the HTTP client's (*Response).Body.Close would try to keep
reading until EOF, hoping to reuse the keep-alive HTTP connection...


In a typical instance, I would use the example shown in the earlier links, like so:

ioutil.ReadAll(resp.Body)


but since I'm pulling the data out of the JSON through code like this:

...

req, _ := http.NewRequest("GET", urlString, nil)
req.Header.Add("Connection", "keep-alive")
resp, err = client.Do(req)
defer resp.Body.Close()

...

decoder := json.NewDecoder(resp.Body)
decoder.Decode(data)


I'm not sure how the two methods would interact.

So the question is, how do I make sure the entire response has been read, so that the connection can later be reused for another request?

Answer

If you want to only decode a single object using the Decoder, then you can use the More() method to see if there is more in the stream that needs to be read.

decoder := json.NewDecoder(resp.Body)
err := decoder.Decode(data)
if err != nil {
    // handle err
}
if decoder.More() {
    // there's more data in the stream, so discard whatever is left
    io.Copy(ioutil.Discard, resp.Body)
}

You can also just defer the Copy on every call, but this way you could more easily handle or log if there was unexpected data or an error.

Comments