KiwiJuicer KiwiJuicer - 2 months ago 24
HTTP Question

Golang - get request max data limit

I'm scraping html pages and have set up a http client like so:

client := *http.Client{
Transport: &http.Transport{
Dial: (&net.Dialer{
Timeout: 30 * time.Second,
KeepAlive: 30 * time.Second,
TLSHandshakeTimeout: 10 * time.Second,
ResponseHeaderTimeout: 10 * time.Second,

Now when I make get requests of multiple urls I don't want to get stuck with urls that deliver massive amount of data.

response, err := client.Get(page.Url)
body, err := ioutil.ReadAll(response.Body)
page.Body = string(body)

Is there a way to limit the amount of data (bytes) the get accepts from an url and stops?


Use an io.LimitedReader

A LimitedReader reads from R but limits the amount of data returned to just N bytes.

limitedReader := &io.LimitedReader{R: response.Body, N: limit}
body, err := ioutil.ReadAll(limitedReader)


body, err := ioutil.ReadAll(io.LimitReader(response.Body, limit))