frankgreco frankgreco - 9 months ago 100
HTTP Question

Very Sporadic Go HTTP Error: multiple response.WriteHeader calls

I wrote Kanali which is an open source Kubernetes Ingress/API management tool and for about 1/200k requests I receive the following fatal error:

2017/08/16 12:40:57 http: multiple response.WriteHeader calls
{"level":"error","method":"GET","msg":"unknown error","time":"2017-08-16T12:40:57Z","uri":"/ommitted/path"}
{"level":"fatal","msg":"write tcp4 192.168.2.160:8443-\u003e192.168.4.0:54554: write: broken pipe","time":"2017-08-16T12:40:57Z"}


I'm having a really hard time reproducing it but here is the relevant code. Kanali is a large project but the td;lr is that after this first code snippet is executed, the second code snipped is executed which handles errors.

func (step WriteResponseStep) Do(ctx context.Context, m *metrics.Metrics, c *controller.Controller, w http.ResponseWriter, r *http.Request, resp *http.Response, trace opentracing.Span) error {
for k, v := range resp.Header {
for _, value := range v {
w.Header().Set(k, value)
}
}
closer, str, err := utils.DupReaderAndString(resp.Body)
if err != nil {
logrus.Errorf("error copying response body, response may not be as expected: %s", err.Error())
}
trace.SetTag("http.status_code", resp.StatusCode)
trace.SetTag("http.response_body", str)
w.WriteHeader(resp.StatusCode)
if _, err := io.Copy(w, closer); err != nil {
return err
}
return nil
}


later in the code...

if err != nil {

w.Header().Set("Content-Type", "application/json")

switch e := err.(type) {
case utils.Error:
logrus.WithFields(logrus.Fields{
"method": r.Method,
"uri": r.RequestURI,
}).Error(e.Error())

errStatus, err := json.Marshal(utils.JSONErr{Code: e.Status(), Msg: e.Error()})
if err != nil {
logrus.Warnf("could not marsah request headers into JSON - tracing data maybe not be as expected")
}
w.WriteHeader(e.Status())
if err := json.NewEncoder(w).Encode(utils.JSONErr{Code: e.Status(), Msg: e.Error()}); err != nil {
logrus.Fatal(err.Error())
}
default:
logrus.WithFields(logrus.Fields{
"method": r.Method,
"uri": r.RequestURI,
}).Error("unknown error")
errStatus, err := json.Marshal(utils.JSONErr{Code: http.StatusInternalServerError, Msg: "unknown error"})
if err != nil {
logrus.Warnf("could not marsah request headers into JSON - tracing data maybe not be as expected")
}
w.WriteHeader(http.StatusInternalServerError)
if err := json.NewEncoder(w).Encode(utils.JSONErr{Code: http.StatusInternalServerError, Msg: "unknown error"}); err != nil {
logrus.Fatal(err.Error())
}
}
}

Answer Source

The error message is telling you that WriteHeader is called more than once (either directly or indirectly by a call to Write). The header can only be written to the network once. Both snippets have a call to WriteHeader.

Recommended from our users: Dynamic Network Monitoring from WhatsUp Gold from IPSwitch. Free Download