Gershom Maes Gershom Maes - 1 month ago 6
Node.js Question

Vanilla node.js response encoding - weird behaviour (heroku)

I'm developing my understanding of servers by writing a webdev framework using vanilla node.js. For the first time a situation arose where a french character was included in a

json
response from the server, and this character showed up as an unrecognized symbol (a question-mark within a diamond on chrome).

The problem was the encoding, which was being specified here:

/*
At this stage we have access to "response", which is an object of the
following format:

response = {
data: 'A string which contains the data to send',
encoding: 'The encoding type of the data, e.g. "text/html", "text/json"'
}
*/

var encoding = response.encoding;
var length = response.data.length;
var data = response.data;

res.writeHead(200, {
'Content-Type': encoding,
'Content-Length': length
});
res.end(data, 'binary'); // Everything is encoded as binary


The problem was that everything sent by the server is encoded as binary, which ruins the ability to display certain characters. The fix seemed simple; include a boolean
binary
value in
response
, and adjust the 2nd parameter of
res.end
accordingly:

/*
At this stage we have access to "response", which is an object of the
following format:

response = {
data: 'A string which contains the data to send',
encoding: 'The encoding type of the data, e.g. "text/html", "text/json"',
binary: 'a boolean value determining the transfer encoding'
}
*/

.
.
.

var binary = response.binary;

.
.
.

res.end(data, binary ? 'binary' : 'utf8'); // Encode responses appropriately


Here is where I have produced some very, very strange behavior. This modification causes french characters to appear correctly, but occasionally causes the last character of a response to be omitted on the client-side!!!

This bug only happens once I host my application on heroku. Locally, the last character is never missing.

I noticed this bug because certain responses (not all of them!) now break the
JSON.parse
call on the client-side, although they are only missing the final
}
character.

I have a horrible band-aid solution right now, which works:

var length = response.data.length + 1;
var data = response.data + ' ';


I am simply appending a space to every single response sent by the server. This actually causes all
text/html
,
text/css
,
text/json
, and
application/javascript
responses to work because they can tolerate the unnecessary whitespace, but I hate this solution and it will break other
Content-Type
s!

My question is: can anyone give me some insight into this problem?

Answer

If you're going to explicitly set a Content-Length, you should always use Buffer.byteLength() on the body to calculate the length, since that method returns the actual number of bytes in the string and not the number of characters like the string .length property will return.

Comments