ZachB ZachB - 4 months ago 28
Node.js Question

Creating a typed array from a byte array stored as a node Buffer

From the node docs regarding the creation of typed arrays from Buffers:


The buffer's memory is interpreted as an array, not a byte array. That
is,
new Uint32Array(new Buffer([1,2,3,4]))
creates a 4-element
Uint32Array
with elements
[1,2,3,4]
, not an
Uint32Array
with a single
element
[0x1020304]
or
[0x4030201]
.


This contrasts to plain javascript, where creating a typed array view from an ArrayBuffer uses the ArrayBuffer's memory as bytes. I need this behavior in node when operating on node Buffers.

I could convert the Buffer to an ArrayBuffer, but this is too slow for my application. (I've tried many methods -- but they're all O(n) time.) (Edit: the fastest method I've found is this, which is a single memmove op and pretty fast, but still has at least momentary 2x memory consumption until the reference to the original Buffer is released.)

Is there any (fast/O(1)) way to get a typed array from a Buffer, using the Buffer's contents as bytes instead of elements? (The needed typed array element size is >1 byte, needless to say.)

Answer

As far as I know, it is not possible to do this without making a copy of the data in memory. Even your example new Uint32Array(new Buffer([1,2,3,4])) internally does this (meaning it's not O(1)).

Note that typed arrays are just views of an ArrayBuffer (not Buffer, that's not possible). new Uint32Array(array) creates an ArrayBuffer of 4 * array.length bytes. You can access it with uint32Array.buffer. The constructor treats your Buffer no different than a normal Array.

The best solution I know is the one you already found.

Another issue with using Uint32Array for what you try to do is that it depends on platform byte order. You could either iterate over the Buffer like this or use DataView if you want to be safe.

Comments