Here's the fiddle for it: https://jsfiddle.net/rujbra6y/3/
I'm already logging the speed so make any changes and just re-run it a couple times to see if the performance increased at all.
Been working on it for a couple hours and I can't think of anything else I can change to make it faster. I'd like it to be as fast as possible because currently there is a small delay when the user uses
There are a couple of things you could do at a brief glance:
Uint32Array. This will save you from unnecessary shifting and ANDing ops
The only thing you need to be aware of for Uint32Array is the byte-order is little-endian, which mean you need to provide the target color in 0xAABBGGRR format (or do an initial bit-shift giving r,g,b as separate values).
With these changes (except the last) the code went down from about 69-75ms to 58-61ms on my computer (i5 at the moment).
I'll leave the typed array for stack as an exercise. :-)