huseyin tugrul buyukisik huseyin tugrul buyukisik - 2 months ago 14
Node.js Question

Using Node-Js as message passing layer


(client) node-js http request <-----------> node-js http listen (server)
^ 100/1000 MBit/s ^
| LAN |
v v
a file in ramdrive a file in ramdrive
^ ^
| |
v v
C# program requesting C# or Java program
compute on data computing

easier than

(client)C# tcp ip <--------------------------------->C#/Java tcp ip (server)
socket 100/1000 Mbit/s LAN socket

in terms of error handling(such as half-open connections, acknowledgement packet management and picking right window/buffer size) and scalability (for 10-100 servers using tree-like connections)?

If node-js layer is to be used just for I/O, would it be better than MPI for a cluster on basic switches/routers(not infiniband for example)?

For now, I'm trying tcp-ip option and already have too many exceptions to handle and far from finished. There are max connections because of desktop os, the serialization of arrays also need checking against endianness of server and client and many more to counterpoise before sending/computing the data. Also everywhere it says async is best. I just started with a 1 thread for 1 connection version. Threaded version also has persistent local data to hold compute device information and compiled kernels(for opencl).

If I understand correctly, both mpi and node-js need their runtime be installed on servers and clients but tcp-ip doesn't need such thing. But this wouldn't balance other difficulties would it?

Adding ramdrive between node-js and C# came into my mind because it looks easy to access a file from node-js(and C#) with much less things to worry about than a tcp connection. Http listening is also nearly one-liner in node-js when it is just passing data.


Without knowing the specifics of what you're doing, it's hard to recommend a specific solution... but I suspect you're re-inventing a solution.

Check out Redis. There are client libraries for most any language, it works over a network, handles all kinds of data types, and even supports pub/sub. It's built to be ridiculously performant... the developers of Redis made a lot of the right trade-offs for many applications.