I want to know how many channels can a selector can handle at maximum? I'm familiar with common I/O algorithms, but I don't know in
NIO
There is no limit other than the number of socket descriptors. Some platforms have underlying limits, but NIO works around them with multiple OS selectors per Selector.
There was a limit up to about Java 1.4.1 but it's long gone.
Should use a number of threads and have a selector in each for handling a specific number of sockets?
It's possible, but I don't really see why you should. Maybe the peers might get more regular service that way, it depends what your code has to do with every request.