ceridwen ceridwen - 1 year ago 99
Linux Question

How to get ZeroMQ to .connect() to localhost using a particular source IP?

I have a Linux machine configured with additional IPs on the loopback interface:

# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet 100.100.100.100/24 scope global lo
valid_lft forever preferred_lft forever
inet 100.100.100.101/24 scope global secondary lo
valid_lft forever preferred_lft forever
...


I'm using ZeroMQ's
PUB/SUB
pattern to connect to remotes using each of these source ips.

socket = zmq.Context().socket(zmq.SUB)
socket.connect('tcp://100.100.100.100:5555;192.168.1.1:5555')


The corresponding server listens on all interfaces.

socket = zmq.Context().socket(zmq.PUB)
socket.bind('tcp://*:5555')


Sending using these addresses works for remote hosts given a properly-configured network, so I know this works. (It's complicated.) Now I want to unit-test this setup, which means checking that setting the source ip works without needing a remote host for the server. I run the server in the same configuration and then try to connect using something like:

socket = zmq.Context().socket(zmq.SUB)
socket.connect('tcp://100.100.100.100:5555;localhost:5555')


The client never makes the connection with the server, but if I remove the source endpoint, it works. Neither localhost, nor
127.0.0.1
work as the destination address in the
.connect()
call. However, if I call
netcat
, using the same source IP,

nc -s 100.100.100.100 -v -z -w 5 localhost 5555


This succeeds, and the server I connected to properly receives the connection as coming from 100.100.100.100. I looked at tshark's output and in the ZeroMQ client case, I don't seen any traffic from 100.100.100.100 over the loopback interface, while when I use
nc
to establish a TCP connection to the ZeroMQ server, I do.

What's going on here? Does ZeroMQ do something special for this kind of hairpin connection, and if so, is there a way to disable it? Is there a good way to test that I'm invoking the source IP functionality of ZeroMQ correctly without using a remote host?

This may be viewed as a follow-up to my previous question.

Answer Source

You can't send and receive on the same port on the same interface.
The best solution is to allow ZeroMQ to pick the port to send on:

socket = zmq.Context().socket(zmq.SUB) socket.connect('tcp://100.100.100.100:*;192.168.1.1:5555')

Unfortunately, the 4.2.2 release of ZeroMQ doesn't support this, though an upcoming release should. For now, the only solution is to hardcode a different port for the sending and receiving address:

socket = zmq.Context().socket(zmq.SUB) socket.connect('tcp://100.100.100.100:6666;192.168.1.1:5555')

Recommended from our users: Dynamic Network Monitoring from WhatsUp Gold from IPSwitch. Free Download