I would like to use Ansible to execute a simple job on several remote nodes concurrently. The actual job involves grepping some log files and then post-processing the results on my local host (which has software not available on the remote nodes).
The command line ansible tools don't seem well-suited to this use case because they mix together ansible-generated formatting with the output of the remotely executed command. The Python API seems like it should be capable of this though, since it exposes the output unmodified (apart from some potential unicode mangling that shouldn't be relevant here).
A simplified version of the Python program I've come up with looks like this:
from sys import argv
runner = ansible.runner.Runner(
results = runner.run()
exarkun@top:/tmp$ time python howlong.py two-hosts.inventory
exarkun@top:/tmp$ time python howlong.py five-hosts.inventory
exarkun@top:/tmp$ time python howlong.py nine-hosts.inventory
ansible all --forks=10 -i five-hosts.inventory -m command -a "sleep 10"
ansible all -c local --forks=10 -i five-hosts.inventory -m command -a "sleep 10"
ansible all -c paramiko --forks=10 -i five-hosts.inventory -m command -a "sleep 10"
Some investigation reveals that ansible is looking for the hosts in my inventory in ~/.ssh/known_hosts. My configuration has HashKnownHosts enabled. ansible isn't ever able to find the host entries it is looking for because it doesn't understand the hash known hosts entry format.
Whenever ansible's ssh transport can't find the known hosts entry, it acquires a global lock for the duration of the module's execution. The result of this confluence is that all execution is effectively serialized.
A temporary work-around is to give up some security and disabled host key checking by putting
host_key_checking = False into
~/.ansible.cfg. Another work-around is to use the paramiko transport (but this is incredibly slow, perhaps tens or hundreds of times slower than the ssh transport, for some reason). Another work-around is to let some unhashed entries get added to the known_hosts file for ansible's ssh transport to find.