divyam jaiswal divyam jaiswal - 1 month ago 19
Python Question

Setting timelimit for sftp.get() of Paramiko module

I am using Paramiko's SFTP client to download a file from remote server to a client(i.e. get operation)

The file to be transferred is a bit huge ~1GB.

So I would like the get operation to timeout if the time is more than 10s.

But setting the timeout value for connect doesn't work, It appears to be the timeout for just creating the SSH connection and not the timeout for the whole ssh connection.

ssh = paramiko.SSHClient()
ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
ssh.connect(host, username=username, password=password, timeout=10.0)
sftp = ssh.open_sftp()
start_time = time.time()
sftp.get(local_path,remote_path)
elapsed_time = time.time()-start_time
print elapsed_time
sftp.close()


I also tried setting the timeout value for the channel but it doesn't work too

sftp.get_channel.settimeout(10.0)


But this timeout is again just for read/write operations

There is a similar question Timeout in paramiko (python) but it only has answer for timeout in creation of SSH-connection

Update 1



Following the comments of @Martin I implemented a callback function which checks for the time-limit for the get operation of sftp:

import paramiko
import time

Class TimeLimitExceeded(Exception):
pass

timelimit = 10
start_time = time.time()

def _timer():
elapsed_time = time.time()-start_time
if elapsed_time > timelimit:
raise TimeLimitExceeded

ssh = paramiko.SSHClient()
ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
ssh.connect(host, username=username, password=password, timeout=10.0)
sftp = ssh.open_sftp()
try:
sftp.get(local_path,remote_path,_timer)
except TimeLimitExceeded:
print "The operation took too much time to complete"
sftp.close()


But the time to except the exception is a lot, the code is blocking somewhere.
I dived into Paramiko source code and found the culprit behind it was the
_close(self,async=False)
method of the
sftp_file.py

Any help to get around this?

Update 2



Trying to close the channel itself if timelimit is exceeded.
Then the exception is flushed to console as the
prefetch
is implemented by separate daemon thread

File "/scratch/divjaisw/python2.7/lib/python2.7/threading.py", line 801, in __bootstrap_inner
self.run()
File "/scratch/divjaisw/python2.7/lib/python2.7/threading.py", line 754, in run
self.__target(*self.__args, **self.__kwargs)
File "/scratch/divjaisw/python_virtual/lib/python2.7/site-packages/paramiko/sftp_file.py", line 488, in _prefetch_thread
num = self.sftp._async_request(self, CMD_READ, self.handle, long(offset), int(length))
File "/scratch/divjaisw/python_virtual/lib/python2.7/site-packages/paramiko/sftp_client.py", line 754, in _async_request
self._send_packet(t, msg)
File "/scratch/divjaisw/python_virtual/lib/python2.7/site-packages/paramiko/sftp.py", line 170, in _send_packet
self._write_all(out)
File "/scratch/divjaisw/python_virtual/lib/python2.7/site-packages/paramiko/sftp.py", line 133, in _write_all
n = self.sock.send(out)
File "/scratch/divjaisw/python_virtual/lib/python2.7/site-packages/paramiko/channel.py", line 715, in send
return self._send(s, m)
File "/scratch/divjaisw/python_virtual/lib/python2.7/site-packages/paramiko/channel.py", line 1081, in _send
raise socket.error('Socket is closed')
error: Socket is closed

Answer

What you ask for in not really a timeout. The term "timeout" is used for limit of waiting for a response.

But your server does not stop responding. The communication is active.


What you ask for is rather a limit for a duration of an operation. You can hardly expect this to be implemented readily for you. It's a rather specific requirement. You have to implement it yourself.

You can use the callback argument of the get method:

def get(self, remotepath, localpath, callback=None):

In the callback, check the duration of the transfer and throw an exception, if the time limit expires.

This won't cancel the transfer immediately. To optimize transfer performance, Paramiko queues up to 100 read requests to the server (see a condition in the sftp_file._write). Once you attempt to cancel the transfer, Paramiko has to wait for (up to 100) responses to those requests to clear the queue.

Comments