I have a problem of buffer using paramiko, I found the same question here and one of the solutions states that:
Rather than using .get(), if you just call .open() to get an SFTPFile
instance, then call .read() on that object, or just hand it to the
Python standard library function shutil.copyfileobj() to download the
contents. That should avoid the Paramiko prefetch cache, and allow you
to download the file even if it's not quite as fast.
Here is a working example that fetches copies a test file on your local machine. The file is much smaller than 1 GIG but gives the general plan.
import paramiko import os import shutil import time import getpass # get params user = getpass.getuser() pwd = getpass.getpass("Enter password: ") bufsize = 2**20 host = 'localhost' test_file_lines = 1000000 # create test file now = time.asctime() testfile_path = os.path.abspath('deleteme') local_path = 'deleteme.copy' print('writing test file...') start = time.time() with open(testfile_path, 'w') as fp: for _ in range(test_file_lines): fp.write(now + '\n') delta = time.time() - start file_size = os.stat(testfile_path).st_size print("file size %d, %d KB/Sec" % (file_size, file_size/1024/delta)) # make connection ssh=paramiko.SSHClient() ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy()) ssh.connect(host,username=user,password=pwd) sftp=ssh.open_sftp() sftp_file=sftp.open(testfile_path, bufsize=bufsize) print('copying file...') start = time.time() shutil.copyfileobj(sftp_file, open(local_path, 'wb', bufsize), bufsize) delta = time.time() - start print('%.3f seconds, %d KB/Sec' % (delta, file_size/1024/delta)) #assert open(testfile_path).read() == open(local_path).read(), "files match"
Running on my machine I got
Enter password: writing test file... file size 25000000, 21017 KB/Sec copying file... 10.225 seconds, 2387 KB/Sec
We expect some slow down because there is a read and a write plus network costs (its to local host so doesn't really touch the wire), but that does seem kinda slow. I am using a low-powered laptop with 2 cores and between this app and sshd, used much of the cpu, presumably to do the encryption. A higher powered machine may work better.