rlartiga rlartiga - 5 months ago 33
Python Question

Paramiko Buffer issue

I have a problem of buffer using paramiko, I found the same question here and one of the solutions states that:

Rather than using .get(), if you just call .open() to get an SFTPFile
instance, then call .read() on that object, or just hand it to the
Python standard library function shutil.copyfileobj() to download the
contents. That should avoid the Paramiko prefetch cache, and allow you
to download the file even if it's not quite as fast.

Now if I have:


How I save this file-like-object in a csv in my local pc? (the original file is a csv also)


Here is a working example that fetches copies a test file on your local machine. The file is much smaller than 1 GIG but gives the general plan.

import paramiko
import os
import shutil
import time
import getpass

# get params
user = getpass.getuser()
pwd = getpass.getpass("Enter password: ")
bufsize = 2**20
host = 'localhost'
test_file_lines = 1000000

# create test file
now = time.asctime()
testfile_path = os.path.abspath('deleteme')
local_path = 'deleteme.copy'

print('writing test file...')
start = time.time()
with open(testfile_path, 'w') as fp:
    for _ in range(test_file_lines):
        fp.write(now + '\n')
delta = time.time() - start
file_size = os.stat(testfile_path).st_size
print("file size %d, %d KB/Sec" % (file_size, file_size/1024/delta))

# make connection
sftp_file=sftp.open(testfile_path, bufsize=bufsize)
print('copying file...')
start = time.time()
    open(local_path, 'wb', bufsize),
delta = time.time() - start
print('%.3f seconds, %d KB/Sec' % (delta, file_size/1024/delta))

#assert open(testfile_path).read() == open(local_path).read(), "files match"

Running on my machine I got

Enter password: 
writing test file...
file size 25000000, 21017 KB/Sec
copying file...
10.225 seconds, 2387 KB/Sec

We expect some slow down because there is a read and a write plus network costs (its to local host so doesn't really touch the wire), but that does seem kinda slow. I am using a low-powered laptop with 2 cores and between this app and sshd, used much of the cpu, presumably to do the encryption. A higher powered machine may work better.