Abhimanyu Khanna Abhimanyu Khanna - 1 year ago 89
Bash Question

Optimizing SCP over network

I have a shell script which sends a file using scp over the network. The script works in an event driven fashion i.e. every time the file is modified,it is sent over the network using scp. I would like to optimize the script such that instead of every time sending the file, I only send the "new appended/new write data" and that gets appended to the file at the other endpoint

The script is as follows

while true
do
inotifywait -e close_write,moved_to,create . |
while read -r directory events filename; do
if [ "$filename" = "keylog.txt" ]; then
sshpass -p "password" scp -o StrictHostKeyChecking=no keylog.txt machine@192.168.151.19:/home/machine/keylog.txt
fi
done
sleep 0.00001
done

Answer Source

I suggest to take a look at rsync. It's designed exactly for this kind of purposes. Quoting from man rsync:

   Rsync is a fast and extraordinarily versatile file copying tool.   It  can
   copy  locally,  to/from  another  host over any remote shell, or to/from a
   remote rsync daemon.  It offers a large number  of  options  that  control
   every aspect of its behavior and permit very flexible specification of the
   set of files to be copied.  It is famous for its delta-transfer algorithm,
   which reduces the amount of data sent over the network by sending only the
   differences between the source files and the existing files in the  desti-
   nation.  Rsync is widely used for backups and mirroring and as an improved
   copy command for everyday use.

On a server that supports scp, typically rsync works too. Try this:

sshpass -p "password" rsync keylog.txt machine@192.168.151.19:/home/machine/keylog.txt 

Even without tuning the parameters, rsync will try to minimize the amount of data transferred. When the destination file already exists, it will transfer only the necessary difference.

Recommended from our users: Dynamic Network Monitoring from WhatsUp Gold from IPSwitch. Free Download