Abhimanyu Khanna Abhimanyu Khanna - 11 months ago 63
Bash Question

Optimizing SCP over network

I have a shell script which sends a file using scp over the network. The script works in an event driven fashion i.e. every time the file is modified,it is sent over the network using scp. I would like to optimize the script such that instead of every time sending the file, I only send the "new appended/new write data" and that gets appended to the file at the other endpoint

The script is as follows

while true
inotifywait -e close_write,moved_to,create . |
while read -r directory events filename; do
if [ "$filename" = "keylog.txt" ]; then
sshpass -p "password" scp -o StrictHostKeyChecking=no keylog.txt machine@
sleep 0.00001

Answer Source

I suggest to take a look at rsync. It's designed exactly for this kind of purposes. Quoting from man rsync:

   Rsync is a fast and extraordinarily versatile file copying tool.   It  can
   copy  locally,  to/from  another  host over any remote shell, or to/from a
   remote rsync daemon.  It offers a large number  of  options  that  control
   every aspect of its behavior and permit very flexible specification of the
   set of files to be copied.  It is famous for its delta-transfer algorithm,
   which reduces the amount of data sent over the network by sending only the
   differences between the source files and the existing files in the  desti-
   nation.  Rsync is widely used for backups and mirroring and as an improved
   copy command for everyday use.

On a server that supports scp, typically rsync works too. Try this:

sshpass -p "password" rsync keylog.txt machine@ 

Even without tuning the parameters, rsync will try to minimize the amount of data transferred. When the destination file already exists, it will transfer only the necessary difference.