I have a shell script which sends a file using scp over the network. The script works in an event driven fashion i.e. every time the file is modified,it is sent over the network using scp. I would like to optimize the script such that instead of every time sending the file, I only send the "new appended/new write data" and that gets appended to the file at the other endpoint
The script is as follows
inotifywait -e close_write,moved_to,create . |
while read -r directory events filename; do
if [ "$filename" = "keylog.txt" ]; then
sshpass -p "password" scp -o StrictHostKeyChecking=no keylog.txt firstname.lastname@example.org:/home/machine/keylog.txt
I suggest to take a look at
It's designed exactly for this kind of purposes.
Rsync is a fast and extraordinarily versatile file copying tool. It can copy locally, to/from another host over any remote shell, or to/from a remote rsync daemon. It offers a large number of options that control every aspect of its behavior and permit very flexible specification of the set of files to be copied. It is famous for its delta-transfer algorithm, which reduces the amount of data sent over the network by sending only the differences between the source files and the existing files in the desti- nation. Rsync is widely used for backups and mirroring and as an improved copy command for everyday use.
On a server that supports
rsync works too.
sshpass -p "password" rsync keylog.txt email@example.com:/home/machine/keylog.txt
Even without tuning the parameters,
rsync will try to minimize the amount of data transferred.
When the destination file already exists,
it will transfer only the necessary difference.