user2058738 user2058738 - 6 months ago 18
Bash Question

Extract file using sftp based from a masterlist

Is it possible to extract certain files based from a master list via sftp.

Example:

directory containts ff files.

aa.txt
bb.txt
cc.txt


masterlist.txt containts:

aa.txt
bb.txt


Files that should be extracted

aa.txt
bb.txt


Edit:

Thanks @shellter for your feedback.
I did try to write my own code but wasn't able to find samples that I could work on(I'm not a unix person btw).

Anyhow, as for your suggestion regarding using while-read-line, I've tried it but I am getting Invalid command error.

#!/bin/ksh
file=MasterList.txt

while IFS= read -r line
do
echo "fetching $line"
sftp user@192.168.1.101
cd /data/EP_files/balex
get "$line"
bye
done <"$file"


Lastly, if my masterfile containts 10k list of files, is this kind of approach ok performance wise?

Thank you

Answer

You need to create the entire sequence of SFTP commands - including the individual get commands for all files in the input list up front, and then invoke sftp only once, passing the command list via stdin (standard input):

#!/usr/bin/env ksh

file=MasterList.txt

sftp -b - user@192.168.1.101 <<EOF
cd /data/EP_files/balex
$(sed -n 's/^file_.*/get "&"/p' "$file")
bye
EOF

The <<EOF ... EOF block is a a so-called here-document, which allows passing multiline strings (optionally with embedded variable references and commands) via stdin.

sed -n 's/^file_.*/get "&"/p' "$file" embeds a get command for each filename in $file that starts with file_, ignoring any other lines (as requested by the OP in a comment).

The above assumes that your sftp utility accepts a list of commands in "batch" mode via the -b option via stdin (-).