Igor Tikhonov Igor Tikhonov - 4 months ago 19
Bash Question

How to download files from my list with wget and ftp

I need to download only defined files with wget and ftp.
For example:

1.I retrieve all files using:

echo ls -R | ftp ftp://user:password@host > ./list.txt

2.Then I will parse the result and get a list with absolute paths for each file:


3.And now I need to download all files from the result list using wget and ftp.

And I don't want to create a separate FTP session for each file download process.

Please give your advice. Thank you.


For recursive download I'm using it:
wget -r ftp://user:password@host:/ -nH -P /download/path
. It works great, but I need to pass a file with a list of remote files for downloading via FTP with one FTP session.


Sorry, I missed the "single session" part when I commented. I think you need to have your script generate a second script to run a single FTP session.

So, your script will not do any FTP itself, it will just write another script that does the transfers. So, it will write a script that does this

get file1 localname1
get file2 localname2
get fileN localnameN

Then it will execute that script, by doing:

bash < thatScript

So your script will look like this:

cat - <<END > $ScriptName
ftp -n <<EOS
quote USER freddy
quote PASS frog

# Your selection code goes here ***PHNQZ***
echo get file1 localname1 >> $ScriptName
echo get file2 localname2 >> $ScriptName
echo get fileN localnameN >> $ScriptName

echo quit >> $ScriptName
echo EOS  >> $ScriptName
echo "Now run bash < $ScriptName"

Then delete the script as it contains your password. Or you can put the password in your .netrc file.

As regards creating directories locally, you can do that in the first script using mkdir -p. The -p has the advantage that it creates all directories in between in one go and doesn't get upset if they already exist.

So, just looking at the area of code where it says ***PHNQZ*** above, let's say your code decides you need file freddy/frog/c.txt, you could do:

localdir=${remotename%/*}         # Get just directory part using "bash Parameter Substitution"
mkdir -p "$localdir"              # make directory and all parts in between