hallettwx hallettwx - 6 months ago 38
Bash Question

Downloading Only Newest File Using Wget / Curl

How would I use wget or curl to download the newest file in a directory?

This seems really easy, however the filename won't always be predictable, and as new data comes in it'll be replaced with a random filename.

Specifically, the directory I wish to download data from has the following naming structure, where the last string of characters is a randomly generated timestamp:

MRMS_RotationTrackML1440min_00.50_20160530-175837.grib2.gz
MRMS_RotationTrackML1440min_00.50_20160530-182639.grib2.gz
MRMS_RotationTrackML1440min_00.50_20160530-185637.grib2.gz


The randomly generated timestamp is in the format of: {hour}{minute}{second}

The directory in question is here: http://mrms.ncep.noaa.gov/data/2D/RotationTrackML1440min/

Could it have to be something with something in the headers, where you'd use curl to sift through the last-modified timestamp?

Any help would be appreciated here, thanks in advance.

Answer

You can just run following command periodically:

wget -r -nc --level=1 http://mrms.ncep.noaa.gov/data/2D/RotationTrackML1440min/

It will download recursively whatever is new in the directory after last run.

Comments