ajwood ajwood - 1 month ago 15
HTTP Question

Getting directory listing over http

There is a directory that is being served over the net which I'm interested in monitoring. Its contents are various versions of software that I'm using and I'd like to write a script that I could run which checks what's there, and downloads anything that is newer that what I've already got.

Is there a way, say with

wget
or something, to get a a directory listing. I've tried using
wget
on the directory, which gives me html. To avoid having to parse the html document, is there a way of retrieving a simple listing like
ls
would give?

Answer

I just figured out a way to do it:

wget --spider -r --no-parent http://some.served.dir.ca/

It's quite verbose, so you need to pipe through grep a couple of times depending on what you're after, but the information is all there. It looks like it prints to stderr, so append 2>&1 to let grep at it. I grepped for "\.tar\.gz" to find all of the tarballs the site had to offer.