Jordan Dea-Mattson Jordan Dea-Mattson - 1 year ago 76
Ruby Question

What are some good Ruby-based web crawlers?

I am looking at writing my own, but I am wondering if there are any good web crawlers out there which are written in Ruby.

Short of a full-blown web crawler, any gems that might be helpful in building a web crawler would be useful. I know this part of the question is touched upon in a couple of places, but a list of gems applicable to building a web crawler would be a great resource as well.

Answer Source

I am building wombat, a Ruby DSL to crawl web pages and extract content. Check it out on github

It is still in an early stage but is already functional with basic functionality. More stuff will be added really soon.

Recommended from our users: Dynamic Network Monitoring from WhatsUp Gold from IPSwitch. Free Download