Jordan Dea-Mattson Jordan Dea-Mattson - 19 days ago 8
Ruby Question

What are some good Ruby-based web crawlers?

I am looking at writing my own, but I am wondering if there are any good web crawlers out there which are written in Ruby.

Short of a full-blown web crawler, any gems that might be helpful in building a web crawler would be useful. I know this part of the question is touched upon in a couple of places, but a list of gems applicable to building a web crawler would be a great resource as well.

Answer

I am building wombat, a Ruby DSL to crawl web pages and extract content. Check it out on github https://github.com/felipecsl/wombat

It is still in an early stage but is already functional with basic functionality. More stuff will be added really soon.

Comments