I'm trying to scrape Reddit with Nokogiri, but a single run of this keeps telling me that I'm putting in too many requests.
url = "https://www.reddit.com/r/all"
redditscrape = Nokogiri::HTML(open(url))
OpenURI::HTTPError: 429 Too Many Requests
Reddit has an API
You could probably query the API for the particular sub-reddit(s) you want to scrape. Attempting to scrape
all of reddit just seems like a nightmare waiting to happen considering the high volume and the nested comments.
It looks like Reddit is blocking the ability to scrape in favor of using their public API.