Ivan Sukhetskyi Ivan Sukhetskyi - 1 month ago 16
HTML Question

Web scraping: How to get specific element from some page?

I'm developing some scrapping software to check if products price are changed or not. To do this I'm sending Web http request to get product page, it returns me whole html page. There are a lot of items to check so I'm doing it in parallel so my internet bandwidth runs out quickly

The main questions:

Is it possible to send Web http request with select query to get only specific element from html page?

Answer

Your best (free / cheap) bet is the Html Agility Pack. There are automated solutions out there that have a UI where you stick in the URL and then just point and click to the elements that you want to extract and they generate the info for you. Big $$$ for that stuff though :(. With the Html Agility Pack, you have to create the "xpath" by hand.