Aishwarya94810 Aishwarya94810 - 21 days ago 7
Python Question

Which Python version to use for using BeautifulSoup and urllib?

I'm working on web crawling using Python. I had issues while using Python version 3. So I wanted to know which version of Python is suitable for web crawling.

Answer

I think there is definitely a counter-argument to making the move to Python 2.7. There is no reason that I can think of, particularly for web crawling, that you would need to revert to Python 2.7.

BeautifulSoup 4 and lxml are both fully ported to Python 3.5.

urllib is fully functional in Python 3.5. You should be aware that there are differences in the implementation of urllib in Python 2.7 and Python 3.5.

However, I would suggest you use the Requests package instead of urllib. Here is a post highlighting some of their differences.

If you need to load pages that require javascript, Selenium also works in Python 3.5. Selenium can also support headless browsing (e.g., PhantomJS).

Also, here is an official post from Python that can help guide you to your decision.