NepUS NepUS - 1 year ago 65
Python Question

retrieve links from web page using python and BeautifulSoup

How can I retrieve the links of a webpage and copy the url address of the links using Python?

ars ars
Answer Source

Here's a short snippet using the SoupStrainer class in BeautifulSoup:

import httplib2
from BeautifulSoup import BeautifulSoup, SoupStrainer

http = httplib2.Http()
status, response = http.request('')

for link in BeautifulSoup(response, parseOnlyThese=SoupStrainer('a')):
    if link.has_attr('href'):
        print link['href']

The BeautifulSoup documentation is actually quite good, and covers a number of typical scenarios:

Edit: Note that I used the SoupStrainer class because it's a bit more efficient (memory and speed wise), if you know what you're parsing in advance.