Kaushik S Kaushik S - 2 months ago 7
Python Question

How do I extract attributes from xml tags using Beautiful Soup?

I am trying to use Beautiful Soup in Django to extract xml tags. This is a sample of the tags that I'm using:

<item>
<title>
Title goes here
</title>
<link>
Link1 goes here
</link>
<description>
Description goes here
</description>
<media:thumbnail url="Image URL goes here" height="222" width="300"/>
<pubDate>Thu, 15 Sep 2016 13:24:48 EDT</pubDate>
<guid isPermaLink="true">
Link2 goes here
</guid>
</item>


I have obtained strings of title,link and description tags. But I'm having trouble obtaining the URL from
media:thumbnail
tag.

This is the snippet where I got the values of rest of the tags:

soup=BeautifulSoup(urlopen(xmllink),'xml')
for items in soup.find_all('item'):
listTitle.append(items.title.get_text())
listURL.append(items.link.get_text())
listDescription.append(items.description.get_text())


Help

Answer

The issue is because not every item has a media:thumbnail so you need to check first:

In [60]: import requests

In [61]: from  bs4 import BeautifulSoup

In [62]: soup = BeautifulSoup(requests.get("https://rss.sciencedaily.com/computers_math/computer_programming.xml").content, "xml")

In [63]: 

In [63]: for item in soup.find_all("item"):
   ....:         thumb = item.find("thumbnail")
   ....:         if thumb:
   ....:                 print(thumb["url"])
   ....:         
https://images.sciencedaily.com/2016/09/160915132448.jpg
https://images.sciencedaily.com/2016/09/160915090018.jpg
https://images.sciencedaily.com/2016/09/160914090327.jpg
https://images.sciencedaily.com/2016/09/160913134149.jpg
https://images.sciencedaily.com/2016/09/160909094844.jpg
https://images.sciencedaily.com/2016/09/160907125004.jpg
https://images.sciencedaily.com/2016/09/160906085157.jpg
https://images.sciencedaily.com/2016/08/160831085055.jpg
https://images.sciencedaily.com/2016/08/160822181811.jpg
https://images.sciencedaily.com/2016/08/160815134941.jpg
https://images.sciencedaily.com/2016/08/160815134817.jpg
https://images.sciencedaily.com/2016/08/160809095640.jpg
https://images.sciencedaily.com/2016/08/160803140137.jpg
https://images.sciencedaily.com/2016/07/160722104135.jpg
https://images.sciencedaily.com/2016/07/160721144139.jpg
https://images.sciencedaily.com/2016/07/160721103855.jpg
https://images.sciencedaily.com/2016/07/160720094641.jpg
https://images.sciencedaily.com/2016/07/160718133206.jpg
https://images.sciencedaily.com/2016/07/160713105850.jpg
https://images.sciencedaily.com/2016/07/160711151055.jpg
https://images.sciencedaily.com/2016/07/160707083258.jpg
https://images.sciencedaily.com/2016/06/160629125823.jpg
https://images.sciencedaily.com/2016/06/160627125140.jpg
https://images.sciencedaily.com/2016/06/160624101050.jpg
https://images.sciencedaily.com/2016/06/160622104810.jpg

A faster alternative would be to use lxml:

from lxml import etree

for item in tree.findall(".//item/media:thumbnail",tree.nsmap):
     parent = item.getparent()
     print(parent.xpath("title/text()")[0])
     print(parent.xpath("link/text()")[0])
     print(item.get("url"))
Comments