PDGill PDGill - 3 months ago 13
HTML Question

Python - BeautifulSoup scrape non-standard web table

I am attempting to scrape data from several webpages in order to create a CSV of the data. The data is just nutritional information of products. I have generated code to access the website, but I can't quite get the code to iterate out properly. The problem is, the website uses DIV tags for the product name, and inside of the DIV either or , it varies between the pages. When I attempt to iterate it out, the product names all show at once, in a list with tags, and then I get the contents of the column I requested, without tags. I am trying to figure out what I am doing wrong.

Source code example:

<div><strong>Product 1 Name</strong></div>

<table>
<tbody>
<tr>
<td>Serving Size</td>
<td>8 (fl. Oz.)</td>
</tr>
<tr>
<td>Calories</td>
<td>122 Calories</td>
</tr>
<tr>
<td>Fat</td>
<td>0 (g)</td>
</tr>
<tr>
<td>Sodium</td>
<td>0.2 (mg)</td>
</tr>
<tr>
<td>Carbs</td>
<td>8.8 (mg)</td>
</tr>
<tr>
<td>Dietary Fiber</td>
<td>0 (g)</td>
</tr>
<tr>
<td>Sugar</td>
<td>8.8 (g)<br />
&nbsp;</td>
</tr>
</tbody>
</table>
&nbsp;

<div><strong>Product 2 Name</strong></div>

<table>
<tbody>
<tr>
<td>Serving Size</td>
<td>8 (fl. Oz.)</td>
</tr>
<tr>
<td>Calories</td>
<td>134 Calories</td>
</tr>
<tr>
<td>Fat</td>
<td>0 (g)</td>
</tr>
<tr>
<td>Sodium</td>
<td>0.0 (mg)</td>
</tr>
<tr>
<td>Carbs</td>
<td>8.4 (mg)</td>
</tr>
<tr>
<td>Dietary Fiber</td>
<td>0 (g)</td>
</tr>
<tr>
<td>Sugar</td>
<td>8.4 (g)<br />
&nbsp;</td>
</tr>
</tbody>
</table>
&nbsp;


Ideally, I would like to be able to output to a CSV that has "Product Name" and Column 1 data in the header row, since it is the same for all of the tables. Then the data rows would be like:
"Product 1 Name, 8, 112, 0, 0.2, 8.8, 0, 8.8"


I know there is some manipulation that needs to be done to the data to get it to that point (to remove the size information).

Here is what I have so far that is starting to drive me crazy:

import requests, bs4, urllib2, csv
from bs4 import BeautifulSoup
from collections import defaultdict


#Loop on URLs to get Nutritional Information from each one.
with open('NutritionalURLs.txt') as f:
for line in f:
r = requests.get('website' + line)
soup=BeautifulSoup(r.text.encode('ascii','ignore'),"html.parser")

#TESTING
with open('output.txt', 'w') as o:
product_list = soup.find_all('b')
product_list = soup.find_all('strong')
print(product_list)
table_list = soup.find_all('table')
for tables in table_list:
trs = tables.find_all('tr')
for tr in trs:
tds = tr.find_all('td')[1:]
if tds:
facts = tds[0].find(text=True)
print(facts)
# o.write("Serving Size: %s, Calories: %s, Fat: %s, Sodium: %s, Carbs: %s, Dietary Fiber: %s, Sugar: %s\n" % \
# (facts[0].text, facts[1].text, facts[2].text, facts[3].text, facts[4].text, facts[5].text, facts[6].text))


This gives me an output like this:

[<strong>Product 1 Name</strong>, <strong>Product 2 Name</strong>]
8 (fl. Oz.)
101 Calories
0 (g)
0.0 (mg)
0 (mg)
0 (g)
0 (g)
8 (fl. Oz.)
101 Calories
0 (g)
0.0 (mg)
0 (mg)
0 (g)
0 (g)
[]

Answer

Find the tables, then extract the text from the previous strong and take the second td from each tr splitting the text once to remove the (g) etc..:

from bs4 import BeautifulSoup

soup = BeautifulSoup(html)

for table in soup.find_all("table"):
    name = [table.find_previous("strong").text]
    amounts = [td.text.split(None, 1)[0] for  td in table.select("tr td + td")])
    print(name + amounts)

Which would give you:

['Product 1 Name', '8', '122', '0', '0.2', '8.8', '0', '8.8']
['Product 2 Name', '8', '134', '0', '0.0', '8.4', '0', '8.4']

select("tr td + td") uses a css selector to get the second td from each tr/row,

Or using find_all and slicing would look like:

for table in soup.find_all("table"):
    name = [table.find_previous("strong").text]
    amounts = [td.find_all("td")[1].text.split(None, 1)[0] for  td in table.find_all("tr")]
    print(name + amounts)

Since it is not always a strong but sometimes a bold tag you want, just look for the strong first and fall back to the bold:

from bs4 import BeautifulSoup
import requests
html = requests.get("http://beamsuntory.desk.com/customer/en/portal/articles/1676001-nutrition-information-cruzan").content
soup = BeautifulSoup(html, "html.parser")
for table in soup.select("div.article-content table"):
    name = table.find_previous("strong") or table.find_previous("b")
    amounts = [td.text.split(None, 1)[0] for  td in table.select("tr td + td")]
    print([name.text] + amounts)

If table.find_previous("strong") finds nothing it will be None so the or will be executed and name will be set to table.find_previous("b").

Now it will work for both:

In [12]: html = requests.get("http://beamsuntory.desk.com/customer/en/portal/articles/1676001-nutrition-information-cruzan").content

In [13]: soup = BeautifulSoup(html, "html.parser")

In [14]: for table in soup.select("div.article-content table"):
   ....:         name = table.find_previous("strong") or table.find_previous("b")
   ....:         amounts = [td.text.split(None, 1)[0] for  td in table.select("tr td + td")]
   ....:         print([name.text] + amounts)
   ....:     
[u'Cruzan Banana Flavored Rum 42 proof', u'1.5', u'79', u'0', u'0.0', u'6.5', u'0', u'6.5']
[u'Cruzan Banana Flavored Rum 55 proof', u'1.5', u'95', u'0', u'0.0', u'6.5', u'0', u'6.5']
[u'Cruzan Black Cherry Flavored Rum 42 proof', u'1.5', u'80', u'0', u'0.0', u'6.9', u'0', u'6.9']
[u'Cruzan Citrus Flavored Rum 42 proof', u'1.5', u'99', u'0', u'0.0', u'2.8', u'0', u'2.6']
[u'Cruzan Coconut Flavored Rum 42 proof', u'1.5', u'78', u'0', u'0.1', u'6.9', u'0', u'6.5']
[u'Cruzan Coconut Flavored Rum 55 proof', u'1.5', u'95', u'0', u'0.1', u'6.1', u'0', u'0']
[u'Cruzan Guaza Flavored Rum 42 proof', u'1.5', u'78', u'0', u'0.1', u'6.5', u'0', u'6.5']
[u'Cruzan Key Lime Flavored Rum 42 proof', u'1.5', u'81', u'0', u'0.0', u'8.1', u'0', u'6']
[u'Cruzan Mango Flavored Rum 42 proof', u'1.5', u'85', u'0', u'0.0', u'8.5', u'0', u'8.5']
[u'Cruzan Mango Flavored Rum 55 proof', u'1.5', u'101', u'0', u'0.0', u'8.5', u'0', u'8.5']
[u'Cruzan Orange Flavored Rum 42 proof', u'1.5', u'76.77', u'0', u'0', u'6.4', u'0', u'6.4']
[u'Cruzan Passion Fruit Flavored Rum 42 proof', u'1.5', u'77', u'0', u'0.0', u'6.3', u'0', u'6.3']
[u'Cruzan Pineapple Flavored Rum 42 proof', u'1.5', u'78', u'0', u'0.0', u'6.5', u'0', u'6.5']
[u'Cruzan Pineapple Flavored Rum 55 proof', u'1.5', u'94', u'0', u'0.0', u'6.5', u'0', u'6.5']
[u'Cruzan Raspberry Flavored Rum 42 proof', u'1.5', u'92', u'0', u'0.0', u'10.1', u'0', u'10.1']
[u'Cruzan Raspberry Flavored Rum 55 proof', u'1.5', u'108', u'0', u'0.0', u'10.1', u'0', u'10.1']
[u'Cruzan Strawberry Flavored Rum 42 proof', u'1.5', u'76', u'0', u'0.0', u'6.1', u'0', u'6']
[u'Cruzan Vanilla Flavored Rum 42 proof', u'1.5', u'78', u'0', u'0.0', u'6.5', u'0', u'6.5']
[u'Cruzan Vanilla Flavored Rum 55 proof', u'1.5', u'94', u'0', u'0.0', u'6.5', u'0', u'6.5']
[u'Cruzan Estate Dark Rum 80 proof', u'1.5', u'101', u'0', u'0.0', u'0', u'0', u'0']
[u'Cruzan Estate Light Rum 80 proof', u'1.5', u'101', u'0', u'0.0', u'0', u'0', u'0']
[u'Cruzan Estate Single Barrel Rum 80 proof', u'1.5', u'99', u'0', u'0.0', u'0.9', u'0', u'0.9']

And the bold:

In [20]: html = requests.get("http://beamsuntory.desk.com/customer/en/portal/articles/1790163-midori-nutrition-information").content

In [21]: soup = BeautifulSoup(html, "html.parser")

In [22]: for table in soup.select("div.article-content table"):
   ....:         name = table.find_previous("strong") or table.find_previous("b")
   ....:         amounts = [td.text.split(None, 1)[0] for  td in table.select("tr td + td")]
   ....:         print([name.text] + amounts)
   ....:     
[u'Midori', u'1.0', u'62.1', u'0', u'0.3', u'7.5', u'0', u'7.0']