paintball247 paintball247 - 3 months ago 16
Python Question

Python ASCII codec can't encode character error during write to CSV

I'm not entirely sure what I need to do about this error. I assumed that it had to do with needing to add .encode('utf-8'). But I'm not entirely sure if that's what I need to do, nor where I should apply this.

The error is:

line 40, in <module>
writer.writerows(list_of_rows)
UnicodeEncodeError: 'ascii' codec can't encode character u'\u2013' in position 1
7: ordinal not in range(128)


This is the base of my python script.

import csv
from BeautifulSoup import BeautifulSoup

url = \
'https://dummysite'

response = requests.get(url)

html = response.content

soup = BeautifulSoup(html)

table = soup.find('table', {'class': 'table'})

list_of_rows = []
for row in table.findAll('tr')[1:]:
list_of_cells = []
for cell in row.findAll('td'):
text = cell.text.replace('[','').replace(']','')
list_of_cells.append(text)
list_of_rows.append(list_of_cells)

outfile = open("./test.csv", "wb")
writer = csv.writer(outfile)
writer.writerow(["Name", "Location"])
writer.writerows(list_of_rows)

Answer

Python 2.x CSV library is broken. You have three options. In order of complexity:

  1. Use the fixed library https://github.com/jdunck/python-unicodecsv (pip install unicodecsv). Use as a drop-in replacement - Example:

    with open("myfile.csv", 'rb') as my_file:    
        r = unicodecsv.DictReader(my_file, encoding='utf-8')
    
  2. Read the CSV manual regarding Unicode: https://docs.python.org/2/library/csv.html (See examples at the bottom)

  3. Manually encode each item as UTF-8:

    for cell in row.findAll('td'):
        text = cell.text.replace('[','').replace(']','')
        list_of_cells.append(text.encode("utf-8"))
    
Comments