Abhishek Sharma Abhishek Sharma - 1 month ago 14
Python Question

UnicodeEncodeError in pulling tweets charmap cannot encode

I am trying to pull tweets from my timeline. But I am able to retrieve only half of them. It is throwing charmap codec cannot encode character u "\u2026". characters map to undefined.
I tried different encodings utf-8,ASCII, latin-1 and cp1252. But I am getting the same result. So I think the encoding is not getting changed.How should I change the encoding and which encoding should I choose for pulling tweets. I am using windows 7 and python 2.7.8. This is my code

import tweepy
import csv
consumer_key = ''
consumer_secret = ''
access_token = ''
access_secret = ''
auth = tweepy.OAuthHandler(consumer_key, consumer_secret)
auth.set_access_token(access_token, access_secret)
api = tweepy.API(auth)
data = api.get_user('')
# Open/Create a file to append data
csvFile = open('hollywood.csv', 'a')
#Use csv Writer
csvWriter = csv.writer(csvFile, delimiter=' ')
i = 0
for tweet in tweepy.Cursor(api.user_timeline).items():
#Write a row to the csv file/ I use encode utf-8
csvWriter.writerow([tweet.created_at, tweet.text.encode('cp1252')])
print tweet.created_at, tweet.text
i+=1
if i%5 == 0:
print i
csvFile.close().

Answer

Try:

tweet.text.encode('utf8')

UTF-8 is bulletproof in this sense. U+2026 can't be encoded in Latin1 and the rest.

Works perfectly:

>>> u"\u2026".encode('utf8')
'\xe2\x80\xa6'