Erik hoeven Erik hoeven - 11 months ago 121
Python Question

Error while fetching Tweets with Tweepy

I have a Python script that fetch tweets. In the script i use the libary Tweepy . I use a valid authentication parameters. After running this script some tweets are stored in my MongoDB and some are refused by the if statement. But still i get the error

requests.packages.urllib3.exceptions.ProtocolError: ('Connection broken: IncompleteRead(0 bytes read, 2457 more expected)'

My question is which part of the script can i improve, so i do not get the error above.

This is my script

from tweepy import Stream
from tweepy import OAuthHandler
from tweepy.streaming import StreamListener
import time
import json
from pymongo import MongoClient

#Mongo Settings
client = MongoClient()
db = client.Sentiment
Tweets = db.Tweet

#Twitter Credentials
ckey ='myckey'
csecret ='mycsecret'
atoken = 'myatoken'
asecret = 'myasecret'

class listener(StreamListener):

def on_data(self, data):

tweet = json.loads(data)

if tweet["lang"] == "nl":
print tweet["id"]

return True
except BaseException, e:
print 'failed on_date,', str(e)

def on_error(self, status):
print status

auth = OAuthHandler(ckey, csecret)
auth.set_access_token(atoken, asecret)
twitterStream = Stream(auth, listener())
twitterStream.filter( track=["geld lenen"
,"Alpha Credit"
,"Advanced Finance"
,"private lease"
,"Nationale Nerderlanden"
,"DGA adviseur"
,"Financieel Attent"
,"De Nederlandse Kredietmaatschappij"
,"De Financiele Makelaar Kredieten"
,"CFSN Kredietendesk"
,"De Graaf Assurantien en Financieel Adviseurs"
,"VDZ Geldzaken"
,"Financium Primae"
], languages="nl"

I hope you can help me...

Answer Source

I had this same problem, solved when I remved languages from the filter function

since it's not yet functional, although Twitter says it is

Instead I keep the check of the language as you did in the on_data(..)

Also I use the on_status(..) instead of on_data(..) as follows:

def on_status(self, status):
    tweet = json.dumps(status)
    if tweet["lang"] == "nl":
        print tweet["id"]

Other people reported that using twitterStream.filter(track=['word'], languages=['nl']), but it didn't with me.