DV Hughes DV Hughes - 2 months ago 13
R Question

Search twitter and obtain tweets by hashtag, maximizing number of returned search results

I am attempting to compile a corpus of all Tweets related to the World Cup on Twitter from their API using the twitteR package in R.

I am using the following code for a single hashtag (for example). However, my problem is that it appears I am only 'authorized' to access a limited set of the tweets (in this case, only the 32 most recent).


reqURL <- "https://api.twitter.com/oauth/request_token"
accessURL <- "https://api.twitter.com/oauth/access_token"
authURL <- "http://api.twitter.com/oauth/authorize"
#consumerKey <- Omitted
#consumerSecret <- Omitted
twitCred <- OAuthFactory$new(consumerKey=consumerKey,
options(RCurlOptions = list(cainfo = system.file("CurlSSL", "cacert.pem", package = "RCurl")))


#save(twitCred, file="twitterAuthentication.Rdata")

FIFA<-searchTwitter("#WorldCup", n=9999, since='2007-10-30')

Returns the following error:

Warning message:
In doRppAPICall("search/tweets", n, params = params, retryOnRateLimit = retryOnRateLimit, :
9999 tweets were requested but the API can only return 32

My question is: How do I access the maximum number of tweets using a specific hashtag? (Also, could someone clarify what the 'max' limit actually is? And why I can't seem to get anywhere close to this value of (~ 1500 tweets)?

I have tested OAuth within the Twitter Developer website and obtained signing results for the Signature base string, authorization header, and cURL commands respectively, indicating to me that I have the appropriate permissions & authorizations to draw the appropriate data from Twitter's servers. Please advise/correct me if I am wrong, or if you need further information on this.

My API Permissions are currently set to: Read, Write & Access direct messages

Session Info:

R version 3.0.2 (2013-09-25)
Platform: x86_64-apple-darwin10.8.0 (64-bit)

[1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8

attached base packages:
[1] stats graphics grDevices utils datasets methods base

other attached packages:
[1] RJSONIO_1.0-3 twitteR_1.1.7 rjson_0.2.12 ROAuth_0.9.3 digest_0.6.3 RCurl_1.95-4.1 bitops_1.0-5
[8] foreign_0.8-55

loaded via a namespace (and not attached):
[1] tools_3.0.2

Additional Resource/Source:

twitter package in R maximum tweets using searchTwitter()

This source states the max is 1500

Twitter api searching tweets for hashtags

This source states the max is 3200


This is not possible,

Using the Twitter Search API

"The Search API is not complete index of all Tweets, but instead an index of recent Tweets. At the moment that index includes between 6-9 days of Tweets."