Mona Jalal Mona Jalal - 1 month ago 7
HTTP Question

imgurpython.helpers.error.ImgurClientRateLimitError: Rate-limit exceeded

I have the following error:

/usr/local/lib/python2.7/dist-packages/requests/packages/urllib3/util/ssl_.py:90: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning.
InsecurePlatformWarning
/usr/local/lib/python2.7/dist-packages/requests/packages/urllib3/util/ssl_.py:90: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning.
InsecurePlatformWarning
Traceback (most recent call last):
File "download.py", line 22, in <module>
search = imgur_client.gallery_search('cat', window='all', sort='time', page=p)
File "/usr/local/lib/python2.7/dist-packages/imgurpython/client.py", line 531, in gallery_search
response = self.make_request('GET', 'gallery/search/%s/%s/%s' % (sort, window, page), data)
File "/usr/local/lib/python2.7/dist-packages/imgurpython/client.py", line 153, in make_request
raise ImgurClientRateLimitError()
imgurpython.helpers.error.ImgurClientRateLimitError: Rate-limit exceeded!


for this code:

1 from imgurpython import ImgurClient
2 import inspect
3 import random
4 import urllib2
5 import requests
6 from imgurpython.helpers.error import ImgurClientError
7
8 client_id = "ABC"
9 client_secret = "ABC"
10 access_token = "ABC"
11 refresh_token = "ABC"
12
13
14
15 image_type = ['jpg', 'jpeg']
16
17 imgur_client = ImgurClient(client_id, client_secret, access_token, refresh_token)
18
19 item_count = 0
20 for p in range(1, 10000):
21 try:
22 search = imgur_client.gallery_search('cat', window='all', sort='time', page=p)
23 for i in range(0, len(search)):
24 item_count +=1
25 print(search[i].comment_count)
26 if search[i].comment_count > 10 and not search[i].is_album:
27 print(search[i].type)
28 if search[i].type[6:] in image_type:
29 count = 0
30 try:
31 image_file = urllib2.urlopen(search[i].link, timeout = 5)
32 image_file_name = 'images/'+ search[i].id+'.'+search[i].type[6:]
33 output_image = open(image_file_name, 'wb')
34 output_image.write(image_file.read())
35 for post in imgur_client.gallery_item_comments(search[i].id, sort='best'):
36 if count <= 10:
37 count += 1
38 output_image.close()
39 except urllib2.URLError as e:
40 print(e)
41 continue
42 except socket.timeout as e:
43 print(e)
44 continue
45 except socket.error as e:
46 print(e)
47 continue
48 except ImgurClientError as e:
49 print(e)
50 continue
51
52 print item_count


Also I see this line almost very often:

/usr/local/lib/python2.7/dist-packages/requests/packages/urllib3/util/ssl_.py:90: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning.
InsecurePlatformWarning


How can I fix the error? Is there any workaround for rate limit error in Imgur? So I am creating this app for academic research use not for commercial and according to https://api.imgur.com/#limits it should be free but I had to register my app to get client_id related stuff. How can I set my application as non-commercial so that I would not get this rate limit error or if all kind of applications get this error how should I handle it? How should I set my code so that it would make only 1250 requests per hour?

Also here's my credit info:

User Limit: 500
User Remaining: 500
User Reset: 2016-10-18 14:32:41
User Client Limit: 12500
User Client Remaining: 9570

Answer

The rate limit refers to how frequently you're hitting the API, not how many calls you're allowed. To prevent hammering, most APIs have a rate limit (eg: 30 requests per minute, 1 every 2 seconds). Your script is making requests as quickly possible, hundreds or even thousands of times faster than the limit.

To prevent your script from hammering, the simplest solution is to introduce a sleep to your for loop.

from time import sleep

for i in range(10000):
    print i
    sleep(2) # seconds

Adjust the sleep time to be at least one second greater than what the API defines as its rate limit.

https://api.imgur.com/#limits

The Imgur API uses a credit allocation system to ensure fair distribution of capacity. Each application can allow approximately 1,250 uploads per day or approximately 12,500 requests per day. If the daily limit is hit five times in a month, then the app will be blocked for the rest of the month. The remaining credit limit will be shown with each requests response in the X-RateLimit-ClientRemaining HTTP header.