Manny Manny - 3 months ago 42
Ruby Question

Ruby Timeout issue

I am working on the following code. My code times out when trying to retrieve all the data. How can I reset the timeout when making a successful call?

Code Below:

require 'rest-client'

module InvisiaCorp
module ApiServices
# Client methods used to fetch data
class Client

def initialize(site, timeout, logger, factory)
@site = site
@timeout = timeout
@logger = logger
@factory = factory
end

def host
@site.url
end

def fetch(sub_url, key = nil)
sub_url = "#{sub_url}/#{URI.encode(key)}" if key
begin
Timeout::timeout(120) do
general_fetch(sub_url, [], 'fetch')
end
rescue Timeout::Error
@logger.warning("General fetch timed out.")
end
end

def fetch_hashed_metrics(sub_url)
begin
Timeout::timeout(120) do
general_fetch(sub_url, {}, 'hashed_metrics')
end
rescue Timeout::Error
@logger.warning("Hashed metrics fetch timed out.")
end
end

private

def general_fetch(sub_url, data_object, type)
uri = URI.parse(sub_url)
has_next_tag = false
result = nil
loop do
uri.query = get_query(uri, result) if has_next_tag && result
result = populate_data_object(type, uri, data_object)
break unless next_tag?(result)
end
data_object
end

def get_query(uri, result)
@logger.debug("Endpoint: <#{uri}>, has additional pages, calling subsequent page.")
URI.encode_www_form('nextTag' => result.next_tag)
end

def populate_data_object(type, uri, data_object)
@logger.debug("In #{type}, calling: <#{uri}>")
result = execute(uri.to_s)
if result && result.status_code == 'SUCCESS'
data_object.merge!(result.hashed_metrics) if type == 'hashed_metrics'
data_object.concat(result.records) if type == 'fetch'
else
log_failed_to_fetch(uri, result)
end
result
end

def danger_uri_processing(uri, container)
@logger.debug("calling: <#{uri_object}>")
result = danger_execute(uri.to_s)
if result && result.status_code == 'SUCCESS'
container.concat(result.records)
else
log_failed_to_fetch(uri, result)
end
result
end

def next_tag?(result)
return false unless result && result.status_code == 'SUCCESS'
result.next_tag?
end

def log_failed_to_fetch(uri_object, result)
@logger.debug("Failed to retrieve information from API Services URI: <#{uri_object}>")
@logger.warning("Call to <#{uri_object}> returned with HTTP Status Code: <#{result.status_code}>") if result
end

def execute(sub_url)
@logger.debug("In execute, calling: <#{sub_url}>")
@factory.client_response @site[sub_url].get(accept: 'application/json', timeout: @timeout)
rescue
@logger.debug("Failed to retrieve information from API Services sub url: <#{sub_url}>")
nil
end

def danger_execute(sub_url)
@factory.client_response @site[sub_url].get(accept: 'application/json', timeout: @timeout)
end
end
end
end


My Error Logs:

[2016-08-28 04:32:12.042] DEBUG: In fetch, calling <192.168.1.12/api/v2/foo/con-ax40>
[2016-08-28 04:32:12.042] DEBUG: In execute, calling <192.168.1.12/api/v2/foo/con-ax40>
[2016-08-28 04:32:12.233] DEBUG: In fetch, calling <192.168.1.12/api/v2/foo/con-cj12>
[2016-08-28 04:32:12.234] DEBUG: In execute, calling <192.168.1.12/api/v2/foo/con-cj12>
[2016-08-28 04:32:13.243] DEBUG: In hashed_metrics, calling <192.168.1.12/api/v2/foo>
[2016-08-28 04:32:13.243] DEBUG: In execute, calling <192.168.1.12/api/v2/foo>
[2016-08-28 04:32:13.249] DEBUG: In hashed_metrics, calling <192.168.1.12/api/v2/foo>
[2016-08-28 04:32:13.249] DEBUG: In execute, calling <192.168.1.12/api/v2/foo>
[2016-08-28 04:32:14.123] DEBUG: In hashed_metrics, calling <192.168.1.12/api/v2/vol>
[2016-08-28 04:32:14.123] DEBUG: In execute, calling <192.168.1.12/api/v2/vol>
[2016-08-28 04:32:14.798] DEBUG: In hashed_metrics, calling <192.168.1.12/api/v2/vol>
[2016-08-28 04:32:14.798] DEBUG: In execute, calling <192.168.1.12/api/v2/vol>
[2016-08-28 04:32:15.400] DEBUG: In hashed_metrics, calling <192.168.1.12/api/v2/vol>
[2016-08-28 04:32:15.400] DEBUG: In execute, calling <192.168.1.12/api/v2/vol>
[2016-08-28 04:32:15.989] DEBUG: In hashed_metrics, calling <192.168.1.12/api/v2/vol>
[2016-08-28 04:32:15.989] DEBUG: In execute, calling <192.168.1.12/api/v2/vol>
[2016-08-28 04:32:16.602] DEBUG: In hashed_metrics, calling <192.168.1.12/api/v2/vol>
[2016-08-28 04:32:16.602] DEBUG: In execute, calling <192.168.1.12/api/v2/vol>
...
[2016-08-28 04:34:12.680] DEBUG: In hashed_metrics, calling <192.168.1.12/api/v2/vol>
[2016-08-28 04:34:12.680] DEBUG: In execute, calling <192.168.1.12/api/v2/vol>
[2016-08-28 04:34:13.212] DEBUG: In hashed_metrics, calling <192.168.1.12/api/v2/vol>
[2016-08-28 04:34:13.212] DEBUG: In execute, calling <192.168.1.12/api/v2/vol>
[2016-08-28 04:34:13.832] DEBUG: In hashed_metrics, calling <192.168.1.12/api/v2/vol>
[2016-08-28 04:34:13.832] DEBUG: In execute, calling <192.168.1.12/api/v2/vol>
[2016-08-28 04:34:14.102] WARN: Hashed metrics fetch timed out.
[2016-08-28 04:34:15.143] DEBUG: In hashed_metrics, calling <192.168.1.12/api/v2/bar>
[2016-08-28 04:34:15.143] DEBUG: In execute, calling <192.168.1.12/api/v2/bar>
[2016-08-28 04:34:16.009] DEBUG: Data collection complete.

Answer

I think you should remove the timeout blocks you have defined (Timeout::timeout(120) do). You probably only want to timeout the HTTP request and that appears to be covered with this line.

 @factory.client_response @site[sub_url].get(accept: 'application/json', timeout: @timeout)

I don't think it is a single request that is timing out. Inside the timeout block you call general_fetch which has loop inside of it. That loop makes multiple requests and even if they are successful, the method as a whole will timeout after 120 seconds.