etayluz etayluz - 5 months ago 19
Ruby Question

RuntimeError (Circular dependency detected while autoloading constant Apps multithreading

I'm receiving this error:
RuntimeError (Circular dependency detected while autoloading constant Apps

when I'm multithreading. Here is my code below. Why is this happening?
The reason I am trying to multithread is because I am writing a HTML scraping app.
The call to Nokogiri::HTML(open()) is a synchronous blocking call that takes 1 second to return, and I have 100,000+ pages to visit, so I am trying to run several threads to overcome this issue. Is there a better way of doing this?

class ToolsController < ApplicationController

def getWebsites
t1=Thread.new{func1()}
t2=Thread.new{func1()}
t3=Thread.new{func1()}
t4=Thread.new{func1()}
t5=Thread.new{func1()}
t6=Thread.new{func1()}
t1.join
t2.join
t3.join
t4.join
t5.join
t6.join
end

def func1
puts Thread.current
apps = Apps.order("RANDOM()").where("apps.website IS NULL").take(1)
while apps.size == 1 do
app = apps[0]
puts app.name
puts app.iTunes
doc = Nokogiri::HTML(open(app.iTunes))
array = doc.css('div.app-links a').map { |link|
url = link['href']
url = Domainatrix.parse(url)
url.domain + "." + url.public_suffix
}
array.uniq!
if (array.size > 0)
app.website = array.join(', ')
puts app.website
else
app.website = "NONE"
end
app.save
apps = Apps.order("RANDOM()").where("apps.website IS NULL").take(1)
end
end


end

Answer

"require" isn't thread-safe

Change your methods so that everything that is to be "required" is done so before the threads start.

For example:

def get_websites
    # values = Apps.all # try uncommenting this line if a second-try is required

    ar = Apps.where("apps.website IS NULL")

    t1 = Thread.new{ func1(ar) }
    t2 = Thread.new{ func1(ar) }

    t1.join
    t2.join
end

def func1( ar )
    apps = ar.order("RANDOM()").limit(1)

    while (apps.size == 1)
      puts Thread.current
    end 
end

But as somebody pointed out, the way you're multithreading within the controller isn't advised.