Tomer Shemesh Tomer Shemesh - 4 months ago 28
Ruby Question

Rails Circular Dependency Errors in rake task only when multithreading without delete_all

I am trying to create a rake task to do some updating of data. the problem is it is kind of slow so i thought i would make it multi threaded to speed it up. the problem is when i run it in 1 thread its fine but when i run it in 5 threads i only sometimes get

rake aborted!
Circular dependency detected while autoloading constant ClassInfo
/home/Tomer/.rvm/gems/ruby-2.2.2/gems/activesupport-4.2.3/lib/active_support/dependencies.rb:492:in `load_missing_constant'
/home/Tomer/.rvm/gems/ruby-2.2.2/gems/activesupport-4.2.3/lib/active_support/dependencies.rb:184:in `const_missing'
/home/Tomer/.rvm/gems/ruby-2.2.2/gems/activesupport-4.2.3/lib/active_support/dependencies.rb:526:in `load_missing_constant'
/home/Tomer/.rvm/gems/ruby-2.2.2/gems/activesupport-4.2.3/lib/active_support/dependencies.rb:184:in `const_missing'
/home/Tomer/workspace/DrexelRegistrationBackend/app/helpers/html_parse_helper.rb:66:in `block in load_class_from_link'
/home/Tomer/.rvm/gems/ruby-2.2.2/gems/nokogiri-1.6.6.2/lib/nokogiri/xml/node_set.rb:187:in `block in each'
/home/Tomer/.rvm/gems/ruby-2.2.2/gems/nokogiri-1.6.6.2/lib/nokogiri/xml/node_set.rb:186:in `upto'
/home/Tomer/.rvm/gems/ruby-2.2.2/gems/nokogiri-1.6.6.2/lib/nokogiri/xml/node_set.rb:186:in `each'
/home/Tomer/workspace/DrexelRegistrationBackend/app/helpers/html_parse_helper.rb:59:in `load_class_from_link'
/home/Tomer/workspace/DrexelRegistrationBackend/app/helpers/html_parse_helper.rb:43:in `block (3 levels) in load_classes_from_catalog'
/home/Tomer/workspace/DrexelRegistrationBackend/app/helpers/html_parse_helper.rb:42:in `each'
/home/Tomer/workspace/DrexelRegistrationBackend/app/helpers/html_parse_helper.rb:42:in `block (2 levels) in load_classes_from_catalog'


I am using nokogiri and opening a url and html parsing it and then inserting it into my database. The weird thing is if i insert ClassInfo.delete_all in the beginning the error goes away, but i dont want to do that. I need to check if its in my database so i check if it exists in this line which is where i crashes

myclass = ClassInfo.find_by(class_id: class_id)


It has no joins or other classes dependent on it this is my migrate

create_table :class_infos do |t|
t.text :class_id, index: true
t.text :subject_code
t.text :subject
t.text :course_number
t.text :title
t.float :credits_lower_range
t.float :credits_upper_range
t.text :description
t.boolean :writing_intensive
t.text :college
t.text :repeat_status
t.text :prereqs
t.text :restrictions
t.text :coreqs
t.text :term_type
t.text :student_type
t.timestamps null: false
end


and this is the model

class ClassInfo < ActiveRecord::Base
end


the very weird thing is that after it crashes i can usually run the rake task again immediately and it works fine the second time. Very weird

Here is mostly my entire module called from my rake

require 'open-uri'

module HtmlParseHelper

def self.parse_catalog
Thread.abort_on_exception=true
@@threads = Array.new
load_classes_from_catalog(url_sections) #an array of 5 arrays contaning urls objects
@@threads.each { |thr| thr.join }
end


def self.load_classes_from_catalog(urlArr)
arrs.each{|urlSection|
@@threads.push(
Thread.new do
urlSection.each{|url|
load_class_from_link(url[0],url[1],url[2],url[3])
}
ActiveRecord::Base.connection.close
end
)
}
end


def self.load_class_from_link(link, term_type, student_type, subject)

begin
doc = Nokogiri::HTML(open(link))do |config|
config.strict.noblanks
end
class_id = doc.css('div.courseblock').children[0].text
drexel_class = ClassInfo.find_by(class_id: class_id) #where it crashes
#.....
drexel_class.save
end
end

Answer

https://github.com/rails/rails/commit/112077c255879351edf4530791cc4bcc7bd4005b

int the end it was eager loading that was screwing me up.

i just added

Rails.application.eager_load!

to the beginning and it worked