TheoretiCAL TheoretiCAL - 26 days ago 10
Python Question

Python httplib2 access RateMyProfessor search results

I'm trying to programatically access RateMyProfessor page responses that contain search results but can't come up with a correct query.

The form submits to http://www.ratemyprofessors.com/search.jsp?
but has an on submit javascript such that it changes to http://www.ratemyprofessors.com/search.jsp?query=&queryoption=TEACHER&search_submit=Search.

I've tried:

h.request("http://www.ratemyprofessors.com/search.jsp?query=&queryoption=TEACHER&search_submit=Search","POST",urllib.urlencode({'Search':''}))


Just to see if the response contains a list of names, but no matter what I do, the table containing search results (id='doc_table') never contains any of the results.

Would anyone know how to use httplib2 to correctly send a search request to ratemyprofessors?

Answer Source

It is an option to just use their private API:

http://www.ratemyprofessors.com/solr/interim.jsp?select?facet=true&q=Edmund&facet.field=schoolname_s&facet.field=teacherdepartment_s&facet.field=schoolcountry_s&facet.field=schoolstate_s&facet.limit=50&rows=20&facet.mincount=1&json.nl=map&fq=content_type_s%3ATEACHER&wt=json

You can access the response data with:

    import ast
    resp, content = h.request("http://www.ratemyprofessors.com/solr/interim.jsp?select?facet=true&q=Christopher+Callam&facet.field=schoolname_s&facet.field=teacherdepartment_s&facet.field=schoolcountry_s&facet.field=schoolstate_s&facet.limit=50&rows=20&facet.mincount=1&json.nl=map&fq=content_type_s%3ATEACHER&wt=json")    

    data = ast.literal_eval(content)
    results = data['response']['docs']

Filtering by school:

http://www.ratemyprofessors.com/solr/interim.jsp?select?facet=true&q=johnson&facet.field=schoolname_s&facet.field=teacherdepartment_s&facet.field=schoolcountry_s&facet.field=schoolstate_s&facet.limit=50&rows=20&facet.mincount=1&json.nl=map&fq=content_type_s:TEACHER&fq=schoolname_s:"collin college"&wt=json

In this example I'm searching for Johnson from collin college.