Priyansh Jain Priyansh Jain - 5 months ago 48
Python Question

How to make a POST request in python wait for other request to complete in requests library?

I want to scrape data from a web page using the requests library. Let's assume I send a POST request 'A' and retrieve a response. Here is the request I use for sending:

da_page =
headers = headers,
data = {
'classId' : course['class_no'],
'courseCode' : course['course_code'],
'slot' : course['slot'],
'fName' : course['faculty_name'],
'type' : course['course_type'],
'title' : course['course_title'],
'option' : course['course_option']},
verify = False,
# stream = True

What happens on the website is that the server code blocks the UI with another request 'B' that has a loading GIF as the response, and once the response is received, it unblocks it.

The js function for the above is as follows:

function myFunction(classId,courseCode,title,type,option,slot,fName) {
var myform = document.getElementById("digitalAssignment");
var fd = new FormData(myform);

// where the ui is blocked
message : '<img src="assets/img/482.GIF"> loading... Just a moment...'

params = "&classId=" + classId
+ "&courseCode=" + courseCode
+ "&title=" + title
+ "&type=" + type
+ "&option=" + option
+ "&slot="+ slot
+ "&fName=" + fName;
url : "examinations/processDigitalAssignment",
type : "POST",
data : "params=" + params,
// where it is unblocked
success : function(response) {



What I infer from this is that the page content is loaded only after the UI is unblocked, which only happens when the request 'B' is completed. The response I get from the former python code is that of the BLOCKED UI, and what I want is the content that loads after the UI is unblocked.

Any ideas on how to proceed? I've tried using stream but with no luck.

Answer Source

Okay, since the web page is a js-rendered page, and since the requests library immediately receives the response, it simply does not support loading dynamic js-rendered web page responses. Hence, the usage of a headless browser like selenium or a library like dryscrape will be of use in this.

EDIT: what @SamHartman said is right, the problem was resolved by sending a request to the page encountered before the page I want to work on, it's probably a server related thing where they keep track of the URLs.

Recommended from our users: Dynamic Network Monitoring from WhatsUp Gold from IPSwitch. Free Download