Mr. A Mr. A - 5 months ago 10
Python Question

"Do like readlines()" to a Session object (Python)

I want to pick out a few lines of information from some webpages. I want (or I am) to open the web pages, iterate down through the lines, checking each for a keyword, copying the information I want when I find it.

The pages require a session.

def getpage()
home = 'website'
exstension1 = '/input/page'
extension2 = '/output/page'
indexnumber = '11100'

sess = requests.Session()
getter = sess.get(home+extension1)
payload = {'foo':'bar','indexnumber':indexnumber}
getter = sess.post(home+extension2,data=payload)

return sess


As I tried to say in the title, I need a readlines() method for a .get()

a.get(somePage)###Now could I put...###.readlines()
####or
a.get(somePage).text.readlines()###?
###I don't think I want the following, for performance reasons, correct me if I am wrong
F = open(someNewFile,mode='w')
F.write(a.get(somePage).text)
F.close()
F = open(thatFileIJustMade).readlines()###All that just to turn it into a File on which I can use readlines?


thanks

When I try

a.get(somePage).readlines()


I get

AttributeError: Response Object Doesn't have attribute readlines

Answer

There are a few ways you can do this, but the most Requests-y way is to use a streaming request along with Response.iter_lines():

r = requests.get(somePage, stream=True)

for line in r.iter_lines(1024):
    # Do stuff on this line.