Cmag Cmag - 10 months ago 59
Javascript Question

Options to parse a JavaScript web app in Python

Have to write a Python app to communicate and fetch some data from a password protected, JavaScript enabled web app.

The webapp unfortunately requires JavaScript to run, and refuses to work if JavaScript is disabled. Therefore, 'urllib2' and 'requests' don't work when trying to fetch data.

What are my options at this point? If I grabbed HTTP traffic via some tool such as Firebug, and tried re-playing this traffic via some Python method, would this work? All I need for the script to do is:

  1. Login into website

  2. Change a dropdown item

  3. Grab a single line of text and logout

Answer Source

Your best bet is to run a headless browser like PyPhantomJS which understands javascript. You have to write your code in javascript and than you can do whatever you want.

It has fast and native support for DOM handling, CSS selector and of course JavaScript.