Konstantin Rusanov Konstantin Rusanov - 1 year ago 146
Python Question

Python flask requests load page

Create script which loads page from my server (with server IP etc.) - all OK, but if I want to click on any link I landed to 404 error page, because link is - some thing like this:


My code:

from flask import Flask
import requests

application = Flask(__name__)

def hello():
result = requests.get("http://ipinfo.info/index.php")
return result.content

if __name__ == "__main__":

Here's a live example :

How I can parse click URL and get this link content?

Answer Source

You are basically trying to make your page act like a proxy for the remote page. In order to do this completely, you need to handle all links in the remote page.

So for example if there is a link like /something/something flask will automatically try to match it with a local url (http://yourserver.com/something/something). Given the fact that you only define a single route ("/") the app will determine that any other page does not exist and will return a 404.

To handle this correctly you can try the following:

import urlparse

def hello(url=None):
    baseurl = "http://ipinfo.info/"
    if not url:
        result = requests.get(urlparse.urljoin(baseurl,"index.php"))
        return result.content
        result = requests.get(urlparse.urljoin(baseurl,url))
        return result.content

A word of warning: this approach might break in various cases (css and js loading for example) so you might need to check the results after the page loads.

Recommended from our users: Dynamic Network Monitoring from WhatsUp Gold from IPSwitch. Free Download