Konstantin Rusanov Konstantin Rusanov - 4 months ago 21
Python Question

Python flask requests load page

Create script which loads page from my server (with server IP etc.) - all OK, but if I want to click on any link I landed to 404 error page, because link is - some thing like this:


...37.139.17.81:5000/html/privacy-check.php


My code:

from flask import Flask
import requests

application = Flask(__name__)

@application.route("/")
def hello():
result = requests.get("http://ipinfo.info/index.php")
return result.content

if __name__ == "__main__":
application.run(host='0.0.0.0')


Here's a live example :


http://37.139.17.81:5000/


How I can parse click URL and get this link content?

Answer

You are basically trying to make your page act like a proxy for the remote page. In order to do this completely, you need to handle all links in the remote page.

So for example if there is a link like /something/something flask will automatically try to match it with a local url (http://yourserver.com/something/something). Given the fact that you only define a single route ("/") the app will determine that any other page does not exist and will return a 404.

To handle this correctly you can try the following:

import urlparse

@application.route("/")
@application.route("/<url:path>")
def hello(url=None):
    baseurl = "http://ipinfo.info/"
    if not url:
        result = requests.get(urlparse.urljoin(baseurl,"index.php"))
        return result.content
    else:
        result = requests.get(urlparse.urljoin(baseurl,url))
        return result.content

A word of warning: this approach might break in various cases (css and js loading for example) so you might need to check the results after the page loads.