Warlock Warlock - 1 year ago 339
Java Question

How to add proxy support to Jsoup (HTML parser)?

I am a newbie to Java and my first task is to parse some 10,000 urls and extract some info outta it, for this I am using Jsoup and its working fine. But now I want to add proxy support to it. The Proxies have a username and password too. Can any1 help me with this.

Answer Source

You don't have to get the webpage data through Jsoup. Here's my solution, it may not be the best though.

  URL url = new URL("http://www.example.com/");
  Proxy proxy = new Proxy(Proxy.Type.HTTP, new InetSocketAddress("", 8080)); // or whatever your proxy is
  HttpURLConnection uc = (HttpURLConnection)url.openConnection(proxy);


    String line = null;
    StringBuffer tmp = new StringBuffer();
    BufferedReader in = new BufferedReader(new InputStreamReader(uc.getInputStream()));
    while ((line = in.readLine()) != null) {

    Document doc = Jsoup.parse(String.valueOf(tmp));

And there it is. This gets the source of the html page through a proxy and then parses it with Jsoup.

Recommended from our users: Dynamic Network Monitoring from WhatsUp Gold from IPSwitch. Free Download