Warlock Warlock - 9 months ago 240
Java Question

How to add proxy support to Jsoup (HTML parser)?

I am a newbie to Java and my first task is to parse some 10,000 urls and extract some info outta it, for this I am using Jsoup and its working fine. But now I want to add proxy support to it. The Proxies have a username and password too. Can any1 help me with this.


You don't have to get the webpage data through Jsoup. Here's my solution, it may not be the best though.

  URL url = new URL("http://www.example.com/");
  Proxy proxy = new Proxy(Proxy.Type.HTTP, new InetSocketAddress("", 8080)); // or whatever your proxy is
  HttpURLConnection uc = (HttpURLConnection)url.openConnection(proxy);


    String line = null;
    StringBuffer tmp = new StringBuffer();
    BufferedReader in = new BufferedReader(new InputStreamReader(uc.getInputStream()));
    while ((line = in.readLine()) != null) {

    Document doc = Jsoup.parse(String.valueOf(tmp));

And there it is. This gets the source of the html page through a proxy and then parses it with Jsoup.