How to add proxy support to Jsoup?

You don't have to get the webpage data through Jsoup. Here's my solution, it may not be the best though.

  URL url = new URL("http://www.example.com/");
  Proxy proxy = new Proxy(Proxy.Type.HTTP, new InetSocketAddress("127.0.0.1", 8080)); // or whatever your proxy is
  HttpURLConnection uc = (HttpURLConnection)url.openConnection(proxy);

  uc.connect();

    String line = null;
    StringBuffer tmp = new StringBuffer();
    BufferedReader in = new BufferedReader(new InputStreamReader(uc.getInputStream()));
    while ((line = in.readLine()) != null) {
      tmp.append(line);
    }

    Document doc = Jsoup.parse(String.valueOf(tmp));

And there it is. This gets the source of the html page through a proxy and then parses it with Jsoup.


You can easily set proxy

System.setProperty("http.proxyHost", "192.168.5.1");
System.setProperty("http.proxyPort", "1080");
Document doc = Jsoup.connect("www.google.com").get();

Jsoup 1.9.1 and above: (recommended approach)

// Fetch url with proxy
Document doc = Jsoup //
               .connect("http://www.example.com/") //
               .proxy("127.0.0.1", 8080) // sets a HTTP proxy
               .userAgent("Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.4; en-US; rv:1.9.2.2) Gecko/20100316 Firefox/3.6.2") //
               .header("Content-Language", "en-US") //
               .get();

You may use also the overload Jsoup#proxy which takes a Proxy class (see below).

Before Jsoup 1.9.1: (verbose approach)

// Setup proxy
Proxy proxy = new Proxy(                                      //
        Proxy.Type.HTTP,                                      //
        InetSocketAddress.createUnresolved("127.0.0.1", 8080) //
);

// Fetch url with proxy
Document doc = Jsoup //
               .connect("http://www.example.com/") //
               .proxy(proxy) //
               .userAgent("Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.4; en-US; rv:1.9.2.2) Gecko/20100316 Firefox/3.6.2") //
               .header("Content-Language", "en-US") //
               .get();

References:

  • Connection#proxy(String,int)
  • Connection#proxy(Proxy)
  • Proxy class

Tags:

Java

Jsoup