Parsing JSON from URL

GSON has a builder that takes a Reader object: fromJson(Reader json, Class classOfT).

This means you can create a Reader from a URL and then pass it to Gson to consume the stream and do the deserialisation.

Only three lines of relevant code.

import java.io.InputStreamReader;
import java.net.URL;
import java.util.Map;

import com.google.gson.Gson;

public class GsonFetchNetworkJson {

    public static void main(String[] ignored) throws Exception {

        URL url = new URL("https://httpbin.org/get?color=red&shape=oval");
        InputStreamReader reader = new InputStreamReader(url.openStream());
        MyDto dto = new Gson().fromJson(reader, MyDto.class);

        // using the deserialized object
        System.out.println(dto.headers);
        System.out.println(dto.args);
        System.out.println(dto.origin);
        System.out.println(dto.url);
    }
    
    private class MyDto {
        Map<String, String> headers;
        Map<String, String> args;
        String origin;
        String url;
    }
}

If you happen to get a 403 error code with an endpoint which otherwise works fine (e.g. with curl or other clients) then a possible cause could be that the endpoint expects a User-Agent header and by default Java URLConnection is not setting it. An easy fix is to add at the top of the file e.g. System.setProperty("http.agent", "Netscape 1.0");.


  1. First you need to download the URL (as text):

    private static String readUrl(String urlString) throws Exception {
        BufferedReader reader = null;
        try {
            URL url = new URL(urlString);
            reader = new BufferedReader(new InputStreamReader(url.openStream()));
            StringBuffer buffer = new StringBuffer();
            int read;
            char[] chars = new char[1024];
            while ((read = reader.read(chars)) != -1)
                buffer.append(chars, 0, read); 
    
            return buffer.toString();
        } finally {
            if (reader != null)
                reader.close();
        }
    }
    
  2. Then you need to parse it (and here you have some options).

    • GSON (full example):

      static class Item {
          String title;
          String link;
          String description;
      }
      
      static class Page {
          String title;
          String link;
          String description;
          String language;
          List<Item> items;
      }
      
      public static void main(String[] args) throws Exception {
      
          String json = readUrl("http://www.javascriptkit.com/"
                                + "dhtmltutors/javascriptkit.json");
      
          Gson gson = new Gson();        
          Page page = gson.fromJson(json, Page.class);
      
          System.out.println(page.title);
          for (Item item : page.items)
              System.out.println("    " + item.title);
      }
      

      Outputs:

      javascriptkit.com
          Document Text Resizer
          JavaScript Reference- Keyboard/ Mouse Buttons Events
          Dynamically loading an external JavaScript or CSS file
      
    • Try the java API from json.org:

      try {
          JSONObject json = new JSONObject(readUrl("..."));
      
          String title = (String) json.get("title");
          ...
      
      } catch (JSONException e) {
          e.printStackTrace();
      }
      

Tags:

Java

Url

Json