0x00 0x00 - 1 month ago 7
Java Question

Parsing JSOUP table

I want to parse a table using jsoup. I have tried to get the flight data but without success!

My code is:

try {
doc = Jsoup.connect("a.html").timeout(13 * 1000).get();
Element table = doc.select("table.arrive-depart-table tbody tr").first();
Iterator<Element> iterator = table.select("td").iterator();
Log.d("log", iterator.next().text());
}


and this is the html:

<table class="arrive-depart-table">
<tbody>
<tr>
<td> string 1</td>
<td> string 2</td>
<td> string 3</td>
<td> string 4</td>
<td> string 5</td>
</tr>
<tr>
<td> string 6</td>
...and more


Exception:

Caused by: java.lang.NullPointerException: Attempt to invoke virtual method 'org.jsoup.select.Elements org.jsoup.nodes.Element.select(java.lang.String)' on a null object reference


I am not able to parse the table flight.

Thank you

Answer

Using the background ajax call (check: chrome dev tools -> Network tab) and JSON.simple to parse the response, it is possible to get the data without JavaScript:

Example Code

try {
    String userAgent = "Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.59 Safari/537.36";
    String url = "https://www.maltairport.com/wp-content/themes/mia/flightsinfo.php?arrivalsDepartures_action=getArrivalsDepartures";
    String referer = "https://www.maltairport.com/passenger/flights-landing/arrivals-departures/";
    String host = "www.maltairport.com";

    Document doc = Jsoup.connect(url).userAgent(userAgent).header("Host", host).header("Referer", referer).ignoreContentType(true).get();

    JSONObject jsonObject = (JSONObject) new JSONParser().parse(doc.body().text());

    JSONArray arrivals = (JSONArray) jsonObject.get("arrivals");
    JSONArray departures = (JSONArray) jsonObject.get("departures");

    System.out.println("departures");

    for (Object object : departures) {
        jsonObject = (JSONObject) object;
        System.out.println("Flight: " + jsonObject.get("flightNumber") + "\n\t" + "To: " + jsonObject.get("airportName")
                + " SCH: " + jsonObject.get("scheduledTime") + " EST: " + jsonObject.get("estimatedTime") + " Status: " + jsonObject.get("remarks"));
    }

    System.out.println("\narrivals");

    for (Object object : arrivals) {
        jsonObject = (JSONObject) object;
        System.out.println("Flight: " + jsonObject.get("flightNumber") + "\n\t" + "To: " + jsonObject.get("airportName")
                + " SCH: " + jsonObject.get("scheduledTime") + " EST: " + jsonObject.get("estimatedTime") + " Status: " + jsonObject.get("remarks"));
    }

} catch (IOException | ParseException e) {
    e.printStackTrace();
}

Truncated Output

departures
Flight: FR 7243
    To: DUBLIN SCH: 15:15 EST: 15:36 Status: AIRBORNE
Flight: LS 650
    To: EAST MIDLANDS SCH: 15:15 EST: 15:33 Status: AIRBORNE    
[...]

arrivals
Flight: KL 3399
    To: AMSTERDAM SCH: 14:50 EST: 14:56 Status: LANDED
Flight: KM 395
    To: AMSTERDAM SCH: 14:50 EST: 14:56 Status: LANDED
[...]
Comments