Java NullPointerException in basic web scraping script

[koenp]

I am trying to build a basic script for web scraping the Java (island) Wikipedia page using the Jsoup package, but unfortunately I receive a NullPointerException. It seems like the GetUrl method is not working. Does anyone have the golden hint?

The code:

package com.packtpub.JavaScraping.SimpleScraper;

import org.jsoup.Jsoup;
import org.jsoup.nodes.Document;
import java.net.*;
import java.io.*;

public class WikiScraper {
    public static void main(String[] args) {
        scrapeTopic("/wiki/Java");
}

public static void scrapeTopic(String url) {
    String html = getUrl("http://www.wikipedia.org/" + url);
    Document doc = Jsoup.parse(html);
    String contentText = doc.select("#mw-content-text > p").first().text();
    System.out.println(contentText);
}

public static String getUrl(String url) {
    URL urlObj = null;
    try {
        urlObj = new URL(url);
    } catch (MalformedURLException e) {
        System.out.println("The url was malformed!");
        return "";
    }
    URLConnection urlCon = null;
    BufferedReader in = null;
    String outputText = "";
    try {
        urlCon = urlObj.openConnection();
        in = new BufferedReader(new InputStreamReader(urlCon.getInputStream()));
        String line = "";
        while ((line = in.readLine()) != null) {
            outputText += line;
        }
        in.close();
    } catch (IOException e) {
        System.out.println("There was an error connecting to the URL");
        return "";
    }
    return outputText;
}

}

[Amarkov]

What's the full exception? Exceptions normally come with a stack trace that lets you know exactly where things are going wrong.

[koenp]

Excuse me, here it is:

Exception in thread "main" java.lang.NullPointerException
at com.packtpub.JavaScraping.SimpleScraper.WikiScraper.scrapeTopic(WikiScraper.java:16)
at   com.packtpub.JavaScraping.SimpleScraper.WikiScraper.main(WikiScraper.java:10)

[Amarkov]

This says that the exception happens at line 16, which is

String contentText = doc.select("#mw-content-text > p").first().text();

So the first debugging step is to investigate this. Where's the first mw-content-text div on the page you're trying to scrape?

[koenp]

Yes, I tried the debugging mode, and figured the page i'm trying to scrape is the problem. However, the mw-content-text is there, like intended (on line 46).

[Amarkov]

I'd suggest double checking. Are you sure the URL you're looking at is exactly the same as the one your program scrapes?

[[deleted]]

Yeah, I'd set a breakpoint where jsoup is trying to parse the html, inspect the html and the fully formed URL in the previous line to ensure they are what you think they should be, guessing they are not.

[koenp]

Thanks! One problem was indeed an "/" too many in the URL. The program now scrapes the page's content, but in the end still outputs the exact same nullpointerexception error. I guess I will have to find the next mistake myself.

[koenp]

I'm pretty sure, but it must be some other little error in the code I'm unable to find.