Web scraping with Java

Your best bet is to use Selenium Web Driver since it

  1. Provides visual feedback to the coder (see your scraping in action, see where it stops)

  2. Accurate and Consistent as it directly controls the browser you use.

  3. Slow. Doesn't hit web pages like HtmlUnit does but sometimes you don't want to hit too fast.

    Htmlunit is fast but is horrible at handling Javascript and AJAX.


HTMLUnit can be used to do web scraping, it supports invoking pages, filling & submitting forms. I have used this in my project. It is good java library for web scraping. read here for more


jsoup

Extracting the title is not difficult, and you have many options, search here on Stack Overflow for "Java HTML parsers". One of them is Jsoup.

You can navigate the page using DOM if you know the page structure, see http://jsoup.org/cookbook/extracting-data/dom-navigation

It's a good library and I've used it in my last projects.