Java Lucene NGramTokenizer

I don't think you'll find what you're looking for trying to find methods returning String. You'll need to deal with Attributes.

Should work something like:

Reader reader = new StringReader("This is a test string");
NGramTokenizer gramTokenizer = new NGramTokenizer(reader, 1, 3);
CharTermAttribute charTermAttribute = gramTokenizer.addAttribute(CharTermAttribute.class);
gramTokenizer.reset();

while (gramTokenizer.incrementToken()) {
    String token = charTermAttribute.toString();
    //Do something
}
gramTokenizer.end();
gramTokenizer.close();

Be sure to reset() the Tokenizer it if it needs to be reused after that, though.

---

Tokenizing grouping of words, rather than chars, per comments:

Reader reader = new StringReader("This is a test string");
TokenStream tokenizer = new StandardTokenizer(Version.LUCENE_36, reader);
tokenizer = new ShingleFilter(tokenizer, 1, 3);
CharTermAttribute charTermAttribute = tokenizer.addAttribute(CharTermAttribute.class);

while (tokenizer.incrementToken()) {
    String token = charTermAttribute.toString();
    //Do something
}

For recent version of Lucene (4.2.1), this is a clean code which works. Before executing this code, you have to import 2 jar files:

• lucene-core-4.2.1.jar
• lucene-analuzers-common-4.2.1.jar

Find these files at http://www.apache.org/dyn/closer.cgi/lucene/java/4.2.1

//LUCENE 4.2.1
Reader reader = new StringReader("This is a test string");      
NGramTokenizer gramTokenizer = new NGramTokenizer(reader, 1, 3);

CharTermAttribute charTermAttribute = gramTokenizer.addAttribute(CharTermAttribute.class);

while (gramTokenizer.incrementToken()) {
    String token = charTermAttribute.toString();
    System.out.println(token);
}