How can I throw error in onNext() with RxJava

The solution is to add an operator in the middle. My suggestion is to use map as it does not generate new Observable object (in comparison to flatMap which does it):

.map(new Func1<Response, Response>() {
    @Override
    public Response call(Response response) {
        if (response.isSuccess()) {
            return response;
        } else {
            throw new Throwable(reponse.getMessage()));
        }
    }
 })

you can flatMap your response to Response or Error

flatMap(new Func1<Response, Observable<Response>>() {
    @Override
    public Observable<Response> call(Response response) {
        if(response.isSuccess()){
            return Observable.just(response);
        } else {
            return Observable.error(new Throwable(response.getMessage()));
        }
    }
})

If FailureException extends RuntimeException, then

.doOnNext(response -> {
  if(!response.isSuccess())
    throw new FailureException(response.getMessage());
})
.subscribe(
    item  -> { /* handle success */ },
    error -> { /* handle failure */ }
);

This works best if you throw the exception as early as possible, as then you can do retries, alternative responses etc. easily.