GWT.setUncaughtExceptionHandler()

I believe what's happening here is that the current JS event cycle is using the DefaultUncaughtExceptionHandler because that was the handler set at the start of the cycle. You'll need to defer further initialization to the next event cycle, like this:

public void onModuleLoad() {
    GWT.setUncaughtExceptionHandler(new ClientExceptionHandler());
    Scheduler.get().scheduleDeferred(new ScheduledCommand() {
        @Override
        public void execute() {
           startApplication();
           Window.alert("You won't see this");
        }
    });
}

private void startApplication() {
    Integer.parseInt("I_AM_NOT_A_NUMBER");
    // or any exception that results from server call
}

Update: And here's the issue that describes why this works, and why it isn't planned to be fixed.

Setting up a default handler can be a tricky proposition some times. I can tell you exactly what is going on. If you get an exception in the onModuleLoad(), the handler will not be called. It is only after the load method is completed that it will ACTUALLY get put into place.