How to force Java to throw arithmetic exception?

I think, you'll have to check manually, e.g.

public double void checkValue(double val) throws ArithmeticException {
    if (Double.isInfinite(val) || Double.isNaN(val))
        throw new ArithmeticException("illegal double value: " + val);
    else
        return val;
}

So for your example

double d = checkValue(b*b - 4*a*c);
double x1 = checkValue((-b - Math.sqrt(d)) / 2 / a);
double x2 = checkValue((-b + Math.sqrt(d)) / 2 / a);

It's not possible to make Java throw exceptions for these operations. Java implements the IEEE 754 standard for floating point arithmetic, which mandates that these operations should return specific bit patterns with the meaning "Not a Number" or "Infinity". Unfortunately, Java does not implement the user-accessible status flags or trap handlers that the standard describes for invalid operations.

If you want to treat these cases specially, you can compare the results with the corresponding constants like Double.POSITIVE_INFINITY (for NaN you have to use the isNAN() method because NaN != NaN). Note that you do not have to check after each individual operation since subsequent operations will keep the NaN or Infinity value. Just check the end result.