Problems casting NAN floats to int

First, a NAN is everything not considered a float number according to the IEEE standard. So it can be several things. In the compiler I work with there is NAN and -NAN, so it's not about only one value.

Second, every compiler has its isnan set of functions to test for this case, so the programmer doesn't have to deal with the bits himself. To summarize, I don't think peeking at the value makes any difference. You might peek the value to see its IEEE construction, like sign, mantissa and exponent, but, again, each compiler gives its own functions (or better say, library) to deal with it.

I do have more to say about your testing, however.

float h = NAN;
printf("%x %d\n", (int)h, (int)h);

The casting you did trucates the float for converting it to an int. If you want to get the integer represented by the float, do the following

printf("%x %d\n", *(int *)&h, *(int *)&h);

That is, you take the address of the float, then refer to it as a pointer to int, and eventually take the int value. This way the bit representation is preserved.

There is a reason for this behavior, but it is not something you should usually rely on.

As you note, IEEE-754 does not specify what happens when you convert a floating-point NaN to an integer, except that it should raise an invalid operation exception, which your compiler probably ignores. The C standard says the behavior is undefined, which means not only do you not know what integer result you will get, you do not know what your program will do at all; the standard allows the program to abort or get crazy results or do anything. You probably executed this program on an Intel processor, and your compiler probably did the conversion using one of the built-in instructions. Intel specifies instruction behavior very carefully, and the behavior for converting a floating-point NaN to a 32-bit integer is to return 0x80000000, regardless of the payload of the NaN, which is what you observed.

Because Intel specifies the instruction behavior, you can rely on it if you know the instruction used. However, since the compiler does not provide such guarantees to you, you cannot rely on this instruction being used.

The result of a cast of a floating point number to an integer is undefined/unspecified for values not in the range of the integer variable (±1 for truncation).

Clause 6.3.1.4:

When a finite value of real floating type is converted to an integer type other than _Bool, the fractional part is discarded (i.e., the value is truncated toward zero). If the value of the integral part cannot be represented by the integer type, the behavior is undefined.

If the implementation defines __STDC_IEC_559__, then for conversions from a floating-point type to an integer type other than _BOOL:

if the floating value is infinite or NaN or if the integral part of the floating value exceeds the range of the integer type, then the "invalid" floating- point exception is raised and the resulting value is unspecified.

(Annex F [normative], point 4.)

If the implementation doesn't define __STDC_IEC_559__, then all bets are off.