Why can't I read fstream's binary data with operator>>?

The iostream extraction operator (>>) attempts to interpret numerical strings separated by whitespace, not binary data. There are many different ways to encode an unsigned integer in binary form (e.g. a 32-bit 2's complement representation in little-endian byte order). That's why you must use the read/write functions to operate on such binary buffers.

However, nothing prevents you from implementing your own class for serializing binary data in whatever form you wish using the insertion and extraction operators. Such a class would likely use the read function of an ifstream object internally. Alternatively, the boost serialization library may already hold exactly what you want.