How is heap size calculated?

Salesforce uses SI megabytes in all of their documentation, not SI mebibytes. This is confusing for Windows users and hardware programmers, who are used to seeing mebibytes called megabytes, and may not even know what a mebibyte is. The difference is that the SI system is a decimal-based system, while the other classic nomenclature is a binary-based system. This means that a string that is 5,110,400 is 5.11 MB, not 4.87 MB. You'll notice that 10.22 MB happens to be exactly double 5.11 MB, which is where you're getting that number from.

However, there's a twist: strings count all Unicode characters under 0x10000 as one "byte", despite them clearly being words (two bytes), and everything 0x10000 or higher as two bytes, despite taking up to 4 bytes of actual memory. So, depending on the contents of your file, your heap may show 10.22 MB while your actual string size may be as large as 20.44 MB, which would exceed the 15 MB (15,000,000 byte) limit.

Also, I believe the response is actually base64 encoded, which would increase the total response size by another 4/3, so 10.22 MB would be closer to 13.63 MB. Realistically, the heap size reported is usually a lot smaller than the actual amount of data that will be transferred over the wire.

You have more in the heap than just file data. Heap can be tricky to measure anyway because of garbage collection but you can see how much data you added by taking a diff:

Integer startingHeap = Limits.getHeapSize();
// add data to heap
Integer addedHeap = Limits.getHeapSize() - startingHeap;