Why doesn't the heat of the Earth's core diffuse to the surface?

It's a bit like when you put a thick jumper on. The inside of your clothing ends up being warmer than the outside of your clothing.

Most of the heat within the earth can be attributed to radioactive decay (of long lived isotopes like potassium). This heat is constantly being conducted out to the surface. (Yes, if you go down into a deep mine, you will get hotter.) It turns out that kilometres of rock works as a reasonably good insulator.

Remember that the difference in temperature affects how quickly heat is transferred. If the surface was nearly as hot as the interior (like when it originally formed) then the surface would radiate heat into the cold night sky much faster, and the crust would conduct internal heat away from the core to the surface even slower, and this imbalance would cause the surface to lose net thermal energy and cool down (while the core heats up even further); this process continues until an equilibrium is reached (where each layer of the earth has its own roughly stable temperature, and each layer is getting rid of excess thermal energy at just the same rate as it acquires it).