Which hard mathematical problems do you have to solve to earn bitcoins ?

Bitcoin mining is based on hash functions. Specifically the SHA-256 hash function, which maps arbitrary bit strings to 256-bit outputs in such a way that nobody knows how to find a collision (two inputs with the same output), although the pigeonhole principle implies collisions exist. Bitcoin mining doesn't involve finding collisions, which would be way too hard. Instead, one has to find inputs that lead to outputs with special properties, namely a lot of consecutive zeros. This is a scaled-down version of inverting the hash function. Of course there's no proof that any of this is actually computationally difficult, and some earlier hash functions have turned out to be weaker than expected (for example, MD5 and SHA-1), but it certainly seems to be.

SHA-256 is not a nice or simple function - it was designed to be hard to analyze - so I'd say this is a devilish problem.

Another way to earn bitcoin is not to mine them, but to have them wired to you from other bitcoin accounts. The transactions are signed using ECDSA. So if you solve the discrete logarithm problem in an elliptic curve (for bitcoin this is the curve Secp256k1), then you potentially can earn many bitcoins (illegally of course...)

Personally I find this problem much more beautiful than finding lot of zeroes in the SHA-256 hash function.