What limits a motherboard from supporting more memory?
In short, this is a mixture of limitations. One is in how many "address lines" the CPU itself supports (different from the 64/32-bit address space/register size), 32 lines would allow 4GiB of RAM, having a 33rd line connected would allow 8GiB of ram and so on.
The other problem is in how many of those address lines the manufacturer actually bothers to wire up to the memory controller.
In order to simplify the design the manufacturers tend to decide on a current realistic amount of memory and wire up as many address lines as are needed to support that amount of memory.
Routing all those address lines on a PCB is painstaking work as all the track lengths have to be as near identical as possible (as at the high frequencies that these things now operate at having a slightly different length can mean that data on one line arrives at a different time to the data sent on another line even though the sender sent them both at the same time) and so manufacturers will keep the amount of work as small as possible and so supporting 65TiB of RAM is nowhere near the same priority as getting the motherboard out this year.
Disclaimer: This is my understanding, I may be wrong, but I have been around the block a bit.
In my experience, the people who write the final "specifications" that customers get to read, are often not necessarily involved with the actual design of the board. As such, you CAN run into occasions where the docs say things like "4GB max supported memory". I've found on many occasions that I can cram more memory on a board than the docs claim is supported, and have things actually work out rather well.
My advice to you is to try it. There's a store I like (just in my small city) that'd let me buy RAM and return it, so I actually could just try it, maybe you can find a store by you that's similarly awesome.
There obviously are various architectural limits of how much memory a machine can address. The most obvious one is whether or not the platform supports 64bit operation - if so, then the addressable memory limit will be fairly massive (~4PB for x86). I think it's safe to assume your board supports 64bit. So then it would come to things like Tomas mentioned, like the memory controller within the CPU. As Tomas also said, it makes sense that BIOS may play a role here too. You should therefore check your motherboard's product page to see if they had any BIOS updates that yap about memory limits.
At the end of the day, there may be various things that limit how much memory your machine will be able to use. We're talking about several variables. And, in my opinion, the best & fastest way to know is to just try it (and run MemTest86 to verify).
Take care & good luck.
There can be other issues, a friend was doing QA on Apple hardware, the system was certified for 768MB max ram (12 64MB modules), in theory, you could double that, but when they were certifying the system, getting 12 128MB was not practical, so it was not certified in that configuration, but it should work.