Google Server for Webfont's or Self-Hosted?
My biggest personal motivation for not using Google's webfont hosting is that it happens to be blocked in China. If loading in China is at all important to you, it's definitely worth it to self-host.
Keep them on Google's servers.
The reason why you're seeing the small file size from Google is because your browser has already downloaded and cached the file. When you uploaded to your own server, your browser had to download it once again as it considers it a new source.
This fact is also one of the reasons you should reference the fonts from Google, as visitors to your site may have already downloaded them from another site that uses the same font from Google.
Hosting them yourself would naturally put additional load on your servers. Having Google host them lets them deal with that load.
Moreover, by referencing from Google you are letting Google handle the maintenance, such as updates to the fonts as well as the syntax that serves the fonts themselves.
Well, here's another take: Host them on your server!
Sleep better at night, especially if you're a web developer.
Reasons.
It's more robust especially for visitors with slower/less-reliable connections. A visitor might resolve your domain but fail to resolve Google's and hence ends up with a lot of latency or without the font file, which is essential of course.
It's safer. Remember Google Reader? Google took it down completely. Google Apps? Google made it paid with no free plans, even tiny free plans were not given. So basically, you never know when Google Fonts will get shut down or made for pay. If you're a web developer, this gets multiplied by the number of websites you develop. Would you like to handle 10 or 20 or more angry clients in a few years when that happens?
Yes it's more work to self-host, but I think it's worth it. Especially for Point #2.
Cheers.