What are the downsides to rebasing topic branches instead of merging?

The main disadvantage with using rebase is that you should (read must) only use rebase locally. That is to say, once something has been pushed DON'T rebase it after that. Rewriting history is fine and dandy, but the instant you start screwing around with the history on a remote, things get really messy.

EDIT

Another disadvantage is the fact that the true history is in fact lost. This means, among other things, that it's impossible to go back to the topic branch (easily) since it would have the appearance of being a part of the main branch. It also makes reverting changes much more painful, because you have to cherry-pick one commit at a time, trying your best to remember which ones came from the original topic branch. If you use an interactive rebase, or worse -- squashed your commits down -- then this could be an enormous headache.

Disadvantages include revisions not matching their actual state during development, losing the ability to hide the commits between merges using flags like --first-parent, and being difficult to share your branch if you ever intend it to be rebased later. If your log is looking crazy, take a look at git help log again. Chances are, there is a combination of flags that will make it look just how you want, without sacrificing flexibility when it's desired.