What advantage do I get by evolving my Pokemon?
The advantage gained is improved stats. The evolved form of a Pokemon has better stats than its earlier forms. However, when you evolve your Pokemon their stats are recalculated from level 1. So your fully evolved Pokemon will have the same stats at level 100 regardless of which level you evolve it at.
There are two disadvantages to evolving a Pokemon though. The first one is pretty minor, and is that a Pokemon takes longer to learn moves in an evolved form (as you noted in your question). The other disadvantage is fairly significant though, and only applies to Pokemon which evolve with an evolution stone. The majority of Pokemon which evolve using a stone won't learn any moves when evolved, so you don't want to evolve those Pokemon until they've learned all their possible moves. Evolving them earlier will really, really limit their usefulness.
The evolved form of a Pokémon always has better (or equally distributed *) base stats than its pre-evolved form (with the exception of Shedninja, because of its 1HP stat). Essentially, your attacks will be dealing more damage, and you will be able to take more damage.
The only reason you would want to stop your Pokémon from evolving is if it learns a move that its evolution doesn't learn (for example, Growlithe learns Flare Blitz but Arcanine doesn't, so you may not want to use a Fire Stone on it right when you acquire one).
* Scizor is interesting in that it has the same total base stats as Scyther – when evolving, Scyther's speed gets reduced, but the points removed go to Scizor's attack and defense stats.
There are two principle reasons to evolve your pokemon.
The first is base stats. Almost without exception, an evolved pokemon's base statistics, that is to say, ATTACK, SP. ATTACK, DEFENSE, SP. DEFENSE, SPEED, and HP are higher than its previous form. This translates to an immediate boost in battling abilities.
The second is that pokemon movepools often expand with evolution - the pokemon is able to learn abilities it would otherwise not be able to (often related to its new type, if applicable). While generally speaking, non-evolved pokemon learn attacks earlier than their evolved counterparts, it is not always true, and depending on the individual scenario, there may be no reason to delay evolution.
Additionally, if the move in question is learn-able via TM or Move Tutor, it can be unnecessary to wait until a certain level in the first place - you're just missing out on the higher stats at that point.