How to counter the effects of aggressive expansion penalty?

You are indeed right that aggressive expansion (AE) is a major obstacle to military expansion: it is what brings balance to the game and allows smaller nations to become successful and beat much more powerful rivals. Otherwise, superpowers can grow unchecked which makes for rather uninteresting games.

At any rate, there are other ways to go:

  • Diplomatic vassalization. This is typically the preferred strategy bar none, if your regional situation allows it. You placate, bribe, ask, threaten your targets into accepting vassalization peacefully.

    Preferably, most of the conditions below need to be fulfilled for it to be effective, or indeed possible:

    • Monarchy type government, as most republics (except Dutch and Noble Republic) cannot enter into royal marriages.
    • Same religion as your intended targets, or at the very least, same religious group.
    • Region with a large number of small countries (Indian subcontinent and Central Europe are prime examples), while you are moderately large or the 800-pound gorilla in the room.
    • Preferably, some diplomatic bonuses from your country's ideas (extra relations and more importantly, diplomatic reputation).
    • Works best if you rarely, if ever, take territory directly in a military conflict.

    The 190 reputation needed can be a challenge to build, particularly if you share a land border with them, but there are many ways to increase your standing with them. Diplomatic vassalization is only possible to propose to countries that have less than 40 total tax base (your own tax base needs to be significantly higher than theirs).

  • Military vassalization. Reserved for countries that just won't see eye to eye to you in terms of your global domination and need to be persuaded gently with a stick, shooting or other. Most of the restrictions applying to diplomatic vassalization are not applicable here: you can vassalize any sufficiently advanced country with any form of government or religion, provided they are small enough to have province score in a war less than 100%.

    Some tips:

    • Fabricate claims to as many provinces as possible: this reduces warscore cost and can mean the difference between success and failure to vassalize the target wholesome.
    • Preferentially look for vassals with many uncontested cores (perhaps lost in previous wars). You cannot wage war for them with other countries, but you can demand they are returned to your vassals as a part of a peace treaty.
    • Full annexation and immediate release of a vassal will wipe any accumulated AE and other negative modifiers, and will spawn them with your state religion.
    • Some casus belli (CBs) require smaller warscore to offer subjugation.

The diplomatic game can be much more satisfying and rewarding in EUIV, and can bring immense variety and depth to the game.

Further information with more strategic and tactical tips is available on EU4wiki:

  • Vassals
  • Casus belli

In addition to what mindcorrosive said, AE is 'proximity' based. That is, if as France you get some stuff from Aragon, Castille and maybe Grenada and Morocco might get upset, but not Burgundy or Milan. So alternate your conquests, while one side of your borders is cooling off, eat from another side. Specially good for large (in area) countries like Muscovy.