What does exactly depend on the character rarity?
There is no difference between 4* and 5* characters, asides from their gacha pull rarity.
People may mistakenly assume that 5* are more powerful than 4*. "Powerful" is an extremely subjective term because of how many different aspects characters can be viewed from, whether it be their base stats, or their relevance in the Spiral Abyss, or for overworld exploration.
5* characters have better base stats on average than 4* characters:
| Rarity | HP Avg | ATK Avg | DEF Avg |
|---|---|---|---|
| 5* | 10434.14 | 257.86 | 676.57 |
| 4* | 8911.5 | 244 | 653.71 |
I've got base numbers from this spreadsheet with all characters stats on 80 lvl, then excluded Traveller (because we cannot get it from Wishes) and did simple average (sum / # of characters).
On average difference is not that big, but let's compare some specific characters - 2 best DPS from each rarity:
| Character | ATK |
|---|---|
| 5* Diluc | 295 |
| 5* Keqing | 285 |
| 4* Fischl | 216 |
| 4* Razor | 207 |
Clearly 5* DPS beat their 4* counterparts by their most important stat.
In theory, constellations for 4* are more easily obtainable and should offset any initial stats difference, but in reality 4* character pool is bigger, meaning you will receive a lot of constellations for characters you don't play.