Does a gym's prestige change if you defeat only some of the opposing Pokemon?

The gyms prestige will go down if you can only defeat some of the Pokémon, but not as much as if you can defeat all of the Pokémon. If you can keep taking down a few Pokémon each time you attack a gym, you will continue to take away some of the prestige each time.

source- personal experience from taking down level 9+ gyms.

Prestige of gym will go down if you defeat at least one defending pokemon, e.g. the first in line.

Thus, stack your team for the fight accordingly - be sure your line up will beat the first.

Note; the autogenerated suggestion is not always the best line up.

Depending on the CP of your pokemon's vs the defending pokemon the amount of prestige lost and XP won will vary.