How does one properly utilize Gyms?

Freekvd's answer covers the basics of gyms, but I think leaves out some important details with regards to Gym mechanics, I have tried to be more thorough in this answer.

Gyms are the only locations (currently) where Pokémon battles can occur. When you are battling at a Pokémon Gym, you are either battling an opposing team, using a full roster of six Pokémon that you have selected, or battling ("training against") your own team, using a single Pokémon that you have selected.

Combat itself is relatively straightforward - single tap repeatedly to perform your Pokémon's basic attack and to charge up your secondary attack, and then long-tap (hold down) to use your secondary attack once it is charged. You can also swipe to the left or right to dodge the enemy's attack, although this is a bit finnicky and the timing can be a bit tricky to get right. Beyond this, the results of a Pokémon battle differ depending on the two types of gyms:

Enemy Gyms

These are gyms that are currently controlled by players who are not on the same team as you (i.e. Instinct (Yellow), Mystic (Blue), or Valor (Red)). When you engage in battle against these gyms, you must pre-select a roster of six Pokémon to fight with. Then you will fight the Pokémon in the gym one by one from weakest to strongest until you have beaten them all (or not). For every enemy Pokémon that you defeat, the prestige of that gym will be reduced. If the prestige is reduced enough, that gym will lose a level and the weakest Pokémon will be kicked out and returned to its trainer. If the gym loses enough prestige and falls below level one, it will turn white (neutral) allowing you to install a Pokémon of your own and claim this gym for your team.

Allied Gyms

These are Gyms that are currently controlled by players who are on the same team as you. When you battle at these gyms, you must select a single Pokémon to fight with, and then combat proceeds as normal against the Pokémon in that gym. Winning these will result in an XP reward, and the prestige of that gym will be increased; however, any Pokémon that you defeat will remain in the gym, unlike battles against enemy gyms. You can also install your own Pokémon in a friendly gym provided there is enough space, which is based on the gym's prestige level (e.g. Prestige 2 gyms can hold two Pokémon). Installing your own Pokémon in a gym can be beneficial because it allows you to collect a reward of Pokécoins and Stardust once every ~20 hours for each Pokémon you have installed in a gym. The maximum any player can have in any singular gym is one Pokémon.

If you get to a gym that is free, you can put one of your pokemon in there. If the gym's occupied by a player from another color, you will have to beat that player first.

If you put your pokemon inside a gym and it gets beaten, it will be returned to you.