When should I use a transistor, mosfet, and when should I use a relay?

Transistors (such as Bipolar Junction Transistors (BJT)) are solid state switchers, that is, they don't have any moving parts. Metal-Oxide-Semiconductor Field-Effect Transistors (MOSFET) are also transistors, but are voltage controlled, whereas classical BJTs are current-controlled. Relays are more expensive, but they have moving parts.

BJTs and MOSFETs produce a lot of heat (with high currents). They both have three pins (usually) called the base, collector and emitter (in the case of BJT) or gate, source, drain (in the case of MOSFET). More info here

Relays use a magnet to control the position of a metal switch. They generally take more power than an Arduino pin can deliver, and therefore they are often controlled by a transistor. Relays can usually switch more current than either a transistor or a MOSFET.

There are also Solid State Relays (SSR), which have a triac (a device similar to a transistor) in them. These can't switch as much as a normal relay, but they generally have better ratings than MOSFETs and BJT. example

Big differences between relays and transistors are:

• relays are all-or-nothing (like switches) whereas transistors can transmit more or less current through the collector based on the current present on their base.
• relays provide isolation between the command circuit (the one with the electromagnet) and the controlled circuit (the one on the switch side of the relay)

If you want to play with arduino, you definitely need to know about transistors; using relays is more akin to special applications.