When a battery is your power source, what is ground?

Yes - just remember that your ground in that case will only be relative to the battery. If you go to connect this to another device (serial interface, etc) you need to link the ground lines so they're a common ground.

So long as it is isolated, you're fine.

The ground pin on the microcontroller is two things: (1) a voltage reference, and (2) a current return.

Voltage Reference: Voltages don't really exist at single points, voltages are differences between points. This means that to speak sensibly about some single point in a circuit being at a given voltage, it has to be relative to some other point in the circuit. The common convention is to pick one node in the circuit and call it 'ground', and then all voltages are specified relative to that. With a microcontroller, the ground pin is the reference against which the inputs are 'measured' to determine whether they are high or low.

Current Return: All the current that flows into a micro-controller has to flow back out again somewhere. You're certainly aware that current flows into the +5V pin. Note that some current will flow out various pins as signal outputs, but then some outputs will actually admit current into the chip. Input pins can also source or sink current. Regardless, whatever current is left over is expected to flow out the ground pin of the micro-controller. In essence, the supply current is returned to the supply through the ground connection.

So when your power supply is a battery, it makes perfect sense to connect the (-) side of the battery to your system's ground pin. Notice that this isn't just a voltage reference though; it is also the supply return. In practical terms, what this means that the wire you use to connect (-) to the board's ground should be at the same size as the wire you use to connect (+) to the board's power input.

The potential is relative. Ground is an arbitrary designation. —Fake Name

This is something I worked out recently, only then realising why the hydraulic analogy was causing me cognitive dissonance. If a "voltage supply" was a waterfall, the image of a 12m top-to-bottom waterfall stuck in the middle of the sky, somewhere above my 5m waterfall which started 5m above the "ground" just didn't make any sense.

And then I tied the negative pin on my Arduino (powered by a 12V wall wart) to the drain on the transistors I was using to switch current to my LED strips and the circuit worked.