Where can I find gravel to collect?

• Villages (post Beta 1.8)

Have roads built out of gravel, and depending on the size of the village you can get quite a bit of gravel from them.

• Seabed

The bottom of the ocean usually has a lot of gravel but, it is obviously a dangerous place to dig due to lack of oxygen, but nonetheless there is gravel.

• Caves (As you mentioned)

Caves sometimes have whole sections of wall just filled with gravel. I'd say this is your best bet at finding gravel as it is as common as it is underground but the tunnels are already dug for you.

• Underground (General)

Underground in general has a lot of gravel that is usually found in clusters like ore, but a lot more of it.

Before 1.8 version you could find Gravel Beaches, which can be (obviously) a good source of gravel:

As sand beaches, there are randomly generated near water, in a lower frequency that sand. Be careful because gravel can be "floating" over a cavern and start falling once you start mining it.