Underground Dirt and Gravel

Dirt and Gravel are found underground, along with stone, water, lava, and occasionally obsidian. All of these are 'natural', with no significance. (You can also sometimes find sand)

Moss Stone indicates Dungeons, and if you find Wood/Rails, that's usually an indication you've wandered into an abandoned mine-- but gravel and dirt alone are inconsequential. They're just there for variety.