Do all eggs give the same rewards when hatched?

Regardless of their contents, all eggs should be hatched if possible.

Unless I'm mistaken, the only way to remove an egg from your inventory is by hatching it. Since you have a limit of 9 eggs and eggs are essentially random, you'll eventually want to hatch all the eggs you can.

The questions then becomes "what order should I hatch them?"

I tend to hatch the longest walking length I can first, since the stronger Pokemon will help me out more in these earlier stages. But that's preference.

As for whether you should run around instead of catching Pokemon, that's a matter of balancing your resources.

If you have a ton of pokeballs, you'll likely want the candy and dust from catching Pokemon so you don't waste the resource. Or if you already have 9 eggs, you may want to rub and hatch a few so you have space for more.

Optimally you should never be sitting on excess eggs or pokeballs or potions etc, unless you have an excess of all of them. In which case, good job!

Here is a chart for egg hatching