What difference do gems make in enchanting weapons?

The difference is how big the maximum charge will be. So, if you use the petty gem, you will have to recharge the weapon all the time. If you use the grand gem (assuming it contains grand soul), you will have to recharge it only rarely.

Source: Skyrim:Enchanting on UESPWiki:

The type of soul gem affects the amount of charges on a weapon or quality of the enchant on apparel.

The gem doesn't matter.

The soul inside the gem does.

You can get a lot more out of a grand soul than you can out of a petty soul -- many more charges (attacks) for weapons, much better stats from armor. A gem can also fit in any soul its size or smaller.

I'd suspect, from the information given in the question, that your grand gem has a lesser or common soul in it, so you're not going to see much difference between that and your petty soul gem with a petty soul in it.