Why bother with Meteorite armour?

Its pretty much just for using the space gun, which Meteor Armor makes cost 0 mana to shoot, plus as it counts as a magic attack the +15% magic attack bonus also applies.

Like you say, the defense it provides is fairly low - if you are using Silver Armor then I'd say that loosing 3 defense might be worth it for the space gun (never used it myself so I'm not sure how good it is), however either way you will probably be getting the Shadow Armor fairly soon which offers a lot more defense, so you won't be wearing it for long.

Personally I'd make a Phaseblade, and possibly some Meteor Shot first, and then make the armor if you have any left over - Meteorite Ore is fairly abundant and isn't really used for that much stuff, so sooner or later you will have enough to make some eventually anyway.