Do the effects of relics like the Vault Hunter's Relic stack in co-op?

Yes, the effect stacks, but unfortunately the Vault Hunter's Relic is a bit vague about what it does. It doesn't actually increase the spawns of things like E-tech, purple or orange items. Rather, it decreases the drop rate of white weaponry while slightly increasing the drop rate of green and blue weaponry:

So I just investigated the Vault Hunter Relic data.

It modifies two attributes:

GD_Balance.Weighting.GearDrops_RareWeightModifier: +0.05 GD_Balance.Weighting.GearDrops_CommonWeightModifier: -0.06

Let's set them to +100.00 and -100.00!

Net result: I have not seen a single white weapon drop, greens drop, seen a few blues. Nothing else.

So, what I can gauge from this, is that it shifts a 5% of your chance for whites into non-whites. Doesn't touch anything else. Very low impact, perhaps bordering on useless.

Thus, even with all four players using it, you wouldn't be guaranteed to find any more very rare weaponry than you would normally.

Moxxi's Endowment on the other hand only increases the XP you get on enemy deaths, not the XP everyone gets. If all four people had it equipped, they'd each get their own XP increase (say, +5%) as opposed to a stacked increase (in this case, +20% XP per player).

In short, the Vault Hunter's Relic stacks but is sadly kind of useless and you'd be better off equipping something else. Moxxi's Endowment won't stack but it's still useful for grinding levels. In my opinion, equipping a health increase or an action skill cooldown relic would be better than both of them.

Also, you asked if other relics stacked. If you were asking about things like health or shield or resistance boosts, then no, none of those stack with other players. They affect only you.