What do gems do?

Gems seem to serve three primary purposes:

• Being worth a boatload of money.
• Being used as an upgrade component.
• Being able to made into accessories.

Some items, like the Berserker set, require 2 diamonds to upgrade. There are likely other sets like this, but I'm uncertain.

As Sam mentioned, you can also use gems to get circlets and earrings from the Gerudo Jeweler. They take up a head slot (so they're not particularly useful in most cases) and there are 6 total:

• Amber (8 amber = +4 def earrings)
• Opal (8 opals = swim speed boost earrings)
• Ruby (5 rubies = ice resist up circlet)
• Sapphire (5 sapphires = fire resist up circlet)
• Diamond (3 diamonds = guardian resist circlet)
• Topaz (5 topaz = electric resist earrings)

This is of course only doable after you give her the 10 Flint.