Do I have to hold all these unusual gems in my inventory?
No, the unusual gems are quest items and therefore cannot be removed.
In terms of Mods available to address this, I found a few.
On Nexus for Skyrim (Oldrim), we have Sell Unusual Gems - Stones of Barenziah by kryptopyr. The description of the mod is:
Allows the player to sell "Unusual Gems" to Ri'saad without joining the Thieves Guild.
On Bethesda Net, we have Stones Of Barenziah DROPPABLE Plus Quest Markers by toaDime.
This is available on: PC, XBox One, and PS4 (believe it or not.)
The description for this mod states:
In addition, the stones are no longer quest items and can be dropped.
Please note:
- As a result, the stones now have WEIGHT, so you will WANT to drop them.
- You will be able to break the quest if you drop them into a respawning container for example. So please don't do that. If you DO break the quest, "setstage tgcrown 30" will fix it.
- The stones still don't stack unfortunately - the not-stacking is caused by the RENAMING. In fact the quest shows signs that its creator tried to fix the not-stacking problem, but was unsuccessful. Anyway, the main concern is how they clutter up the inventory and this is fixed
- The stones have no physics - if you drop them in mid-air they will float.