Do followers benefit from gems that do not boost their primary stat?

Yes.

Items equipped on your followers do benefit from socketed gems. The wiki entry on followers doesn't go into detail, but followers do benefit from equipment and gems just like your character does. You can test this by letting your follower attack enemies with and without emeralds, and counting the damage of criticals (which I have done, sadly).

Your play style, and that of your follower, depends on the gear you and your follower use, and vice versa. E.g., rubies will always increase strength (which is useless for say, a wizard), and emeralds will always give critical damage (which is great for everyone, unless you want to focus on something else). In my experience, critical strike chance gives you the highest damage increase, pound for pound, even more than critical damage. If your follower dies a lot though, maybe go for vitality, life on hit, etc (or opt for a legendary follower accessory with "Your follower cannot die.")

It should be noted however that for followers, the critical hit damage bonus is capped at +100% (beyond that natural bonus), so if you have a follower accessory with high CHD, a ruby in the weapon socket will result in higher damage.

Yes... but not very much

First of all, an emerald in a weapon does not increase anybody's critical hit chance, but merely the critical hit damage. This is the same for you and your follower.

For the other slots (shield/jewelry), you should stick the one that gives a bonus to their primary stat, as it is the only one that will increase their damage.

However, it's not strictly true to say that there is no benefit from using the other gems, as strength and dexterity still increases armor for an intelligence-based character, and intelligence increases all resist. But since the primary stat has both the offensive and defensive perks, you should go with that one.

Ideally though...

Apart from the weapon (which is debatable), you shouldn't really be using socketed gear on your followers, since they benefit more from other stats (cooldown reduction, for example), unless this is all you have to give to them. Sockets in jewelry are only good if you can use legendary gems, which followers unfortunately cannot.