Are enchanted books or direct enchanting better for rare enchantments?

If you're looking for something specific, you have to enchant a lot more books to get it than enchanting a specific tool, since there are more possibilities.

• If you enchant a book at level 30, your chance of getting Looting II or III is approximately .02 (1 in 50), and so you will have to enchant about 50 books before you get it. If you have XP and time to burn, this is the way to go as books can be made from easy farming.

• On the other hand, if you enchant a diamond sword at level 30, you have a .15 (1 in 7) chance of getting Looting II or III. (This should actually be even better because I'm not working in the effect of multiple enchantments.) Thus, you will get it much sooner — but you will also get (on average) six swords that don't have Looting on them. Note that you can then recycle those swords by repair-crafting them with almost-used-up diamond swords.

Therefore, given that you say you want to spend diamonds rather than XP, you should definitely enchant swords.

(all numbers from minecraftenchantmentcalculator, back when it existed, which is a sloppy tool because it just does a simulation rather than calculating true probabilities, but it was handy and should be roughly correct)