How was non-decimal money represented in software?

PL/I had a type specifically for British currency - I don't know about COBOL. The British currency at one time incorporated farthings, or a quarter of a penny; I'm not sure though that computers had to deal with those, just with half pennies or ha'pennies.

Accurate accounting usually uses special types - representing decimals exactly. The new IEEE 754 has support for floating-point decimals, and some chips (notably IBM pSeries) have such support in hardware.