How to use Advanced Currency Management (ACM) in Custom Objects?

Short answer: you can't do this without a major investment.

Longer answer: my job is developing FinancialForce's PSA application, which requires supporting dated exchange rates and other ACM features on custom objects. Salesforce has confirmed to us more than once that there is no native way to do this, so we've implemented our own currency conversion from the ground up to support our needs. We do support reading the dated exchange rates from ACM's tables, but we still have to do all our own math on the conversions in apex logic. At the very least this means that you can tap into the same currency data for your calculations, even if the implementation of the logic must be done independently.

The net effect is that if you want ACM functionality outside of the officially supported objects you're going to have to build you own currency conversion system, and your own apex-managed rollups rather than using the native functionality, for this to work. If you do go this route you'll probably want to turn off parenthesized currency conversions as they'd use the non-dated rates and effectively lie to users of your custom currency system.

This is obviously a non-trivial solution to the problem, but it's the only one that's currently possible today.