How to Add a viewModel to a core block?

It is possible and it works as expected!

The reason why it was not working was because I was not implementing the right interface in my viewModel class.

All viewModels need to to implement ArgumentInterface.

namespace Vendor\CustomModule\ViewModel;

class Class implements \Magento\Framework\View\Element\Block\ArgumentInterface
{
    public function __construct()
    {

    }
}