How hot is too hot for a transformer?

The maximum allowed winding temperature VS. insulation class:

The transformer should have a specification sheet that details such things are max. temperature. Since this one is custom, it seems that you should have that available.