How to completely avoid localized .NET exception messages

Remove any .NET language packs. According to this ("The .NET Framework 4.5 Language Pack contains translated error messages and other UI text for languages other than English. If you don't install a language pack, this text is displayed in English"), it should be enough.