How to check if a .txt file is in ASCII or UTF-8 format in Windows environment?

Text files in Windows don't have a format. There's an unofficial convention that if the file starts with the BOM codepoint in UTF-8 format that it's UTF-8, but that convention isn't universally supported. That would be the 3 byte sequence "\xef\xbf\xbe", i.e. ￾ in the Latin-1 character set.

Open the file in Notepad. Click 'Save As...'. In the 'Encoding:' combo box you will see the current file format.

Open the file using Notepad++ and check the "Encoding" menu, you can check the current Encoding and/or Convert to a set of encodings available.