UnicodeDecodeError in Python 3 when importing a CSV file
From the python docs, you have to set the encoding for the file. Here is an example from the site:
import csv
with open('some.csv', newline='', encoding='utf-8') as f:
reader = csv.reader(f)
for row in reader:
print(row)
Edit: Your problem appears to happen with printing. Try using pretty printer:
import csv
import pprint
with open('some.csv', newline='', encoding='utf-8') as f:
reader = csv.reader(f)
for row in reader:
pprint.pprint(row)
Another option is to cover up the errors by passing an error handler:
with open('some.csv', newline='', errors='replace') as f:
reader = csv.reader(f)
for row in reader:
print(row)
which will replace any undecodable bytes in the file with a "missing character".
It seems your problem boils down to:
print("α")
You could fix it by specifying PYTHONIOENCODING
:
$ PYTHONIOENCODING=utf-8 python3 test.py > output.txt
Note:
$ python3 test.py
should work as is if your terminal configuration supports it, where test.py
:
import csv
with open('myfile.csv', newline='', encoding='utf-8') as file:
for row in csv.reader(file):
print(row)
If open()
has no encoding
parameter above then you'll get UnicodeDecodeError
with LC_ALL=C
.
Also with LC_ALL=C
you'll get UnicodeEncodeError
even if there is no redirection i.e., PYTHONIOENCODING
is necessary in this case (before PEP 538: Legacy C Locale Coercion implemented in Python 3.7+).