Pandas read csv file with float values results in weird rounding and decimal digits
Pandas uses a dedicated dec 2 bin
converter that compromises accuracy in preference to speed.
Passing float_precision='round_trip'
to read_csv
fixes this.
Check out this page for more detail on this.
After processing your data, if you want to save it back in a csv file, you can passfloat_format = "%.nf"
to the corresponding method.
A full exemple:
import pandas as pd
df_in = pd.read_csv(source_file, float_precision='round_trip')
df_out = ... # some processing of df_in
df_out.to_csv(target_file, float_format="%.3f") # for 3 decimal places
I realise this is an old question, but maybe this will help someone else:
I had a similar problem, but couldn't quite use the same solution. Unfortunately the float_precision
option only exists when using the C engine and not with the python engine. So if you have to use the python engine for some other reason (for example because the C engine can't deal with regex literals as deliminators), this little "trick" worked for me:
In the pd.read_csv
arguments, define dtype='str'
and then convert your dataframe to whatever dtype you want, e.g. df = df.astype('float64')
.
Bit of a hack, but it seems to work. If anyone has any suggestions on how to solve this in a better way, let me know.