Pandas, read CSV ignoring extra commas

You can use re.sub to replace the first few commas with, say, the '|', save the intermediate results in a StringIO then process that.

import pandas as pd
from io import StringIO
import re

for_pd = StringIO()
with open('MikeS159.csv') as mike:
    for line in mike:
        new_line = re.sub(r',', '|', line.rstrip(), count=7)
        print (new_line, file=for_pd)

for_pd.seek(0)

df = pd.read_csv(for_pd, sep='|', header=None)
print (df)

I put the two lines from your question into a file to get this output.

       0       1  2                    3  4  5   6  \
0  061AE  Active  1  2017_02_24 15_18_01  6  1  13   
1  061AE  Active  1  2017_02_24 15_18_01  6  1  13   

                             7  
0                 some message  
1  longer message, with commas  

You can use the parameter usecols in the read_csv function to limit what columns you read in. For example:

import pandas as pd
pd.read_csv(path, usecols=range(8))

if you only want to read the first 8 columns.

Tags:

Python

Pandas