Python & JSON: ValueError: Unterminated string starting at:

If someone is here just like I am and if you're handling json from the form requests then check if there is any Content-Length header set or not. I was getting this error because of that header. I used the JSON beautification and found the json became quite large which raised this error.

I got the same problem. As it turned out, the last line of the file was incomplete probably due to the abrupt halt of the download as I found there was enough data and simply stopped the process on the terminal.

So I figured it out... I post this answer just in case someone else makes the same error.

First, I found a work around but I wasn't sure why this worked. From my original code, here is my file_get_contents function:

def file_get_contents(fname):
    if s.stripos(fname, 'http://'):
        import urllib2
        return urllib2.urlopen(fname).read(maxUrlRead)
    else:
        return open(fname).read(maxFileRead)

I used it via:

tmp = json.loads(f.file_get_contents(aggFile))

This failed, over and over and over again. However, as I was attempting to get Python to at least give me the JSON string to put through a JSON validator I came across mention of json.load vs json.loads. So I tried this instead:

a = open('D://Server Data//eagle805//emmetrics//forms//leads\July_2014.cd.lead.agg')
b = json.load(a)

While I haven't tested this output in my overall code this code chunk does in fact read in the file, decode the JSON, and will even display the data without crashing Spyder. The variable explorer in Spyder shows that b is a dict of size 1465 and that is exactly how many records it should have. The portion of the displayed text from the end of the dict all looks good. So overall I have a reasonably high level confidence that the data was parsed correctly.

When I wrote the file_get_contents function I saw several recommendations that I always provide a max number of bytes to read so as to prevent Python from hanging on a bad return. The value of maxReadFile was 1E7. When I manually forced maxReadFile to be 1E9 everything worked fine. Turns out the file is just under 1.2E7 bytes. So the resulting string from reading the file was not the full string in the file and as a result was invalid JSON.

Normally I would think this is a bug but clearly when opening and reading a file you need to be able to read just a chunk at a time for memory management. So I got bit by my own shortsightedness with regards to the maxReadFile value. The error message was correct but sent me off on a wild goose chase.

Hopefully this could save someone else some time.