What does 'killed' mean when a processing of a huge CSV with Python, which suddenly stops?
Exit code 137 (128+9) indicates that your program exited due to receiving signal 9, which is SIGKILL. This also explains the killed message. The question is, why did you receive that signal?
The most likely reason is probably that your process crossed some limit in the amount of system resources that you are allowed to use. Depending on your OS and configuration, this could mean you had too many open files, used too much filesytem space or something else. The most likely is that your program was using too much memory. Rather than risking things breaking when memory allocations started failing, the system sent a kill signal to the process that was using too much memory.
As I commented earlier, one reason you might hit a memory limit after printing finished counting is that your call to counter.items() in your final loop allocates a list that contains all the keys and values from your dictionary. If your dictionary had a lot of data, this might be a very big list. A possible solution would be to use counter.iteritems() which is a generator. Rather than returning all the items in a list, it lets you iterate over them with much less memory usage.
So, I'd suggest trying this, as your final loop:
for key, value in counter.iteritems():
writer.writerow([key, value])
Note that in Python 3, items returns a "dictionary view" object which does not have the same overhead as Python 2's version. It replaces iteritems, so if you later upgrade Python versions, you'll end up changing the loop back to the way it was.
There are two storage areas involved: the stack and the heap.The stack is where the current state of a method call is kept (ie local variables and references), and the heap is where objects are stored. recursion and memory
I gues there are too many keys in the counter dict that will consume too much memory of the heap region, so the Python runtime will raise a OutOfMemory exception.
To save it, don't create a giant object, e.g. the counter.
1.StackOverflow
a program that create too many local variables.
Python 2.7.9 (default, Mar 1 2015, 12:57:24)
[GCC 4.9.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> f = open('stack_overflow.py','w')
>>> f.write('def foo():\n')
>>> for x in xrange(10000000):
... f.write('\tx%d = %d\n' % (x, x))
...
>>> f.write('foo()')
>>> f.close()
>>> execfile('stack_overflow.py')
Killed
2.OutOfMemory
a program that creats a giant dict includes too many keys.
>>> f = open('out_of_memory.py','w')
>>> f.write('def foo():\n')
>>> f.write('\tcounter = {}\n')
>>> for x in xrange(10000000):
... f.write('counter[%d] = %d\n' % (x, x))
...
>>> f.write('foo()\n')
>>> f.close()
>>> execfile('out_of_memory.py')
Killed
References
- 7. Memory : Stack vs Heap
- recursion and memory
Most likely, you ran out of memory, so the Kernel killed your process.
Have you heard about OOM Killer?
Here's a log from a script that I developed for processing a huge set of data from CSV files:
Mar 12 18:20:38 server.com kernel: [63802.396693] Out of memory: Kill process 12216 (python3) score 915 or sacrifice child
Mar 12 18:20:38 server.com kernel: [63802.402542] Killed process 12216 (python3) total-vm:9695784kB, anon-rss:7623168kB, file-rss:4kB, shmem-rss:0kB
Mar 12 18:20:38 server.com kernel: [63803.002121] oom_reaper: reaped process 12216 (python3), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
It was taken from /var/log/syslog.
Basically:
PID 12216 elected as a victim (due to its use of +9Gb of total-vm), so oom_killer reaped it.
Here's a article about OOM behavior.