Establishing why an object can't be pickled

I would use dill, which has tools to investigate what inside an object causes your target object to not be picklable. See this answer for an example: Good example of BadItem in Dill Module, and this Q&A for an example of the detection tools in real use: pandas.algos._return_false causes PicklingError with dill.dump_session on CentOS.

>>> import dill
>>> x = iter([1,2,3,4])
>>> d = {'x':x}
>>> # we check for unpicklable items in d (i.e. the iterator x)
>>> dill.detect.baditems(d)
[<listiterator object at 0x10b0e48d0>]
>>> # note that nothing inside of the iterator is unpicklable!
>>> dill.detect.baditems(x)
[]

However, the most common starting point is to use trace:

>>> dill.detect.trace(True)
>>> dill.detect.errors(d)
D2: <dict object at 0x10b8394b0>
T4: <type 'listiterator'>
PicklingError("Can't pickle <type 'listiterator'>: it's not found as __builtin__.listiterator",)
>>>

dill also has functionality to trace pointers referrers and referents to objects, so you can build a hierarchy of how objects refer to each other. See: https://github.com/uqfoundation/dill/issues/58

Alternately, there's also: cloudpickle.py and debugpickle.py, which are for the most part no longer developed. I'm the dill author, and hope to soon merge any functionality in these codes that is missing in dill.


You may want to read the python docs and check your API's Object class afterwards.

With respect to the "internal structure of the object", usually instance attributes are stored in the __dict__ attribute (and since class attributes are not pickled you only care about the instance attributes) - but note that you'll also have to recursively inspect the __dict__s for each attribute.


I tried Dill but it didn't explain my issue. Instead, I used the following code from https://gist.github.com/andresriancho/15b5e226de68a0c2efd0, which happened to show a bug in my __getattribute__ override:

def debug_pickle(instance):
  """
  :return: Which attribute from this object can't be pickled?
  """
  attribute = None

  for k, v in instance.__dict__.iteritems():
      try:
          cPickle.dumps(v)
      except:
          attribute = k
          break

  return attribute

Edit: Here's a reproduction of my code, using pickle and cPickle:

class myDict(dict):

    def __getattribute__(self, item):
        # Try to get attribute from internal dict
        item = item.replace("_", "$")

        if item in self:
            return self[item]

        # Try super, which may leads to an AttribueError
        return super(myDict, self).__getattribute__(item)

myd = myDict()

try: 
    with open('test.pickle', 'wb') as myf:
        cPickle.dump(myd, myf, protocol=-1)
except:
    print traceback.format_exc()


try:
    with open('test.pickle', 'wb') as myf:
        pickle.dump(myd, myf, protocol=-1)
except:
    print traceback.format_exc()

Output:

Traceback (most recent call last):
File "/Users/myuser/Documents/workspace/AcceptanceTesting/ingest.py", line 35, in <module>
  cPickle.dump(myd, myf, protocol=-1)
UnpickleableError: Cannot pickle <class '__main__.myDict'> objects

Traceback (most recent call last):
File "/Users/myuser/Documents/workspace/AcceptanceTesting/ingest.py", line 42, in <module>
  pickle.dump(myd, myf, protocol=-1)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 1370, in dump
  Pickler(file, protocol).dump(obj)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 224, in dump
  self.save(obj)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pickle.py", line 313, in save
  (t.__name__, obj))
PicklingError: Can't pickle 'myDict' object: {}

You'll see that the reason is because attribute names are being mangled by __getattribute__

Tags:

Python

Pickle

Recent Posts