Python 3.5 dill pickling/unpickling on different servers: "KeyError: 'ClassType'"
The culprit is cloudpickle. By default in Python 3.5, types.ClassType is left unset.
>>> import types
>>> dir(types)
['BuiltinFunctionType', 'BuiltinMethodType', 'CodeType', ...]
When cloudpickle is imported, suddenly, types.ClassType becomes defined.
>>> import cloudpickle
>>> dir(types)
['BuiltinFunctionType', 'BuiltinMethodType', 'ClassType', 'CodeType', ...]
Server A uses dill to serialize objects, and also imports cloudpickle. Therefore it includes a reference to ClassType during serialization.
Server B does NOT import cloudpickle, and then tries to find a reference to ClassType during deserialization and fails. Raising the error:
Traceback (most recent call last):
File "/home/streamsadmin/git/streamsx.topology/test/python/topology/deleteme2.py", line 40, in <module>
a = dill.loads(base64.b64decode(a.encode()))
File "/home/streamsadmin/anaconda3/lib/python3.5/site-packages/dill/dill.py", line 277, in loads
return load(file)
File "/home/streamsadmin/anaconda3/lib/python3.5/site-packages/dill/dill.py", line 266, in load
obj = pik.load()
File "/home/streamsadmin/anaconda3/lib/python3.5/site-packages/dill/dill.py", line 524, in _load_type
return _reverse_typemap[name]
KeyError: 'ClassType'
On our system, we can't remove cloudpickle from our environment, so we had to do the following workaround.
On server B, right after we import dill and sometime before the first call to dill.loads, we invoke the following line of code:
dill._dill._reverse_typemap['ClassType'] = type
This defines ClassType appropriately. And causes dill deserialization to work as expected.
I am sure cloudpickle is causing the problem. You can debug it step by step.
First Check if classType Exists in your builtin types
import types dir(types)
if it exist than it should have worked for you, if not than move to next steps.
import cloudpickle and now check again. You will have classType in buildin types
excute below code
dill.dill._reverse_typemap['ClassType'] = type
it should work for you :)
But if you are still getting error AttributeError: module 'dill' has no attribute 'dill'
than use this one dill._dill._reverse_typemap['ClassType'] = type because dill.dill is moved to dill._dill