close

[Solved] Python multiprocessing PicklingError: Can’t pickle

Hello Guys, How are you all? Hope You all Are Fine. Today I get the following error Python multiprocessing PicklingError: Can’t pickle in python. So Here I am Explain to you all the possible solutions here.

Without wasting your time, Let’s start This Article to Solve This Error.

How Python multiprocessing PicklingError: Can’t pickle Error Occurs?

Today I get the following error Python multiprocessing PicklingError: Can’t pickle in python.

How To Solve Python multiprocessing PicklingError: Can’t pickle Error ?

  1. How To Solve Python multiprocessing PicklingError: Can't pickle Error ?

    To Solve Python multiprocessing PicklingError: Can't pickle Error I have found that I can also generate exactly that error output on a perfectly working piece of code by attempting to use the profiler on it.

  2. Python multiprocessing PicklingError: Can't pickle

    To Solve Python multiprocessing PicklingError: Can't pickle Error I have found that I can also generate exactly that error output on a perfectly working piece of code by attempting to use the profiler on it.

Solution 1


Here is a list of what can be pickled. In particular, functions are only picklable if they are defined at the top-level of a module.

This piece of code:

import multiprocessing as mp

class Foo():
    @staticmethod
    def work(self):
        pass

if __name__ == '__main__':   
    pool = mp.Pool()
    foo = Foo()
    pool.apply_async(foo.work)
    pool.close()
    pool.join()

yields an error almost identical to the one you posted:

Exception in thread Thread-2:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 552, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 505, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/usr/lib/python2.7/multiprocessing/pool.py", line 315, in _handle_tasks
    put(task)
PicklingError: Can't pickle <type 'function'>: attribute lookup __builtin__.function failed

The problem is that the pool methods all use a mp.SimpleQueue to pass tasks to the worker processes. Everything that goes through the mp.SimpleQueue must be pickable, and foo.work is not picklable since it is not defined at the top level of the module.

It can be fixed by defining a function at the top level, which calls foo.work():

def work(foo):
    foo.work()

pool.apply_async(work,args=(foo,))

Notice that foo is pickable, since Foo is defined at the top level and foo.__dict__ is picklable.

Solution 2

I have found that I can also generate exactly that error output on a perfectly working piece of code by attempting to use the profiler on it.

Note that this was on Windows (where the forking is a bit less elegant).

I was running:

python -m profile -o output.pstats <script> 

And found that removing the profiling removed the error and placing the profiling restored it. Was driving me batty too because I knew the code used to work. I was checking to see if something had updated pool.py… then had a sinking feeling and eliminated the profiling and that was it.

Posting here for the archives in case anybody else runs into it.

Summery

It’s all About this issue. Hope all solution helped you a lot. Comment below Your thoughts and your queries. Also, Comment below which solution worked for you? Thank You.

Also, Read