close

[Solved] Multiprocessing example giving AttributeError

Hello Guys, How are you all? Hope You all Are Fine. Today I get the following error Multiprocessing example giving AttributeError in python. So Here I am Explain to you all the possible solutions here.

Without wasting your time, Let’s start This Article to Solve This Error.

How Multiprocessing example giving AttributeError Error Occurs?

Today I get the following error Multiprocessing example giving AttributeError in python.

How To Solve Multiprocessing example giving AttributeError Error ?

  1. How To Solve Multiprocessing example giving AttributeError Error ?

    To Solve Multiprocessing example giving AttributeError Error Of course, externalizing the code as suggested in this answer works as well, but I find it very inconvenient: That is not why (and how) I use IPython environments.

  2. Multiprocessing example giving AttributeError

    To Solve Multiprocessing example giving AttributeError Error Of course, externalizing the code as suggested in this answer works as well, but I find it very inconvenient: That is not why (and how) I use IPython environments.

Solution 1

This problem seems to be a design feature of multiprocessing.Pool. For some reason Pool does not always work with objects not defined in an imported module. So you have to write your function into a different file and import the module.

File: defs.py

def f(x):
    return x*x

File: run.py

from multiprocessing import Pool
import defs

 if __name__ == '__main__':
    with Pool(5) as p:
        print(p.map(defs.f, [1, 2, 3]))

If you use print or a different built-in function, the example should work. If this is not a bug (according to the link), the given example is chosen badly.

Solution 2

The multiprocessing module has a major limitation when it comes to IPython use:

Functionality within this package requires that the __main__ module be importable by the children. […] This means that some examples, such as the multiprocessing.pool.Pool examples will not work in the interactive interpreter. [from the documentation]

Fortunately, there is a fork of the multiprocessing module called multiprocess which uses dill instead of pickle to serialization and overcomes this issue conveniently.

Just install multiprocess and replace multiprocessing with multiprocess in your imports:

import multiprocess as mp

def f(x):
    return x*x

with mp.Pool(5) as pool:
    print(pool.map(f, [1, 2, 3, 4, 5]))

Of course, externalizing the code as suggested in this answer works as well, but I find it very inconvenient: That is not why (and how) I use IPython environments.

<tl;dr> multiprocessing does not work in IPython environments right away, use its fork multiprocess instead.

Summery

It’s all About this issue. Hope all solution helped you a lot. Comment below Your thoughts and your queries. Also, Comment below which solution worked for you? Thank You.

Also, Read