Monday, 15 June 2015

Python multiprocessing: is it possible to have a pool inside of a pool? -


I have a module e that can take data and analyze it for module B, C, D, etc. and Then shake your results together.

But it appears that the module B, C, D, etc. themselves can not make a multi processing pool, otherwise I will

  AssertionError: children for demonic procedures Is not it allowed to parallel   

Is it possible to parallel these jobs in any other way?

For clarity, here's a (n is bad) bad example. (I usually try / catch up, but you get a summary.)

  Import from multiprocessing import: ACP: import B pool def main (): p = pool () result = p .map (bfu, range (10)) p. Close () p. Jane () Return Results B.py: Pool Diff Fu (X) from Multiprocessing Import: P = Pool () Results = PAMP (St, X) P.close () p.join () Return Results    

What is it possible to have a pool inside the pool?

Yes, this is possible, although this may not possibly be a good idea unless you want to mobilize. From:

  import multiprocessing.pool functools imported from imported partial class from Nodine process (multiprocessing.Process): # make 'daemon' attribute always returned the wrong def _get_daemon (self): back Return def defset_daemon (self, value): pass the daemon = property (_get_daemon, _set_daemon) # We instead of multi-processing multi-process multiplication. Pool.Pool # Because the latter is only one wrapper function, not a proper class. Class Pool (Multiprocessing. Pool.Pool): Process = Nodamon Process Def EF (X, Depth = 0): If Depth == 0: Returns X and: with Closing (Pool ()) as P: Return P. MAP (Partial (

output

  [[ 0], [0, 1], [0, 1, 2], [0, 1, 2, 3], [0, 1, 2, 3, 4], [0, 1, 2, 3, 4, 5], [0, 1, 2, 3, 4, 5, 6], [0, 1, 2, 3, 4, 5, 6, 7] [0, 1, 2, 3, 4, 5, 6 , 7, 8], [0, 1, 2, 3, 4, 5, 6, 7, 8, 9], [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10 ]]   

supports it by default:

  # $ pip installation futures # come from Python 2 concurrent.futures ProcopsPool Exxecutor functools imports as pool in the form of partial def foo (X, Depth = 0): If Depth == 0: Return x and: With Pool: As P: Return List (PMAP (partial (foo , Depth = depth 1), range (x + 1)) if __name__ == "__main__": from pprint import pprint pprint (foo (10, Depth = 2))   

Generates the same output.

Is it possible to parallel these jobs in any other way?

Yes, for example, see how.

No comments:

Post a Comment