Python Multiprocessing With A Single Function

I have a simulation that is currently running, but the ETA is about 40 hours -- I'm trying to speed it up with multi-processing. It essentially iterates over 3 values of one variab

Solution 1:

• Wrap the data for each iteration up into a tuple.
• Make a list data of those tuples
• Write a function f to process one tuple and return one result
• Create p = multiprocessing.Pool() object.
• Call results = p.map(f, data)

This will run as many instances of f as your machine has cores in separate processes.

Edit1: Example:

from multiprocessing import Pool

data = [('bla', 1, 3, 7), ('spam', 12, 4, 8), ('eggs', 17, 1, 3)]

deff(t):
    name, a, b, c = t
    return (name, a + b + c)

p = Pool()
results = p.map(f, data)
print results

Edit2:

Multiprocessing should work fine on UNIX-like platforms such as OSX. Only platforms that lack os.fork (mainly MS Windows) need special attention. But even there it still works. See the multiprocessing documentation.

Solution 2:

Use Pool().imap_unordered if ordering is not important. It will return results in a non-blocking fashion.

Solution 3:

Here is one way to run it in parallel threads:

import threading

L_a = []

for L inrange(0,6,2):
    for a inrange(1,100):
        L_a.append((L,a))
        # Add the rest of your objects heredefRunParallelThreads():
    # Create an index list
    indexes = range(0,len(L_a))
    # Create the output list
    output = [Nonefor i in indexes]
    # Create all the parallel threads
    threads = [threading.Thread(target=simulate,args=(output,i)) for i in indexes]
    # Start all the parallel threadsfor thread in threads: thread.start()
    # Wait for all the parallel threads to completefor thread in threads: thread.join()
    # Return the output listreturn output

defsimulate(list,index):
    (L,a) = L_a[index]
    list[index] = (a,L) # Add the rest of your objects here

master_list = RunParallelThreads()