Hey! I tweaked my code a little bit to improve the speed of communications by replacing PMI invokes in some places with MPI scatter. The thing is, PMI invokes distribute data using MPI bcast, which is perfectly fine if all workers need to work with the same sets of data. However, if each worker has to process only a chunk of this data that corresponds to the node, it would be more logical to use MPI scatter that sends chunks of data to nodes (instead of the full data set, with each node taking its chunk from it) - that's what I thought. PMI does not have mechanisms for scattering, so I had to reimplement my invocation as a call, that breaks data down into chunks on node 0 and scatters it around the rest of nodes. Wondering if my tweak was at all useful, I constructed a simple test that checks how long it would take to send the same chunk of data using MPI bcast and scatter (I was only interested in communication time):
from mpi4py import MPI
from numpy.random import beta
from time import time
comm = MPI.COMM_WORLD
size = comm.Get_size()
rank = comm.Get_rank()
big_array = [beta(a=1,b=1000,size=(500,500)) for x in range(size)]
if rank == 0:
data = big_array
else:
data = None
scatter_time = 0
bcast_time = 0
tries = 3
for j in range(tries):
t0 = time()
for i in range(100):
x = comm.scatter(data, root=0)
scatter_time += time() - t0
t0 = time()
for i in range(100):
x = comm.bcast(data, root=0)
bcast_time += time() - t0
scatter_time = comm.gather(scatter_time/tries, root=0)
bcast_time = comm.gather(bcast_time/tries, root=0)
if rank==0:
print "COMM_WORLD of size", size
print "Scatter time:", sum(scatter_time) / len(scatter_time)
print "Bcast time:", sum(bcast_time) / len(bcast_time)
And the results of testing on Jupiter:
COMM_WORLD of size 2
Scatter time: 1.22514196237
Bcast time: 1.80673313141
COMM_WORLD of size 4
Scatter time: 3.10921456416
Bcast time: 5.65636410316
COMM_WORLD of size 6
Scatter time: 4.18686661455
Bcast time: 11.574133065
COMM_WORLD of size 8
Scatter time: 6.34783770641
Bcast time: 18.6762983203
Interesting, isn't it? That's one issue to take into consideration when programming with PMI.
Sounds like it's worth considering improving PMI to add scatter-based functions. I'd contact the author first before working on that, though! He/she might already have some ideas or suggestions.
ReplyDeleteJim
Indeed, I do.
ReplyDeleteThe reason why scattering is not in there (yet?) is, that PMI was intended mostly to provide a simple way to call parallel functions within an MPI framework, so that inside these functions MPI can be used for communication. That PMI itself also provides certain communication patterns is more or less a side effect for convenience. PMI should not replace the communication via MPI!
It would be no problem to extend the interface to allow for scattering, too. However, I wonder whether this would start something that doesn't really end. Are there other communication patterns that one would like to have? And does that mean that in the end I have to mimick all of MPIs communication patterns in PMI?
So, for now, I think it is probably the best to do the scattering inside the parallel functions and call them via pmi.call(), using __pmictr_-arguments.