Quick summary for the day

Upgrade to new versions of mpi4py and OpenMPI went successfully. Then I decided to check whether my assumption that I could get some performance out of replacing the currently used in Topographica collective routines with different versions was correct. In particular, I wanted to see whether Scatter would work faster than scatter and Gather - than gather. The assumption was based on this conversation on mpi4py googlegroup: http://groups.google.com/group/mpi4py/browse_thread/thread/a1741896517c7d12/5c5e604047365348?lnk=gst&q=konstantin#5c5e604047365348. This way I though I would re-familiarise myself with topographica code and also hope to get some performance improvement.

To summarise, scatter/gather involve pickling while Scatter/Gather just make use of the underlying C routines MPI_Gather() and MPI_Scatter(). Thus, the advantage of the first way is that it can communicate arbitrary Python objects while the latter is meant to be faster.

I decided to write a simple Python application that gathers and scatters (later on added broadcasting) large arrays of floats in a loop, 100 times each, and measures the execution  time for each routine. N controlled the size of individual chunks, so the master node (rank 0) would gather/scatter N*P elements, where P is the number of processes used. After having finished the coding I played around with different values of N and P, also running my code in SMP mode and MPP (i.e. using a cluster of 3 machines). Here are some results:


N=10,000
SMP, 3 nodes:


scatter time: 0.0482320785522
Scatter time: 0.0166721343994
bcast time: 0.0553240776062
Bcast time: 0.041867017746
gather time: 0.0940809249878
Gather time: 0.0110039710999


MPP, 3 nodes:

scatter time: 0.177076816559

Scatter time: 0.176827907562

bcast time: 0.211786031723

Bcast time: 0.213672876358

gather time: 0.22430896759

Gather time: 0.168394088745

N=50,000

SMP, 3 nodes:

scatter time: 0.220118045807

Scatter time: 0.0812599658966

bcast time: 0.321494102478

Bcast time: 0.180558919907

gather time: 0.190404176712

Gather time: 0.0511260032654

MPP, 3 nodes:

scatter time: 3.98046803474

Scatter time: 0.681133985519

bcast time: 1.60409212112

Bcast time: 1.39482522011

gather time: 0.849080085754

Gather time: 0.734710931778

N=200,000

SMP, 3 nodes

scatter time: 1.79250884056

Scatter time: 0.383674144745

bcast time: 2.98570203781

Bcast time: 0.743601083755

gather time: 1.25536489487

Gather time: 0.288822889328

MPP, 3 nodes

scatter time: 3.97047996521

Scatter time: 2.73933315277

bcast time: 6.40952301025

Bcast time: 5.25424790382

gather time: 3.54560399055

Gather time: 3.84726691246

As we can see, SMP benefits from non-pickling routines the most, and the difference in performance seems to increase with the amount of data in each packet sent. On the contrary, MPP hasn't shown any significant difference between the performance of two routines. Except in the Scatter/scatter case. I think there must be a hint there: what if the performance is limited by the interconnect bandwidth? Thing is, in my experiment, data is sent in loops, without any interruption or work done between the communication routine calls. This is quite far from what a real application would look like, so I decided to put some work there. Now each node on each gathering/scattering iteration will do some computations. To make it a bit similar to topographica, each node will compute a dot-product of the distributed array and a local set of data. I'll measure both the computation and gathering/scattering time.