Introduction

The objective of this assignment is 2 fold.

1. Solidify the concepts needed to distribute data between PEs given a ghost cell region of depth 1 using MPI.
2. Develop an intuition for the overhead incurred by using interprocess communication between every timestamp of a calculation.

To achieve each of these you will be performing a series of programming tasks followed by a series of experimental tasks. The coding is quite tricky. I suggest you follow the steps outlined below. You will be implementing the communication required for a 2D data decomposition of Jacobi2D in MPI. The vast majority of the code is supplied. There is a comment in the code in the locations that you need to update. Take note that you will be given the opportunity to turn in a partial result. If you are able to get the vertical communication working, but unable to get the horizontal communication working you make copy your file to one named Jacobi2D-Block-MPI-VERT.test.c. We will use this one for automated testing in the vertical direction if it exists. This file is not required. Recall from lecture that we are making some simplifying assumptions. The data domain is square ( -p problemSize). Your block may or may not be square, but each PE only gets a single block. This means that there is a mathematical relationship between the problem size the PE count and the block sizes (-x xsize -y ysize). You need to figure that out (and put it in your report).

Programming Tasks

1. Insert the code needed to exchange ghost cells between neighbors to the north and south of each block. Remember to be congnizant of the order of your sends and recieves as well as making sure you don't attemp to send to a rank that does not exist (for instance -1). If this is all the further you get you can turn this in as per the note above. This will be less than 50% credit on the coding portion of the exercise.
2. Insert the code needed to pack a column of a 2D array as implemented in the code into a single vector. There is an empty function at the top of the file for you to use for this purpose.(packColToVec). You will need to call this function before MPI_Send when sending data east or west.
3. Insert the code needed to read a vector and unpack to to a column of a 2D array. There is an empty function at the top of the file for you to use for this purpose.(unpackVecToCol). You will need to call this function after MPI_Recv when getting data from the east or the west.
4. Insert the code needed to exchange ghost cells between neighbors to the east and the west of each block. Keep in mind the same gotchas that you ran into for the north and south neighbors.

Experimental Tasks

This assignment is purposely leaving this more vague than previous assignments. You should come up with a hypothesis about which tile shape is going to perform best for this algorithm and test that hypothesis. Your hypothesis should be the first section of your report. You then need to choose (on your own) three different sizes of p to run the tests on. You have some information to help you. First, think about what a real application (maybe a CFD application being run within a PDE Frameowrk) will have as a footprint. Also, consider cases that will spill out of on chip cache. Run your experiments on the Cray and report result up to a number of PEs that demonstrates where your code stops scaling. Stop at 64 if you have perfectly scaling code. Use -T 100 for your experimental results. (I would debug with -T 1 and -T 2) Note that the compiler options that are currently in Jacobi2D-Block-MPI.perfcompopts are NOT appropriate for performance results. You need to change this file.

Code Submission

Submit your PA3.tar which should contain your Makefile, src directory with at least 2 files (Jacobi2D-Block-MPI.test.s, Jacobi2D-Block-MPI.perfcompopts). You may also include Jacobi2D-Block-MPI-VERT.test.c if you would like your vertical tiling only code to be tested seperatly. Do not change src/util.h.Grading will be done using ours. Submit your tarball through the class website. This will give you some initial testing results.

Report Submission

Outline

1. Hypothesis
2. Algorithm Description
3. Blocking description: inlcude details on ghost cell exchanges.
4. Experimental Setup (Including chosen data sizes and tile shapes.)
5. Results
6. Conclusions and Analysis (was your hypothesis correct?).

Some Notes on Getting Started

Dowload the Starter code. I am providing you with the MPI code complete with validation code that is lacking any ghost cell exchange. When you open the tarball attempt to make the executable by typing the following:

make Jacobi2D-BlockParallel-MPI

Then attempt to run the code and make sure that it validates for serial execution.

$ mpiexec -np 1 bin/Jacobi2D-BlockParallel-MPI -p 8 -x 8 -y 8 -v
Time: 0.000094
SUCCESS

Note the command line arguments.
p is the problem size.
x is the length of one tile in the x direction.
y is the length of one tile in the y direction.
v indicates that validation should take place.
Do NOT change the command line argument parsing code.
Do NOT run your experiments with validation on