MPI Programming in Practice: from Hello to Parallel Reduce

Abstract: Last article only printed; this one computes. Using a cluster-wide summation, we move from point-to-point (Send/Recv) to collective communication (MPI_Reduce), with complete compilable code, run instructions, and result checks.

1. The Problem: Parallel Summation

N ranks each hold $x_{\text{rank}} = \text{rank}+1$; compute the total.

• Serial: one person adds
• Parallel: each rank computes its value → aggregate to rank 0

2. Version 1: Point-to-Point (MPI_Send / MPI_Recv)

#include <mpi.h>
#include <stdio.h>

int main(int argc, char* argv[])
{
    MPI_Init(NULL, NULL);

    int size, rank;
    MPI_Comm_size(MPI_COMM_WORLD, &size);
    MPI_Comm_rank(MPI_COMM_WORLD, &rank);

    int value = rank + 1;
    int sum   = 0;

    if (rank == 0) {
        sum = value;
        for (int i = 1; i < size; i++) {
            int recv;
            MPI_Recv(&recv, 1, MPI_INT, i, 0, MPI_COMM_WORLD, MPI_STATUS_IGNORE);
            sum += recv;
        }
        printf("Total sum = %d\n", sum);
    } else {
        MPI_Send(&value, 1, MPI_INT, 0, 0, MPI_COMM_WORLD);
    }

    MPI_Finalize();
    return 0;
}

mpicc -o mpi_sum mpi_sum.c
mpirun -np 8 ./mpi_sum
# Total sum = 36        (1+2+...+8)

Clear logic, but rank 0 receives serially — a bottleneck as ranks grow.

3. Version 2: Collective (MPI_Reduce)

#include <mpi.h>
#include <stdio.h>

int main(int argc, char* argv[])
{
    MPI_Init(NULL, NULL);

    int size, rank;
    MPI_Comm_size(MPI_COMM_WORLD, &size);
    MPI_Comm_rank(MPI_COMM_WORLD, &rank);

    int value = rank + 1;
    int sum   = 0;

    MPI_Reduce(&value, &sum, 1, MPI_INT, MPI_SUM, 0, MPI_COMM_WORLD);

    if (rank == 0)
        printf("Total sum (Reduce) = %d\n", sum);

    MPI_Finalize();
    return 0;
}

mpicc -o mpi_sum2 mpi_sum2.c
mpirun -np 8 ./mpi_sum2
# Total sum (Reduce) = 36

One MPI_Reduce replaces all Send/Recv with a tree aggregation — far better scaling. That's why real HPC applications are built on collectives.

4. Collective Communication Quick Reference

Call	Purpose	Common use
MPI_Bcast	broadcast root → all	distribute parameters/matrices
MPI_Reduce	all → root, aggregated	sum, max, global error
MPI_Allreduce	reduce then broadcast to all	everyone needs the result
MPI_Scatter	split data among ranks	data partitioning
MPI_Gather	collect per-rank data at root	result collection
MPI_Barrier	synchronization	phase alignment

5. Run Notes

5.1 Ranks and nodes

# 8 ranks: 4 nodes × 2 ranks/node
mpirun -perhost 2 -machinefile hostfile -np 8 ./mpi_sum2

5.2 Multi-node fabric (same as hello world)

# Ethernet
mpirun -perhost 2 -np 8 -genv I_MPI_FABRICS shm:ofi -genv FI_PROVIDER tcp -machinefile hostfile ./mpi_sum2
# InfiniBand
mpirun -perhost 2 -np 8 -genv I_MPI_FABRICS shm:ofi -genv FI_PROVIDER mlx  -machinefile hostfile ./mpi_sum2

5.3 Verification

• Serial check: $1+2+\cdots+N = N(N+1)/2$ — compare with program output.
• Run with different -np values and confirm identical results — the first correctness gate of MPI.

6. From Summation to Real Workloads

The same pattern extends directly:

• Global error/norm: MPI_Allreduce(MPI_SUM) over per-rank local errors
• Parallel linear algebra: MPI_Scatter blocks + local compute + MPI_Gather (the seed of HPL)
• Numerical simulation: each rank owns a grid; MPI_Sendrecv exchanges boundaries

7. Summary

• Point-to-point suits small, targeted traffic; collectives scale.
• Parallel program verification: small-scale smoke → consistent results across -np → match serial reference.

This completes the CPU practical chain: theory → estimation → HPL measurement → MPI environment → hello → programming. Related guide: CPU HPL Benchmark Guide (Intel).