Throughput measurements¶
The throughput \(\dot N\) is defined as the number of grid point updates per second:
This metric allows us to compare the performance of different solvers and configurations. The higher the throughput, the better the performance.
The throughput is expected to be:
for the NumPy solver: almost constant with respect to the domain size
for the MPI solver at a constant number of CPU cores: almost constant with respect to the domain size if the domain is large enough to amortize the communication overhead
for the OpenCL solver: increasing with respect to the domain size, as the GPU is better utilized with larger domains.
We run the llg3d.bench.throughput script that executes the
llg3d.benchmarks.throughput module to measure the
throughput of the different solvers on different hardware configurations:
Parallelipedic domains of size \((J_x \times 24 \times 24)\) with \(J_x = 128, 256, ..., 4096\)
solvers:
NumPy solver
MPI solver with 8 or 32 processes
OpenCL solver with a GPU device
The number of iterations is varied according to the domain size to keep the total computation time. Tests are performed in single precision to be compatible with the Apple M3 GPU.
Performance on an Apple M3 Max¶
We run the benchmark with 8 MPI processes and 10 repeats for each domain size.
$ llg3d.bench.throughput run --np 8 --repeats 10 --csv m3/bench_throughput.csv
It produces the CSV file m3/bench_throughput.csv than can
be used to generate the report from the m3 directory:
$ llg3d.bench.throughput report bench_throughput.csv
CPU: Apple M3 Max | GPU: Apple M3 Max
Domain size NumPy (1 CPU core) MPI (8 CPU cores) (Accel) OpenCL (1 GPU) (Accel)
------------- -------------------- --------------------------- ------------------------
128 2.5e+07 1.6e+08 ( 6.4x) 1.2e+09 ( 48.9x)
256 2.2e+07 1.5e+08 ( 7.0x) 2.2e+09 (101.4x)
512 2e+07 1.5e+08 ( 7.3x) 3.6e+09 (176.0x)
1024 2e+07 1.3e+08 ( 6.6x) 4.1e+09 (206.2x)
2048 1.8e+07 9.7e+07 ( 5.4x) 3.1e+09 (171.3x)
4096 2e+07 7.9e+07 ( 3.9x) 2.8e+09 (136.0x)
and the resulting plot using:
$ llg3d.bench.throughput plot bench_throughput.csv
The NumPy solver shows an almost constant throughput, as expected.
The MPI solver with 8 processes shows a decreasing throughput with the domain size. The reason for this behavior is not clear.
The OpenCL solver shows an exponentially increasing throughput when the domain size increases from 128 to 512. A maximum is reached around 1024, then the throughput stabilises, remaining much higher than the NumPy solver.
Acceleration
For \(J_x = 1024\), OpenCL is \(200 \times\) faster than NumPy!
Performance on a multi-core server with GPUs¶
We now test the performance on a server with 32 CPU cores equipped with 3 AMD
Instinct MI210 GPUs (only one GPU is used here) using the following
sbatch.slurm script:
#!/bin/bash
#SBATCH -p gpu # targeting the gpu partition
#SBATCH --ntasks-per-core=1 # disabling multithreading
#SBATCH -n 32 # number of CPU cores
#SBATCH --gres=gpu:1 # ask for 1 GPU
#SBATCH -J llg3d-gpu # job name
# Activating the Python virtual environment
source ../../../../.venv/bin/activate
# Launching the computation
llg3d.bench.throughput run --repeats 10 --csv gaya/bench_throughput.csv
We submit the job with:
$ sbatch sbatch.slurm
It produces the CSV file gaya/bench_throughput.csv than can
be used to generate the report from the gaya directory:
$ llg3d.bench.throughput report bench_throughput.csv
CPU: AMD EPYC 7313 16-Core Processor | GPU: AMD Instinct MI210
Domain size NumPy (1 CPU core) MPI (32 CPU cores) (Accel) OpenCL (1 GPU) (Accel)
------------- -------------------- ---------------------------- ------------------------
128 1.1e+07 1.5e+08 ( 13.1x) 2.3e+09 (208.5x)
256 9.6e+06 2.0e+08 ( 21.2x) 3.7e+09 (386.7x)
512 9.3e+06 2.3e+08 ( 24.4x) 4.6e+09 (501.7x)
1024 1e+07 2.7e+08 ( 26.8x) 5.7e+09 (572.1x)
2048 1e+07 2.4e+08 ( 23.4x) 6.7e+09 (659.4x)
4096 9.6e+06 2.3e+08 ( 24.3x) 7.2e+09 (749.9x)
and the resulting plot using:
$ llg3d.bench.throughput plot bench_throughput.csv
The NumPy solver shows an almost constant throughput, as expected.
The MPI solver with 32 processes shows an increasing throughput with the domain size up to \(J_x = 1024\) then a slight decrease. Compared to the NumPy executions, the acceleration is close to 32 in the 512-4096 range.
On this MI210 GPU, the OpenCL solver shows a monotonic increasing throughput in a ratio of 3 from 128 to 4096.
Acceleration
For \(512 \leq J_x \leq 4096\), OpenCL is between \(500\times\) and \(750\times\) faster than NumPy!
Comparison of three GPU platforms¶
We compare the throughput of the OpenCL solver on three different GPU platforms in single precision:
Apple M3 Max (64GB RAM, 40 GPU cores)
AMD Instinct MI210
NVIDIA V100 (16GB RAM PCIe)
$ llg3d.bench.throughput compare m3/bench_throughput.csv gaya/bench_throughput.csv v100/bench_throughput.csv --solver opencl
Saved comparison plot to bench_throughput_comparison_opencl.png
Note
The throughput benchmark runs with no data recording: only the compute kernels are measured. In practice, the total execution time of a simulation includes additional overheads such as data recording, which can significantly impact the overall performance, especially for OpenCL where the compute time is very low.