Benchmarking suite – Final Report

This is the final report of the project “Automated benchmark suite for numerical libraries in Gentoo” for the Google Summer of Code 2011.

Project description

The project aims to develop a simple yet powerful automated system of benchmarking for numerical libraries. The Gentoo software system provides many implementations of widely used standards such as BLAS, CBLAS, LAPACK, ScaLAPACK and some other numerical library such as FFTW, MKL. The developed tools will aid the system maintener to choose the best suited implementation with respect to the machine hardware and to test the same implementation or different ones with different compilers, compiler versions and compile flags. Read more of this post

Advertisements

Benchmarking suite – Report 12

This is the report of the project “Automated benchmark suite for numerical libraries in Gentoo” for the week 8 – 15 August.

Project description

The project aims to develop a simple yet powerful automated system of benchmarking for numerical libraries. The Gentoo software system provides many implementations of widely used standards such as BLAS, CBLAS, LAPACK, ScaLAPACK and some other numerical library such as FFTW, MKL. The developed tools will aid the system maintener to choose the best suited implementation with respect to the machine hardware and to test the samer implementation or different ones with different compilers, compiler versions and compile flags. Read more of this post

Testing numbench

This article explains how to install the numbench script and to run the benchmarks.

Install app-benchmarks/numbench

First of all, the science overlay has to be added through layman; therefore, install layman on your Gentoo system if you don’t have it already, and do:

layman -L
layman -a science

After that you will be able to install the package app-benchmarks/numbench. Remember that numbench is still unstable and therefore I will need to install the ~x86/~amd64 version.

The package installs the executable “numbench”, some Python data and a man page numbench(1).

I recommend to add the bicatali overlay through layman too, because it contains many numerical libraries that can be benchmarked, even if we are migrating them into the science overlay.

Run the benchmarks

In order to run the benchmarks you have to provide a configuration file. The man page explains how to write one, and you will find some examples under /usr/share/numbench/samples. Once you have your configuration file (say conf.in), and you have chosen the module to test (e.g. blas, lapack or lapack_accuracy; see man numbench or numbench -h), just run the command

numench module conf.in -s

The documentation explains how to run the test with more parameters in order to choose the tests that have to be performed.

After the execution you will find interesting directories under ~/.benchmarks:

  • log contains the log, obviously; they are divided in subfolders in case of multiple runs
  • packages contains the packages that are useful if you decide to install some tested one (the documentation here lacks)
  • reports contains for each run a set of images, an HTML page and a copy of the logs; they are ready to be published somewhere, just copy the whole folder in your www directory
  • roots and tests are two directories which are used by the script for storing data; they are keeped in order not tu run the tests again if the results already exist
Please let me know if you find any bugs!

Benchmarking suite – Report 11

This is the report of the project “Automated benchmark suite for numerical libraries in Gentoo” for the week 1 – 7 August.

Project description

The project aims to develop a simple yet powerful automated system of benchmarking for numerical libraries. The Gentoo software system provides many implementations of widely used standards such as BLAS, CBLAS, LAPACK, ScaLAPACK and some other numerical library such as FFTW, MKL. The developed tools will aid the system maintener to choose the best suited implementation with respect to the machine hardware and to test the samer implementation or different ones with different compilers, compiler versions and compile flags. Read more of this post

Benchmarking suite – Report 10

This is the report of the project “Automated benchmark suite for numerical libraries in Gentoo” for the week 25 July-1 August.

Project description

The project aims to develop a simple yet powerful automated system of benchmarking for numerical libraries. The Gentoo software system provides many implementations of widely used standards such as BLAS, CBLAS, LAPACK, ScaLAPACK and some other numerical library such as FFTW, MKL. The developed tools will aid the system maintener to choose the best suited implementation with respect to the machine hardware and to test the samer implementation or different ones with different compilers, compiler versions and compile flags. Read more of this post

Benchmarking suite – Report 9

This is the report of the project “Automated benchmark suite for numerical libraries in Gentoo” for the week 18-24 July.

Project description

The project aims to develop a simple yet powerful automated system of benchmarking for numerical libraries. The Gentoo software system provides many implementations of widely used standards such as BLAS, CBLAS, LAPACK, ScaLAPACK and some other numerical library such as FFTW, MKL. The developed tools will aid the system maintener to choose the best suited implementation with respect to the machine hardware and to test the samer implementation or different ones with different compilers, compiler versions and compile flags. Read more of this post

Benchmarking suite – Report 8

A very short log of what has been done during the last days. I am
working on the PBLAS and ScaLAPACK benchmarks, which is a very
challenging topic, because it is very difficult to debug such
applications.

* I changed some parts of the BTL framework, adapting it to the
distributed memory benchmarks. This has required writing two new
perfanalyzers — one for the root process, one for the other (node)
processes. The nodes do not perform any measurement, while the root
process broadcasts the needed informations, measures the time and
manages the output (both std{out,err} and resulting file).

* I added a BLACS library that provides an useful interface which
scatters and gathers matrices and vectors. I also added a PBLAS
library that inherits the BLACS one and will support the most common
operations (at the moment just the parallel matrix-vector
multiplication).

* I added an action for the parallel matrix-vector multiplication
which makes use of the two described interfaces.

The matrix-vector multiplication is a case study for now. If
everything goes fine (and it seems so, now), then more actions will be
provided, for both PBLAS and ScaLAPACL, which share the same concepts.
I plan to have tomorrow a working (but incomplete) Python module for
PBLAS, too.

Milestones for the next week:
* Having working PBLAS and ScaLAPACK modules
* Do some benchmarks using these modules and publish the results
* Start the implementation of the advanced FFTW benchmarks, as
previously described

Best regards
Andrea Arteaga