Benchmarking suite – Report 11

This is the report of the project “Automated benchmark suite for numerical libraries in Gentoo” for the week 1 – 7 August.

Project description

The project aims to develop a simple yet powerful automated system of benchmarking for numerical libraries. The Gentoo software system provides many implementations of widely used standards such as BLAS, CBLAS, LAPACK, ScaLAPACK and some other numerical library such as FFTW, MKL. The developed tools will aid the system maintener to choose the best suited implementation with respect to the machine hardware and to test the samer implementation or different ones with different compilers, compiler versions and compile flags.

Progrees during the week

This week has been devoted to the following features:

  • Stabilization of the PBLAS/ScaLAPACK tests
  • Introduction of the METIS tests
  • Implementation of the LAPACK accuracy tests
  • Refinement of the LAPACK performance tests


METIS is a package — a set of executables and a library — that perform some common preliminary numerical work: graph partitioning, mesh partitioning, sparse matrix reordering. The module taht has been introduced benchmarks the run times of the executables pmetis and kmetis when doing the most important task: graph partitioning — the other task rely on this one. In order to keep the benchmarks fair, the same input data is processed by every “implementation” (i.e. for each different compilation of the sci-libs/metis package). The module is already stable and usable, although some more tests could be added.

LAPACK performance tests

We decided to benchmark matrix decompositions rather than full solvers, as the decompositions are the real core of each LAPACK solver, while the substitutions and transposed matrix-vector multiplication are BLAS tasks that have to be benchmarked separately. Therefore the following tests have been deprecated — they are still available, but are not part of the standard set:

  • general_solve: General linear system of equations solver
  • least_squares: General linear least squares solver
  • symm_ev: Symmetric matrix eigensolver — eigenvalues only
Some new tests have been added, and the following are the resulting standard tests:
  • lu_decomp: LU decomposition
  • cholesky: Cholesky decomposition of a SPD matrix
  • qr_decomp (new): QR decomposition
  • svd_decomp (new): Singular Values Decomposition
  • syev (new): Symmetric matrix eigensolver (eigenvalues and eigenvectors), full diagonalization
  • stev (new): Tridiagonal matrix eigensolver (eigenvalues and eigenvectors), full diagonalization

LAPACK accuracy tests

Following the module strategy, a new much more interesting module has been added for testing the different LAPACK implementations for accuracy. This includes every LAPACK standard tests (see above section). The matrix decomposition are tested by multiplying the results and comparing this to the original matrix; the eigensolvers are treated as decompositions (diagonalization) and tested the same way. Some reports are available on my homepage.

Plan for the next weeks

The GSoC programme is reaching the end. Next week is the soft “pencils down” date. Therefore I will spend the next week doing only tests and bugfixes, writing documentation and performing other “administrative” tasks (ebuild refinment, repository management,…). On Friday the unstable branch of the repository will be merged to the stable. Then a full report will be written and sent to the mailing list.

After the soft “pencils down” date, only critical bugs will be fixed and more documentation will be added. Then, I will release a stable, tested, documented and supported version of my suite just before the hard “pencils down” date.

I would appreciaty very much some helping hand in testing my script! If somebody is interested in having a look of it, it very easy to install the required packages and run some tests. So please if you have some time, let it run and report bugs that you find (if any) to me; if you don’t find bugs, please write to me as well! I will write a short howto for run the tests and send the link to the mailing list.

Best regards
Andrea Arteaga


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: