Benchmarking suite – Final Report

This is the final report of the project “Automated benchmark suite for numerical libraries in Gentoo” for the Google Summer of Code 2011.

Project description

The project aims to develop a simple yet powerful automated system of benchmarking for numerical libraries. The Gentoo software system provides many implementations of widely used standards such as BLAS, CBLAS, LAPACK, ScaLAPACK and some other numerical library such as FFTW, MKL. The developed tools will aid the system maintener to choose the best suited implementation with respect to the machine hardware and to test the same implementation or different ones with different compilers, compiler versions and compile flags.

Release

This report refers to the 0.1 release, which is the commit tagged with “0.1” on the git repository.

Archieved objectives

In the original project description, I pointed out that I would have delivered a benchmarking suite and a script, acting respectively as “data provider” for the benchmarking results and as system organizer for the work to do, user interface, base for the customization, … During the summer I decided to adopt the BTL (Bench Template Library) as benchmarking suite, and actually much work has been devoted to this; the BTL was initially designed to be extensible, and I actually used almost all extensible features of the BTL in order to adapt it for my purposes. In facts, I discovered that the BTL is not quite perfect and decided that, after the GSoC, I will spend some time trying to figure out how this very powerful library could improve even more. Anyway, it has performed well for my GSoC project, and the modified version is available on my hg repository.

The script is in facts a Python library that:

• Uses the featured provided by portage to search, compile, emerge packages in a separate root; resolves the dependencies (this part is still unstable); follows the user’s instruction on the environment to use during the emerge process; stores the generated binary packages for future usage (instruction to the user are printed at the end).
• Writes everything onto comprehensive and organized logs.
• Interprets an highly customizable configuration file provided by the user to define the tests that have to be run.
• Compiles the benchmarking suite with the correct options, flags, libraries,…
• Wraps the suite execution and provides user-friendly output; stores its results in an organized fashion.
• Collects the results and plots them.
• Saves the plots and generates a comprehensive HTML report; the report contains, along with the images, information about the system and the time, the logs, the configuration file and optionally a summary figure.
• Cleans the system.

The script is modular: for each different library a module has to be provided. The module specifies where the particular benchmarking suite main source is, how to compile it, how to run it, how to interpret the results,… As most of the modules are tested through the modified BTL library, a generic code is present in an abstract BTL module and the specific modules can just inherit this module and add a few information. The following modules make use of the BTL:

• blas
• cblas
• lapack
• scalapack
• fftw

The following modules make instead use of a different benchmarking suite, that is now part of the modified BTL, but follows a different implementation paradigm:

• blas_accuracy
• lapack_accuracy

These test the accuracy of the implementations instead of the computational speed.

The following module does not use a benchmarking suite, but relies on the information provided by the executable that is contained in the package:

• metis

A total of 8 modules are provided. This exceeds the initial expectations; to be honest, I have to say that some of them (in particular blas_accuracy and metis) are very basic modules and lack of some features, while the module scalapack was not tested much and is to be considered unstable.

Documentation

A web page is present at http://soc.dev.gentoo.org/~spiros/. Here some documentation about how to install the numbenc package is present. Since numbench is much Gentoo-specific, only instruction on how to install it on Gentoo are provided. In facts, it is completely useless without the emerge and equery commands. The page also gives an overview on hot to run the script.

The package installs a man page numbench(1) that explains in more detail how to configure and run a test, where the logs and the results are and gives some more information. A set of sampl configuration files come with the package, too, and are installed into /usr/share/numbench/samples.

The source (which is available on the auto-numerical-bench git repository for the script and on the cited mercurial repository for the benchmarking suite) contains some comments that could be useful for developers that want to write new modules or improve the project. In any case, if you plan adding features, please contact me!

License

The modified BTL has not changed license and is therefore re-released with the GPL -2 license. The same license is adopted for the Python part.

Results

On the web page a set of results are available. They cover almost all available module and also give examples of configuration files.

Acknowledgements

Fisrt of all, thanks to Google for sponsorizing the such a programme. This is of course useful for the students that have the possibility to do a real-work job (as student I often feel the need of a similar experience in my academic activities), earn something and, above all, at least for us Gentoo students, to get in touch with the FLOSS community.

Thanks to Donnie and the metors that make Gentoo being a perfect choice for a student who wants to participate to the SoC.

Many thanks to the whole Gentoo community that is always responsive and helpful. The forums, IRC channels and mailing-lists are full of experts that are ready to help you just as you need. Being a SoC student make you feel like a privileged and very respected person, which is just wonderful.

And a very big thank to my mentor Sébastien, who helped me a lot almost daily during the whole summer. Working with Sébastien is very motivating and his testing work has been very helpful. Thanks very much!

That’s all!

Best regards
Andrea Arteaga