The Scalable HeterOgeneous Computing (SHOC) benchmark suite was released in 2010 as a tool to evaluate the stability and performance of emerging heterogeneous architectures and to compare different programming models for compute devices used in those architectures. Since then, high-performance computing (HPC) system architectures have increasingly incorporated both discrete and fused multi-core and many-core processors. The TOP500 list illustrates this trend: heterogeneous systems grew from a 3.4% to 18.0% share of the list between June 2010 and June 2015. Not only are there more heterogeneous systems on the TOP500 list today, those machines are responsible for a disproportionately large percentage of list systems' aggregate performance: as of June 2015, the performance share for heterogeneous systems has grown to 33.7%.
Part of this shift toward heterogeneous architectures has stemmed from new products in the hardware accelerator market, such as Intel's Xeon Phi coprocessor, and improvements in the approaches for programming such accelerators. Existing approaches such as CUDA and OpenCL have become more powerful and easy to use, and directive-based programming models such as OpenACC, OpenMP 4.0, and Intel's Language Extensions for Offload (LEO) are gaining traction among users targeting systems with heterogeneous architectures. These hardware and software advances are not limited to HPC as they are increasingly used for "big data" applications that analyze large data sets.
We extended SHOC to support evaluation of recent heterogeneous architectures and programming models such as OpenACC and LEO, and we added new benchmarks to increase SHOC's coverage of application domains. In this paper, we describe our modifications to the stock SHOC distribution and present several examples of using our augmented version of SHOC for evaluation of recent heterogeneous architectures and programming models.