• CompilationSpeed

Method

Update to C++ compilation test done previously, and supplement continuous timing results.

Five compilers are tested:

gcc-4.1 gcc-4.4 gcc-4.6 clang-2.8 icc-2.8

For each, run through 2557 individual translation units comprised of select files from the libstdc++ testsuite directory using scripts/check_compile_only. Of the initial 4789 files, all C++0x files were removed, and then other problematic files were removed based on early results. The goal is to have a select number of files, including a limited number of known negative tests, that can be compiled by a wide variety of compilers. No attempt is made to cache or parallelize compiles, indeed the point is to compile sequentially.

The test machine is a corei7/12GB machine running F14/x86_64.

Compile flags were varied to try and measure various parts of the compiler in isolation.

Times were collected for various options:

• -S -O0
  • Times the C++ FE.
• -S -O0 -g
  • Times the C++ FE and debug information generation.
• -S -O2
  • Times the C++ FE and the optimizers.
• -S -O0 -include stdc++.h "prefix"
  • Times the C++ FE using a prefix file included before all sources.
• -S -O0 "iwyu"
  • Times the C++ FE with unmodified sources. "Include What You Use."

When PCH files are used, the size of the generated file is noted. Time is in seconds. A percentage is given, and that is the ratio of elapsed seconds over the "iwyu" time.

Opinions vary widely on what style of C++ constitutes a good test case for measuring compilation speed. Users of C++ compilers no doubt have their own favorite pathological code examples. For that reason, we should ideally include more than one set of sources, instead of relying on one particular project. No doubt code samples can be constructed that highlight the deficiencies of any compiler, so examples should be taken with a grain of salt, and should be re-sampled to account for new designs and techniques. For instance, meta-programming techniques, which already tax C++ front ends, might have difference performance footprints on varying implementations.

Candidates for base source material for additional tests include parts of KDE or webkit or testsuites thereof, boost or boost::mpl, others.

Compilation Speed

ONE

TWO

THREE

FOUR

FIVE

Observations

GCC Notes: PCH timings remarkably consistent through generations. Worrisome compilation speed with -std=gnu++0x as features mature. Runtime sources tested are as per GCC version under test.

ICC Notes: weakref means extern. Runtime sources tested are as per GCC 4.1. Has explicit creation and use flags, unable to use manual pch files via -pch-create/-pch-use nor via deprecated -create_pch/-use_pch. Has automatic PCH mode via -pch flag. Can be used with runtime sources more recent than GCC 4.1. For the 4.4 release, see 137 total fails. (New fails from 4.3+ variadic template use in tr1/type_traits and tr1/tuple). For the 4.6 release, see 551 total fails. (New fails from float = float complex, invalid multibyte sequence, is_trivial, decltype)

Clang Notes: weakref means static. Runtime sources tested are as per GCC 4.1. No equivalent to -fpermissive. Template specialization parse error, -Wbool-conversions, dependant type lookup vs. typename, template argument deduction, template template parameters. Has explicit creation and use flags (-emit-pch/-include-pch) and supports other header caching schemes. Parse is strict and includes more syntax checking on template bodies than other compilers, but less flexibility to selectively turn off diagnostics.