Measuring build timings with mathbench
Fast iteration times are something that many game developers consider to be of utmost importance. Keeping build times short is a major component of quick iteration for a programmer. Aside from the actual time spent compiling, any time you have to wait long enough that you start to lose focus on the activity you are working on, or you start to get distracted or lose track of what you were doing which costs you more time.
Thus one of my goals when writing
glam was to ensure it was fast to compile.
Rust compile times are known to be a bit slow compared to many other languages,
and I didn’t want to pour fuel on to that particular fire.
As part of writing
glam I also wrote
mathbench so I could compare
performance with similar libraries. I also always wanted to include build time
comparisons as part of
mathbench and I’ve finally got around to doing that
with a new tool called
Introducing build bench
buildbench uses the unstable
-Z timings feature of
cargo, thus you
need a nightly build installed to use it.
buildbench generates a
Cargo.toml and empty
src/lib.rs in a temporary
directory for each bench crate, recording some build timing information from
cargo which is included in the summary table below. The temporary directory is
created every time the tool is run so this is a full build from a clean state.
Each bench is only built once so you may wish to run
times to ensure results are consistent.
By default crates are built using the
release profile with default features
enabled. There are options for building the
dev profile or without default
buildbench --help for more information.
The columns outputted include the total build time, the self build time which is the time it took to build the crate excluding dependencies, and the number of units which is the number of dependencies (this will be 2 at minimum).
When comparing build times keep in mind that each library has different feature sets and that naturally larger libraries will take longer to build. For many crates tested the dependencies take longer than the main crate. Also keep in mind if you are already using one of the dependencies in your project you won’t pay the build cost twice (unless it’s a different version).
|crate||version||total (s)||self (s)||units||full report link|
|cgmath||0.17.0||6.5||3.0||17||cgmath build timings|
|euclid||0.20.5||3.2||1.1||4||euclid build timings|
|glam||0.8.6||0.8||0.5||3||glam build timings|
|nalgebra||0.21.0||32.1||17.8||29||nalgebra build timings|
|pathfinder_geometry||0.5.0||5.6||0.3||8||pathfinder build timings|
|ultraviolet||0.4.5||2.4||1.2||4||ultraviolet build timings|
|vek||0.10.1||38.0||10.6||16||vek build timings|
These benchmarks were performed on an Intel i7-4710HQ CPU with 16GB RAM and a Toshiba MQ01ABD100 HDD (SATA 3Gbps 5400RPM) on Linux.
ultraviolet has been included in the build timings but I want to
resolve issue #21 before including it in performance benchmarks.
Considering the results
It seems I achieved my goal of making
glam fast to build! As
glam grows and
gets more features build times will of course increase, but a few seconds is the
ballpark I’m hoping to stay in.
What isn’t being measured
buildbench is only measuring the time to do a full build. Incremental build
times would be very interesting since that’s mostly what people will be doing
during development. Especially from the point of view of iteration time.
Unfortunately I think it would be quite time consuming to write a meaningful
test of incremental build times across all crates in
mathbench. To do that I
think I’d need to write the equivalent amount of code for each crate, enough
that changing it shows up in build times. It would be great to see the results
of that, but it’s a lot more effort. For now I’ll have to live with comparing
full build times.
Related to that, Maik Klein pointed out that one thing I am not measuring is
the cost of generics in user code. When using generics a lot of the cost is
shifted into the crate that is instantiating them. That is not something that
buildbench is attempting to measure at the moment.
One big difference between all of these crates is their code size. Most are
oriented towards game development with the exception of
nalgebra which has
much broader design goals and supports many more features than
about 8.5K lines of Rust,
nalgebra is more like 40K.
glam is much closer to
cgmath but without generics. Aside
from the use of generics one big difference between
indeed most of the other crates is what dependencies are included by default.
The most recent release of
cgmath on crates.io has non-optional dependencies
rand. I noticed on
rand is now
optional, but there hasn’t been a release made in a while.
In addition to making most
glam dependencies optional I also made all optional
features opt-in rather than opt-out. That is
glam optional dependencies are
not enabled by default. That is part of why
glam’s build time is lower than
the others, by default its only dependency is
glam does has have
rand but you have to enable those features if
you want to use them.
vek stands out as taking a significant amount of time compared to the others.
One thing to note though is the self time at 10.7 seconds is a lot less than
17.9 seconds of
nalgebra. A large portion of the time building
No default features surprise
You could argue that for a fair comparison I should be building all these crates
default-features = false. That is true and
buildbench does support
this, but unfortunately with some crates one does not simply disable default
features. Many libraries that support no_std do so by making a
which is included in the
default feature list. Thus
no_std can be supported
by disabling default features on a dependency using
default-features = false.
Because of this building with
default-features = false can give surprising
Early versions of
mathbench disabled default features for all the math
libraries in an attempt to improve build times. I encountered a few issues
taking this approach and eventually went back to using default features.
One problem was
nalgebra was getting really poor results for some benchmarks
in a way that didn’t make much sense. The dot product benchmark was similar to
other libraries but vector length was really slow, the only difference between
those is a square root. Fortunately some members of the Rust Community Discord
prompted the author of
nalgebra to investigate the issue and submitted a PR.
The problem was that disabling defaults effectively put
nalgebra in no_std
mode. In this mode it changed the way math libraries were linked which meant
calls to functions like
sin were no longer inlined, which has a big
performance impact. I also had an issue raised to benchmark with default
features enabled as that’s what most people will use so between these two things
I started building all libraries with default features enabled.
nalgebra no_std issue was quite surprising to me. I hadn’t realised the
no_std and If I wasn’t writing benchmarks I don’t think I would have
noticed that something strange was going on. This is mentioned in the
documentation under web assembly and embedded programming but I think most
people aren’t going to go looking for that when disabling default features.
vek also has some surprising issues around disabling default features and
support for no_std.
vek has the largest total build time of all the crates
tested, but it’s self time is only 25% of the total build time and building
dependences are the other 75% or 30 seconds on my laptop. Looking at vek build
serde_derive crates are a large chunk of that 30
seconds, According to crates.io the in the 0.10.1 release that I was using,
serde is an optional dependency. OK, so I’ll build
default-features = false, but this is intended for use with no_std so if you
just disable default features
vek doesn’t build at all. To build with no_std
you need to manually add the
libm feature, which I assume will link the
necessary math routines. I imagine that this will have the same performance
implications that it did for
nalgebra. On the bright side this bought the
total build time down to 7.54 seconds!. If you are still building for std though
things get a bit trickier. You need to manually add the
feature. Doing this I was able to build in 6.93 seconds. Initially I tried
disabling default features and enabling
std again but this adds the
serde/std feature, pulling in the
serde_derive crates which were
supposed to be optional. I removed the
serde/std from the
std feature of
vek and it compiled, so perhaps it isn’t necessary. Removing it certainly
makes the default build a lot faster.
I’m glad it is possible but it’s really not obvious how to build with default features disabled.
This isn’t really a criticism of
vek. I feel that this confusion
has arisen due to the convention of using
default-features = false to build
for no_std. If all you want to do is reduce unused feature dependencies
default-features = false should be the right lever to push but in reality due
to this being conflated with building for no_std it’s often not that simple.
One criticism I do have of many crates is this kind of behaviour is not well
vek do document how to build for no_std but
if you just want to disable optional features in a std build you seem to be on
No default features
With all of that out of the way, I think that making the optional features of
glam not default features was a good choice. As I’ve discussed above turning
off default features has not always been an easy thing to do. I think having
them off by default and documenting what features are available and how they can
be enabled might be a better approach for the majority of users.
While providing benchmarks for the default features is the most useful benchmark
it doesn’t always represent the minimal build time for many crates.
Unfortunately it seems I will need to tune the minimal set of features for each
crate to achieve this. That’s something I was trying to avoid doing but I think
in fairness I will support it in a future version of
adding support for building all dependencies might also be informative.
As usual trying to compare libraries on some metric turned out to be not that simple.
Turning off default features is harder than it should be. This setting is often
an alias for
#![no_std] support. I think that’s unfortunate and perhaps there
should be more explicit flags added to
cargo to build for no_std. The Rust
language is a very explicit language, there is not a lot of hidden behaviour.
The pattern of using default features to control std versus no_std builds feels
contrary to that.
Ultimately the point of this exercise was to provide another metric to consider when choosing a math library. Are you paying for features you aren’t using?