This notebook shows the usage of BenchmarkLite.jl, a lightweight Julia package for performance benchmarking.
Suppose we want to compare the performance of several math functions (applied in batch to vectors). We can do this in several steps:
Like other packages, one can load a package with either import
or using
. Most of the methods in this package are extended from Julia Base. Hence, import
should be good enough in typical cases. However, if you want to access the types like Proc
and BenchmarkTable
more conveniently, you may use using
.
The package is very lightweight. So the package should load very fast.
using BenchmarkLite
All procedures to be benchmarked should be defined as subtypes of Proc
, which is an abstract type defined in the BechmarkLite
module. Several methods need to be defined for a procedure. Each procedure can be run under differen configs.
string(proc)
:
a short name to identify procedure (this will be used when showing the benchmark table)
length(proc, cfg)
:
the size of the problem under certain configuration. For example, if the procedure is to run computation of some function over n
elements, then this function is to return n
.
isvalid(proc, cfg)
:
whether the procedure can be run under the given configuration cfg
.
s = start(proc, cfg)
:
initialize states to support the procedure (e.g. allocating necessary memory, or connecting to a database). This part is not counted in the run-time of the procedure.
run(proc, cfg, s)
:
run the procedure under certain given config (together with initialized states)
done(proc, cfg, s)
:
de-initialize the run-time states (e.g. closes a file or database connection)
Note: all these methods are extended from Julia Base.
Now we define a VecMath
subtype to represent the procedures:
type VecMath{Op} <: Proc end
Here, the type parameter Op
can be Sqrt
, Exp
etc, as we defined below, to represent the calculation we want to perform on each scalar. Using types to represent functions, allow specific computation to be inlined without incurring runtime overheads.
type Sqrt end
calc(::Sqrt, x) = sqrt(x)
type Exp end
calc(::Exp, x) = exp(x)
type Log end
calc(::Log, x) = log(x);
Define procedure names:
Base.string{Op}(::VecMath{Op}) = string("vec-", lowercase("$Op"));
string(VecMath{Sqrt}())
"vec-sqrt"
To preclude the memory allocation time from the benchmark, we need to allocate arrays of specific sizes in advance, and store them as the initialized states. Particularly, we need to vectors, one for input, and the other for output. We use FVecPair
as a shortname to represent such a bi-vector state:
typealias FVecPair (Vector{Float64},Vector{Float64});
The configuration is vector length, which can be simply represented by an integer. Then, we can define the procedures as follows:
Base.length(p::VecMath, n::Int) = n
Base.isvalid(p::VecMath, n::Int) = (n > 0)
Base.start(p::VecMath, n::Int) = (rand(n), zeros(n))
function Base.run{Op}(p::VecMath{Op}, n::Int, s::FVecPair)
x, y = s
op = Op()
for i = 1:n
@inbounds y[i] = calc(op, x[i])
end
end
Base.done(p::VecMath, n, s) = nothing;
Collect all procedures into a Proc
-vector, as
procs = Proc[ VecMath{Sqrt}(),
VecMath{Exp}(),
VecMath{Log}() ];
Collect all configurations into an Int
-vector, as
cfgs = 2 .^ (4:10)
7-element Array{Int64,1}: 16 32 64 128 256 512 1024
Now, we call run
to actually run the benchmark. For each procedure under each configuration, there are three stages of running:
warming up:
it runs the procedure under the given configuration once, which triggers the pre-compilation of the function.
probing:
it runs the procedure again to roughly estimate the time needed to run it once. Then the total number of runs is determined such that the entire duration of measuring takes about 1 second. If you want to change this duration, you may set it using the duration
keyword argument. For example, duration = 0.5
means having each procedure under each configuration run for about 0.5 second.
measuring:
it runs the procedure a number of times (the number of times is decided in the probing stage), and records the elapsed time.
rtable = run(procs, cfgs);
Benchmarking vec-sqrt ... vec-sqrt with cfg = 16: nruns = 2816902, elapsed = 0.181171339 secs vec-sqrt with cfg = 32: nruns = 2949853, elapsed = 0.383015011 secs vec-sqrt with cfg = 64: nruns = 2096437, elapsed = 0.541707007 secs vec-sqrt with cfg = 128: nruns = 1394701, elapsed = 0.707816771 secs vec-sqrt with cfg = 256: nruns = 800000, elapsed = 0.809421738 secs vec-sqrt with cfg = 512: nruns = 431407, elapsed = 0.872099936 secs vec-sqrt with cfg = 1024: nruns = 224568, elapsed = 0.908604125 secs Benchmarking vec-exp ... vec-exp with cfg = 16: nruns = 675220, elapsed = 0.830725251 secs vec-exp with cfg = 32: nruns = 372440, elapsed = 0.90710495 secs vec-exp with cfg = 64: nruns = 194402, elapsed = 0.945588553 secs vec-exp with cfg = 128: nruns = 99286, elapsed = 0.96417331 secs vec-exp with cfg = 256: nruns = 34761, elapsed = 0.680242924 secs vec-exp with cfg = 512: nruns = 24515, elapsed = 0.956955982 secs vec-exp with cfg = 1024: nruns = 11338, elapsed = 0.881633733 secs Benchmarking vec-log ... vec-log with cfg = 16: nruns = 670691, elapsed = 0.808676817 secs vec-log with cfg = 32: nruns = 378788, elapsed = 0.900356022 secs vec-log with cfg = 64: nruns = 198689, elapsed = 0.946116462 secs vec-log with cfg = 128: nruns = 100827, elapsed = 0.965859782 secs vec-log with cfg = 256: nruns = 49604, elapsed = 0.948375143 secs vec-log with cfg = 512: nruns = 24885, elapsed = 0.983452456 secs vec-log with cfg = 1024: nruns = 12500, elapsed = 0.96543341 secs
The result is stored in an instance of BenchmarkTable
, which can be shown in different units. For example, you can show how many milliseconds each procedure takes (under various configuration):
show(rtable; unit=:msec)
BenchmarkTable [unit = msec] config | vec-sqrt vec-exp vec-log -------------------------------------- 16 | 0.0001 0.0012 0.0012 32 | 0.0001 0.0024 0.0024 64 | 0.0003 0.0049 0.0048 128 | 0.0005 0.0097 0.0096 256 | 0.0010 0.0196 0.0191 512 | 0.0020 0.0390 0.0395 1024 | 0.0040 0.0778 0.0772
If millisecond is not precise enough, you may try showing in terms of microseconds:
show(rtable; unit=:usec)
BenchmarkTable [unit = usec] config | vec-sqrt vec-exp vec-log -------------------------------------- 16 | 0.0643 1.2303 1.2057 32 | 0.1298 2.4356 2.3769 64 | 0.2584 4.8641 4.7618 128 | 0.5075 9.7111 9.5794 256 | 1.0118 19.5691 19.1189 512 | 2.0215 39.0355 39.5199 1024 | 4.0460 77.7592 77.2347
Sometimes, you may want to watch the results in terms of speed (e.g., MPS, million numbers per second):
show(rtable; unit=:mps)
BenchmarkTable [unit = mps] config | vec-sqrt vec-exp vec-log -------------------------------------- 16 | 248.7724 13.0049 13.2699 32 | 246.4533 13.1386 13.4627 64 | 247.6836 13.1577 13.4403 128 | 252.2146 13.1808 13.3620 256 | 253.0201 13.0818 13.3899 512 | 253.2742 13.1163 12.9555 1024 | 253.0889 13.1689 13.2583
Here is a list of supported units:
unit |
description |
---|---|
:sec |
seconds per run |
:msec |
milliseconds per run |
:usec |
microseconds per run |
:nsec |
nanoseconds per run |
:ups |
how many items/numbers per second. note: the number of items per run is determined by length(proc, cfg) |
:kps |
how many thousand items/numbers per second |
:mps |
how many million items/numbers per second |
:gps |
how many trillion items/numbers per seoncd |