This notebook shows the usage of BenchmarkLite.jl, a lightweight Julia package for performance benchmarking.

Suppose we want to compare the performance of several math functions (applied in batch to vectors). We can do this in several steps:

  • Load the BenchmarkLite package
  • Define the procedures to be tested
  • Run the benchmark
  • See the results

Loading the package

Like other packages, one can load a package with either import or using. Most of the methods in this package are extended from Julia Base. Hence, import should be good enough in typical cases. However, if you want to access the types like Proc and BenchmarkTable more conveniently, you may use using.

The package is very lightweight. So the package should load very fast.

In [1]:
using BenchmarkLite

Defining the procedures

All procedures to be benchmarked should be defined as subtypes of Proc, which is an abstract type defined in the BechmarkLite module. Several methods need to be defined for a procedure. Each procedure can be run under differen configs.

  • string(proc):

    a short name to identify procedure (this will be used when showing the benchmark table)

  • length(proc, cfg):

    the size of the problem under certain configuration. For example, if the procedure is to run computation of some function over n elements, then this function is to return n.

  • isvalid(proc, cfg):

    whether the procedure can be run under the given configuration cfg.

  • s = start(proc, cfg):

    initialize states to support the procedure (e.g. allocating necessary memory, or connecting to a database). This part is not counted in the run-time of the procedure.

  • run(proc, cfg, s):

    run the procedure under certain given config (together with initialized states)

  • done(proc, cfg, s):

    de-initialize the run-time states (e.g. closes a file or database connection)

Note: all these methods are extended from Julia Base.

Now we define a VecMath subtype to represent the procedures:

In [2]:
type VecMath{Op} <: Proc end

Here, the type parameter Op can be Sqrt, Exp etc, as we defined below, to represent the calculation we want to perform on each scalar. Using types to represent functions, allow specific computation to be inlined without incurring runtime overheads.

In [3]:
type Sqrt end
calc(::Sqrt, x) = sqrt(x)

type Exp end
calc(::Exp, x) = exp(x)

type Log end
calc(::Log, x) = log(x);

Define procedure names:

In [4]:
Base.string{Op}(::VecMath{Op}) = string("vec-", lowercase("$Op"));
In [5]:
string(VecMath{Sqrt}())
Out[5]:
"vec-sqrt"

To preclude the memory allocation time from the benchmark, we need to allocate arrays of specific sizes in advance, and store them as the initialized states. Particularly, we need to vectors, one for input, and the other for output. We use FVecPair as a shortname to represent such a bi-vector state:

In [6]:
typealias FVecPair (Vector{Float64},Vector{Float64});

The configuration is vector length, which can be simply represented by an integer. Then, we can define the procedures as follows:

In [8]:
Base.length(p::VecMath, n::Int) = n

Base.isvalid(p::VecMath, n::Int) = (n > 0)

Base.start(p::VecMath, n::Int) = (rand(n), zeros(n))

function Base.run{Op}(p::VecMath{Op}, n::Int, s::FVecPair)
    x, y = s
    op = Op()
    for i = 1:n
        @inbounds y[i] = calc(op, x[i])
    end
end

Base.done(p::VecMath, n, s) = nothing;

Running the benchmark

Collect all procedures into a Proc-vector, as

In [9]:
procs = Proc[ VecMath{Sqrt}(), 
              VecMath{Exp}(), 
              VecMath{Log}() ];

Collect all configurations into an Int-vector, as

In [10]:
cfgs = 2 .^ (4:10)
Out[10]:
7-element Array{Int64,1}:
   16
   32
   64
  128
  256
  512
 1024

Now, we call run to actually run the benchmark. For each procedure under each configuration, there are three stages of running:

  • warming up:

    it runs the procedure under the given configuration once, which triggers the pre-compilation of the function.

  • probing:

    it runs the procedure again to roughly estimate the time needed to run it once. Then the total number of runs is determined such that the entire duration of measuring takes about 1 second. If you want to change this duration, you may set it using the duration keyword argument. For example, duration = 0.5 means having each procedure under each configuration run for about 0.5 second.

  • measuring:

    it runs the procedure a number of times (the number of times is decided in the probing stage), and records the elapsed time.

In [11]:
rtable = run(procs, cfgs);
Benchmarking vec-sqrt ...
  vec-sqrt with cfg = 16: nruns = 2816902, elapsed = 0.181171339 secs
  vec-sqrt with cfg = 32: nruns = 2949853, elapsed = 0.383015011 secs
  vec-sqrt with cfg = 64: nruns = 2096437, elapsed = 0.541707007 secs
  vec-sqrt with cfg = 128: nruns = 1394701, elapsed = 0.707816771 secs
  vec-sqrt with cfg = 256: nruns = 800000, elapsed = 0.809421738 secs
  vec-sqrt with cfg = 512: nruns = 431407, elapsed = 0.872099936 secs
  vec-sqrt with cfg = 1024: nruns = 224568, elapsed = 0.908604125 secs
Benchmarking vec-exp ...
  vec-exp with cfg = 16: nruns = 675220, elapsed = 0.830725251 secs
  vec-exp with cfg = 32: nruns = 372440, elapsed = 0.90710495 secs
  vec-exp with cfg = 64: nruns = 194402, elapsed = 0.945588553 secs
  vec-exp with cfg = 128: nruns = 99286, elapsed = 0.96417331 secs
  vec-exp with cfg = 256: nruns = 34761, elapsed = 0.680242924 secs
  vec-exp with cfg = 512: nruns = 24515, elapsed = 0.956955982 secs
  vec-exp with cfg = 1024: nruns = 11338, elapsed = 0.881633733 secs
Benchmarking vec-log ...
  vec-log with cfg = 16: nruns = 670691, elapsed = 0.808676817 secs
  vec-log with cfg = 32: nruns = 378788, elapsed = 0.900356022 secs
  vec-log with cfg = 64: nruns = 198689, elapsed = 0.946116462 secs
  vec-log with cfg = 128: nruns = 100827, elapsed = 0.965859782 secs
  vec-log with cfg = 256: nruns = 49604, elapsed = 0.948375143 secs
  vec-log with cfg = 512: nruns = 24885, elapsed = 0.983452456 secs
  vec-log with cfg = 1024: nruns = 12500, elapsed = 0.96543341 secs

The result is stored in an instance of BenchmarkTable, which can be shown in different units. For example, you can show how many milliseconds each procedure takes (under various configuration):

In [12]:
show(rtable; unit=:msec)
BenchmarkTable [unit = msec]
config |  vec-sqrt  vec-exp  vec-log  
--------------------------------------
16     |    0.0001   0.0012   0.0012  
32     |    0.0001   0.0024   0.0024  
64     |    0.0003   0.0049   0.0048  
128    |    0.0005   0.0097   0.0096  
256    |    0.0010   0.0196   0.0191  
512    |    0.0020   0.0390   0.0395  
1024   |    0.0040   0.0778   0.0772  

If millisecond is not precise enough, you may try showing in terms of microseconds:

In [14]:
show(rtable; unit=:usec)
BenchmarkTable [unit = usec]
config |  vec-sqrt  vec-exp  vec-log  
--------------------------------------
16     |    0.0643   1.2303   1.2057  
32     |    0.1298   2.4356   2.3769  
64     |    0.2584   4.8641   4.7618  
128    |    0.5075   9.7111   9.5794  
256    |    1.0118  19.5691  19.1189  
512    |    2.0215  39.0355  39.5199  
1024   |    4.0460  77.7592  77.2347  

Sometimes, you may want to watch the results in terms of speed (e.g., MPS, million numbers per second):

In [15]:
show(rtable; unit=:mps)
BenchmarkTable [unit = mps]
config |  vec-sqrt  vec-exp  vec-log  
--------------------------------------
16     |  248.7724  13.0049  13.2699  
32     |  246.4533  13.1386  13.4627  
64     |  247.6836  13.1577  13.4403  
128    |  252.2146  13.1808  13.3620  
256    |  253.0201  13.0818  13.3899  
512    |  253.2742  13.1163  12.9555  
1024   |  253.0889  13.1689  13.2583  

Here is a list of supported units:

unit description
:sec seconds per run
:msec milliseconds per run
:usec microseconds per run
:nsec nanoseconds per run
:ups how many items/numbers per second. note: the number of items per run is determined by length(proc, cfg)
:kps how many thousand items/numbers per second
:mps how many million items/numbers per second
:gps how many trillion items/numbers per seoncd

Export

You can also export the result table to a CSV file using writecsv:

writecsv(io, rtable)