Basic statistics

This notebook is an element of the free risk-engineering.org courseware. It can be distributed under the terms of the Creative Commons Attribution-ShareAlike licence.

Author: Eric Marsden [email protected].


This notebook contains an introduction to use of Python and the NumPy library for basic statistical calculations. It uses Python3.

We start by importing the numpy library, which makes it possible to use functions and variables from the library, prefixed by numpy..

In [1]:
import numpy

We can use Python as simple interactive calculator:

In [2]:
2 + 3 + 4
Out[2]:
9

Here we call the sqrt function from the numpy library.

In [3]:
numpy.sqrt(2 + 2)
Out[3]:
2.0

Some useful constants are predefined.

In [4]:
numpy.pi
Out[4]:
3.141592653589793
In [5]:
numpy.sin(numpy.pi)
Out[5]:
1.2246467991473532e-16

The notation e-16 above means $10^{-16}$; the number above is very very small.

We can generate a random number from a uniform distribution between 20 and 30. If you evaluate this several times (in most Jupyter interfaces, press Shift-Enter or press on the Run button in the toolbar above), it will generate a different random number each time.

In [6]:
numpy.random.uniform(20, 30)
Out[6]:
25.842934168520067
In [7]:
numpy.random.uniform(20, 30)
Out[7]:
26.730778225953763

We can generate an array of random numbers by passing a third argument to the numpy.random.uniform function, saying how many random numbers we want. We store the array in a variable named obs.

In [8]:
obs = numpy.random.uniform(20, 30, 10)
obs
Out[8]:
array([24.82376244, 28.5778536 , 27.64506311, 23.00748749, 21.78162667,
       22.87893526, 29.28029561, 21.05215274, 26.00044959, 20.96768435])

The builtin function len in Python tells us the length of an array or a list.

In [9]:
len(obs)
Out[9]:
10

We can do arithmetic on arrays, adding them together or subtracting a constant from each element.

In [10]:
obs + obs
Out[10]:
array([49.64752488, 57.15570721, 55.29012621, 46.01497498, 43.56325335,
       45.75787052, 58.56059122, 42.10430548, 52.00089919, 41.93536871])
In [11]:
obs - 25
Out[11]:
array([-0.17623756,  3.5778536 ,  2.64506311, -1.99251251, -3.21837333,
       -2.12106474,  4.28029561, -3.94784726,  1.00044959, -4.03231565])

The array has methods, a kind of function that acts on the array.

In [12]:
obs.mean()
Out[12]:
24.601531087103975
In [13]:
obs.sum()
Out[13]:
246.01531087103976
In [14]:
obs.min()
Out[14]:
20.967684353383454

There are similar functions in the numpy library that take an array as argument:

In [15]:
numpy.mean(obs)
Out[15]:
24.601531087103975
In [16]:
numpy.sum(obs)
Out[16]:
246.01531087103976
In [17]:
numpy.min(obs)
Out[17]:
20.967684353383454

Simple plotting

The matplotlib library allows you to generate many types of plots and statistical graphs in a convenient way. The online gallery shows the variety of plots available, and the documentation is also available online. We import the pyplot component of matplotlib and give it an alias plt.

In [18]:
import matplotlib.pyplot as plt
%config InlineBackend.figure_formats=['svg']
In [19]:
X = numpy.random.uniform(20, 30, 10)
Y = numpy.random.uniform(50, 100, 10)
plt.scatter(X, Y);
In [20]:
x = numpy.linspace(-2, 10, 100)
plt.plot(x, numpy.sin(x));

We can add two vectors together, assuming that all their dimensions are identical. Our array $x$ has one dimension of size 100. We can add another random vector of size 100 to it, containing numbers drawn from a uniform probability distribution between -0.1 and 0.1 (these represent some random “noise” which is added to our sine curve).

In [21]:
x = numpy.linspace(0, 10, 100)
obs = numpy.sin(x) + numpy.random.uniform(-0.1, 0.1, 100)
plt.plot(x, obs);

The central limit theorem states that the sum of a number of independent random variables tends toward a normal distribution even if the original variables themselves are not normally distributed. We illustrate this result by examining the distribution of the sums of 1000 realizations of a uniformly distributed random variable, plotting the distribution as a histogram.

In [22]:
N = 10000
sim = numpy.zeros(N)
for i in range(N):
    sim[i] = numpy.random.uniform(30, 40, 1000).mean()
plt.hist(sim, bins=20, alpha=0.5, density=True);