from IPython.core.display import HTML
css_file = '../styles.css'
HTML(open(css_file, "r").read())
- Define a function that takes parameters.
- Return a value from a function.
- Set default values for function parameters.
- Explain why we should divide programs into small, single-purpose functions.
At this point, we've written code to draw some interesting features in our star data, loop over all our data files to quickly draw these plots for each of them, and have Python make decisions based on what it sees in our data.
But our code is getting pretty long and complicated; what if we had thousands of datasets, and didn't want to generate a figure for every single one? Commenting out the figure-drawing code is a nuisance.
Also, what if we want to use that code again, on a different dataset or at a different point in our program? Cutting and pasting it is going to make our code get very long and very repetative, very quickly.
We'd like a way to package our code so that it is easier to reuse, and Python provides for this by letting us define things called 'functions' - a shorthand way of re-executing longer pieces of code.
Let's start by defining a function fahr_to_kelvin
that converts temperatures from Fahrenheit to Kelvin:
def fahr_to_kelvin(temp):
return ((temp - 32) * (5/9)) + 273.15
The function definition opens with the word def
, which is followed by the name of the function and a parenthesized list of parameter names.
The body of the function - the
statements that are executed when it runs - is indented below the definition line, typically by four spaces.
When we call the function, the values we pass to it are assigned to those variables so that we can use them inside the function. Inside the function, we use a return statement to send a result back to whoever asked for it.
Let's try running our function. Calling our own function is no different from calling any other function:
print ('freezing point of water:', fahr_to_kelvin(32) )
print ('boiling point of water:', fahr_to_kelvin(212) )
freezing point of water: 273.15 boiling point of water: 373.15
Now that we've seen how to turn Fahrenheit into Kelvin, it's easy to turn Kelvin into Celsius:
def kelvin_to_celsius(temp):
return temp - 273.15
print( 'absolute zero in Celsius:', kelvin_to_celsius(0.0) )
absolute zero in Celsius: -273.15
What about converting Fahrenheit to Celsius? We could write out the formula, but we don't need to. Instead, we can compose the two functions we have already created:
def fahr_to_celsius(temp):
temp_k = fahr_to_kelvin(temp)
result = kelvin_to_celsius(temp_k)
return result
print( 'freezing point of water in Celsius:', fahr_to_celsius(32.0) )
freezing point of water in Celsius: 0.0
This is our first taste of how larger programs are built: we define basic operations, then combine them in ever-large chunks to get the effect we want. Real-life functions will usually be larger than the ones shown here - typically half a dozen to a few dozen lines - but they shouldn't ever be much longer than that, or the next person who reads it won't be able to understand what's going on.
Now that we know how to wrap bits of code up in functions,
we can make our star brightness analysis easier to read and easier to reuse. First, let's make an process
function that takes our data and processes it to remove the cloud-induced variability:
def process(filename):
data = numpy.loadtxt(fname=filename, delimiter=',')
ave_brightness = data.mean(axis=0)
processed_data = data / ave_brightness
return processed_data
and another function that produces a plot of the processed data
def plot(processed_data):
image = matplotlib.pyplot.imshow(processed_data)
matplotlib.pyplot.show(image)
and one more that checks for variable stars.
def detect_variables(processed_data):
# variation of each star
deviations = processed_data.std(axis=1)
# calculate the 'mean' of the variations
mean_deviation = deviations.mean()
std_deviations = deviations.std()
for star_deviation in deviations:
if (star_deviation - mean_deviation > 3.0*std_deviations):
print('variable star in this data')
Notice that rather than jumbling this code together in one giant for
loop, we can now read and reuse both ideas separately.
We can reproduce the previous analysis with a much simpler for
loop:
import numpy
import matplotlib.pyplot
import glob
%matplotlib inline
filenames = glob.glob('data/*.csv')
for f in filenames:
print(f)
processed_data = process(f)
plot(processed_data)
detect_variables(processed_data)
data/star_data_01.csv
data/star_data_02.csv
data/star_data_03.csv
data/star_data_04.csv
variable star in this data data/star_data_05.csv
data/star_data_06.csv
data/star_data_07.csv
data/star_data_08.csv
data/star_data_09.csv
data/star_data_10.csv
data/star_data_11.csv
data/star_data_12.csv
By giving our functions human-readable names, we can more easily read and understand what is happening in the for
loop. Even better, if at some later date we want to use either of those pieces of code again,
we can do so in a single line.
Once we start putting things in functions so that we can re-use them, we need to start testing that those functions are working correctly. To see how to do this, let's write a function to center a dataset around a particular value:
def center(data, desired):
return (data - data.mean()) + desired
We could test this on our actual data, but since we don't know what the values ought to be, it will be hard to tell if the result was correct. Instead, let's use NumPy to create a matrix of 0's and then center that around 3:
z = numpy.zeros((2,2))
print ( center(z,3) )
[[ 3. 3.] [ 3. 3.]]
That looks right,
so let's try center
on our real data:
data = numpy.loadtxt(fname='data/star_data_01.csv', delimiter=',')
print ( center(data, 0) )
[[-3.6718427 -3.81570581 -4.13173512 ..., 0.2458925 -0.90905404 0.44061367] [-4.05148132 -3.47573063 -2.45566153 ..., 0.73122172 -0.87609992 1.22656046] [-4.90548289 -4.63515785 -3.45843892 ..., -0.84650214 0.08481151 0.2557072 ] ..., [-3.35356354 -4.94257997 -4.77925868 ..., 0.58308457 0.75383582 0.04129991] [-4.58199832 -4.98773945 -3.27734984 ..., -0.09449615 -0.29142883 1.1928766 ] [-3.50721851 -4.32961005 -2.06341214 ..., 1.32277885 0.30183067 0.63087012]]
It's hard to tell from the default output whether the result is correct, but there are a few simple tests that will reassure us:
print ('original min, mean, and max are:', data.min(), data.mean(), data.max())
centered = center(data, 0)
print ('min, mean, and and max of centered data are:', centered.min(), centered.mean(), centered.max())
original min, mean, and max are: 16.4477700722 22.5002003719 26.9747210312 min, mean, and and max of centered data are: -6.0524302997 -1.07284918023e-15 4.47452065932
That seems right. The mean of the centered data isn't quite zero - we'll explore why not in the challenges - but it's pretty close.
We can even go further and check that the standard deviation hasn't changed:
print ('std dev before and after:', data.std(), centered.std())
std dev before and after: 1.71095670561 1.71095670561
It looks like our function works. It's still possible that our function is wrong, but it seems unlikely enough that we should probably get back to doing our analysis. We have one more task first, though: we should write some documentation for our function to remind ourselves later what it's for and how to use it.
The usual way to put documentation in software is to add comments like this:
# return a new array containing the original data centered around the desired value.
def center(data, desired):
return (data - data.mean()) + desired
There's a better way, though. If the first thing in a function is a string that isn't assigned to a variable, that string is attached to the function as its documentation:
def center(data, desired):
'''Return a new array containing the original data centered around the desired value.'''
return (data - data.mean()) + desired
This is better because we can now ask Python's built-in help system to show us the documentation for the function:
help(center)
Help on function center in module __main__: center(data, desired) Return a new array containing the original data centered around the desired value.
A string like this is called a docstring. We don't need to use triple quotes when we write one, but if we do, we can break the string across multiple lines:
def center(data, desired):
'''Return a new array containing the original data centered around the desired value.
Example: center([1, 2, 3], 0) => [-1, 0, 1]'''
return (data - data.mean()) + desired
help(center)
Help on function center in module __main__: center(data, desired) Return a new array containing the original data centered around the desired value. Example: center([1, 2, 3], 0) => [-1, 0, 1]
We have passed parameters to functions in two ways: directly, as in center(data)
, and by name, as in numpy.loadtxt(fname='something.csv', delimiter=',')
.
In fact, we can pass the filename to loadtxt
without the fname=
:
numpy.loadtxt('inflammation-01.csv', delimiter=',')
but we still need to say delimiter=
:
numpy.loadtxt('data/star_data_01.csv', ',')
--------------------------------------------------------------------------- TypeError Traceback (most recent call last) <ipython-input-20-343613368fe6> in <module>() ----> 1 numpy.loadtxt('data/star_data_01.csv', ',') /usr/local/lib/python3.4/site-packages/numpy/lib/npyio.py in loadtxt(fname, dtype, comments, delimiter, converters, skiprows, usecols, unpack, ndmin) 803 try: 804 # Make sure we're dealing with a proper dtype --> 805 dtype = np.dtype(dtype) 806 defconv = _getconv(dtype) 807 TypeError: data type "," not understood
To understand what's going on, and make our own functions easier to use, let's re-define our center
function like this:
def center(data, desired=0.0):
'''Return a new array containing the original data centered around the desired value (0 by default).
Example: center([1, 2, 3], 0) => [-1, 0, 1]'''
return (data - data.mean()) + desired
The key change is that the second parameter is now written desired=0.0
instead of just desired
.
If we call the function with two arguments,
it works as it did before:
test_data = numpy.zeros((2, 2))
print (center(test_data, 3))
[[ 3. 3.] [ 3. 3.]]
But we can also now call it with just one parameter,
in which case desired
is automatically assigned the default value of 0.0:
more_data = 5 + numpy.zeros((2, 2))
print ('data before centering:')
print (more_data)
print ('centered data:')
print (center(more_data))
data before centering: [[ 5. 5.] [ 5. 5.]] centered data: [[ 0. 0.] [ 0. 0.]]
This is handy: if we usually want a function to work one way, but occasionally need it to do something else, we can allow people to pass a parameter when they need to but provide a default to make the normal case easier.
The example below shows how Python matches values to parameters:
def display(a=1, b=2, c=3):
print ('a:', a, 'b:', b, 'c:', c)
print ('no parameters:')
display()
print ('one parameter:')
display(55)
print ('two parameters:')
display(55, 66)
no parameters: a: 1 b: 2 c: 3 one parameter: a: 55 b: 2 c: 3 two parameters: a: 55 b: 66 c: 3
As this example shows, parameters are matched up from left to right, and any that haven't been given a value explicitly get their default value. We can override this behavior by naming the value as we pass it in:
print ('only setting the value of c')
display(c=77)
only setting the value of c a: 1 b: 2 c: 77
With that in hand,
let's look at the help for numpy.loadtxt
:
help(numpy.loadtxt)
Help on function loadtxt in module numpy.lib.npyio: loadtxt(fname, dtype=<class 'float'>, comments='#', delimiter=None, converters=None, skiprows=0, usecols=None, unpack=False, ndmin=0) Load data from a text file. Each row in the text file must have the same number of values. Parameters ---------- fname : file or str File, filename, or generator to read. If the filename extension is ``.gz`` or ``.bz2``, the file is first decompressed. Note that generators should return byte strings for Python 3k. dtype : data-type, optional Data-type of the resulting array; default: float. If this is a record data-type, the resulting array will be 1-dimensional, and each row will be interpreted as an element of the array. In this case, the number of columns used must match the number of fields in the data-type. comments : str, optional The character used to indicate the start of a comment; default: '#'. delimiter : str, optional The string used to separate values. By default, this is any whitespace. converters : dict, optional A dictionary mapping column number to a function that will convert that column to a float. E.g., if column 0 is a date string: ``converters = {0: datestr2num}``. Converters can also be used to provide a default value for missing data (but see also `genfromtxt`): ``converters = {3: lambda s: float(s.strip() or 0)}``. Default: None. skiprows : int, optional Skip the first `skiprows` lines; default: 0. usecols : sequence, optional Which columns to read, with 0 being the first. For example, ``usecols = (1,4,5)`` will extract the 2nd, 5th and 6th columns. The default, None, results in all columns being read. unpack : bool, optional If True, the returned array is transposed, so that arguments may be unpacked using ``x, y, z = loadtxt(...)``. When used with a record data-type, arrays are returned for each field. Default is False. ndmin : int, optional The returned array will have at least `ndmin` dimensions. Otherwise mono-dimensional axes will be squeezed. Legal values: 0 (default), 1 or 2. .. versionadded:: 1.6.0 Returns ------- out : ndarray Data read from the text file. See Also -------- load, fromstring, fromregex genfromtxt : Load data with missing values handled as specified. scipy.io.loadmat : reads MATLAB data files Notes ----- This function aims to be a fast reader for simply formatted files. The `genfromtxt` function provides more sophisticated handling of, e.g., lines with missing values. Examples -------- >>> from StringIO import StringIO # StringIO behaves like a file object >>> c = StringIO("0 1\n2 3") >>> np.loadtxt(c) array([[ 0., 1.], [ 2., 3.]]) >>> d = StringIO("M 21 72\nF 35 58") >>> np.loadtxt(d, dtype={'names': ('gender', 'age', 'weight'), ... 'formats': ('S1', 'i4', 'f4')}) array([('M', 21, 72.0), ('F', 35, 58.0)], dtype=[('gender', '|S1'), ('age', '<i4'), ('weight', '<f4')]) >>> c = StringIO("1,0,2\n3,0,4") >>> x, y = np.loadtxt(c, delimiter=',', usecols=(0, 2), unpack=True) >>> x array([ 1., 3.]) >>> y array([ 2., 4.])
There's a lot of information here, but the most important part is the first couple of lines:
loadtxt(fname, dtype=<type 'float'>, comments='#', delimiter=None, converters=None, skiprows=0, usecols=None,
unpack=False, ndmin=0)
This tells us that loadtxt
has one parameter called fname
that doesn't have a default value,
and eight others that do.
If we call the function like this:
numpy.loadtxt('data/star_data_01.csv', ',')
then the filename is assigned to fname
(which is what we want),
but the delimiter string ','
is assigned to dtype
rather than delimiter
,
because dtype
is the second parameter in the list. However ',' isn't a known dtype
so
our code produced an error message when we tried to run it.
When we call loadtxt
we don't have to provide fname=
for the filename because it's the
first item in the list, but if we want the ',' to be assigned to the variable delimiter
,
we do have to provide delimiter=
for the second parameter since delimiter
is not
the second parameter in the list.
"Adding" two strings produces their concatenation:
'a' + 'b'
is'ab'
. Write a function calledfence
that takes two parameters calledoriginal
andwrapper
and returns a new string that has the wrapper character at the beginning and end of the original. A call to your function should look like this:print (fence('name', '*')) *name*
# Write your own code here
If the variable
s
refers to a string, thens[0]
is the string's first character ands[-1]
is its last. Write a function calledouter
that returns a string made up of just the first and last characters of its input. A call to your function should look like this:print ( outer('helium') ) hm
# Write your own code here
Write a function
rescale
that takes an array as input and returns a corresponding array of values scaled to lie in the range 0.0 to 1.0. (Hint: If $L$ and $H$ are the lowest and highest values in the original array, then the replacement for a value $v$ should be $(v-L) / (H-L)$.)
# Write your own code here
Run the commands
help(numpy.arange)
andhelp(numpy.linspace)
to see how to use these functions to generate regularly-spaced values, then use those values to test yourrescale
function. Once you've successfully tested your function, add a docstring that explains what it does.
Rewrite the
rescale
function so that it scales data to lie between 0.0 and 1.0 by default, but will allow the caller to specify lower and upper bounds if they want. Compare your implementation to your neighbour's: do the two functions always behave the same way?
What does the following piece of code display when run - and why?
For discussion and understanding, the following two examples at http://pythontutor.com may be illuminating
If you find this example confusing - try reading the following chapter of [How to Think Like a Computer Scientist](
f = 0
k = 0
def f2k(f):
k = ((f-32)*(5.0/9.0)) + 273.15
return k
f2k(8)
f2k(41)
f2k(32)
print (k)
0