In this tutorial, we will cover:
conda
My info:
Course:
Tutorials:
Homework:
Python is a great general-purpose programming language on its own and with the addition of a few
popular libraries such as numpy
, pandas
, scikit-learn
, matplotlib
and others it becomes an
effective scientific computing environment.
Recently many Deep Learning frameworks have emerged for python. Arguably the most notable ones in 2019 are TensorFlow (with the Keras frontend) and PyTorch. In this course we'll use PyTorch, which is currently the leading DL framework for research.
Many of you may have some experience with Python and numpy; for the rest of you, this section will serve as a quick crash course both on the Python programming language and on the use of Python for scientific computing.
If you have previous knowledge in Matlab, we recommend the numpy for Matlab users page as a useful resource.
Credit: Some parts of the Python tutorial here were adapted from the CS231n Python tutorial by Justin Johnson.
To install and manage all the necessary packages and dependencies for the course tutorials and assignments, we use conda, a popular package-manager for python.
environment.yml
file which defines which third-party libraries we depend on.python installation you may have.
Install the python3 version of miniconda. Follow the installation instructions for your platform.
Install all dependencies with conda
:
conda env update -f environment.yml
To activate the virtual environment (set up $PATH
):
conda activate cs236781
To check what conda environments you have and which is active, run
conda env list
From a terminal, enter the folder contaning the tutorial notebooks.
Make sure that the active conda environment is cs236781
, and run
jupyter lab
This will start a jupyter lab server and open your browser at the local server's url. You can now start working with the notebooks.
If you're new to jupyter notebooks, you can get started by reading the UI guide and also about how to use notebooks in JupyterLab.
Jupyter notebooks consist mainly of code and markdown cells.
The code cells contain code that is run by a kernel
, an
interpreter for some programming language, python in our case.
# This is a code cell; it can contain arbitrary python code.
foo = 'bar'
print(foo)
def the_answer():
return 42
# The output of the last expression in a cell is shown
2*the_answer()
the_answer()
bar
42
Variables and functions defined in a code cell are available in subsequent cells.
ans = the_answer()
ans
42
This is a markdown cell. You can use markdown syntax to format your text, and also include equations written in $\LaTeX$:
$$ e^{i\pi} - 1 = 0 $$Other useful things to know about:
%matplotlib inline
%load_ext autoreload
%autoreload 2
The autoreload extension is already loaded. To reload it, use: %reload_ext autoreload
Python is a high-level, dynamically typed multiparadigm programming language. Python code is often said to be almost like pseudocode, since it allows you to express very powerful ideas in very few lines of code while being very readable. As an example, here is an implementation of the classic quicksort algorithm in Python:
def quicksort(arr):
if len(arr) <= 1:
return arr
pivot = arr[len(arr) // 2]
left = [x for x in arr if x < pivot]
middle = [x for x in arr if x == pivot]
right = [x for x in arr if x > pivot]
return quicksort(left) + middle + quicksort(right)
print(quicksort([3,6,8,10,1,2,1]))
[1, 1, 2, 3, 6, 8, 10]
Python has great documentation! Use it often.
A python module is simply a python file (.py
), which can contain functions, classes and even top-level code.
A package is a collection of modules within a directory. Python comes with a standard library which includes many useful packages.
A package must be imported before use. They can be imported like so:
# Import packages from the python standard library
import math
import sys
Additionally, we can define our own packages and modules. This tutorial comes with a demo_package
which includes a demo_module
.
Any object can be imported from a module like so:
# Import just a specific function from a specific module from a specific package
from demo_package.demo_module import demo_func
demo_func(math.pi)
print('FOO', file=sys.stdout)
this is a demo, a=17 this is a demo, a=3.141592653589793 FOO
Integers and floats work as you would expect from other languages:
x = 3
print(x, type(x))
3 <class 'int'>
print(x + 1) # Addition;
print(x - 1) # Subtraction;
print(x * 2) # Multiplication;
print(x ** 2) # Exponentiation;
4 2 6 9
x += 1
print(x)
x *= 2
print(x)
4 8
y = 2.5
print(type(y))
print(y, y + 1, y * 2, y ** 2, y / 2, y // 2)
<class 'float'> 2.5 3.5 5.0 6.25 1.25 1.0
Note that unlike many languages, Python does not have unary increment (x++) or decrement (x--) operators.
Python also has built-in types for long integers and complex numbers; you can find all of the details in the documentation.
Python implements all of the usual operators for Boolean logic, but uses English words rather than symbols (&&
, ||
, etc.):
t, f = True, False
Now we let's look at the operations:
print(t and f) # Logical AND
print(t or f ) # Logical OR
print(not t ) # Logical NOT
print(t != f ) # Logical XOR
False True False True
hello = 'hello' # String literals can use single quotes
world = "world" # or double quotes; it does not matter.
hello, len(hello)
('hello', 5)
# String concatenation
'aaa ' + 'bbb'
'aaa bbb'
There are several way to created formatted strings, here are a couple:
s = 'hello'
a = [1,2,3]
# sprintf style string formatting
print('%s %s: pi=%.5f' % (s, a, math.pi))
# formatting with f-string literals (python 3.6+)
print(f'{s} {a}: pi={math.pi:.5f}')
hello [1, 2, 3]: pi=3.14159 hello [1, 2, 3]: pi=3.14159
String objects have a bunch of useful methods; for example:
s = "hello"
print(s.capitalize() ) # Capitalize a string; prints "Hello"
print(s.upper() ) # Convert a string to uppercase; prints "HELLO"
print(s.rjust(7) ) # Right-justify a string, padding with spaces; prints " hello"
print(s.center(7) ) # Center a string, padding with spaces; prints " hello "
print(s.replace('l', '(ell)')) # Replace all instances of one substring with another
print(' world '.strip()) # Strip leading and trailing whitespace; prints "world"
Hello HELLO hello hello he(ell)(ell)o world
You can find a list of all string methods in the documentation.
Python includes several built-in container types: lists, dictionaries, sets, and tuples.
A list is the Python equivalent of an array, but is resizeable and can contain elements of different types:
xs = [3, 1, 2] # Create a list
print(xs)
print(xs[2], xs[-1]) # Negative indices count from the end of the list; prints "2"
[3, 1, 2] 2 2
xs[2] = 'foo' # Lists can contain elements of different types
print(xs)
[3, 1, 'foo']
xs.append('bar') # Add a new element to the end of the list
print(xs)
[3, 1, 'foo', 'bar']
x = xs.pop() # Remove and return the last element of the list
x, xs
('bar', [3, 1, 'foo'])
In addition to accessing list elements one at a time, Python provides concise syntax to access sublists; this is known as slicing:
nums = list(range(5))
nums
[0, 1, 2, 3, 4]
nums[2:4] # Get a slice from index 2 to 4 (exclusive)
[2, 3]
nums[2:] # Get a slice from index 2 to the end
[2, 3, 4]
nums[:2] # Get a slice from the start to index 2 (exclusive)
[0, 1]
nums[:] # Get a slice of the whole list
[0, 1, 2, 3, 4]
nums[:-1] # Slice indices can be negative
[0, 1, 2, 3]
nums[0:4:2] # Can also specify slice step size
[0, 2]
nums[2:4] = [8, 9] # Assign a new sublist to a slice
nums
[0, 1, 8, 9, 4]
# Delete elements from a list
nums[0:1] = []
del nums[-1]
nums
[1, 8, 9]
You can loop over the elements of a list like this:
animals = ['cat', 'dog', 'monkey']
for animal in animals:
print(animal)
cat dog monkey
If you want access to the index of each element within the body of a loop, use the built-in enumerate
function:
animals = ['cat', 'dog', 'monkey']
for idx, animal in enumerate(animals):
print(f'#{idx+1}: {animal}')
#1: cat #2: dog #3: monkey
When programming, frequently we want to transform one type of data into another. As a simple example, consider the following code that computes square numbers:
nums = [0, 1, 2, 3, 4]
squares = []
for x in nums:
squares.append(x ** 2)
squares
[0, 1, 4, 9, 16]
You can make this code simpler using a list comprehension:
squares = [x ** 2 for x in nums]
squares
[0, 1, 4, 9, 16]
List comprehensions can also contain conditions:
even_squares = [x ** 2 for x in nums if x % 2 == 0]
even_squares
[0, 4, 16]
List comprehensions can be nested:
nums2 = [-1, 1]
[x * y for x in nums for y in nums2]
[0, 0, -1, 1, -2, 2, -3, 3, -4, 4]
A dictionary stores (key, value) pairs. In other languages this is known as a Map
or Hash
.
d = {'cat': 'cute', 'dog': 'furry'} # Create a new dictionary with some data
print(d['cat']) # Get an entry from a dictionary
print('cat' in d) # Check if a dictionary has a given key
cute True
d['fish'] = 'wet' # Set an entry in a dictionary
d
{'cat': 'cute', 'dog': 'furry', 'fish': 'wet'}
# Trying to access a non-existing key raises a KeyError
try:
d['monkey']
except KeyError as e:
print(e, file=sys.stderr)
'monkey'
print(d.get('monkey', 'N/A')) # Get an element with a default
print(d.get('fish', 'N/A')) # Get an element with a default
N/A wet
del d['fish'] # Remove an element from a dictionary
d
{'cat': 'cute', 'dog': 'furry'}
# Iteration over keys
d = {'person': 2, 'cat': 4, 'spider': 8}
for animal in d:
print(f'A {animal} has {d[animal]} legs')
A person has 2 legs A cat has 4 legs A spider has 8 legs
# Iterate over key-value pairs
d = {'person': 2, 'cat': 4, 'spider': 8}
for animal, num_legs in d.items():
print(f'A {animal} has {num_legs} legs')
A person has 2 legs A cat has 4 legs A spider has 8 legs
# Create a dictionary using the built-in dict() function
dict(foo=1, bar=2, baz=3)
{'foo': 1, 'bar': 2, 'baz': 3}
These are similar to list comprehensions, but allow you to easily construct dictionaries. For example:
nums = [0, 1, 2, 3, 4]
even_num_to_square = {x: x ** 2 for x in nums if x % 2 == 0}
even_num_to_square
{0: 0, 2: 4, 4: 16}
A set is an unordered collection of distinct elements
animals = {'cat', 'dog'}
print(animals)
print('cat' in animals ) # Check if an element is in a set
print('fish' in animals) # prints "False"
{'dog', 'cat'} True False
animals.add('fish') # Add an element to a set
print('fish' in animals)
len(animals) # Number of elements in a set
True
3
animals.add('cat') # Adding an element that is already in the set does nothing
animals
{'cat', 'dog', 'fish'}
Loops: Iterating over a set has the same syntax as iterating over a list; however since sets are unordered, you cannot make assumptions about the order in which you visit the elements of the set:
animals = {'cat', 'dog', 'fish'}
for idx, animal in enumerate(animals):
print(f'#{idx}: {animal}')
#0: fish #1: dog #2: cat
Like lists and dictionaries, we can easily construct sets using set comprehensions:
from math import sqrt
s = {int(sqrt(x)) for x in range(37)}
s
{0, 1, 2, 3, 4, 5, 6}
A tuple is an immutable ordered list of values.
t = (1, 2, 'three')
t
(1, 2, 'three')
It can be used in ways similar to a list:
t[0:1], t[1:3], t[-1], len(t)
((1,), (2, 'three'), 'three', 3)
A tuple can be used a key in a dictionary and as an element of a sets, while lists cannot.
d = {(x, x + 1): x for x in range(10)} # Create a dictionary with tuple keys
d
{(0, 1): 0, (1, 2): 1, (2, 3): 2, (3, 4): 3, (4, 5): 4, (5, 6): 5, (6, 7): 6, (7, 8): 7, (8, 9): 8, (9, 10): 9}
A tuple (and also a list) can be unpacked:
one, two, three = t
one, two, three
(1, 2, 'three')
Note that when retuning multiple values from a function (or code block in a jupyter notebook, as above) your values get wrapped in a tuple, and the tuple is what's returned. Unpacking the return value of a function can make it seem as if multiple values were returned.
Python functions are defined using the def
keyword. For example:
def sign(x):
if x > 0:
return 'positive'
elif x < 0:
return 'negative'
else:
return 'zero'
for x in [-1, 0, 1]:
print(sign(x))
negative zero positive
We will often define functions to take optional keyword arguments, like this:
def hello(name, loud=False):
if loud:
print('HELLO, %s' % name.upper())
else:
print('Hello, %s!' % name)
hello('Bob')
hello('Fred', loud=True)
Hello, Bob! HELLO, FRED
Python functions are very flexible in the way they accept arguments. Both positional (regular) and keyword
arguments are supported and can be mixed in the same definition. Additionally, extra arguments can be passed in with the *args
and **kwargs
constructs.
Here's a function with three positional arguments and three keyword arguments which also accepts extra positional and keyword arguments.
def myfunc(a1, a2, a3, *extra_args, kw1='foo', kw2='bar', kw3=3, **extra_kwargs):
print(f'Got positional args: {(a1, a2, a3)}')
print(f'Got keyword args : {dict(kw1=kw1, kw2=kw3, kw3=kw3)}')
print(f'Got extra positional args: {extra_args}')
print(f'Got extra keyword args: {extra_kwargs}')
It can be called in many ways:
myfunc(1,2,3,4,5,6)
Got positional args: (1, 2, 3) Got keyword args : {'kw1': 'foo', 'kw2': 3, 'kw3': 3} Got extra positional args: (4, 5, 6) Got extra keyword args: {}
my_args = [1,2,3,4]
myfunc(*my_args)
Got positional args: (1, 2, 3) Got keyword args : {'kw1': 'foo', 'kw2': 3, 'kw3': 3} Got extra positional args: (4,) Got extra keyword args: {}
myfunc(1,2,3, kw3=3, kw2=2, foo='bar')
Got positional args: (1, 2, 3) Got keyword args : {'kw1': 'foo', 'kw2': 3, 'kw3': 3} Got extra positional args: () Got extra keyword args: {'foo': 'bar'}
my_kwargs = dict(kw1=1, kw2=2, kw3=3, kw4=4)
myfunc(1,2,3, **my_kwargs)
Got positional args: (1, 2, 3) Got keyword args : {'kw1': 1, 'kw2': 3, 'kw3': 3} Got extra positional args: () Got extra keyword args: {'kw4': 4}
Note that keyword args can be omitted, while positional args cannot:
try:
myfunc(1,2)
except TypeError as e:
print(e, file=sys.stderr)
myfunc() missing 1 required positional argument: 'a3'
The syntax for defining classes in Python is straightforward:
class Greeter:
# Constructor
def __init__(self, name):
self.name = name # Create an instance variable
# Instance method
def greet(self, loud=False):
if loud:
print('HELLO, %s!' % self.name.upper())
else:
print('Hello, %s' % self.name)
g = Greeter('Fred') # Construct an instance of the Greeter class
g.greet() # Call an instance method
g.greet(loud=True) # Call an instance method
Hello, Fred HELLO, FRED!
Classes can implement special magic functions that enable them to be integrated nicely with other python code. Magic functions have special names that start and end with __
.
For example, here's a class that can be indexed with []
and iterated over with a for
loop.
class ExampleCollection(object):
def __init__(self):
self.items = [100, 200, 300]
def __len__(self):
return len(self.items)
def __getitem__(self, idx):
return self.items[idx]
def __iter__(self):
class ExampleIter():
def __init__(self, collection):
self.idx = 0
self.collection = collection
def __next__(self):
if self.idx >= len(self.collection):
raise StopIteration()
x = self.collection[self.idx]
self.idx += 1
return x
return ExampleIter(self)
example = ExampleCollection()
print('length=', len(example)) # invokes __len__
print('example[0]=', example[0]) # invokes __getitem__
length= 3 example[0]= 100
for x in example: # invokes __iter__ and it's __next__
print(x)
100 200 300
Many other magic functions exist. Consult the docs and see if you can catch 'em all!
Numpy is a core library for scientific computing in Python. It provides a high-performance multidimensional array object, and tools for working with these arrays.
We'll refer to such n-dimentional arrays as tensors in accordance with the deep learning terminology.
Although we'll mainly use PyTorch tensors for implementing our Deep Learning systems, it's still important to be proficient with numpy
, since:
To use Numpy, we first need to import the numpy
package:
import numpy as np
A numpy array represents an n-dimentional grid of values, all of the same type, and is indexed by a tuple of nonnegative integers.
We can initialize numpy arrays from nested Python lists, and access elements using square brackets:
a = np.array([1, 2, 3]) # Create a rank 1 array
a
array([1, 2, 3])
a[0]
1
Two very important properties of any numpy array are its shape
and dtype
.
def print_arr(arr, pre_text=''):
print(f'{pre_text}{arr} shape={arr.shape} dtype={arr.dtype}')
print_arr(a)
[1 2 3] shape=(3,) dtype=int64
a[0] = 5 # Change an element of the array
a
array([5, 2, 3])
b = np.array([[1,2,3],[4,5,6.7]]) # Create a rank 2 array
print_arr(b)
[[1. 2. 3. ] [4. 5. 6.7]] shape=(2, 3) dtype=float64
b[0, 0], b[0, 1], b[1, 0]
(1.0, 2.0, 4.0)
Numpy also provides many functions to create arrays:
np.zeros((2,2)) # Create an array of all zeros
array([[0., 0.], [0., 0.]])
np.ones((1,10)) # Create an array of all ones
array([[1., 1., 1., 1., 1., 1., 1., 1., 1., 1.]])
np.full((3,3), 7.2) # Create a constant array
array([[7.2, 7.2, 7.2], [7.2, 7.2, 7.2], [7.2, 7.2, 7.2]])
np.eye(4, dtype=np.int) # Create an identity matrix of integers
array([[1, 0, 0, 0], [0, 1, 0, 0], [0, 0, 1, 0], [0, 0, 0, 1]])
t = np.random.random((4,4,3)) # Create a 3d-array filled with U[0,1] random values
t
array([[[0.21921227, 0.39286303, 0.89934625], [0.72141811, 0.1582272 , 0.14031527], [0.20468505, 0.38861605, 0.44066535], [0.50377059, 0.06713136, 0.75217024]], [[0.40337263, 0.10336229, 0.22444997], [0.82814678, 0.48742751, 0.53362932], [0.74819915, 0.63329761, 0.31033589], [0.8924808 , 0.85084316, 0.50349368]], [[0.88498439, 0.74481972, 0.40691951], [0.34428811, 0.57900335, 0.47587481], [0.69307823, 0.51344594, 0.92398386], [0.5347534 , 0.12659492, 0.57205981]], [[0.05397933, 0.1507 , 0.92718578], [0.2778349 , 0.51612588, 0.24952806], [0.43105188, 0.54110875, 0.07947924], [0.81656111, 0.87409979, 0.79965167]]])
t[1,1,2]
0.5336293174054009
In numpy
rank means number of dimensions.
rank-0 arrays are scalars.
a0 = np.array(17)
print_arr(a0)
17 shape=() dtype=int64
# Get sclar as a python float
a0.item()
17
rank-1 arrays of length n
have a shape of (n,)
.
# A rank-1 array
a1 = np.array([1,2,3])
print_arr(a1)
[1 2 3] shape=(3,) dtype=int64
# A rank-1 array scalar
print_arr(np.array([3.14]))
[3.14] shape=(1,) dtype=float64
rank-2 arrays have a shape of (n,m)
.
a2 = np.array([[1,2,3], [4,5,6]])
print_arr(a2)
[[1 2 3] [4 5 6]] shape=(2, 3) dtype=int64
# A column vector is also rank-2!
a_col = a1.reshape(-1, 1)
print_arr(a_col)
[[1] [2] [3]] shape=(3, 1) dtype=int64
# A row vector is also rank-2
a_row = a1.reshape(1, -1)
print_arr(a_row)
[[1 2 3]] shape=(1, 3) dtype=int64
rank-k arrays have a shape of (n1,...,nk)
.
print_arr(np.zeros((2,3,4)))
[[[0. 0. 0. 0.] [0. 0. 0. 0.] [0. 0. 0. 0.]] [[0. 0. 0. 0.] [0. 0. 0. 0.] [0. 0. 0. 0.]]] shape=(2, 3, 4) dtype=float64
print_arr(np.ones((2,2,2,2)))
[[[[1. 1.] [1. 1.]] [[1. 1.] [1. 1.]]] [[[1. 1.] [1. 1.]] [[1. 1.] [1. 1.]]]] shape=(2, 2, 2, 2) dtype=float64
Basic mathematical functions operate elementwise on arrays, and are available both as operator overloads and as functions in the numpy module:
x = np.array([[1,2],[3,4]], dtype=np.float64)
y = np.array([[5,6],[7,8]], dtype=np.float64)
# Elementwise basic math
print(x + y)
print(x - y)
print(x * y)
print(x / y)
[[ 6. 8.] [10. 12.]] [[-4. -4.] [-4. -4.]] [[ 5. 12.] [21. 32.]] [[0.2 0.33333333] [0.42857143 0.5 ]]
# Elementwise functions
print(np.sqrt(x))
print(np.exp(x))
print(np.log(x))
[[1. 1.41421356] [1.73205081 2. ]] [[ 2.71828183 7.3890561 ] [20.08553692 54.59815003]] [[0. 0.69314718] [1.09861229 1.38629436]]
There are of course many more elementwise operations inmplemented by numpy
.
Unlike MATLAB, *
is elementwise multiplication, not matrix multiplication (as we saw above).
We can instead use the dot()
function to:
The dot()
function is available both as a function in the numpy module and as an instance
method of array objects.
v = np.array([9,10])
w = np.array([11, 12])
# Inner product of vectors; both produce 219
print(v.dot(w))
print(np.dot(v, w))
219 219
Rank-1 arrays arrays are somewhat special in that numpy
can treat them both as column or as row vectors.
Arrays of different rank have different semantics when using them in vector-vector or vector-matrix products, so always make sure you known what shapes you're working with:
print_arr(np.dot(a1, a1), 'a1 * a1 =\t')
print_arr(np.dot(a_row, a1), 'a_row * a1 =\t')
print_arr(np.dot(a1, a_col), 'a1 * a_col =\t')
print_arr(np.dot(a_row, a_col), 'a_row * a_col =\t')
print_arr(np.dot(a_col, a_row), 'a_col * a_row =\n')
a1 * a1 = 14 shape=() dtype=int64 a_row * a1 = [14] shape=(1,) dtype=int64 a1 * a_col = [14] shape=(1,) dtype=int64 a_row * a_col = [[14]] shape=(1, 1) dtype=int64 a_col * a_row = [[1 2 3] [2 4 6] [3 6 9]] shape=(3, 3) dtype=int64
Numpy provides many useful functions for performing computations on arrays.
x = np.array([[1,2],[3,4]])
print_arr(x)
[[1 2] [3 4]] shape=(2, 2) dtype=int64
print(np.sum(x)) # Compute sum of all elements
print(np.mean(x, axis=0)) # Compute mean of each column
print(np.prod(x, axis=1) ) # Compute product of each row
10 [2. 3.] [ 2 12]
You can find the full list of mathematical functions provided by numpy in the documentation.
Numpy offers several ways to index into arrays.
Slicing
Similar to Python lists, numpy arrays can be sliced. Since arrays may be multidimensional, you must specify a slice for each dimension of the array:
a = np.array([[1,2,3,4], [5,6,7,8], [9,10,11,12]])
print_arr(a)
[[ 1 2 3 4] [ 5 6 7 8] [ 9 10 11 12]] shape=(3, 4) dtype=int64
b = a[:2, 1:3]
print_arr(b)
[[2 3] [6 7]] shape=(2, 2) dtype=int64
A slice of an array is a view into the same in-memory data, so modifying it will modify the original array.
# Changing a view
b[0, 0] = 77777
# ...modifies original
a
array([[ 1, 77777, 3, 4], [ 5, 6, 7, 8], [ 9, 10, 11, 12]])
You can also mix integer indexing with slice indexing. However, doing so will yield an array of lower rank than the original array.
a = np.array([[1,2,3,4], [5,6,7,8], [9,10,11,12]])
a
array([[ 1, 2, 3, 4], [ 5, 6, 7, 8], [ 9, 10, 11, 12]])
Two ways of accessing the data in the middle row of the array.
row_r1 = a[1, :] # Rank 1 view of the second row of a
row_r2 = a[1:2, :] # Rank 2 view of the second row of a
row_r3 = a[[1], :] # Rank 2 view of the second row of a
print_arr(row_r1)
print_arr(row_r2)
print_arr(row_r3)
[5 6 7 8] shape=(4,) dtype=int64 [[5 6 7 8]] shape=(1, 4) dtype=int64 [[5 6 7 8]] shape=(1, 4) dtype=int64
# We can make the same distinction when accessing columns of an array:
col_r1 = a[:, 1]
col_r2 = a[:, 1:2]
print_arr(col_r1)
print_arr(col_r2)
[ 2 6 10] shape=(3,) dtype=int64 [[ 2] [ 6] [10]] shape=(3, 1) dtype=int64
Integer array indexing
a = np.array([[1,2], [3, 4], [5, 6]])
print_arr(a)
[[1 2] [3 4] [5 6]] shape=(3, 2) dtype=int64
# An example of integer array indexing.
# The returned array will have shape (3,)
print_arr(a[ [0, 1, 2], [0, 1, 0] ])
[1 4 5] shape=(3,) dtype=int64
# The above example of integer array indexing is equivalent to this:
print_arr(np.array([a[0, 0], a[1, 1], a[2, 0]]))
[1 4 5] shape=(3,) dtype=int64
# When using integer array indexing, you can reuse the same
# element from the source array:
print(a[[0, 0], [1, 1]])
# Equivalent to the previous integer array indexing example
print(np.array([a[0, 1], a[0, 1]]))
[2 2] [2 2]
One useful trick with integer array indexing is selecting or mutating one element from each row of a matrix:
# Create a new array from which we will select elements
a = np.array([[1,2,3], [4,5,6], [7,8,9], [10, 11, 12]])
a
array([[ 1, 2, 3], [ 4, 5, 6], [ 7, 8, 9], [10, 11, 12]])
# Create an array of indices
b = np.array([0, 2, 0, 1])
# Select one element from each row of a using the indices in b
a[np.arange(4), b]
array([ 1, 6, 7, 11])
# Mutate one element from each row of a using the indices in b
a[np.arange(4), b] += 1000
a
array([[1001, 2, 3], [ 4, 5, 1006], [1007, 8, 9], [ 10, 1011, 12]])
Boolean array indexing
This type of indexing is used to select the elements of an array that satisfy some condition (similar to MATLAB's logical indexing).
a = np.array([[1,2], [3, 4], [5, 6]])
print_arr(a)
[[1 2] [3 4] [5 6]] shape=(3, 2) dtype=int64
bool_idx = (a > 2) # Find the elements of a that are bigger than 2;
# this returns a numpy array of Booleans of the same
# shape as a, where each slot of bool_idx tells
# whether that element of a is > 2.
bool_idx
array([[False, False], [ True, True], [ True, True]])
# We use boolean array indexing to construct a rank 1 array
# consisting of the elements of a corresponding to the True values
# of bool_idx
a[a>2]
array([3, 4, 5, 6])
For brevity we have left out a lot of details about numpy array indexing; if you want to know more you should read the documentation.
Every numpy array is a grid of elements of the same type. Numpy provides a large set of numeric datatypes that you can use to construct arrays. Numpy tries to guess a datatype when you create an array, but functions that construct arrays usually also include an optional argument to explicitly specify the datatype. Here is an example:
x = np.array([1, 2]) # Let numpy choose the datatype
y = np.array([1.0, 2.0]) # Let numpy choose the datatype
z = np.array([1, 2], dtype=np.int64) # Force a particular datatype
x.dtype, y.dtype, z.dtype
(dtype('int64'), dtype('float64'), dtype('int64'))
You can read all about numpy datatypes in the documentation.
You can transpose dimensions within an array using arbitrary axis permutations.
a = np.ones((3, 5))
print_arr(a.transpose()) # also a.T
[[1. 1. 1.] [1. 1. 1.] [1. 1. 1.] [1. 1. 1.] [1. 1. 1.]] shape=(5, 3) dtype=float64
a = np.ones((2, 4, 6))
a[1,2,3] = 777
print_arr(a.transpose(1,0,2))
[[[ 1. 1. 1. 1. 1. 1.] [ 1. 1. 1. 1. 1. 1.]] [[ 1. 1. 1. 1. 1. 1.] [ 1. 1. 1. 1. 1. 1.]] [[ 1. 1. 1. 1. 1. 1.] [ 1. 1. 1. 777. 1. 1.]] [[ 1. 1. 1. 1. 1. 1.] [ 1. 1. 1. 1. 1. 1.]]] shape=(4, 2, 6) dtype=float64
Note that an element [x,y,z]
moves to position [y,x,z]
after a transpose with this permutation (1,0,2).
Another important feature is reshaping an array into different dimensions.
a = np.ones((3, 6))
print_arr(np.reshape(a, (2, 9)))
[[1. 1. 1. 1. 1. 1. 1. 1. 1.] [1. 1. 1. 1. 1. 1. 1. 1. 1.]] shape=(2, 9) dtype=float64
When reshaping, we need to make sure to preserve the same number of elements.
Use -1
in one of the dimensions to tell numpy to "figure it out".
You can also combine multiple arrays with concatenation along an arbitrary axis.
a = np.array([[1, 2], [3, 4]])
b = np.array([[5, 6]])
print_arr(a)
print_arr(b)
[[1 2] [3 4]] shape=(2, 2) dtype=int64 [[5 6]] shape=(1, 2) dtype=int64
print_arr(np.concatenate((a, b), axis=0))
[[1 2] [3 4] [5 6]] shape=(3, 2) dtype=int64
print_arr(np.concatenate((a, b.T), axis=1))
[[1 2 5] [3 4 6]] shape=(2, 3) dtype=int64
print_arr(np.concatenate((a, b), axis=None))
[1 2 3 4 5 6] shape=(6,) dtype=int64
Broadcasting is a powerful mechanism that allows numpy to work with arrays of different shapes when performing arithmetic operations.
Frequently we have a smaller array and a larger array, and we want to use the smaller array multiple times to perform some operation on the larger array.
For example, suppose that we want to add a constant vector to each row of a matrix.
# We will add the vector v to each row of the matrix x,
# storing the result in the matrix y
x = np.array([[1,2,3], [4,5,6], [7,8,9], [10, 11, 12]])
v = np.array([1, 0, 1])
y = np.empty_like(x) # Create an empty matrix with the same shape as x
print_arr(x,'x=\n')
print_arr(v, 'v=')
x= [[ 1 2 3] [ 4 5 6] [ 7 8 9] [10 11 12]] shape=(4, 3) dtype=int64 v=[1 0 1] shape=(3,) dtype=int64
Naïve approach: Use a loop.
# Add the vector v to each row of the matrix x with an explicit loop
for i in range(4):
y[i, :] = x[i, :] + v
y
array([[ 2, 2, 4], [ 5, 5, 7], [ 8, 8, 10], [11, 11, 13]])
This works; however computing explicit loops in Python is slow.
Naïve approach 2: adding the vector v to each row of the matrix x
is equivalent to forming a matrix vv
by stacking multiple copies of v
vertically, then performing elementwise summation of x
and vv
.
We could implement this approach like this:
vv = np.tile(v, (4, 1)) # Stack 4 copies of v on top of each other
vv
array([[1, 0, 1], [1, 0, 1], [1, 0, 1], [1, 0, 1]])
y = x + vv # Add x and vv elementwise
y
array([[ 2, 2, 4], [ 5, 5, 7], [ 8, 8, 10], [11, 11, 13]])
Nice, but a new array was allocated and memory was copied.
Numpy broadcasting allows us to perform this computation without actually creating multiple copies of v. Consider this version, using broadcasting:
x = np.array([[1,2,3], [4,5,6], [7,8,9], [10, 11, 12]])
v = np.array([1, 0, 1])
# Add v to each row of x using broadcasting
y = x + v
print('shapes: ', x.shape, v.shape)
y
shapes: (4, 3) (3,)
array([[ 2, 2, 4], [ 5, 5, 7], [ 8, 8, 10], [11, 11, 13]])
The line y = x + v
works even though x
has shape (4, 3)
and v
has shape (3,)
due to broadcasting; this line works as if v actually had shape (4, 3)
, where each row was a copy of v
, and the sum was performed elementwise.
Broadcasting two arrays together follows these rules:
In our example:
x
has shape (4,3)
v
has shape (3,)
.Following the Broadcasting logic, we can say the following is equivalent to what happened:
v
has less dims than x
so a dimension of 1
is prepended -> v
is now (1, 3)
.(max(1,4), max(3,3)) = (4,3)
.v
matches exactly (3); dim 0 is exactly 1, so we can use the first data entry (row 0) for each time any row is accessed. This is effectively like converting v
from (1,3)
to (4,3)
by replicating.Broadcasting is incredibly useful and necessary for writing vectorized code, i.e. code that avoids explicit python loops which are very slow. Instead, this approach leveraged the underlying C implementation of numpy.
For more on broadcasting, see the documentation or this explanation.
Functions that support broadcasting are known as universal functions. You can find the list of all universal functions in the documentation.
Here are some applications of broadcasting:
# Compute outer product of vectors
v = np.array([1,2,3]) # v has shape (3,)
w = np.array([4,5]) # w has shape (2,)
print_arr(v)
print_arr(w)
[1 2 3] shape=(3,) dtype=int64 [4 5] shape=(2,) dtype=int64
# To compute an outer product, we first reshape v to be a column
# vector of shape (3, 1); we can then broadcast it against w to yield
# an output of shape (3, 2), which is the outer product of v and w:
# (3,1) * (2,) -> (3,1) * (1, 2) -> (3, 2) * (3, 2)
np.reshape(v, (3, 1)) * w
array([[ 4, 5], [ 8, 10], [12, 15]])
# Multiply a matrix by a constant:
x = np.ones((2,3))
# x has shape (2, 3). Numpy treats scalars as arrays of shape ();
# these can be broadcast together to shape (2, 3).
# (2,3) * () -> (2,3) * (1,1) -> (2,3) * (2,3)
x * 2
array([[2., 2., 2.], [2., 2., 2.]])
Broadcasting typically makes your code more concise and faster, so you should strive to use it where possible.
This brief overview has touched on many of the important things that you need to know about numpy, but is far from complete. Check out the numpy reference to find out much more about numpy.
import torch
torch.__version__
'1.3.0'
PyTorch is a relatively new yet widely used deep learning framework.
During the course we'll use it extensively and learn many parts of its API. You should also familiarize yourself with the PyTorch Documentation as it will greatly assist you when implementing your own models.
This notebook will show only a small part of PyTorch's API, the Tensor
class.
This class is very similar to numpy's ndarray
, and provides much of the same functionality.
However, it also has two important distinctions:
In the next tutorials we will examine these concepts further.
This notebook will show some brief examples, just to get a feel for it and compare it to the usual numpy ndarray
s.
You will be using both PyTorch tensors and numpy ndarray
s extensively throughout the course homework assignments, and in general when implementing deep learning algorithms.
# Basic tensor creation
t = torch.tensor([[1, 2, 3], [4, 5, 6]], dtype=torch.float32)
print_arr(t)
tensor([[1., 2., 3.], [4., 5., 6.]]) shape=torch.Size([2, 3]) dtype=torch.float32
Some other tensor construction methods:
torch.zeros((1,2))
tensor([[0., 0.]])
torch.ones((3,2))
tensor([[1., 1.], [1., 1.], [1., 1.]])
torch.randint(10, (4,1))
tensor([[6], [5], [1], [4]])
As in numpy, tensors can be reshaped
t.reshape(3, 2)
tensor([[1., 2.], [3., 4.], [5., 6.]])
In many cases, PyTorch provides the same methods on both the Tensor
object and via the torch
package.
torch.reshape(t, (3, -1))
tensor([[1., 2.], [3., 4.], [5., 6.]])
As in numpy, many functions are provided for computations over tensors, for example,
print_arr(torch.mean(t, dim=1))
tensor([2., 5.]) shape=torch.Size([2]) dtype=torch.float32
Sometime it's necessary to preserve original dimensions:
print_arr(torch.mean(t, dim=1, keepdim=True))
tensor([[2.], [5.]]) shape=torch.Size([2, 1]) dtype=torch.float32
Properties of a tensor:
print('shape:', t.size()) # t.shape also works, like numpy
print('dimensions:', t.dim())
print('type:', t.type())
print('device:', t.device)
shape: torch.Size([2, 3]) dimensions: 2 type: torch.FloatTensor device: cpu
Slicing works just like in numpy:
t = torch.Tensor([[1, 2, 3], [4, 5, 6], [7, 8, 9]])
t
tensor([[1., 2., 3.], [4., 5., 6.], [7., 8., 9.]])
t[:, -1]
tensor([3., 6., 9.])
t[:2, :]
tensor([[1., 2., 3.], [4., 5., 6.]])
t[-1:, -1:]
tensor([[9.]])
Unlike numpy, PyTorch tensors can be moved to GPU memory (if a GPU is available on the machine).
if torch.cuda.is_available():
t = t.cuda()
# Better way to use GPU tensors (and later - models)
device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
t = t.to(device)
Broadcasting also works (with almost identical semantics):
x = torch.tensor([[1,2,3], [4,5,6], [7,8,9], [10, 11, 12]])
v = torch.tensor([1, 0, 1])
print_arr(x)
print_arr(v)
tensor([[ 1, 2, 3], [ 4, 5, 6], [ 7, 8, 9], [10, 11, 12]]) shape=torch.Size([4, 3]) dtype=torch.int64 tensor([1, 0, 1]) shape=torch.Size([3]) dtype=torch.int64
y = x + v # Add v to each row of x using broadcasting
y
tensor([[ 2, 2, 4], [ 5, 5, 7], [ 8, 8, 10], [11, 11, 13]])