Before we start with Numpy...
Because Numpy is based on Python, we want to make sure you know the basics of Python3 in order to understand the materials we will dicuss throughout the course without any significant difficulties. Let's begin our learning with basic Python3 syntax before we dig into the actual data science.
If you are an exprienced Python programmer, you can skip this section and start from the What is Numpy? section.
When we do a programming, we need to know what variables are, when we create them, and how we use them. A variable is a memory location that stores certain value, object, or data type. Let's see an example:
x = 3 y = 2
What did we just do above? We just created two variables, one called x which stores a numerical value of 3, and the other one called y which stores a numerical value of 2. We call it assignment because we assigned a value of 3 to x and a value of 2 to y. We can also do multiple assignments like:
x, y, z = 1, 2, 3
This will assign x as 1, y as 2, and z as 3. Really intuitive isn't it? :)
Numbers are not the only type we can assign. See different types of variables we can create:
# We can write any statement as "comment" if we put a hashtag in front of a line. # These comments will be ignored by computer when it tries to read the code, so you can write anything here @#!#$ name = "Jared Lim" # any combination of characters called String graduation_year = 2020 # Integer CDS_Member = True # is it true or false? called Boolean list_of_qualities = ["Fun", "Cool"] # a list of things
The most important data type we will discuss is list, because it can be interpreted as a one-dimensional vector, which can be seen as a matrix with only one row (Numpy is a package that deals with matrix and matrix-related operations).
As shown above, list is a data type that stores a sequence of items separated by commas and encolsed by two square brackets . One big advantage if list is that the items of a list can be of different types: for example, a list can contain number, string, and boolean all at the same time.
cds_list = [1998, "cds", True] #notice in the list there is an Int, String, and Boolean bigger_list = ["abcd", 1234, cds_list] # list within a list
So far, we have only discussed about initializing a list. Now, let's learn how to deal with items inside of a list. Python is a 0-indexed language, which means we start counting from 0. We can access the values in a list by using these indexes:
bigger_list #returns 'abcd', which is the 0th element in the list
bigger_list # returns 1234, which is our "first" element
bigger_list #returns cds_list which is the 2nd element within the bigger_list
[1998, 'cds', True]
You can also set the values to a specific index:
my_list = ["A", "B", "C"] my_list = "D" my_list # returns ['D', 'B', 'C']
['D', 'B', 'C']
A more advanced technique to access the elements in the list involve slice operator ([ ] and [:]) with indexes starting at 0 in the beginning of the list and working their way to end -1.
a_list = ["Abby", "Ann", "Cameron", "Shubhom", "Ethan"] a_list[0:2] # the first index is inclusive, the second index is exclusive # returns ["Abby", "Ann"]
You can use the plus (+) sign to concatenate lists.
You can use the asterisk (*) to repeat the list multiple times.
For more details, check the Python list operations: click me
b_list=["Dae Won", "Jared"] ab_list=a_list + b_list # Using the plus (+) sign I can put 2 lists together ab_list
['Abby', 'Ann', 'Cameron', 'Shubhom', 'Ethan', 'Dae Won', 'Jared']
b_list=b_list*3 # Using the asterisk (*) I can repeat the list. b_list
['Dae Won', 'Jared', 'Dae Won', 'Jared', 'Dae Won', 'Jared']
What is Numpy?
Numerical Python, or "Numpy" for short, is a foundational package on which many of the most common data science packages are built. Numpy provides us with multi-dimensional arrays, called ndarrays, which can be created as vectors or matrices. We can use numpy to manipulate datasets to make them easier to work with. Numpy also comes with a number of helpful statistical methods.
The key features of numpy are:
Additional Recommended Resources:
Python for Data Analysis by Wes McKinney
Python Data science Handbook by Jake VanderPlas
Intro to ndarrays
ndarrays are time and space-efficient multidimensional arrays at the core of numpy. One important thing to note is that all elements in an ndarray must be of the same type. Let's get started by creating ndarrays using the numpy package.
Creating and modifying Rank 1 ndarrays:
The "as" keyword in the import statement allows us to give a local name to the numpy package, so that we can refer to it as "np" rather than "numpy" in subsequent code. In the following lines of code, we use a couple of methods in the numpy package:
np.array([comma-separated elements here]) creates a rank 1 array (like a vector) with the elements specified between the brackets
nameOfArray.shape() returns a list of integers that represent the size of the array in each dimension
import numpy as np # makes local name for package numpy as np arr = np.array([3, 2, 1]) # Create a rank 1 array print(type(arr)) # The type of an ndarray is "<class 'numpy.ndarray'>"
# the shape of arr print(arr.shape)
# access each element in the array using its index print(arr, arr, arr)
(3, 2, 1)
Ndarrays are mutable, which means that the contents of an array can be changed after it is created.
arr = 100 # change the first element of the array (the element at index 0) print(arr)
[100 2 1]
Creating a Rank 2 ndarray:
A rank 2 ndarray has two dimensions. Notice the format below of [ [row] , [row] ]. 2 dimensional arrays are great for representing matrices which are often useful in data science. We use the same methods as before to analyze rank 2 arrays as well.
arr2 = np.array([[1,2,3],[6,5,4]]) # Create a rank 2 array print(arr2) # print the array print(arr2.shape) # print number of rows, columns print(arr2[0, 0], arr2[0, 1], arr2[1, 0]) # print the elements at the specified indices [row, column]
[[1 2 3] [6 5 4]] (2L, 3L) (1, 2, 6)
Different ways to create ndarrays:
In the code below, we create a number of different sized arrays with different shapes and values. Numpy has some built in methods (listed below) which help us quickly and easily create multidimensional arrays with pre-filled values.
np.zeros((dimensions)) creates an array of zeros with the specified dimensions
np.full((dimensions), value) creates an array with the specified dimensions where every element is the specified value
np.eye(dimensions) creates an array where the elements on the diagonal are 1s and all other elements are 0
np.ones(dimensions) creates an array of ones
np.random.random(dimensions) generates an array of random floating-point values between 0 and 1
import numpy as np # create a 3x4 array of zeros ex1 = np.zeros((3, 4)) print(ex1)
[[ 0. 0. 0. 0.] [ 0. 0. 0. 0.] [ 0. 0. 0. 0.]]
# create a 2x3 array filled with 9.0 ex2 = np.full((2,3), 6.0) print(ex2)
[[ 6. 6. 6.] [ 6. 6. 6.]]
# create a 2x3 array of random floating-point numbers between 0 and 1 ex3 = np.random.random((2,3)) print(ex3)
[[ 0.33070061 0.42710688 0.7277987 ] [ 0.63052933 0.12942678 0.86988264]]
Using Array Indexing
It's often more convenient to look only at specific sections of arrays. We can accomplish this using array indexing.
Slice indexing (slicing):
We can use slice indexing to pull out sub-regions of ndarrays. The general syntax for this is array[start index:end index]. Note that the start index is included in the slice, while the last index is not.
import numpy as np # Rank 2 array of shape (4, 4) an_array = np.array([[11,12,13,14], [21,22,23,24], [31,32,33,34], [41,42,43,44]]) print(an_array)
[[11 12 13 14] [21 22 23 24] [31 32 33 34] [41 42 43 44]]
array_slice = an_array[:2, 2:] # Use array slicing to get a subarray of the first 2 rows x the last 2 columns print(array_slice)
[[13 14] [23 24]]
When you modify a slice, you actually modify the underlying array. This is because when you use array slicing, you aren't creating a new array; instead, you're creating a reference to the slice of the array that you've selected. Also, note that the element at a given index in a slice often does not correspond to the element at that index in the original array, since a slice is a section of the original array. For example:
print('Initial element at 0, 1: ', an_array[0, 2]) # print the element at 0, 1 array_slice[0, 0] = 55 # array_slice[0, 0] is the same piece of data as an_array[0, 2] print('After modification: ', an_array[0, 2])
Initial element at 0, 1: 55 After modification: 55
Using integer indexing & slice indexing
Integer indexing, as the name implies, simply selects the elements of an array at the specified indices. We can use combinations of integer indexing and slice indexing to create different shaped matrices.
# create the same 4x4 array as above an_array = np.array([[11,12,13,14], [21,22,23,24], [31,32,33,34], [41,42,43,44]]) print(an_array)
[[11 12 13 14] [21 22 23 24] [31 32 33 34] [41 42 43 44]]
When slicing, [:] with no start or end indices selects all the elements in that row or column. In the following example, the combination of integer and slice indexing selects all elements in the last row of the original array.
# Using integer indexing with slicing generates an array of lower rank row_rank1 = an_array[2, :] print(row_rank1) # notice the 
[31 32 33 34]
Note that when you try to do the same thing using slicing alone, the subarray that you create will be of the same rank as the original array, even though it may actually be of a smaller dimension.
# Using slicing alone generates an array of the same rank as the original array row_rank2 = an_array[2:3, :] print(row_rank2) # Notice the [[ ]]
[[31 32 33 34]]
#We see the same thing when we work with the columns instead of the rows of the array: col_rank1 = an_array[:, 3]# turns to single brackets col_rank2 = an_array[:, 3:4]# turns to double brackets print(col_rank1) #signle brackets print() print(col_rank2) #double brackets
[14 24 34 44] [   ]
Using Array Indexing to change elements
Sometimes it's useful to use an array of indices to access or change elements in our larger matrix.
# Create a new 4x4 array an_array = np.array([[11,12,13,14], [21,22,23,24], [31,32,33,34], [41,42,43,44]]) print('Starting Array:') print(an_array)
Starting Array: [[11 12 13 14] [21 22 23 24] [31 32 33 34] [41 42 43 44]]
In the following code, we create an array of indices, called col_indices, with the values zero, one, two, and zero. We then use the np.arange function to create an ndarray with the values zero, one, and two, and three.
# Create an array of indices col_indices = np.array([0, 2, 1, 3]) print('Column indices picked : ', col_indices) row_indices = np.arange(4) print('Row indices picked : ', row_indices)
Column indices picked : [0 2 1 3] Row indices picked : [0 1 2 3]
Using a for loop and the zip function, we can see how these values might pair up if used as row and column indices. The corresponding 2D indices are printed below.
# print the indices (row, column) in the arrays created above for row,col in zip(row_indices,col_indices): print('(',row,',',col,')')
( 0 , 0 ) ( 1 , 2 ) ( 2 , 1 ) ( 3 , 3 )
When we inspect the contents at those pairs of indices, we get back the values 11, 23, 32, and 44. This technique is very useful, as it allows us to can access elements using arrays as indices.
# print the values in the array at the indices specified above print('Values in the array at specified indices: ',an_array[row_indices, col_indices])
('Values in the array at those indices: ', array([11, 23, 32, 44]))
We can also change elements in the array the same way. Here, we add 1,000 to an_array for our row and column indices. Looking at the new array, we see that (0, 0), (1, 2), (2, 1), and (3, 3) have all been incremented.
# change the elements at the selected indices an_array[row_indices, col_indices] = 1000 print('\nChanged Array:') print(an_array) print('\nNew values in the array at specified indices: ',an_array[row_indices,col_indices])
Changed Array: [[1000 12 13 14] [ 21 22 1000 24] [ 31 1000 33 34] [ 41 42 43 1000]] ('\nNew values in the array at those indices: ', array([1000, 1000, 1000, 1000]))
Using Boolean Indexing
We can also use boolean indexing to filter out elements of an array based on whether or not they fulfill some condition. This is very useful when we only want to look at a specific portion of a dataset, ex. where some entries have a certain characteristic we want to explore.
Using Array Indexing to change elements
# create a 2x3 array an_array = np.array([[11,12,13], [21,22,23]]) print(an_array)
[[11 12 13] [21 22 23]]
The following code creates a boolean array of the same dimensions as the original array, where True indicates that the corresponding element of the original array fulfills the condition, and False indicates that the corresponding element does not fulfill the condition.
# create a filter of boolean values indicating whether each element meets the condition filter = (an_array < 15) filter
array([[ True, True, True], [False, False, False]], dtype=bool)
# using the filter, we can select just those elements which meet that criteria print(an_array[filter])
[21 22 23]
The following code accomplishes the same thing without the intermediate step of creating the filter array:
an_array[(an_array < 15)]
array([21, 22, 23])
We can also change elements in the array using a filter. The following code adds 200 to all elements < 15 in the array:
an_array[an_array < 15] += 200 print(an_array)
[[ 11 12 13] [121 122 123]]
Arithmetic Array Operations:
There are also a number of useful arithmetic operations that may be used on numpy arrays. These include:
add, which adds the corresponding elements of different arrays. You can use np.add(array1, array2) or simply use the plus sign. Subtraction, multiplication, and division work the same way. See Numpy documentation for more details.
np.sqrt(array) returns an array where each element is the square root of the corresponding element in the original array.
np.exp(array) returns an array where each element is e raised to the power of the corresponding element in the original array.
a = np.array([[11,12],[21,22]], dtype=np.int) b = np.array([[11.1,12.1],[21.1,22.1]], dtype=np.float64) print(a) print() print(b)
[[11 12] [21 22]] [[ 11.1 12.1] [ 21.1 22.1]]
# add print(a + b) print() print(np.add(a, b)) # the plus sign does the same thing as the numpy function "add"
[[ 22.1 24.1] [ 42.1 44.1]] [[ 22.1 24.1] [ 42.1 44.1]]
# subtract print(a - b) print() print(np.subtract(a, b))
[[-0.1 -0.1] [-0.1 -0.1]] [[-0.1 -0.1] [-0.1 -0.1]]
# multiply print(a * b) print() print(np.multiply(a, b))
[[ 122.1 145.2] [ 443.1 486.2]] [[ 122.1 145.2] [ 443.1 486.2]]
# divide print(a / b) print() print(np.divide(a, b))
[[ 0.99099099 0.99173554] [ 0.99526066 0.99547511]] [[ 0.99099099 0.99173554] [ 0.99526066 0.99547511]]
# square root print(np.sqrt(a))
[[ 3.31662479 3.46410162] [ 4.58257569 4.69041576]]
# exponent (e ** a) print(np.exp(a))
[[ 5.98741417e+04 1.62754791e+05] [ 1.31881573e+09 3.58491285e+09]]
Intro to Statistical Methods, Sorting, and Set Operations
Getting Started with Statistical Operations
There are many useful statistical operations for numpy arrays, some of which are:
array.mean(), which computes the mean of all elements in a matrix. array.mean(axis = 1) returns an array containing the mean values of each row, while arr.mean(axis = 0) returns an array containing the mean values of each column.
array.sum() returns the sum of all of the elements in the array.
np.median(array, axis=) computes the median of the elements in an array. Similar to the mean function, the axis argument specifies whether the medians should be computed by row or by column.
There are many other statistical methods out there; check out the numpy reference below if you need a function that isn't listed here or if you're looking for more detailed information about the functions above. Numpy Reference
# setup a random 3x3 matrix arr = 10 * np.random.randn(3,3) print(arr) # compute the mean for all elements in the array print('\n',arr.mean()) # set the axis value to 1 compute the means for each row print('\n',arr.mean(axis = 1)) # set the axis value to 0 compute the means for each column print('\n',arr.mean(axis = 0)) # sum all the elements in the array print('\n',arr.sum()) # compute the median for all elements in the array print('\n',np.median(arr)) # compute the medians for each row print('\n',np.median(arr, axis = 1))
[[-14.23672556 -0.78790432 4.5445005 ] [ 8.69446704 -0.5144282 3.5245827 ] [ -2.45896255 1.9621798 -26.3136989 ]] -2.84288772214 [-3.49337646 3.90154051 -8.93682722] [-2.66707369 0.21994909 -6.08153857] -25.5859894993 -0.514428200974 [-0.78790432 3.5245827 -2.45896255]
Using the Unique method
The NumPy method unique is very useful in data science. It allows us to pull out only the values that are unique in an array. Note that in the following example, the array has a number of duplicate 8s, 12s, and 13s. The output after calling unique on it is just 8, 12, and 13.
an_array = np.array([8,12,13,13,12,8,13,12]) print(np.unique(an_array))
[ 8 12 13]
Set Operations on ndarrays
We can use set routines in numpy to perform operations on and compare two arrays. In the code below, we use the following set methods:
np.intersect1d(array1, array2) returns an array with the values that array1 and array2 both have.
np.union1d(array1, array2) returns an array with all the unique values from both array1 and array2.
np.setdiff1d(array1, array2) returns an array with elements in array1 that are not in array2.
np.in1d(array1, array2) returns a boolean array of whether each element of array1 is also present in array2.
*Note that the two arrays can have different numbers of elements, but must both be rank 1 arrays.
ar1 = np.array(['dog','cat','bird','turtle']) ar2 = np.array(['cat','bird','horse']) print(ar1, ar2)
['dog' 'cat' 'bird' 'turtle'] ['cat' 'bird' 'horse']
print( np.intersect1d(ar1, ar2) )
print( np.union1d(ar1, ar2) )
['bird' 'cat' 'dog' 'horse' 'turtle']
print( np.setdiff1d(ar1, ar2) )
print( np.in1d(ar1, ar2) )
[False True True False]
Intro to Broadcasting:
Broadcasting is one of the more advanced features of NumPy, and it can help make array operations much more convenient. The term broadcasting describes how numpy treats arrays with different shapes during arithmetic operations. Subject to certain constraints, the smaller array is “broadcast” across the larger array so that they have compatible shapes. During the operation, no copy is involved in the process and both arrays retain their original shapes, making broadcasting very memory and computationally efficient.
For more details on broadcasting, please see <a href= https://docs.scipy.org/doc/numpy-1.10.1/user/basics.broadcasting.html>this resource.</a>
import numpy as np array = np.zeros((5,3)) print(array)
[[ 0. 0. 0.] [ 0. 0. 0.] [ 0. 0. 0.] [ 0. 0. 0.] [ 0. 0. 0.]]
# create a rank 1 ndarray with 3 values add_rows = np.array([5, 3, 9]) print(add_rows)
[5 3 9]
x = start - add_rows # subtract from each row of 'array' using broadcasting print(x)
[[-5. -3. -9.] [-5. -3. -9.] [-5. -3. -9.] [-5. -3. -9.]]
# create a 5x1 ndarray to broadcast across columns add_cols = np.array([[10,20,30,40,50]]) add_cols = add_cols.T print(add_cols)
[    ]
# add to each column of 'start' using broadcasting x = array + add_cols print(x)
[[ 10. 10. 10.] [ 20. 20. 20.] [ 30. 30. 30.] [ 40. 40. 40.] [ 50. 50. 50.]]
# this will broadcast in both dimensions scalar = np.array() print(array+scalar)
[[ 3. 3. 3.] [ 3. 3. 3.] [ 3. 3. 3.] [ 3. 3. 3.] [ 3. 3. 3.]]
Other Common ndarray Operations
Below, you'll find some other useful functions for ndarrays. There are a myriad of these, so we encourage you to go through these and explore the numpy documentation linked below.
Dot Product and Inner Product
array1.dot(array2) or np.dot(array1, array2) returns the dot or inner product of two arrays.
*Note that if the two arrays are 2D (matrices), dot returns the dot product, and if they are 1D (vectors), it returns the inner product.
# determine the dot product of two matrices arr1_2d = np.array([[2,2],[2,2]]) arr2_2d = np.array([[1,1],[1,1]]) print(arr1_2d.dot(arr2_2d)) print() print(np.dot(arr1_2d, arr2_2d))
[[4 4] [4 4]] [[4 4] [4 4]]
# determine the inner product of two vectors arr1_1d = np.array([9 , 9 ]) arr2_1d = np.array([10, 10]) print(arr1_1d.dot(arr2_1d)) print() print(np.dot(arr1_1d, arr2_1d))
# dot product on an array and vector print(arr1_2d.dot(arr1_1d)) print() print(np.dot(arr1_2d, arr1_1d))
[36 36] [36 36]
In the following code, we explore the various uses of the sum() method.
# sum elements in the array arr1 = np.array([[10,15],[20,25]]) print(np.sum(arr1)) # sum of all elements
print(np.sum(arr1, axis=0)) # sum of elements in each column
print(np.sum(arr1, axis=1)) # sum of elements in each row
np.maximum(array1, array2) compares two arrays and returns a new array containing the element-wise maxima. For more element-wise functions, see the numpy documentation.
# create a random array a = np.random.randn(3,2) print(a)
[[ 0.2464078 1.08978049] [-2.32407039 -1.26413564] [ 0.65640993 -0.08689985]]
# create another random array b = np.random.randn(3,2) print(b)
[[-0.23497313 -0.65355001] [-1.44739306 -0.24026659] [ 0.50512202 -1.92570741]]
# return the element wise maxima between two arrays print(np.maximum(a, b))
[[ 0.2464078 1.08978049] [-1.44739306 -0.24026659] [ 0.65640993 -0.08689985]]
array.reshape(dimensions) gives a new shape to an array without changing its data.
# put values 0 through 14 in an array arr = np.arange(15) print(arr)
[ 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14]
# reshape to be a 5 x 3 matrix new_arr = arr.reshape(5,3) print(new_arr)
[[ 0 1 2] [ 3 4 5] [ 6 7 8] [ 9 10 11] [12 13 14]]
[[ 0 1 2 3 4] [ 5 6 7 8 9] [10 11 12 13 14]]
np.transpose(array) returns the transpose of an array with its dimensions permuted.
# transpose arr = np.array([[11,12],[21,22]]) new_arr1 = np.transpose(arr) print(new_arr1)
[[11 21] [12 22]]
# another way to call the method new_arr2 = arr.T print(new_arr2)
[[11 21] [12 22]]
Indexing using where():
np.where(condition, array1, array2) returns elements, either from array1 or array2, depending on the condition. The output array contains elements of array1 where the condition is True, and elements from array2 elsewhere.
array1 = np.array([1,2,3,4,5]) array2 = np.array([10,20,30,40,50]) filter = np.array([True, False, True, False, True])
out = np.where(filter, array1, array2) print(out)
[ 1 20 3 40 5]
ran_arr = np.random.rand(3,3) print(ran_arr)
[[ 0.19950743 0.02384889 0.61389011] [ 0.17116865 0.7800457 0.19325871] [ 0.35606818 0.16259557 0.06632833]]
new_arr = np.where( ran_arr > 0.5, 1000, -1) print(new_arr)
[[ -1 -1 1000] [ -1 1000 -1] [ -1 -1 -1]]
Using any() and all()
np.any() tests whether any element in an array evaluates to True.
np.all() tests whether all elements in an array evaluate to True.
arr_bools = np.array([ True, False, True, True, False ])
Random Number Generation:
np.random.normal(mean, standard deviation, dimensions) draws random samples from a normal (Gaussian) distribution using information provided in parameters.
np.random.randint(low, high, dimensions) returns an array with specified dimensions of random integers from low (inclusive) to high (exclusive).
np.random.permutation(array) returns a new array with original array elements shuffled randomly.
np.random.uniform(low, high, dimensions) draws samples from a uniform distribution using information provided in parameters.
arr1 = np.random.normal(size = (3,4)) print(arr1)
[-2.71998444 -0.09117765 -0.42421087 2.65954844]
arr2 = np.random.randint(low=3,high=30,size=5) print(arr2)
[18 28 13 6 25]
np.random.permutation(arr2) # reorder elements in arr2
array([28, 6, 25, 13, 18])
np.random.uniform(size=3) # uniform distribution
array([ 0.31604841, 0.98754604, 0.73381728])
np.random.normal(size=3) # normal distribution
array([ 0.9300743 , 1.11163927, 0.90843176])
Merging two data sets:
np.vstack((array1, array2)) takes a sequence of arrays and stacks them vertically to make a single array.
np.hstack((array1, array2)) takes a sequence of arrays and stacks them horizontally to make a single array.
np.concatenate((array1, array2), axis) joins a sequence of arrays along a the specified axis.
arr1 = np.random.randint(low=5,high=30,size=(3,3)) print(arr1) print() arr2 = np.random.randint(low=5,high=30,size=(3,3)) print(arr2)
[[ 7 6 17] [ 7 21 9] [ 8 21 7]] [[ 6 19 13] [ 9 13 28] [28 7 24]]
varr = np.vstack((arr1,arr2)) print(varr)
[[ 7 6 17] [ 7 21 9] [ 8 21 7] [ 6 19 13] [ 9 13 28] [28 7 24]]
harr = np.hstack((arr1,arr2)) print(harr)
[[ 7 6 17 6 19 13] [ 7 21 9 9 13 28] [ 8 21 7 28 7 24]]
np.concatenate([arr1, arr2], axis = 0)
array([[ 7, 6, 17], [ 7, 21, 9], [ 8, 21, 7], [ 6, 19, 13], [ 9, 13, 28], [28, 7, 24]])
np.concatenate([arr1, arr2.T], axis = 1)
array([[ 7, 6, 17, 6, 9, 28], [ 7, 21, 9, 19, 13, 7], [ 8, 21, 7, 13, 28, 24]])