You can read an overview of this Numerical Linear Algebra course in this blog post. The course was originally taught in the University of San Francisco MS in Analytics graduate program. Course lecture videos are available on YouTube (note that the notebook numbers and video numbers do not line up, since some notebooks took longer than 1 video to cover).

You can ask questions about the course on our fast.ai forums.

0. Course Logistics

Ask Questions

Let me know how things are going. This is particularly important since I'm new to MSAN, I don't know everything you've seen/haven't seen.

Intro

My background and linear algebra love:

  • Swarthmore College: linear algebra convinced me to be a math major! (minors in CS & linguistics) I thought linear algebra was beautiful, but theoretical
  • Duke University: Math PhD. Took numerical linear algebra. Enjoyed the course, but not my focus
  • Research Triangle Institute: first time using linear algebra in practice (healthcare economics, markov chains)
  • Quant: first time working with lots of data, decided to become a data scientist
  • Uber: data scientist
  • Hackbright: taught software engineering. Overhauled ML and collaborative filtering lectures
  • fast.ai: co-founded to make deep learning more accessible. Deep Learning involves a TON of linear algebra

Teaching

Teaching Approach

I'll be using a top-down teaching method, which is different from how most math courses operate. Typically, in a bottom-up approach, you first learn all the separate components you will be using, and then you gradually build them up into more complex structures. The problems with this are that students often lose motivation, don't have a sense of the "big picture", and don't know what they'll need.

If you took the fast.ai deep learning course, that is what we used. You can hear more about my teaching philosophy in this blog post or in this talk.

Harvard Professor David Perkins has a book, Making Learning Whole in which he uses baseball as an analogy. We don't require kids to memorize all the rules of baseball and understand all the technical details before we let them play the game. Rather, they start playing with a just general sense of it, and then gradually learn more rules/details as time goes on.

All that to say, don't worry if you don't understand everything at first! You're not supposed to. We will start using some "black boxes" or matrix decompositions that haven't yet been explained, and then we'll dig into the lower level details later.

To start, focus on what things DO, not what they ARE.

People learn by:

  1. doing (coding and building)
  2. explaining what they've learned (by writing or helping others)

Text Book

The book Numerical Linear Algebra by Trefethen and Bau is recommended. The MSAN program has a few copies on hand.

A secondary book is Numerical Methods by Greenbaum and Chartier.

Basics

Office hours: 2:00-4:00 on Friday afternoons. Email me if you need to meet at other times.

My contact info: [email protected]

Class Slack: #numerical_lin_alg

Email me if you will need to miss class.

Jupyter Notebooks will be available on Github at: https://github.com/fastai/numerical-linear-algebra Please pull/download before class. Some parts are removed for you to fill in as you follow along in class. Be sure to let me know THIS WEEK if you are having any problems running the notebooks from your own computer. You may want to make a separate copy, because running Jupyter notebooks causes them to change, which can create github conflicts the next time you pull.

Check that you have MathJax running (which renders LaTeX, used for math equations) by running the following cell:

$$ e^{\theta i} = \cos(\theta) + i \sin(\theta)$$

check that you can import:

In [2]:
import numpy as np
import sklearn

Grading Rubric:

Assignment Percent
Attendance 10%
Homework 20%
Writing: proposal 10%
Writing: draft 15%
Writing: final 15%
Final Exam 30%

Honor Code

No cheating nor plagiarism is allowed, please see below for more details.

On Laptops

I ask you to be respectful of me and your classmates and to refrain from surfing the web or using social media (facebook, twitter, etc) or messaging programs during class. It is absolutely forbidden to use instant messaging programs, email, etc. during class lectures or quizzes.

Syllabus

Topics Covered:

1. Why are we here?

  • Matrix and Tensor Products
  • Matrix Decompositions
  • Accuracy
  • Memory use
  • Speed
  • Parallelization & Vectorization

2. Topic Modeling with NMF and SVD

  • Topic Frequency-Inverse Document Frequency (TF-IDF)
  • Singular Value Decomposition (SVD)
  • Non-negative Matrix Factorization (NMF)
  • Stochastic Gradient Descent (SGD)
  • Intro to PyTorch
  • Truncated SVD, Randomized SVD

3. Background Removal with Robust PCA

  • Robust PCA
  • Randomized SVD
  • LU factorization

4. Compressed Sensing for CT scans with Robust Regression

  • L1 regularization

5. Predicting Health Outcomes with Linear Regression

  • Linear regression
  • Polynomial Features
  • Speeding up with Numba
  • Regularization and Noise
  • Implementing linear regression 4 ways

6. PageRank with Eigen Decompositions

  • Power Method
  • QR Algorithm
  • Arnoldi Iteration

7. QR Factorization

  • Gram-Schmidt
  • Householder
  • Stability

Writing Assignment

Writing Assignment: Writing about technical concepts is a hugely valuable skill. I want you to write a technical blog post related to numerical linear algebra. A blog is like a resume, only better. Technical writing is also important in creating documentation, sharing your work with co-workers, applying to speak at conferences, and practicing for interviews. (You don't actually have to publish it, although I hope you do, and please send me the link if you do.)

  • List of ideas here
  • Always cite sources, use quote marks around quotes. Do this even as you are first gathering sources and taking notes. If you plagiarize parts of someone else's work, you will fail.
  • Can be done in a Jupyter Notebook (Jupyter Notebooks can be turned into blog posts) or a Kaggle Kernel

For the proposal, write a brief paragraph about the problem/topic/experiment you plan to research/test and write about. You need to include 4 sources that you plan to use: these can include Trefethen, other blog posts, papers, or books. Include a sentence about each source, stating what it's in it.

Feel free to ask me if you are wondering if your topic idea is suitable!

Excellent Technical Blogs

Examples of great technical blog posts:

Deadlines

Assignment Dates
Homeworks TBA
Writing: proposal 5/30
Writing: draft 6/15
Writing: final 6/27
Final Exam 6/29

Linear Algebra

We will review some linear algebra in class. However, if you find there are concepts you feel rusty on, you may want to review on your own. Here are some resources:

USF Policies

Academic Integrity

USF upholds the standards of honesty and integrity from all members of the academic community. All students are expected to know and adhere to the University’s Honor Code. You can find the full text of the code online. The policy covers:

  • Plagiarism: intentionally or unintentionally representing the words or ideas of another person as your own; failure to properly cite references; manufacturing references.
  • Working with another person when independent work is required.
  • Submission of the same paper in more than one course without the specific permission of each instructor.
  • Submitting a paper written (entirely or even a small part) by another person or obtained from the internet.
  • Plagiarism is plagiarism: it does not matter if the source being copied is on the Internet, from a book or textbook, or from quizzes or problem sets written up by other students.
  • The penalties for violation of the policy may include a failing grade on the assignment, a failing grade in the course, and/or a referral to the Academic Integrity Committee.

Students with Disabilities

If you are a student with a disability or disabling condition, or if you think you may have a disability, please contact USF Student Disability Services (SDS) at 415 422-2613 within the first week of class, or immediately upon onset of disability, to speak with a disability specialist. If you are determined eligible for reasonable accommodations, please meet with your disability specialist so they can arrange to have your accommodation letter sent to me, and we will discuss your needs for this course. For more information, please visit this website or call (415) 422-2613.

Behavioral Expectations

All students are expected to behave in accordance with the Student Conduct Code and other University policies. Open discussion and disagreement is encouraged when done respectfully and in the spirit of academic discourse. There are also a variety of behaviors that, while not against a specific University policy, may create disruption in this course. Students whose behavior is disruptive or who fail to comply with the instructor may be dismissed from the class for the remainder of the class period and may need to meet with the instructor or Dean prior to returning to the next class period. If necessary, referrals may also be made to the Student Conduct process for violations of the Student Conduct Code.

Counseling and Psychological Services

Our diverse staff offers brief individual, couple, and group counseling to student members of our community. CAPS services are confidential and free of charge. Call 415-422-6352 for an initial consultation appointment. Having a crisis at 3 AM? We are still here for you. Telephone consultation through CAPS After Hours is available between the hours of 5:00 PM to 8:30 AM; call the above number and press 2.

Confidentiality, Mandatory Reporting, and Sexual Assault

As an instructor, one of my responsibilities is to help create a safe learning environment on our campus. I also have a mandatory reporting responsibility related to my role as a faculty member. I am required to share information regarding sexual misconduct or information about a crime that may have occurred on USFs campus with the University. Here are other resources:

  • To report any sexual misconduct, students may visit Anna Bartkowski (UC 5th floor) or see many other options by visiting this website
  • Students may speak to someone confidentially, or report a sexual assault confidentially by contacting Counseling and Psychological Services at 415-422-6352
  • To find out more about reporting a sexual assault at USF, visit USF’s Callisto website
  • For an off-campus resource, contact San Francisco Women Against Rape 415-647-7273