Jupyter notebooks provide a useful environment for interactive exploration of data. A common question I get, though, is how you can progress from this nonlinear, interactive, trial-and-error style of exploration to a more linear and reproducible analysis based on organized, packaged, and tested code. This series of videos presents a case study in how I personally approach reproducible data analysis within the Jupyter notebook.
Each video is approximately 5-8 minutes; the videos are available in a YouTube Playlist. Alternatively, below you can find the videos with some description and links to relevant resources
# Quick utility to embed the videos below from IPython.display import YouTubeVideo def embed_video(index, playlist='PLYCpMb24GpOC704uO9svUrihl-HY1tTJJ'): return YouTubeVideo('', index=index - 1, list=playlist, width=600, height=350)
In this video, I introduce the dataset, and use the Jupyter notebook to download and visualize it.
Fremont Bridge Bike Counter: the website where you can explore the data
A Whirlwind Tour of Python: my book introducing the Python programming language, aimed at scientists and engineers.
Python Data Science Handbook: my book introducing Python's data science tools, including an introduction to the IPython, Pandas, and Matplotlib tools used here.
In this video, I do some slightly more sophisticated visualization with the data, using matplotlib and pandas.
In this video, I set up a repository on GitHub and commit the notebook into version control.
In this video, I refactor the data download script so that it only downloads the data when needed
In this video, I move the data download utility into its own separate package
In this video, I refactor the data download function to be a bit faster
In this video, I discover that my refactoring has caused a bug. I debug it and fix it.
In this video, I discover a bug in the scikit-learn codebase, and go through the process of submitting a GitHub Pull Request fixing the bug
In this video, I apply unsupervised learning techniques to the data to explore what we can learn from it
In this video, I clean-up the unsupervised learning analysis to make it more reproducible and presentable.