Scraping NBA team information from Wikipedia (Revisited)¶

Update (March 6, 2018): This simple Wikipedia table is the subject of a lot of editing! Somebody modified it again in early February. In this case, the editor felt that US state postal codes would be unfamiliar to international NBA fans. Instead, the table current uses the full name of the US states (and Ontario, in the case of Toronto). The URL below was changed to reflect an historical snapshot of the table prior to this change.

In this notebook, we are going to take another look at scraping NBA team information from Wikipedia. We will also see how generate a map of NBA arena locations.

In an earlier notebook, we scraped the table using the Requests and BeautifulSoup packages.

Unfortunately, somebody modified the table in mid-December. As a result, the original code in that notebook no longer works.

Web content changes all the time, which will occasionally break web scraping code. This is particularly true of Wikipedia, where pages are open to edits by the community.

In this particular case, we could just move on and ignore the table changes. The NBA team data are basically unchanged. We could just use the saved CSV file from the prior scraping. That's a major reason why you should always save the result of web scraping.

On the other hand, I think this is a good opportunity to try to scrape the table in a more robust and general way. You will also see examples of some useful pandas techniques to clean up the Wikipedia data. I think you will find these techniques useful in your own sports analytics projects.

I also wanted to do something useful with the Wikipedia information, beyond using it as an example to learn web scraping. Later in this post, we'll discuss why arena location data can be useful in sports analytics. Drawing a map is a perfect way to learn how to start using geographic data in Python.

What Changed¶

The change the person made to the table was relatively simple. This person decided to group together certain cells in the table for the two New York teams (the Knicks and the Nets) and the LA teams (the Clippers and the Lakers). In particular, this person added HTML rowspan tags in the City columns, as well as in the Arena column for the LA teams.

Think of rowspan and colspan tags in an HTML table as being similar to merged cells in a spreadsheet program like Microsoft Excel or Google Sheets.

You can look at the Wikipedia page prior to the table change here, and compare to the current table. Try to use your browser's inspection tools to find the rowspan tags that changed.

We want to figure out how to read these merged cells, and "unspan" them to make the table layout simpler.

Scraping the Table, Again¶

We are going to use a general approach for scraping HTML tables. This approach will work for Wikipedia and other web pages, and will automatically handle the spanning that broke our original code.

The scraping code is part of the pracpred package, which you can find on GitHub or find on from PyPI. You can install the package using the command pip install pracpred in your sports analytics environment.

As usual, we will do our data analysis using pandas. We will also use the Matplotlib Basemap package for plotting a map at the end of this notebook.

You'll notice we are going to use the warnings module from the Python standard library. This is purely cosmetic, because as you'll see toward the end of this notebook, Basemap emits some warning messages that I want to suppress.

Getting the Raw HTML Table¶

f you inspect the HTML for the Wikipedia page, you'll see that it has 5 HTML tables. We really only want the one for the NBA teams. If you inspect this table in your browser, you'll see that it has the HTML tag <table class="navbox wikitable">. We can specify this class to make sure we only get back the table we want.

Now we can call the scraping code. You can find the source code on GitHub here. The package defines two Python classes, HTMLTables and HTMLTable. The HTMLTables class is basically a wrapper on top of Requests and BeautifulSoup. This class gets and stores the HTML for one or more tables from a URL. The HTMLTable class has the code to unspan the table and convert it to a pandas DataFrame.

We got back one table, which has 33 rows and 9 columns. Notice that the table dimensions are the largest number of rows in any column, and the largest number of columns in any row. This is the key to getting the unspanning to work. We want to view the table as a grid of cells to remove the spanning structure.

Now, let's convert the HTML table to a pandas DataFrame. For this particular table, we want to have any spanned cells repeat the values when we unspan the table.

Cleaning Up the Table¶

Now let's clean up the raw information in the table.

Column Headers¶

First, notice that our generic scraping function doesn't know anything about what columns are in the table. We need to create useful column headers.

NBA Conference Information¶

Next, notice that the Eastern and Western Conference repeat across the entire row. What we want is to remove those rows, and create a new column showing the conference for each team.

City and Postal Code¶

Next, we want to split the city and the postal code into two separate columns.

To do this, we need to use pandas string-handling methods.

Arena Latitude and Longitude¶

Lastly, we need to clean up the arena latitude and longitude. This is a little tricky, since there is a lot of content packed into the Coordinates column in the DataFrame. Let's focus on one row to see what's going on.

There are 3 elements per row, with different formats for the latitude and longitude. In case you were wondering, the \ufeff appearing in the text strings are a special Unicode character. We are going to ignore the first two elements and just get the third.

We need to split this third element into latitude and longitude by the semi-colon (;) and extract the numbers. Again, we will use pandas string-handling methods, along with regular expressions. Regular expressions are a very general way to find and extract text in many computer languages, including Python. In this particular case, the regular expression just gets numbers with a decimal point, potentially starting with a negative sign.

Putting It All Together¶

Now we'll just put all these steps into one function. This function will combine all the steps, do a few more simple cleanups, and drop columns that we don't need at the end. We also want to save the final, cleaned-up results.

You can use these HTML scraping tools and techniques in your own sports analytics projects. But we're not done yet.

A Map of NBA Arenas¶

We haven't done anything useful with the NBA team data from Wikipedia. One nice thing we can do is to draw a map of NBA arena locations using the latitude and longitude information.

There's a more practical use for this arena data. Most serious strength of schedule analysis in the NBA looks at road games, rest and distance traveled. In a future post, we'll see how to incorporate this geographic information to estimate travel distance between games.

Another HTML Table to Scrape¶

We are going to use Python's Basemap package to draw a map of North America with NBA arenas. We are also going to going to fill in the U.S. states having NBA arenas using a different color for each NBA Division. Sorry Toronto and Washington fans. Any coloring for Washington, D.C. wouldn't be visible anyway, and this example won't fill in the province of Ontario.

In order to do this coloring, we need to use shapefiles. These files contain information about the shapes of various geographic features (in this case, U.S. states). We will overlay these shapes on our map filled with the correct color. The shapefiles we will use come from the U.S. Census Bureau.

In order to use these particular shapefiles, we need to be able to move between state names and postal abbreviations. Our arena data has only the postal abbreviations, and the shapefiles use state names.

There are plenty of ways to get this information (including typing it in to a Python program yourself). However, since this technical guide is about scraping HTML tables, we can use it as another opportunity to scrape a Wikipedia table.

Let's scrape Wikipedia's list of U.S. state abbreviations.

We can use the same HTML table scraping function as before. In this case, we want to use a table class of 'sortable' to get the right table.

Cleaning an Ugly Table¶

This is a relatively ugly table. Notice that many of the cells are blank. One of the reasons I wanted to use this example is to show how this general web scraping framework works, even for ugly tables.

In this case, we just want the name, status and USPS columns. We can also filter out any obsolete postal codes.

This simple table is just what we need. Now, we can build a function which will return the postal abbreviation given the state name.

Take another look at what this function does. We used an inner function, which we return. This inner function "remembers" the DataFrame which was passed in to the outer function. It will be very easy to use this simple function as a wrapper to the DataFrame in our map-drawing code below.

Drawing the Map¶

Now we can start putting the pieces of the map together.

First, we need a function to create a Basemap of the lower 48 U.S. states, along with portions of Canada and Mexico.

Reading the Shapefiles¶

Next, we read in our shapefiles for the U.S. states.

Making a Colormap for NBA Divisions¶

Next, we create a colormap with a distinct color for each of the NBA Divisions.

We want to assign a color to each state that has an NBA arena. Of course, we can't use the U.S. state shapefiles for Toronto.

Next, we need to get the information from the shapefile for each U.S. state. This is where we need to use our function to look up the postal abbreviation given a U.S. state name from the shapefile.

Putting It All Together and Drawing the Map¶

To draw the map, we need to perform the following steps:

• Create the Basemap;
• Read in the state shapefiles and assign the colors for the states that need to be filled in;
• Fill in the states with the correct colors
• Draw markers for the arenas using the latitude and longitude information
• Create text labels for the arenas using the team names
• Show the map

As I mentioned above, Basemap may emit some warnings when you run this code. The warnings I get are harmless, and I've filtered them out using the warnings module. You can run this code without the warnings module if you want, and the map should still be fine.

This simple example only scratches the surface of what you can do with geographical data in Python.