Scraping NBA team information from Wikipedia¶

Update (January 20, 2018): Somebody updated the Wikipedia page for NBA teams in mid-December. This person changed the table layout to use rowspan tags for the cities of both the New York and the Los Angeles teams. This is similar to merging cells in a spreadsheet, and breaks the scraping example from the original notebook. To fix this, I changed the URL in this revised notebook to point to the historical version of the Wikipedia page, prior to the table change. In a separate post, we will look at a more general approach to handle rowspan and colspan tags in HTML tables.

We are going to scrape some basic NBA team information from Wikipedia. In particular, we are going to scrape the table listing each team's division, city, arena information, the year the team joined the NBA, and the current head coach.

Before we get started, it's worth pointing out that for a table this small, you don't really need to build an automated web scraper. You could just copy the data to a spreadsheet and save it to your computer for use in later analysis. Sometimes manual labor is the right solution. In this case, the small data size and relatively simple structure make this an ideal example for learning the basics of web scraping.

Let's get started.

Importing Libraries¶

It's good Python practice to put all your import statements at the beginning of your code. That way, if you or somebody else needs to figure out which packages are used in your code, they're easy to find.

First come some imports from the Python standard library.

We are going to use IPython's rich display functionality to embed screenshots in this notebook.

We are going to use Requests and BeautifulSoup for web scraping.

We are going to do our data analysis using pandas, which is built on top of the numpy package. These are among the most popular and central libraries in the Python ecosystem. You'll see them imported in virtually every code example on this site.

Before we start scraping, we are going to define some directory paths to allow us to access and store data and images. We are using the pathlib module from the Python standard library in order to create paths that should work on all platforms, whether you are running this on a Mac, Windows or Linux machine.

We'll build a list of screenshots here in advance.

Getting the HTML¶

In order to get the Wikipedia page, Requests will need the web address, which is more formally referred to as a Uniform Resource Locator, or URL.

Now let's ask Requests to to issue a GET request for the page.

The parameters after the URL tell Requests that we don't want to allow the web site (Wikipedia in this case) to redirect us to another web page, and to not wait more than 10 seconds before returning an error if the web site is not responding. It's good practice to always include these parameters when using Requests.

If you run this code on your computer, this will print out status code 200 unless a problem occured (maybe the Wikipedia website is down or the URL has changed since I wrote this).

All of the text, images, links and other content from the web page now lurk inside that object named response.

Let's see what sort of information Wikipedia returned.

Now let's see how much text (measured by number of characters) is on the web page.

That's a lot of content. If you are trying this out on your computer and want to scroll through the whole thing, feel free to print(response.content). If you do, you will see a large amount of text which probably doesn't make a lot of sense.

We are going to use BeautifulSoup to make some sense of the HTML. We are going to use the default HTML parser, which is appropriately enough named html.parser.

The soup object now contains all of the original HTML, but structured in a way that will make it much easier to extract the information we want.

The <title> and </title> strings are examples of HTML tags, which tell your browser about the structure and components of a document. An HTML element typically consists of a start tag (<title> in this example) and an end tag (</title> in this example), with the content in between the tags.

The document structure is created by nesting elements inside other elements. We'll see an example of document structure in scraping the Wikipedia page.

To access the content without the tags, use the text attribute.

Note that this text is what the browser displays for the window or tab containing the Wikipedia page.

Examining the HTML Tags¶

We can use BeautifulSoup's find_all() method to extract the tags we want. First, let's get all of the HTML tags from inside the body of the page.

There are over 5300 HTML tags representing the body of this page. Let's use the Counter() class from Python's standard library to count up how many of each tag type are present.

There are 35 distinct tag types. Let's look at the most common ones.

So roughly 97% of the tags on this Wikipedia page are from the above 15 tag types. This represents some progress, but it's still not obvious which elements are important to us and how we're supposed to access them.

Fortunately, it turns out that there is a relatively simple trick to immediately find the interesting elements on an HTML-rendered web page.

Use Your Browser's Developer Tools¶

The trick is to use your browser's developer tools to inspect the web page's Document Object Model, or DOM. Most major browsers have tools that allow you to click on elements in the web page and jump directly to the associated HTML, to see what tags correspond to particular elements. Here's a screenshot of how it looks in my Chrome browser, as I inspect the table of NBA teams.

See the instructions for opening and using Chrome's developer tools. In Chrome, you first select the Elements group of tools and use the "Select an element in the page to inspect it" tool (the left-most symbol, with a square and arrow, as shown in the screen shot above).

The easier way to get the same result in Chrome is to hover over an element, right-click (or control-click on a Mac), and choose Inspect from the menu that pops up.

Other popular browsers have their own versions of these tools which work similarly. You can read about Safari's Web Inspector, Firefox's Inspector and Microsoft's DOM Explorer.

From the above screen shot, it's easy to see that the table we care about is inside a <table> tag. We saw from our Counter analysis above that there are 38 <table> tags in the page body, so which one in particular do we want?

Extracting the Table¶

HTML elements can have attributes which provide additional information, such as class, id or style. According to the element inspector tool, we want the particular table having class="navbox wikitable". We can specify attributes in BeautifulSoup's find_all() and find() methods to narrow down the elements we extract.

Notice that the class_ parameter has a trailing underscore, so it's not confused with the Python reserved word class.

Also notice that find_all() returns a list of tables, which in this case happens to only have one item. From this, we know that there is only one table on this page with class=navbox wikitable.

Let's continue to use the element inspector tool to see how we can work our way through the table to extract the useful information.

Each row in the table is bracketed by a <tr> tag. The first row contains the column headers. Let's extract all the rows from the table into a new variable and work with that.

It's a good sign that there are 33 rows in our results. There are 30 NBA teams, each of which take up one row in the table. There is one extra row for the column headers, and two rows for each of the conferences, making up 33 total rows. That means the rows variable has all the information we need.

Getting the Column Headers¶

Let's look at the first row and confirm it has the column headers.

This confirms that the elements with the <th> tags are the headers. We can use find_all() to extract the text and save the headers in a list for later use.

Write Python Functions to Store Knowledge¶

We've come a long way in figuring out how to scrape this table. It's time to start putting this knowledge into Python functions. This will have a lot of benefits.

First, coding all the steps in a function forces us to think clearly, since we are turning over control of the process to the computer.

Second, we can reuse our function in the future. If a team builds a new arena or changes the coach, we can run our function again. It's much easier to run a function than to remember all the steps we took. The important thing is that writing the function documents our methods.

This leads to the third benefit, which is that other people can more easily replicate our analysis if we share the function with them.

Although we're scraping a web page in this example, the general concept of using functions to store knowledge will apply to many aspects in your sports analytics projects. Each time you figure something out, take the time to record it properly in a function. You'll thank yourself every time you decide to rerun or enhance an analysis, or share it with somebody else.

Let's start by writing a small function to scrape the column headers and return them.

Let's test the function.

This is a very easy function. However, by the time we get to the end of this post, I hope you'll see the value in creating these small, understandable functions as you go.

Getting the Conference and Division Information¶

Now, let's use the element inspector tool to figure out how the get the NBA conference and division information.

Let's look at the second row in the table.

This row is relatively simple, even though it has a few elements. It has a row header with tag <th> that spans multiple columns to stretch across the entire table. That is the meaning of the colspan="9" attribute. This row also has a link (the <a> element) pointing to the Wikipedia page for the Eastern Conference. The text and color of the row is specified in the <span> element.

Let's write a small function to scrape the conference name, clean it up a bit, and return it.

This function is slightly tricky, since it will return Python's special value None if the row doesn't have a <th> tag with a colspan attribute. This means we can be sure this function will only return a value if the row does in fact contain a conference name. Let's test that out on some different rows.

The function doesn't return anything except for the two rows which contain the conference name, which is what we want.

The division information is a bit more complicated. As you can see from the screen shot below showing the Atlantic division, the division is stored in a row header (<th> tag). This row header has a rowspan="5" attribute, since each division has 5 NBA teams.

Notice that the Atlantic division header only appears in the same row as the Boston Celtics (the first team listed in the division), even though it is displayed as covering all 5 rows because of the rowspan="5" attribute. This will be important later.

Let's focus first on the row header for the third row (the Celtics).

We can get the division name easily using the text attribute.

However, if we try this for the subsequent row (the Nets), we don't get anything back.

This shows that the division information is only available on the row containing the first team in that division. We'll need to incorporate this fact into our scraping strategy for the whole table later.

Let's write a small function to scrape the division name for a row, if it's there, and return it. Similar to the conference function above, if the row doesn't have a division name, the function will return None.

Let's test out our function.

Understanding the Team Information¶

Let's move on for the moment to examine the team data in each row. From the screen shot, it's clear that the table cells correspond to <td> elements.

Let's figure out how to process this structure for each team. Let's start by checking how many <td> tags there are in each row.

There are 3 rows with zero <td> tags, which correspond to the column header row and the two conference name rows. Those rows have no team data. Of the 30 team rows, there are 21 rows with 7 <td> tags, and 9 rows with 8 <td> tags.

The first thing to notice is that the team rows only have either 7 or 8 data fields, whereas there are 9 column headers. One of the columns is the division, which as we discussed above is actually a row header (<tr> tag).

The other reason for the discrepancy in column numbers is in in the "year founded" and "year joined" columns. Looking at the screen shot above, we see that an original BAA team such as the Celtics has one value (1946) spanning both columns, since the year joined is the same as the year founded. An ABA-NBA merger team such as the Nets has two fields with different values for year founded and year joined.

We need to be careful about the different row structures for the various teams. Even though we might not care about the year founded and year joined at this point, it's a good learning exercise to see how to scrape a table like this.

Let's focus on the Celtics for now.

We can extract all the fields into a list.

Let's look at the fields in detail.

The first field is the team name, which is easy to extract. We will also save the Wikipedia link to the team information using the href attribute of the underlying <a> tag. Saving the Wikipedia link will be useful in case we ever decide we want to scrape the detailed Wikipedia pages for each team.

Similarly, we can easily get the team's home city and postal code. Wikipedia stores two links to the city and state (or province in the case of Toronto), except in the case of Washington, D.C. We will only store the city Wikipedia link.

Next comes the arena name and Wikipedia link, which are similar to the team name.

The arena capacity requires some work, since Wikipedia stores the numbers are strings with a comma as thousands separators. We need to strip out the comma and convert to a Python int to store the capacity as a number.

The arena geographic coordinates require the most work. They are stored in a <span> tag with class="geo". We can extract the text very easily, but we have to separate the text into latitude (North/South) and longitude (East/West) and store the results as numbers with decimals (Python float).

Now we will skip to the last field, which is the coach's name. The only wrinkle in this field is that the field number will change depending upon whether the team has only a year founded, or both a year founded and a year joined. We can avoid any confusion by using Python's [-1] list index syntax to always get the last field.

The only remaining information we need to extract is year founded (and year joined, if applicable). We know that year founded is always the sixth field. Remember that lists in Python are zero-indexed, so the sixth field is fields[5]. We also see from the table that some of the years have an asterisk (*) which needs to be removed.

To get the year joined, we can use a simple trick. If we get the second-to-last field value, it will either be the year joined (for rows with 8 fields), or the year founded (for rows with 7 fields). Either way, getting the second-to-last field value is what we want to use for year joined. (For rows with only 7 fields, we want to have the same value for both year founded and year joined).

Notice how looking from right-to-left in the row made this particular scraping problem simpler. Python makes it easy to use these simpler tricks in a clear, readable way.

Processing the Team Information with a Function¶

Now it's time to put our understanding of the team information into a function we can automate and reuse. First, let's create a simple structure to hold each team information using the namedtuple class from Python's standard library. We'll put the most interesting fields first in the data structure.

Now let's write a function to bring together all the steps we followed above.

Notice that we store the conference and division values in the record with default values of None specified in the function signature. This means we can use our function, even if we don't have the conference and division values yet. Let's test our function on the Wizards.

It worked. Notice that the namedtuple object respected the ordering of the fields we set when we defined WikiNBATeamInfo type.

Now let's try the function for the Nets, which has both year founded and year joined cells in the table.

One nice thing about using the namedtuple class is that we can easily transform the data to other Python data types, such as an OrderedDict. Since OrderedDict looks like the standard Python dict for most purposes, we can use our namedtuple data with a lot of other libraries that expect a dict.

Putting It All Together¶

Now we can put together the overall web scraping process, including the conference and division information. Our function should be self-contained, so it gets the HTML again from scratch.

Here's a quick outline of the process.

1. Get the table from the web page and split into rows.
2. Get the column headers from the first row.
3. Loop through each of the remaining rows.

• If the row has a conference name, store that and use it for each team until we find a new conference name. Continue on to the next row, since we know there is no other information on a conference row.
• If the row has a division name, store that and use it for each team until we find a new division name.
• Scrape the team information (using the get_team_info() function). We have the team's conference and division, so pass them into the function.
• Store all the team information.

Now we'll see how easy Python is to read and write.

See how the Python code is almost as readable as the outline?

Notice that we didn't actually need to use the column header information in this simple example. It's useful to extract the headers in any case, to make sure that the table columns match up with the assumed field ordering in our scraping code.

Let's run the top-level function on the entire table.

It looks like it worked, since we have 30 rows for 30 teams. Let's look at the first and last items in the scraped data.

Each item is an OrderedDict, with the column headers as keys included in each item.

Create a DataFrame using pandas¶

If we want to analyze data, the best way to do it in Python is to use the pandas package. In pandas, the DataFrame is the main structure for storing and analyzing tabular data. You can think of a DataFrame as similar to spreadsheet programs such as Microsoft Excel or Google Sheets, which let you store and access information arranged in a table with named columns and identifiable rows.

We're going to use DataFrame objects and the other capabilities of pandas a lot on this site, and we will have a number of technical guides and links to external tutorials about pandas. For now, we're just going to show you how to create a DataFrame, save the contents to disk and do some simple analysis.

Fortunately, pandas makes it very easy to use our scraped data. It can figure out how to create a DataFrame based upon several possible input data structures, including a list of dict-like items, which is how our scraped data is currently store.

The shape attribute tells us the DataFrame has 30 rows and 16 columns. Let's use the head() method to see the first few rows of our DataFrame.

We can select the subset of columns we want to easily screen out the less-interesting _wiki columns.

Save the Data to a CSV File and Do Some Quick Analysis¶

Let's save the results of our work to a comma-separated value (CSV) text file. We will be able to use this text file in future analyses, rather than having to rescrape the Wikipedia page each time we want to use the team information. The great thing about CSV files is that it's very easy to share data with other people or use a variety of tools in your analysis. Spreadsheet programs such as Microsoft Excel and Google Sheets can read and write CSV files, and most programming languages make it very easy to read and write them as well.

Fortunately, pandas makes it very easy to save or load DataFrames in various formats, including CSV.

We're not going to do more than scratch the surface of what pandas can do in this post. Here's just a few quick examples of some of the things you can easily do. Let's examine arena seating capacity across the NBA.

Which teams have the smallest and largest arenas?

What's the current total seating capacity in the NBA arenas?

How does seating capacity vary across the NBA divisions?

We're done for now. In future posts, we'll see how easy it is to read the CSV file back into a DataFrame to continue our analysis.