Elements of Data Science

by Allen Downey

MIT License


In the previous notebook we used a for loop to read a file and count the words. In this notebook, you'll learn about a new type called a "dictionary", and we will use it to count the number of unique words and the number of times each one appears.

Along the way, you will also see how to use an index to select an element from a sequence (tuple, list, or array). And you will learn a little about Unicode, which is used to represent letters, numbers, and punctuation for almost every language in the world.


Suppose you have a variable named t that refers to a list or tuple. You can select an element using the bracket operator, [].

For example, here's a tuple of strings:

In [1]:
t = 'zero', 'one', 'two'

To select the first element, we put 0 in brackets:

In [2]:

To select the second element, we put 1 in brackets:

In [3]:

To select the third element, we put 2 in brackets:

In [4]:

The number in brackets is called an "index" because it indicates which element we want.

Tuples and lists use zero-based numbering; that is, the index of the first element is 0. Some other programming languages use one-based numbering. There are pros and cons of both systems.

The index in brackets can also be a variable:

In [5]:
i = 1

Or an expression with variables, values, and operators:

In [6]:

But if the index goes past the end of the list or tuple, you get an error:

In [7]:

The index has to be an integer; if it is any other type, you get an error.

In [8]:
In [9]:

Exercise: You can use negative integers as indices. Try using -1 and -2 as indices, and see if you can figure out what they do.

In [10]:
# Solution goes here
In [11]:
# Solution goes here


A dictionary is similar to a tuple or list, but in a dictionary, the index can be almost any type, not just an integer.

We can create an empty dictionary like this:

In [12]:
d = {}

Then we can add elements like this:

In [13]:
d['one'] = 1
d['two'] = 2

If you display the dictionary, it shows each index and the corresponding value.

In [14]:

Instead of creating an empty dictionary and then adding elements, you can create a dictionary and specify the elements at the same time:

In [15]:
d = {'one': 1, 'two': 2, 'three': 3}

When we are talking about dictionaries, an index is usually called a "key". In this example, the keys are strings and the corresponding values are integers.

A dictionary is also called a "map", because it represents a mapping, in the sense of a correspondence, between keys and values. So we might say that this dictionary "maps from" English number names to the corresponding integers.

You can use the bracket operator to select an element from a dictionary, like this:

In [16]:

But don't forget the quotation marks. If you write something like this:

In [17]:

Python looks for a variable named two and doesn't find one.

To check whether a particular key is in a dictionary, you can use the special word in:

In [18]:
'one' in d
In [19]:
'zero' in d

The word in is actually an operator in Python, so you can't use it as a variable name:

In [20]:
in = 5

If a key is already in a dictionary, adding it again has no effect:

In [22]:
In [23]:
d['one'] = 1

But you can also change the value associated with a key:

In [25]:
d['one'] = 100

You can loop through the keys in a dictionary like this:

In [26]:
for key in d:

If you want the keys and the values, a simple way to get them is to loop through the keys and look up the values:

In [27]:
for key in d:
    print(key, d[key])

Or you can loop through both at the same time, like this:

In [28]:
for key, value in d.items():
    print(key, value)

The items method loops through the key-value pairs in the dictionary; each time through the loop, they are assigned to key and value.

Exercise: Make a dictionary with the numbers 1, 2, and 3 as keys and strings as values. The strings should be the numbers "one", "two", and "three" in any language you know.

Write a loop that prints just the values from the dictionary.

In [31]:
# Solution goes here
In [32]:
# Solution goes here

Unique words

In the previous notebook we downloaded War and Peace from Project Gutenberg and counted the number of lines and words.

Now that we have dictionaries, we can also count the number of unique words and the number of times each one appears.

First, let's download the book again. If you run the following cell, it checks to see whether you already have a file named 2600-0.txt; if not, it uses wget to copy the file from Project Gutenberg to your computer.

In [33]:
import os

if not os.path.exists('2600-0.txt'):
    !wget https://www.gutenberg.org/files/2600/2600-0.txt

Now we can read the file and count the words as we did in the previous notebook:

In [34]:
fp = open('2600-0.txt')
count = 0
for line in fp:
    count += len(line.split())

To count the number of unique words, I'll loop through the words in each line and add as keys in a dictionary:

In [35]:
fp = open('2600-0.txt')
unique_words = {}
for line in fp:
    for word in line.split():
        unique_words[word] = 1

This is the first example we've seen with one loop inside another.

  • The outer loop runs through the lines in the file.

  • The inner loops runs through the words in each line.

Each time through the inner loop, we add a word as a key in the dictionary, with the value 1. If the same word appears more than once, it gets added to the dictionary again, which has no effect. So the dictionary contains only one copy of each unique word in the file.

At the end of the loop, we can display the first 10 keys:

In [36]:
i = 0
for key in unique_words:
    i += 1
    if i == 10:

So far, it looks like all the words in the file, in order.

But each word only appears once, so the number of keys is the number of unique words:

In [37]:

It looks like there are about 42,000 different words in the book, which is substantially less than the total number of words, about 560,000.

But that's not quite right, because we have not taken into account capitalization and punctuation.

Exercise: Before we deal with those problems, let's practice with "nested loops", that is, one loop inside another.

Suppose you have a list of words, like this:

In [4]:
line = ['War', 'and', 'Peace']

Write a nested loop that iterates through each word in the list, and each letter in each word, and prints the letters on separate lines.

In [5]:
# Solution goes here


When we count unique words, we probably want to treat The and the as the same word. We can do that by converting all words to lower case, using the lower function:

In [38]:
word = 'The'
In [39]:
word = 'the'

lower creates a new string; it does not modify the original string.

In [40]:
word = 'THE'
In [41]:

However, you can assign the new string back to the existing variable, like this:

In [42]:
word = 'THE'
word = word.lower()

Now if we can display the new value of word, we get the lowercase version:

In [43]:

Exercise: Modify the previous loop so it makes a lowercase version of each word before adding it to the dictionary. How many unique words are there, if we ignore the difference between uppercase and lowercase?

In [44]:
# Solution goes here


To remove punctuation from the words, we can use strip, which removes specified characters from the beginning and end of a string. Here's an example:

In [45]:
word = 'abracadabra'

In this example, strip removes all instances of a and b from the beginning and end of the word, but not from the middle.

But note that it makes a new word; it doesn't modify the original:

In [46]:

To remove punctuation, we can use the string library, which provides a variable named punctuation.

In [47]:
import string


string.punctuation contains the most common punctuation marks, but as we'll see, not all of them.

Nevertheless, we can use it to handle most cases. Here's an example:

In [48]:
line = "It's not given to people to judge what's right or wrong."

for word in line.split():
    word = word.strip(string.punctuation)

Notice that strip does not remove the apostrophe from the middle of don't, which is probably what we want.

To see how well it works, I'll select an arbitrary line from the file:

In [49]:
fp = open('2600-0.txt')
count = 0
for line in fp:
    if count == 1000:
    count += 1

And try to remove punctuation from the words:

In [50]:
for word in line.split():
    word = word.strip(string.punctuation)

It words pretty well, but the last word is a problem because it ends with a quotation mark that is not in string.punctuation. To fix this problem, we'll use the following loop, which

  1. Reads the file and builds a dictionary that contains all punctuation marks that appear in the book, then

  2. It uses the join function to concatenate the keys of the dictionary in a single string.

You don't have to understand everything about how it works, but you should read it and see how much you can figure out. You can read the documentation of the unicodedata library here.

In [52]:
import unicodedata

fp = open('2600-0.txt')
punc_marks = {}
for line in fp:
    for x in line:
        category = unicodedata.category(x)
        if category[0] == 'P':
            punc_marks[x] = 1
all_punctuation = ''.join(punc_marks)

Exercise: Modify the word-counting loop from the previous section to convert words to lower case and strip punctuation before adding them to the dictionary. Now how many unique words are there?

Optional: You might want to skip over the frontmatter and start with the text of Chapter 1, and skip over the license at the end, as we did in the previous notebook.

In [55]:
# Solution goes here

Word frequencies

In the previous section we counted the number of unique words, but we might also want to know how often each word appears. Then we can find the most common and least common words in the book.

To count the frequency of each word, we'll make a dictionary that maps from each word to the number of times it appears.

Here's an example that loops through a string and counts the number of times each letter appears.

In [56]:
line = 'If everyone fought for their own convictions there would be no war.'

letter_counts = {}
for x in line:
    if x in letter_counts:
        letter_counts[x] += 1
        letter_counts[x] = 1

The if statement in the previous example uses a feature we have not seen before, an else clause.

Here's how it works.

  1. First, it checks whether the letter is already a key in the dictionary, letter_counts.

  2. If so, it runs the first statement, letter_counts[x] += 1, which increments the value associated with the letter, x.

  3. Otherwise, it runs the second statement, letter_counts[x] = 1, which adds x as a new key, with the value 1 indicating that we have seen the new letter once.

The result is a dictionary that maps from each letter to the number of times it appears.

To get the most common letters, we can use a Counter, which is similar to a dictionary. To use it, we have to import a library called collections:

In [58]:
import collections

Then we use collections.Counter as a function to convert the dictionary to a Counter value:

In [59]:
counter = collections.Counter(letter_counts)

Counter provides a function called most_common we can use to get the most common characters:

In [60]:

The result is a list of tuples, where each tuple contains a character and an integer.

The most common character is a space, followed by o and e.

Exercise: Modify the loop from the previous exercise to count the frequency of the words in War and Peace; then print the 20 most common words and the number of times each one appears.

In [61]:
# Solution goes here

Exercise: You can run most_common with no value in parentheses, like this:

word_freq_pairs = counter.most_common()

The result is a list of tuples, with one tuple for every unique word in the book. Assign the result to a variable so it doesn't get displayed. Then answer the following questions:

  1. How many times does the #1 ranked word appear (that is, the first element of the list)?

  2. How many times does the #10 ranked word appear?

  3. How many times does the #100 ranked word appear?

  4. How many times does the #1000 ranked word appear?

  5. How many times does the #10000 ranked word appear?

Do you see a pattern in the results? We will explore this pattern more in the next notebook.

In [62]:
# Solution goes here
In [63]:
# Solution goes here
In [64]:
# Solution goes here
In [65]:
# Solution goes here
In [66]:
# Solution goes here

Exercise: Write a loop that counts how many words appear 200 times. What are they? How many words appear 100 times, 50 times, and 20 times?

Optional: If you know how to define a function, write a function that takes a Counter and a frequency as arguments, prints all words with that frequency, and returns the number of words with that frequency.

In [72]:
# Solution goes here
In [73]:
# Solution goes here
In [69]:
# Solution goes here
In [70]:
# Solution goes here
In [71]:
# Solution goes here
In [ ]: