Loops and files

Elements of Data Science

by Allen Downey

MIT License

Goals

This notebook presents loops, which are used to express repeated computation, and files, which are used to store data. As an example, we will download the famous book War and Peace from Project Gutenberg and write a loop that reads the book and counts the words.

In the next notebook, we'll extend this example to count the number of unique words and the number of times each word appears.

This example is meant to introduce some computational tools you will need; it is also an introduction to working with textual data.

Loops

One of the most important elements of computation is repetition, and the most common way to express repetition is a for loop.

As a simple example, suppose we want to display the elements of a tuple. Here's a tuple of three integers:

In [1]:
t = 1, 2, 3

And here's a for loop that prints the elements.

In [2]:
for x in t:
    print(x)

The first line of the loop is a "header" that specifies the tuple, t, and a variable name x. The tuple already exists, but x does not; the loop will create it. The header has to end with a colon, :.

Inside the loop is a print statement, which displays the value of x.

So here's what happens:

  1. When the loop starts, it gets the first element of t, which is 1, and assigns it to x. It executes the print statement, which displays the value 1.

  2. Then it gets the second element of t, which is 2, and displays it.

  3. Then it gets the third element of t, which is 3, and displays it.

After printing the last element of the tuple, the loop ends.

We can also loop through the letters in a string:

In [3]:
word = 'Data'

for letter in word:
    print(letter)

When the loop begins, word already exists, but letter does not. Again, the loop creates letter and assign values to it.

The variable created by the loop is called the "loop variable". You can give it any name you like; in this example, I chose letter to remind me what kind of value it contains.

After the loop ends, the loop variable contains the last value.

In [4]:
letter

Exercise: Create a list with four elements, of any type, and write a for loop that prints the elements. Call the list sequence and the loop variable element.

In [5]:
# Solution goes here

You might wonder why I didn't call the list list. I avoided it because Python has a function named list that makes new lists:

In [6]:
list('string')

If you create a variable named list, you can't use the function any more.

In [7]:
list = [1, 2, 3]
list('string')

In this example, Python tries to use your list as a function, which doesn't work because you can't "call" a list like a function.

Exercise: After running that cell, we won't be able to use the list function until we restart the Python interpreter. So this is a good time to find out how.

  1. "Comment out" the previous cell; that is, add the comment symbol, #, to the beginning of the first line so it will be treated as a comment.
# list = [1, 2, 3]
list('string')
  1. Now restart the Python interpreter. If you are running this notebook on Colab, click the Runtime menu and select Restart all runtimes...; if you are running in another Jupyter environment, click on the Kernel menu and select Restart.

When you restart the interpreter, your variables disappear, so you should start again at the beginning of the notebook and run the cells again, in order.

Looping and counting

War and Peace is a famously long book; let's see how long it is. To count the words in a book, we need to

  1. Loop through a book and

  2. Count.

We'll start with counting.

We've already seen that you can create a variable and give it a value, like this:

In [8]:
count = 0
count

If you assign a different value to the same variable, the new value replaces the old one.

In [9]:
count = 1
count

You can increase the value of a variable by reading the old value, adding 1, and assigning the result back to the original variable.

In [10]:
count = count + 1
count

Increasing the value of a variable is called an "increment"; decreasing the value is called a "decrement". It is such a common operation that there is a special operator for it.

In [11]:
count += 1
count

In this example, the += operator reads the value of count, adds 1, and assigns the result back to count.

Python also provides -= and other update operators like *= and /=.

Exercise: The following is a number trick from Learn With Math Games

"Finding Someone’s Age

  • Ask the person to multiply the first number of his or her age by 5.

  • Tell them to add 3.

  • Now tell them to double this figure.

  • Finally, have the person add the second number of his or her age to the figure and have them tell you their answer.

  • Deduct 6 and you will have their age."

Test this algorithm using your age. Use a single variable and update it using += and other update operators.

In [12]:
# Solution goes here

Files

Now we can use a loop to read lines from a file and use a variable to count them.

We can download War and Peace from Project Gutenberg, which is a repository of free books.

The following cell imports os, which stands for "operating system"; it is a Python library that provides functions for interacting with files and other parts of the operating system.

In [2]:
import os

If you run the following cell, it checks to see whether you already have a file named 2600-0.txt, which is the name of the file that contains the text of War and Peace.

If not, it uses wget to copy the file from Project Gutenberg to your computer.

In [3]:
if not os.path.exists('2600-0.txt'):
    !wget https://www.gutenberg.org/files/2600/2600-0.txt

If you are running this notebook on Colab, it will copy the file to a "virtual file system" on Colab, which means it will disappear if you leave the notebook idle for a while. In that case, you can download it again later.

In order to read the contents of the file, you have to "open" it, which you can do with the open function.

In [15]:
fp = open('2600-0.txt')
fp

The result is a TextIOWrapper, which is a type of "file pointer".

The file pointer contains the name of the file, the mode (which is r for "reading") and the encoding (which is UTF for "Unicode Transformation Format").

A file pointer is like a bookmark; it keeps track of which parts of the file you have read.

If you use a file pointer in a for loop, it loops through the lines in the file.

So we can count the number of lines like this:

In [16]:
fp = open('2600-0.txt')
count = 0
for line in fp:
    count += 1

And then display the result.

In [17]:
count

There are about 66,000 lines in this file.

if statements

We've already see comparison operators, like > and <, which compare values and produce a Boolean result, True or False.

For example, we can compare the final value of count to a number:

In [19]:
count > 60000

We can use a comparison operator in an if statement to check for a condition and take action accordingly.

In [20]:
if count > 60000:
    print('Long book!')

The first line of the if statement specifies the condition we're checking for. Like the header of a for statement, the first line of an if statement has to end with a colon.

If the condition is true, the indented statement runs; otherwise, it doesn't.

In the previous example, the condition is true, so the print statement runs.

In the following example, the condition is false, so the print statement doesn't run.

In [21]:
if count < 1000:
    print('Novella!')

We can put a print statement inside a for loop. In this example, we only print a line from the book when count is 1.

The other lines are read, but not displayed.

In [23]:
fp = open('2600-0.txt')
count = 0
for line in fp:
    if count == 1:
        print(line)
    count += 1

Notice the indentation in this example:

  • Statements inside the for loop are indented.

  • The statement inside the if statement is indented.

  • The statement count += 1 is "outdented" from the previous line, so it ends the if statement. But it is still inside the for loop.

It is legal in Python to use spaces or tabs for indentation, but the most common convention is to use four spaces, never tabs. That's what I'll do in my code and I strongly suggest you follow the convention.

break statement

If we display the final value of count, we see that the loop reads the entire file, but only prints one line:

In [24]:
count

We can avoid reading the whole file by using a break statement, like this:

In [25]:
fp = open('2600-0.txt')
count = 0
for line in fp:
    if count == 1:
        print(line)
        break
    count += 1

The break statement ends the loop immediately, skipping the rest of the file. We can confirm that by checking the last value of count:

In [26]:
count

Exercise: Write a loop that prints the first 5 lines of the file and then breaks out of the loop.

In [27]:
# Solution goes here

Whitespace

If we run the loop again and display the final value of line, we see that there are two extra characters at the end:

In [28]:
fp = open('2600-0.txt')
count = 0
for line in fp:
    if count == 1:
        print(line)
        break
    count += 1
In [29]:
line

The sequence \n represents a single character, called a "newline", that puts vertical space between lines.

If we use a print statement to display line, we don't see the special sequence, but we do see extra space after the line.

In [30]:
print(line)

In other strings, you might see the sequence \t, which represents a "tab" character.

When you print a tab character, it adds enough space to make the next character appear in a column that is a multiple of 8.

In [31]:
print('01234567' * 6)
print('a\tbc\tdef\tghij\tklmno\tpqrstu')

In the previous cell, the first line uses a feature we have not seen before: if you multiply a string by an integer, Python repeats the string the given number of times.

Newline characters, tabs, and spaces are called "whitespace" because when they are printed they leave white space on the page.

Counting words

So far we've managed to count the lines in a file, but each line contains several words.

In [32]:
line

To split a line into words, we can use a function called split that returns a list of words.

To be more precise, split doesn't actually know what a word is; it just splits the line wherever there's a space or other whitespace character.

In [33]:
line.split()

Notice that the syntax for split is different from other functions we have seen. Normally when we call a function, we name the function and provide values in parentheses. So you might have expected to write split(line). Sadly, that doesn't work:

In [34]:
split(line)

The problem is that the split function belongs to the string line; in a sense, the function is attached to the string, so we can only refer to it using the string and the dot operator (the period between line and split).

For historical reasons I won't try to explain, functions like this are called "methods".

Anyway, now that we can split a line into a list of words, we can use len to get the number of words in each list, and increment count accordingly.

In [35]:
fp = open('2600-0.txt')
count = 0
for line in fp:
    count += len(line.split())
In [36]:
count

By this count, there are more than half a million words in War and Peace.

Actually, there aren't quite that many, because the file we got from Project Gutenberg has some introductory text and a table of contents before the text. And it has some license information at the end.

To skip this "frontmatter", we can use one loop to read lines until we get to CHAPTER I, and then a second loop to count the words in the remaining lines.

The file pointer, fp, keeps track of where it is in the file, so the second loop picks up where the first loop leaves off.

In the second loop, we check for the end of the book and stop, so we ignore the "endmatter" at the end of the file.

In [37]:
first_line = "CHAPTER I\n"
last_line = "End of the Project Gutenberg EBook of War and Peace, by Leo Tolstoy\n"

fp = open('2600-0.txt')
for line in fp:
    if line == first_line:
        break

count = 0
for line in fp:
    if line == last_line:
        print(line)
        break
    count += len(line.split())
In [38]:
count

Two things to notice about this program:

  • When we compare two values to see if they are equal, we use the == operator, not to be confused with =, which is the assignment operator.

  • The string we compare line to has a newline at the end. If we leave that out, it doesn't work.

Exercise:

  1. In the previous program, replace == with = and see what happens. This is a common error, so it is good to see what the error message looks like.

  2. Correct the previous error, then remove the newline character after CHAPTER I, and see what happens.

The first error is a "syntax error", which means that the program violates the rules of Python. If your program has a syntax error, the Python interpreter prints an error message, and the program never runs.

The second error is a "logic error", which means that there is something wrong with the logic of the program. The syntax is legal, and the program runs, but it doesn't do what we wanted. Logic errors can be hard to find because we don't get any error messages.

If you have a logic error, here are two strategies for debugging:

  1. Add print statements so the program displays additional information while it runs.

  2. Simplify the program until it does what you expect, and then gradually add more code, testing as you go.

In [ ]: