Data is central to GIS, and the tabular arrangement of data is very common. Accordingly, Python provides a large number of ways to read in tabular data. These vary depending on how the data are stored, where they are located, etc. To help keep things as simple as possible, the 'pandas' Python library provides an operator, read_csv()
that allows you to access data files in tabular format on your computer as well as data stored in online repositories, or one that a course instructor might set up for his or her students.
If you successfully ran the install script, then you already have 'pandas' installed. Now, you simply need to import pandas
in order to to use read_csv()
, as well as a variety of other 'pandas' operators that you will encounter later (it is also usually a good idea to import numpy as np
at the same time that we import pandas as pd
).
import pandas as pd
import numpy as np
You need do this only once in each session of Python, and on systems such as IPython, the library will sometimes be reloaded automatically (if you get an error message, it’s likely that the 'pandas' library has not been installed on your system.)
Reading in a data table with read_csv()
is simply a matter of knowing the name (and location) of the data set. For instance, one data table I use in my statistics classes is "swim100m.csv"
. To read in this data table and create an object in Python that contains the data, use a command like this:
swim = pd.read_csv("http://www.mosaic-web.org/go/datasets/swim100m.csv")
The csv part of the name in "swim100m.csv"
indicates that the file has been stored in a particular data format, comma-separated values that is handled by spreadsheet software as well as many other kinds of software. The part of this command that requires creativity is choosing a name for the Python object that will hold the data. In the above command it is called swim
, but you might prefer another name (e.g., s
or sdata
or even ralph
). Of course, it's sensible to choose names that are short, easy to type and remember, and remind you what the contents of the object are about.
The type of Python object created by read_csv()
is called a data frame and is essentially a tabular layout. To illustrate, here are the first several cases of the swim
data frame created by the previous use of read_csv()
:
swim.head()
Note that the head()
function, one of several functions built-into 'pandas' data frames, is a function of the Python object (data frame) itself; not from the main 'pandas' library.
Data frames, like tabular data generally, involve variables and cases. In 'pandas' data frames, each of the variables is given a name. You can refer to the variable by name in a couple of different ways. To see the variable names in a data frame, something you might want to do to remind yourself of how names a spelled and capitalized, use the columns
attribute of the data frame object:
swim.columns
Note that we have not used brackets ()
in the above command. This is because columns
is not a function; it is an attribute of the data frame. Attributes add 'extra' information (or metadata) to objects in the form of additional Python objects. In this case, the attributes describe the names (and data types) of the columns. Another way to get quick information about the variables in a data frame is with describe()
:
swim.describe()
This provides a numerical summary of each of the variables contained in the data frame. To keep things simple, the output from describe()
is itself a data frame.
There are lots of different functions and attributes available for data frames (and any other Python objects). For instance, to see how many cases and variables there are in a data frame, you can use the shape
attribute:
swim.shape
Perhaps the most common operation on a data frame is to refer to the values in a single variable. The two ways you will most commonly use involve referring to a variable by string-quoted name (swim["year"]
) and as an attribute of a data frame without quotes (swim.year
).
Most of the statistical/mathematical functions you will encounter in this tutorial are designed to work with data frames and allow you to refer directly to variables within a data frame. For instance:
swim.year.mean()
swim["year"].min()
It is also possible to combine 'numpy' operators with 'pandas' variables:
np.min(swim["year"])
np.min(swim.year)
The swim
portion of the above commands tells Python which data frame we want to operate on. Leaving off that argument leads to an error:
year.min()
The advantage of referring to variables by name becomes evident when you construct statements that involve more than one variable within a data frame. For instance, here's a calculation of the mean year, separately for (grouping by) the different sexes:
swim.groupby('sex')['year'].mean()
Both the mean()
and min()
functions have been arranged by the 'pandas' library to look in the data frame when interpreting variables, but not all Python functions are designed this way. For instance:
swim.year.sqrt()
When you encounter a function that isn't supported by data frames, you can use 'numpy' functions and the special apply
function built-into data frames (note that the func
argument is optional):
swim.year.apply(func=np.sqrt).head() # There are 62 cases in total
Alternatively, since columns are basically just arrays, we can use built-in numpy functions directly on the columns:
np.sqrt(swim.year).head() # Again, there are 62 cases in total
Sometimes you will compute a new quantity from the existing variables and want to treat this as a new variable. Adding a new variable to a data frame can be done similarly to accessing a variable. For instance, here is how to create a new variable in swim
that holds the time
converted from seconds to units of minutes:
swim['minutes'] = swim.time/60. # or swim['time']/60.
By default, columns get inserted at the end. The insert
function is available to insert at a particular location in the columns.
swim.insert(1, 'mins', swim.time/60.)
You could also, if you want, redefine an existing variable, for instance:
swim['time'] = swim.time/60.
As always, we can take a quick look at the results of our operations by using the head()
fuction of our data frame:
swim.head()
Such assignment operations do not change the original file from which the data were read, only the data frame in the current session of Python. This is an advantage, since it means that your data in the data file stay in their original state and therefore won’t be corrupted by operations made during analysis.