Pandas
Pandas is a primary data analysis library in Python. It offers a number of operations to aid in data exploration, cleaning and transformation, making it one of the most popular data science tools. To name a few examples of these operations, Pandas enables various methods to handle missing data and data pivoting, easy data sorting and description capabilities, fast generation of data plots, and Boolean indexing for fast image processing and other masking operations.
Some of the key features of Pandas are:
Pandas also builds upon numpy and other Python packages to provide easy-to-use data structures and data manipulation functions with integrated indexing.
Additional Recommended Resources:
Introduction to Pandas Data Structures
Series in Pandas
Pandas Series are one-dimensional labeled arrays. Since they act like ndarrays, they are valid arguments to most Numpy methods. Series support many data types, including integers, strings, floating point numbers, Python objects, etc. Their axis labels are collectively referred to as the index, and we can get and set values using these index labels. You can think of a Series as a flexible dictionary-like object.
Let's look at some code examples with the Pandas Series.
# import the Pandas package
import pandas as pd
# create a Series called sr
# syntax is: pd.Series([data elements], [index elements])
# note that the elements in the data and index sets do not have to be the same
sr = pd.Series([10, 'foo', 30, 90.4], ['peach', 'plum', 'dog', 'band'])
# view the Series
sr
peach 10 plum foo dog 30 band 90.4 dtype: object
# view the indices
sr.index
Index(['peach', 'plum', 'dog', 'band'], dtype='object')
# access the data at an index
sr['plum']
'foo'
# OR
sr.loc['plum']
'foo'
# access the data at multiple indices
sr[['peach', 'band']]
peach 10 band 90.4 dtype: object
# OR
sr.loc[['peach', 'band']]
peach 10 band 90.4 dtype: object
You can see that the data is represented so that you can access it like a list with numeric indices (list[x]) or more like a dictionary (dic['key']).
# access a data element by position in the list
sr[2]
30
# OR
sr.iloc[2]
30
# access multiple data elements by positions in the list
sr[[0, 1, 2]]
peach 10 plum foo dog 30 dtype: object
# OR
sr.iloc[[1, 2, 3]]
plum foo dog 30 band 90.4 dtype: object
# is the index 'peach' in the Series?
'peach' in sr
True
We can also use basic Python operations like multiplication on a Series. In the code below, we multiply the whole Series by 2. Note that this operation is performed on all data types, even strings, where the string is doubled.
sr *2 #Notice foo turns to foofoo when multiplied by 2
peach 20 plum foofoo dog 60 band 180.8 dtype: object
sr # Because we did no set sr = sr*2, sr doesn't change values
peach 10 plum foo dog 30 band 90.4 dtype: object
We can square the numerical index values in a Series. If we tried to square an index that's not a numeric data type, however, we would get an error.
sr[['peach', 'band']] ** 2 # you cannot square a string, so if you include 'foo' you will get an error
peach 100 band 8172.16 dtype: object
DataFrames in Pandas
Pandas DataFrames are flexible 2-dimensional labeled data structures. They also support heterogeneous data and have labeled axes for rows and columns. We can think of a DataFrame as a container for Series objects, where each row is a Series.
Below we give some examples of things you can do with the Pandas DataFrame. You can find the full documentation for the DataFrames here.
Creating a DataFrame
There are many ways to create Pandas DataFrames. We often just read and ingest data into a data frame, but in this example, we create the DataFrame manually by starting with a dictionary of Series. Note that we are adding another dimensions to our data structure, so we need to label each Series. Here, we label the first Series 'a' and the second 'b'.
# create a dictionary called df_data
df_data = {'a' : pd.Series([1., 2., 3., 4.], index=['dog', 'cat', 'fruit', 'bird']),
'b' : pd.Series([10., 20., 30.], index=['cake', 'fruit', 'ice cream'])}
# create and output the DataFrame
df = pd.DataFrame(df_data)
df
a | b | |
---|---|---|
bird | 4.0 | NaN |
cake | NaN | 10.0 |
cat | 2.0 | NaN |
dog | 1.0 | NaN |
fruit | 3.0 | 20.0 |
ice cream | NaN | 30.0 |
Series 'a' and 'b' don't share the all of same indices. When we print the DataFrame, we see NaN values, which indicate that the Series does not contain a certain index.
df.index
Index(['bird', 'cake', 'cat', 'dog', 'fruit', 'ice cream'], dtype='object')
df.columns
Index(['a', 'b'], dtype='object')
We can also create a smaller DataFrame using a subset of the same data, this time specifying which indices we want to be included.
pd.DataFrame(df_data, index=['dog', 'fruit', 'bird'])
a | b | |
---|---|---|
dog | 1.0 | NaN |
fruit | 3.0 | 20.0 |
bird | 4.0 | NaN |
By specifying the column parameter, you can select which columns you'd like the new DataFrame to include. In the code below, we ask the DataFrame to include column 'e', which doesn't exist in the original dictionary. Because of this, a new column 'e' will be created with all its entries as NaN.
pd.DataFrame(df_data, index=['dog', 'fruit', 'bird'], columns=['a', 'e'])
a | e | |
---|---|---|
dog | 1.0 | NaN |
fruit | 3.0 | NaN |
bird | 4.0 | NaN |
Creating a DataFrame from a list of Python dictionaries
Another way to create a DataFrame is to use a list of Python dictionaries as your data. In the code below, we create a list of Python dictionaries called 'df_data2' and use this to make a DataFrame called 'df2'. We then use many of the same techniques as above to explore the DataFrame.
Please see this link for a reminder on Python dictionaries.
# create a Python dictionary
df_data2 = [{'apple': 5, 'cherry': 10}, {'peter': 1, 'emily': 2, 'brian': 6}]
# labels get created as column headers
pd.DataFrame(df_data2)
apple | brian | cherry | emily | peter | |
---|---|---|---|---|---|
0 | 5.0 | NaN | 10.0 | NaN | NaN |
1 | NaN | 6.0 | NaN | 2.0 | 1.0 |
# rename the rows from 0 and 1 to 'blue' and 'yellow' by specifying the index parameter
pd.DataFrame(df_data2, index=['blue', 'yellow'])
apple | brian | cherry | emily | peter | |
---|---|---|---|---|---|
blue | 5.0 | NaN | 10.0 | NaN | NaN |
yellow | NaN | 6.0 | NaN | 2.0 | 1.0 |
# create a smaller DataFrame by specifying the columns
pd.DataFrame(df_data2, columns=['cherry', 'emily','brian'])
cherry | emily | brian | |
---|---|---|---|
0 | 10.0 | NaN | NaN |
1 | NaN | 2.0 | 6.0 |
Exploring some basic DataFrame operations
Now let's look into how we can get data out of a DataFrame with some basic DataFrame operations. In the following code, we perform some operations on our DataFrame df.
# our DataFrame of interest
df
a | b | |
---|---|---|
bird | 4.0 | NaN |
cake | NaN | 10.0 |
cat | 2.0 | NaN |
dog | 1.0 | NaN |
fruit | 3.0 | 20.0 |
ice cream | NaN | 30.0 |
# display only column 'a' of df (subsetting)
df['a']
bird 4.0 cake NaN cat 2.0 dog 1.0 fruit 3.0 ice cream NaN Name: a, dtype: float64
# create a new column 'c' by adding 'a' and 'b' together
df['c'] = df['a'] + df['b']
df
a | b | c | |
---|---|---|---|
bird | 4.0 | NaN | NaN |
cake | NaN | 10.0 | NaN |
cat | 2.0 | NaN | NaN |
dog | 1.0 | NaN | NaN |
fruit | 3.0 | 20.0 | 23.0 |
ice cream | NaN | 30.0 | NaN |
Note that since NaN values cannot be added to floating point values, the resulting values in 'c' are NaN. For index 'fruit', however, both 'a' and 'b' are floating point values and can be added together.
# create a new column 'd' of boolean values indicating whether or not an index's value in 'a' is greater than 2.0
# NaN values evaluate to False
df['d'] = df['a'] > 2.0
df
a | b | c | d | |
---|---|---|---|---|
bird | 4.0 | NaN | NaN | True |
cake | NaN | 10.0 | NaN | False |
cat | 2.0 | NaN | NaN | False |
dog | 1.0 | NaN | NaN | False |
fruit | 3.0 | 20.0 | 23.0 | True |
ice cream | NaN | 30.0 | NaN | False |
# set cee equal to the 'c' column in the DataFrame
cee = df.pop('c')
cee
bird NaN cake NaN cat NaN dog NaN fruit 23.0 ice cream NaN Name: c, dtype: float64
# the pop method has removed 'c' from df
df
a | b | d | |
---|---|---|---|
bird | 4.0 | NaN | True |
cake | NaN | 10.0 | False |
cat | 2.0 | NaN | False |
dog | 1.0 | NaN | False |
fruit | 3.0 | 20.0 | True |
ice cream | NaN | 30.0 | False |
# delete column 'b' from the DataFrame
del df['b']
df
a | d | |
---|---|---|
bird | 4.0 | True |
cake | NaN | False |
cat | 2.0 | False |
dog | 1.0 | False |
fruit | 3.0 | True |
ice cream | NaN | False |
# insert a new column that is a copy of column 'a'
df.insert(2, 'copy_of_a', df['a'])
df
a | d | copy_of_a | |
---|---|---|---|
bird | 4.0 | True | 4.0 |
cake | NaN | False | NaN |
cat | 2.0 | False | 2.0 |
dog | 1.0 | False | 1.0 |
fruit | 3.0 | True | 3.0 |
ice cream | NaN | False | NaN |
# insert a new column that is a copy of 'a' up to excluding the value at the third position of the Series. The rest of the
# column is NaNs
df['a_upper_half'] = df['a'][:3]
df
a | d | copy_of_a | a_upper_half | |
---|---|---|---|---|
bird | 4.0 | True | 4.0 | 4.0 |
cake | NaN | False | NaN | NaN |
cat | 2.0 | False | 2.0 | 2.0 |
dog | 1.0 | False | 1.0 | NaN |
fruit | 3.0 | True | 3.0 | NaN |
ice cream | NaN | False | NaN | NaN |
Note that while both methods above (df.insert and df['col']) allowed us to insert new columns into the DataFrame, only df.insert lets us specify which position we want the column to be in.
Data Manipulation with Pandas
There are 5 main data manipulation tools that Pandas covers:
Let's look at how we can use the iris dataset to use these tools.
# Load the iris dataset from sklearn and create a corresponding DataFrame
from sklearn import datasets
iris = datasets.load_iris()
iris_data = pd.DataFrame(iris.data,columns = ['Sepal Length','Sepal Width','Petal Length','Petal Width'])
iris_data.head()
Sepal Length | Sepal Width | Petal Length | Petal Width | |
---|---|---|---|---|
0 | 5.1 | 3.5 | 1.4 | 0.2 |
1 | 4.9 | 3.0 | 1.4 | 0.2 |
2 | 4.7 | 3.2 | 1.3 | 0.2 |
3 | 4.6 | 3.1 | 1.5 | 0.2 |
4 | 5.0 | 3.6 | 1.4 | 0.2 |
First, let's filter the data so we have only samples with Petal Length > 1.0.
filtered = iris_data[iris_data['Petal Width']>1.0]
filtered.head()
Sepal Length | Sepal Width | Petal Length | Petal Width | |
---|---|---|---|---|
50 | 7.0 | 3.2 | 4.7 | 1.4 |
51 | 6.4 | 3.2 | 4.5 | 1.5 |
52 | 6.9 | 3.1 | 4.9 | 1.5 |
53 | 5.5 | 2.3 | 4.0 | 1.3 |
54 | 6.5 | 2.8 | 4.6 | 1.5 |
Next, we use subsetting to find the variance in the Petal Width.
petal_width = iris_data['Petal Width']
petal_width.var(axis=0)
0.5824143176733784
In pandas, it's especially easy to combine datasets by column because you can write dataframe['columnname']. However, say we want to make a new dataframe that has a categorical column based on which type of iris flower it is.
labels = pd.DataFrame(iris.target,columns=['Flower Type'])
labels
result = pd.concat([iris_data,labels],axis=1) # requires iterable argument so the DataFrames are in a list
result.head()
Sepal Length | Sepal Width | Petal Length | Petal Width | Flower Type | |
---|---|---|---|---|---|
0 | 5.1 | 3.5 | 1.4 | 0.2 | 0 |
1 | 4.9 | 3.0 | 1.4 | 0.2 | 0 |
2 | 4.7 | 3.2 | 1.3 | 0.2 | 0 |
3 | 4.6 | 3.1 | 1.5 | 0.2 | 0 |
4 | 5.0 | 3.6 | 1.4 | 0.2 | 0 |
Lastly, mutations let you modify the data in an entire row or column. We will use mutation to do a simple standard score normalization, which is commonly used to scale the data (though as you'll see, it's not as statistically applicable in this case).
mean = iris_data['Sepal Width'].mean()
std = iris_data['Sepal Width'].std()
iris_data['Sepal Width']=(iris_data['Sepal Width']-mean)/std
iris_data['Sepal Width'].head()
0 1.028611 1 -0.124540 2 0.336720 3 0.106090 4 1.259242 Name: Sepal Width, dtype: float64