Baseball Data Analysis

Introduction¶

The data that we pulled had a lot of information. The file readME2014.txt has information about the CSV files and basic information of all the CSV files. The following analysis was not on all the CSV files. The files that was put at the centre of all the analysis was salary.csv because a lot of the analysis is centered around the salary. The data cleaning operations were performed on what is present on the salary.csv file. On the way of exploring the Baseball data, some other csv files are also explored, i.e,

• Master.csv
• Teams.csv
• Batting.csv
• Pitching.csv

All the CSVs are given with the project files. Downloand links of all the versions are provided in GitHub README in the form of PDF, HTML, MarkDown, ipython Notebook and JPEG images for plots

The code is strictly written in Python 3.6 with a conda environment. It is recommended to run the code is Python > 3.4 and using Python2 is not at all recommended.

NOTE: The actual ipynb file may not have all the details in written and the styling may not kick in as it is in PDF or MarkDown. If the intention is to skim through the analysis only instead of running the code, the PDF version is recommended.

First get all the imports down the line. I am alsi configuring my own css file to improve the look and feel of the Jupyter Notebook. The CSS file is present in styles directory in the project.

Now before we start lets first see what are the questions that we can answer about the dataset from the analysis. These question are of various type and varies from different range and different levels. Hence after every question, the level of question is also mentioned in bold. The questions are

• How teams invested in players in terms of player's salary? (Basic)
• How players recieved salary over their career? (Basic)
• What are the most common salary ranges teams preferred to compensate their players? This reveals the salary standard of BaseBall. (Intermediate)
• Is there any abrupt changes (ups/downs) teams faced in terms of the salary they paid? This shows if any year was good or bad for Baseball players or was their any effect of recessions on players' salary. (Intermediate)
• Is there any abrupt chnages (ups/downs) in players' performance in any year and what could be the probable reason? (Intermediate)
• How batters' homeruns (HR) or a pitchers' shutouts (SHO) is affected by thier height and weight? (Intermediate)
• How salary affects players' performance? (Advanced)
• What makes a team secure first rank or at least be in top 3? (Advanced)

Configuring our RC file is important to give plots my own customized look and feel. For more info on rcParams and visual parameter configuration, please click here

Now some basic path resolve according to the project directory. Here is the documentation for os module for python.

Let's start the Data cleaning !!¶

The data cleaning that we are going to perform would be based on Salary.csv. If we open the Salary.csv, we see no players' name but only their IDs. Now we would need players' name in furute. The players' name is only available in Master.csv. Hence we need to pullin the corresponding names of the players along with their IDs. Moreover in the salary csv file, entries are noted only from 1985 to 2014. Salary info of any other year previous to this is not present, may be because the recodings of Salary is started from 1985 and not prior to that. Hence players who used to play prior to 1985 had no entry in the Salary.csv file. Hence we need to take only those entries whose record can be found in Salary.csv.

This plot is the basic plot to show a bargraph of how the teams spent on player's salary. This also ensures the data cleaning was performed correctly.

Now we know that NYA, BOS, LAN, PHI, NYN are the top 5 teams in terms of spending money on their players. The next thing that we should do is to check what are top players in terms of salary and logically they should be from these top teams.

The following section shows the top 5 players in terms of total salary and the teams they played for throughout their career. This also proves the above data wrangling was performed correctly. Note that their_clubs is not an OrderedDict and hence the 5 players mentioned are among to 5 but may not be in any ascending or descending order. To check top 5 players in descending order, please visit the following barchart of top 20 players.

and here is the chart of top 20 players in terms of their career earnings.

We would revisit this later on. Now before we move ahead, let's visualize the distruibution desdity of player's salary. From this we can visualize that what is the most common salary ranges teams prefer to compensate their players.

Were they all thrifty ??¶

The above distribution is skewed and it is the distribution of salary. This shows the most density distribution is located around 0 to 5000000. This proves that high salaries were not so frequent. What does this shows?? This shows a common human nature; most people are thrifty when it comes it money, atleast the baseball teams were.

Did teams think similar when it comes to money ??¶

The above timeseries shows how people liked to invest over the passage of time. Each teams represent a single line. The idea is not to show how each teams are investing over time, rather it shows what is the investment pattern of all the teams. We can see that upto 2001 all the teams invested almost similarly. Beyond 2001 to around 2010 the diffence between the investment of teams increased substantially with some team investing a lot than others. But beyong 2011 this difference got really big and we can also see the top 3 highest investments were made in the year of 2014. For example, we can see the the one team at 2005 invested a lot more than others and kept on investing till the end. The following code shows this club name an the money that they invested.

data_1[data_1.yearID == 2005].groupby('teamID', 
        as_index=False)['salary'].sum() \
        .sort_values('salary', ascending=False) \
        .reset_index().drop('index', axis=1).ix[0]

This prints

teamID          NYA
salary    208306817
Name: 0, dtype: object

This is logical too because we have already seen that NYA invested most out of all the teams.

Interestingly, in the year of 2014, NYA didn't remain at number 1 position and it changed to LAN. This code shows it below

data_1[data_1.yearID == 2014].groupby('teamID', 
        as_index=False)['salary'].sum() \
        .sort_values('salary', ascending=False) \
        .reset_index().drop('index', axis=1).ix[0:2]

this prints,

teamID	salary
0	LAN	214014600
1	NYA	197543907
2	PHI	180944967

Howerver the trend that we have seen, we can say unless the team is an extraordinary, most teams spent likely over the passage of time

Now in the above plot I have tried to show how different metric changed over time form 1985 to 2014. We can see for example that there is an abrupt decrese in the total runs scored in the year of 1994 as compared to other years. Let's verify if the plot shows the right value? If we execute plot_data.R.argmin() it returns 9 which is the row number of the plot_data table which is further derived from batting. So to get the year we execute plot_data.ix[9][['yearID', 'R']] and indeed the year is 1994 and the run is 15752 which can also be verified from the scale. Evidently all other performance of Batters and Pitchers in the year of 1994 was poor as compared to other years. ** SO I GUESS 1994 WAS A DARK YEAR FOR BASEBALL FANS..:)**

Time for another data cleaning¶

Now as we have colleced some data into the DataFrame called data_1, we need to match it with our new DataFrame: batting. In other words we need to keep only those entries from batting Dataframe whose entry is found in the data_1 DataFrame. The resaon we want to do this because we don't want to keep any entries of any players who salary, name, surname, height, weight, pitching hand and batting hand is not present. The common way to achieve this is to merge(..) on both the DataFrames on=playerID

Now I would go into much more interesting and fun stuffs we I would bring in some statistics and hypothesis tests into this but for now, let's focus on the the data table that we have already. Here all the batters staticstics form 1985 to 2014 is avaialbale with their salary and other game related performances.

Let's merge the pitching table too as we have done with the batting table. The data cleaning process would be exactly similar except the coulumns that needs to be merged

What are the facts the above plot shows ??¶

• More players play with their RIGHT hand as compared to the LEFT hand because there is a lot green as compared to red and blue
• Players with most homeruns are of medium height and medium weight category. Tall or heavy weight players performed not so well.

The below figure shows number of batters for each hands

And we can see that what we predicted was true there are more than double players playing with their right hand as compared to others.

Wow we see some wonderful trends here too¶

• Number of players with shutouts is far less than the number of players with homeruns. This means players like to bat more than pitch.
• There are no pitchers who like to throw with both of their hand as comared to the batters. Which means piching can either be done by left hand or right hand but not with both the hand.
• The weight and the height trend is similar to the batting. Most of the players performed well if they are medium weight and medium height. We can verify that the bigger marker sizes are all in the middle.
• One more interesting thing is that even some of the outliers are also above 280 in weight category whcich is I guess natural because if you are heavy weight, you can not run well and hence as a pitcher, you perform poorly

Let's move into some Statistics now!!¶

First let's group all the enries in the DataFrame by the player's ID. Then sum all player's Total Runs Scored and their Total Salary

Interestingly some players have scored 0 runs but they were paid a lot. How is it possible? It's not necessary that these players are bad as these players can be pitchers and they might have some good Saves against some batters. So the logical solution is to combine the SV column with this group by table. But the problem is that they don't share the same scale hence to combine, we need to standardise these two columns and then both the values and create a new column say performance.

Now I guess I may not need the Runs_Scored and the SV columns. So we can drop them.

Now it's time to prepare our data. We are going to perform an Hypothesis test to figure out ** If above average salary resulted in above average performance??** However we are taking median here instead of mean to reject all the outliers.

Now before we move ahead, let's perofrom a quick plot see the total number of people in both the groups

As we expected, there are a lot less high performers as compared low performers.

Decision on the Hypothesis¶

So as we can see that the t-stat is huge as compared to the t-critical and the t-stat is positive. This means we reject the null hypothesis in favour of alternative hypothesis. Now we can surely say that The players who were paid more performed better Below is a plot which displays it more clearly

In the above plot we can see that the category Salary below Average is only concentrated within the performance range 0 to 2.5 with an exception of few points as true otliers. While in the right for the category Salary above average is distributed upto 7.5 and having some outliers upto 20.

Another plot below is more descriptive about two categories above relating to the wide range of performance keys distributed within the sample data.

We can see that in all of the cases players above average salary did well in all those performance metrics as compared to players below average salary. This proves even more that players with better salary played better too. This proves very well that in almost all the metrics players performed well when they were payed more.

Phew!! I have worked really hard for this..¶

The above is not just a heatmap but it combines so many things togather. But before that lets first see what are the questions we are trying to answer with this.

• Can we say that most expensive players in total are quick team switchers?
• Are most expensive players are best peformers too?
• When player played for a less amount of time, are their performace and salary is that good as compared to when they played for long?

The above heatmap is shows the way top 5 players ( in terms of total Salary ) switched their teams over their career. It also shows 3 numbers for each cells.

The first Number shows how much money that players has earned while playing for that team.

The second Number shows a performace metric of that player when he played in that team.

The third Number shows How long the player played for that team.

With all these we can see that some arrow (up or down) after the first two numbers. These arrows shows trends. For example an up arrow after the salary means that THE PLAYER'S SALARY WHILE HE WAS PLAYING FOR THAT TEAM IS WITHIN THE TOP 5 FOR THE AMOUNT FOR YEARS HE PLAYED FOR THAT TEAM. The same is for performance metric. Consequently, the down arrow means it is not in top 5.

The performance metric itself is curated very carefully giving preference to almost all the factors that makes baseball player a baseball player. The metrics that are used is

# G           = Number of total games played    ( For all )
# Runs_Scored = Total number of runs scored     ( For batters  )
# HR_Scored   = Total number of homeruns scored ( For batters  )
# SHO         = Total number of shutouts        ( For Pitchers )
# SV          = Total number of saves           ( For Pitchers )
metrics = ['G', 'Runs_Scored', 'HR_Scored', 'SHO', 'SV']

As these metrics lie in different range from each other and combining these direcly would not work. Hence the metrics were standardized like this

gTemp = temp.groupby('fullName', as_index=False)[metrics].sum()
gTemp['G'] /= temp.G.std()
gTemp['Runs_Scored'] /= temp.Runs_Scored.std()
gTemp['HR_Scored'] /= temp.HR_Scored.std()
gTemp['SHO'] /= temp.SHO.std()
gTemp['SV'] /= temp.SV.std()
gTemp['Gross_Metric_Score'] = gTemp['G'] + \
     gTemp['Runs_Scored'] + gTemp['HR_Scored'] + \
     gTemp['SHO'] + gTemp['SV']

Because the gross metric is taken, it it pretty balanced out for only batters, only pitchers or both batter and pitchers.

In the above heatmap we can see that most of the players switched their teams atleast once. Some even switched to even 4 teams. Some players played for a team for less than one year which is pretty stange. We can that Derek Jeter is a player who played for only one club and never switched in his long 18 years of career. Also for both his salary and performance we see an up arrow. This means he was happy while playing for NYA and paid very well for his long career.

Barry Bonds was a remarkable player though. While he was playing for PIT for 6 long years, even his salary was not within the top 5, but his performance was always in top 5 range. Probably that's why he switched the team to SFN where he played for 14 long years. This can be verified further by running the following command

for k, g in data_1[data_1.fullName == 'Barry_Bonds'].groupby('teamID'):

    display(g)

We can see that Barry Bonds played in PIT from 1986 to 1992 and then he switched to SFN where he played from 1993 to 2007

But interestingly more money playes got, more teams they changed. The code below proves that

number_of_switches_by_low = []

number_of_switches_by_high = []

for k, g in data_1.groupby('fullName'):

    if g.salary.mean() <= data_1.salary.mean():

        number_of_switches_by_low.append(len(g.teamID))
    else:

        number_of_switches_by_high.append(len(g.teamID))
print(np.mean(number_of_switches_by_low))
print(np.mean(number_of_switches_by_high))

This prints

4.1706527155
10.8418918919

This means players who got more salary than the average, switched more than two times than those who got less salary than average. This trend is also true for number of games played(G). More games player played, more teams(almost 2 times more) they switched. This means more active players switched more teams although Derek Jeter was truly an exceptional case.

We can also see that the players who have been in top 5 for their performance in any team for a long time could not retain their top 5 position in some other team when they played there for short time.

All about teams !!¶

Here we are pulling the Teams.csv file to get some info about the Teams. The following section can answer some of the important questions like,

• How different metrics like Total Homeruns or Number of errors made affect the winning ?
• What makes a team rank first as compared to others ?

The above four plot shows different things. These are grouped between World Series Winners and Others. Scatter plots with world series winners are in Blue while others are in Green. This shows the correlation of Team's Winning (W) with two important metrics, i.e., Total Homeruns Scored(HR) and Numbers of errors made(E). The scatter plot marker sizes are also controlled by down sclaed value of Attendance.

As we can see that the most of the teams showed a upward trend while correlating with Total homeruns scored while they showed a downward treand correlating with Errors made. This pattern was quite expected. We also see that bigger balls are scattered all over which means that there is hardly any relation of the attendance with the correlation group of Wins and Errors or Wins and Homeruns.

What ensures first rank ??¶

We can see that as it was expected more run team scored, better was their ranks. For example teams with rank 1 scored an average of 793.778 which is almost 20 runs more than the second ranking teams having 772.441 runs. The same trend is visible for homeruns. Consequently, teams with rank 1, have less runs allowed(RA) and homeruns allowed(HRA) as compared to teams with rank 2 or 3.

Interestingly, Fielding Percentage(FP) is almost same for teams with any ranks which means Fielding Percentage doesn't affect the rank of any teams. This is logical also because any team would try to field as much as possible irrespective of their expected/desired rank.

The even more interesting part is Division Wins(DivWin), League Wins(LgWin), Wildcard Wins(WCWin) and World Series Wins(WSWin). We can see that World Series Wins affect a lot in rank which was logical but Division Wins is only constrained to first ranking teams. What does this mean?? This means if any team wins a world series, there is a probable chance that team can be in the list of rank 1 but there is no gurantee because even second ranking players are also world series winners. But if a team gets a Divisional Win, there is a gurantee that the team would be in the rank 1 list because divisional wins is only constrained within the first ranking teams. The first ranking teams are never Wild card winners which may mean that to become a first ranker, a team has to perform from the beginning and there can't be any abrupt or special case first rankers. You have to be a clear winner to be first ranker, NO WILDCARD, NO SPECIAL CASES.

Final Hypothesis ??¶

Now before I end this analysis, lets induge into some hypothesis and a plot again. The Hypothesis test would be based on the Teams statistics.

I think the less number of complete game(CG) players the team had, the better their chances were to be in top 3 in rankings

In baseball the complete game(CG) means when a pitcher pitches a complete game without any rest. I believe that if a pitcher is pitching for the complete game, his efficiency would decrease than those players who are sitting idle. So if a team would have more CG players, their performance would suffer and lesser their chance would be in top 3. I may be wrong but there is no better way to find this out other than a hypothesis test.

Statistically we would first take two data frames; first having entries of all top three teams only and second, having entries for rest of the teams. Then we take two sample series. One would have number of CG players for top ranking teams and other is numner of CG players for rest of the teams. Then we would perform One tailed Independent t-Test where,

Oops!! I was wrong.¶

Looks like I was completely wrong. The t-stats is a lot more than the one tailed t-critical value. This mean we have to accept the NULL Hypothesis. But NULL hypothesis was that their means would be same. But looking at the t-stats we see that if our ALTERNATIVE hypothesis would have been

The above bar-chart shows what we have found using the Hypothesis. Rank 1, 2 & 3 holders much widely distributed as compared to other rankers. With in I tentatively end my Analysis here.

Conclusions !!¶

There were several coclusions that we can draw. Some were solid, some were casual and some limitations. Let's lool into these

• The first plot displays total money spent by all teams. This actually solves our first basic question of how teams invested money on players. The inference/conclusion shows Team NYA invested a lot more than even the 2nd ranking team BOS (almost $1B more). This makes NYA being rich or well funded team of all.

• The second plot shows the lifetime earning of top 20 players which answers the second question How players recieved salary over their career? But this show one more thing that big difference between the players' salary is only limited within first few players. After first few top palyers, the difference between salary is very less. This concludes Except some few exceptional players, most of the baseball players were payed almost similarly. The next plot proves that even more and assures our colclusion better as discussed below.

• The next plot shows the disribution of different salary ranges from 1985 to 2014. This shows the most popular salary range is within 5000000. This probably answers our third question. This also shows the salary standard. But most importantly this proves our conclusion discussed in the last point. The distribution is heavily skewed and this proves most of the players were payed similarly.

• The next quesion was Is there any abrupt changes (ups/downs) teams faced in terms of the salary they paid?. Only a timeseries can reveal the answer to it and the All teams investment each year just did that. The most important conclusion that we can find is Except some few teams most of the teams' investment trend are almost similar. This means most of the teams were very competitive in terms of salary/investment.

• The next plot shows a lot of things and it speaks for itself. It shows basically how different performance metrics changes in different year. This may not show any generic trend as it is true for any timeseries but it shows any incedent that affected the output. In this case for example we see that there is definitely something bad happed in the year of 1994 which affected all the performance of all the players. With little bit of googleing, I found that there was a stike for all players in the year of 1994. Here is a links as a reference.

• The next two scatter plots of weight and height with HR/SHO as bubble size shows few things that fall under the question category What makes a batter or pitcher perform better? or more precisely what makes `How BMI (Weight and Height ratio) of a player affected their performance? We concluded so many other things but the most important one was Players with medium weight and height performed best as a player as compared to others.

• The count plot just shows that there are more under-performing players as compared to above average players. So does the plot for two salary groups after the first hypothesis test. This brings us to the first hypothesis - Does better payed players performed better??

• The first Hypothesis was quite clear. It strongly showed that better payed players performed better. The following Pi chart also proves it with more detail

Limitations in Analysis :-(¶

Correlation never implies Causation!!

This is true when we run the following commands

hrmean = teams.HRA.mean()
gwmean = teams.W.mean()

print(teams[(teams.HRA < hrmean) & (teams.W > gwmean)].shape[0])

print(teams[(teams.HRA > hrmean) & (teams.W > gwmean)].shape[1])```

This prints,

147 48

This is sad because it shows there are **147 times** teams won games more than average but had less than average homeruns. While we have **only 48 times** when teams won more than average games and had more than average homeruns. This means even though there was a **strong positive correlation** visible between **Homeruns and Games Won** but higher homeruns **don't assure** higher number of wins.


The second problem was with the dataset `Teams.csv`. The reason that caused the main problem was the data cleaning process on Teams.csv file. Here while data cleaning, `dropna()` was used in `core.py`. Now unfortuantely most of the entries/index were NAN and hence most of the entries were removed and then the Hypothesis was performed about CG players' numbers and Top 3 Ranks on the trimmed version of Teams.csv. But in reality if we reject the NAN values by bypassing the cleaning process. Then if we perform the Hypothesis test, the **result is exactly opposite** of being 

`degrees of freedom = 2773`

`t-Satistics = -7.8253361462186195`

`t-critical at α=0.5 is -1.64540335`

This would have forced us to reject the NULL Hypothesis and ACCEPT the Alternative Hypothesis. this is more visible if we plot this 

The above plot confirms that teams with better rankings had small values of CG players as compared to teams with lower ranks. But the question is can we rely on a dataset having so many NULL values or in other words was dropna() relevanat or not?

What's Ahead ??¶

As section is all about what I can do with this analysis in future, this is a good time I cogitate on the findings and the limitations of these. Based on that analysis, I have fonud few improvements that are categorised into the following sections.

1. Improvements and progress in data analysis
2. Improvements and progress in data visualisation
3. Improvements and progress in code implementations

Improvements and progress in data analysis

• We can pull in the `Manager.csv` file and see how many teams were managed in each year
• We can see some correlation of Manager.csv table which are interesting
• We can then merge this table with player's salary and performance table and perform Hypothesis test to see If better managed palyers performed better
• We can also pull in and merge the VotesCastPlayers table to see if high performing palyer were given more votes or not.
• We may compare it with those players who may not have high performance but played for a long time in a team

<h4>Improvements and progress in data visualisation</h4>
<li>Improving the visual quality of weight vs height scatter plots with HR and SHO as bubble size</li>
<li>We can perform correlation plot of Manager's performance VS Their Player's Performance</li>
<li>We can create a heatmap of Vote's casted of all team VS All years</li>

<h4>Improvements and progress in code implementations</h4>
<li>Need to create more generic and reusable codes that can be used for future projects</li>
<li>Need to create a generic data cleaning class for better code readability in ipython notebooks</li>