The goal of the project is to analyze the Exit Surveys collected from the Employees of Department of Education, Training and Employment (DETE) and the Technical and Further Education (TAFE) institute in Queensland, Australia.
The Department of Employment, Education and Training was an Australian government department that existed between July 1987 and March 1996. At its creation, the Department was responsible for the following:
In Australia, technical and further education or TAFE institutions provide a wide range of predominantly vocational courses, mostly qualifying courses under the National Training System/Australian Qualifications Framework/Australian Quality Training Framework. Fields covered include business, finance, hospitality, tourism, construction, engineering, visual arts, information technology and community work.
The purpose of the analysis is to answer the following questions -
Are employees who only worked for the institutes for a short period of time resigning due to some kind of dissatisfaction? What about employees who have been there longer?
Are younger employees resigning due to some kind of dissatisfaction? What about older employees?
The datasets from both the institutes are surveys collected from the out going employees. They have large number of columns, predominantly columns that are questions asked to the employees and the answer either boolean or on the Likert
scale. A few of the columns, enough to get started, from both the datasets are described below:-
dete_survey.csv 's :
ID: An id used to identify the participant of the survey
SeparationType: The reason why the person's employment ended
Cease Date: The year or month the person's employment ended
DETE Start Date: The year the person began employment with the DETE
tafe_survey.csv 's :
Record ID: An id used to identify the participant of the survey
Reason for ceasing employment: The reason why the person's employment ended
LengthofServiceOverall. Overall Length of Service at Institute (in years): The length of the person's employment (in years)
%matplotlib inline
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
from pywaffle import Waffle
df_dete = pd.read_csv('dete_survey.csv')
df_tafe = pd.read_csv('tafe_survey.csv')
df_dete.head(5)
ID | SeparationType | Cease Date | DETE Start Date | Role Start Date | Position | Classification | Region | Business Unit | Employment Status | ... | Kept informed | Wellness programs | Health & Safety | Gender | Age | Aboriginal | Torres Strait | South Sea | Disability | NESB | |
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
0 | 1 | Ill Health Retirement | 08/2012 | 1984 | 2004 | Public Servant | A01-A04 | Central Office | Corporate Strategy and Peformance | Permanent Full-time | ... | N | N | N | Male | 56-60 | NaN | NaN | NaN | NaN | Yes |
1 | 2 | Voluntary Early Retirement (VER) | 08/2012 | Not Stated | Not Stated | Public Servant | AO5-AO7 | Central Office | Corporate Strategy and Peformance | Permanent Full-time | ... | N | N | N | Male | 56-60 | NaN | NaN | NaN | NaN | NaN |
2 | 3 | Voluntary Early Retirement (VER) | 05/2012 | 2011 | 2011 | Schools Officer | NaN | Central Office | Education Queensland | Permanent Full-time | ... | N | N | N | Male | 61 or older | NaN | NaN | NaN | NaN | NaN |
3 | 4 | Resignation-Other reasons | 05/2012 | 2005 | 2006 | Teacher | Primary | Central Queensland | NaN | Permanent Full-time | ... | A | N | A | Female | 36-40 | NaN | NaN | NaN | NaN | NaN |
4 | 5 | Age Retirement | 05/2012 | 1970 | 1989 | Head of Curriculum/Head of Special Education | NaN | South East | NaN | Permanent Full-time | ... | N | A | M | Female | 61 or older | NaN | NaN | NaN | NaN | NaN |
5 rows × 56 columns
df_tafe.head(5)
Record ID | Institute | WorkArea | CESSATION YEAR | Reason for ceasing employment | Contributing Factors. Career Move - Public Sector | Contributing Factors. Career Move - Private Sector | Contributing Factors. Career Move - Self-employment | Contributing Factors. Ill Health | Contributing Factors. Maternity/Family | ... | Workplace. Topic:Does your workplace promote a work culture free from all forms of unlawful discrimination? | Workplace. Topic:Does your workplace promote and practice the principles of employment equity? | Workplace. Topic:Does your workplace value the diversity of its employees? | Workplace. Topic:Would you recommend the Institute as an employer to others? | Gender. What is your Gender? | CurrentAge. Current Age | Employment Type. Employment Type | Classification. Classification | LengthofServiceOverall. Overall Length of Service at Institute (in years) | LengthofServiceCurrent. Length of Service at current workplace (in years) | |
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
0 | 6.341330e+17 | Southern Queensland Institute of TAFE | Non-Delivery (corporate) | 2010.0 | Contract Expired | NaN | NaN | NaN | NaN | NaN | ... | Yes | Yes | Yes | Yes | Female | 26 30 | Temporary Full-time | Administration (AO) | 1-2 | 1-2 |
1 | 6.341337e+17 | Mount Isa Institute of TAFE | Non-Delivery (corporate) | 2010.0 | Retirement | - | - | - | - | - | ... | Yes | Yes | Yes | Yes | NaN | NaN | NaN | NaN | NaN | NaN |
2 | 6.341388e+17 | Mount Isa Institute of TAFE | Delivery (teaching) | 2010.0 | Retirement | - | - | - | - | - | ... | Yes | Yes | Yes | Yes | NaN | NaN | NaN | NaN | NaN | NaN |
3 | 6.341399e+17 | Mount Isa Institute of TAFE | Non-Delivery (corporate) | 2010.0 | Resignation | - | - | - | - | - | ... | Yes | Yes | Yes | Yes | NaN | NaN | NaN | NaN | NaN | NaN |
4 | 6.341466e+17 | Southern Queensland Institute of TAFE | Delivery (teaching) | 2010.0 | Resignation | - | Career Move - Private Sector | - | - | - | ... | Yes | Yes | Yes | Yes | Male | 41 45 | Permanent Full-time | Teacher (including LVT) | 3-4 | 3-4 |
5 rows × 72 columns
df_dete.info()
<class 'pandas.core.frame.DataFrame'> RangeIndex: 822 entries, 0 to 821 Data columns (total 56 columns): # Column Non-Null Count Dtype --- ------ -------------- ----- 0 ID 822 non-null int64 1 SeparationType 822 non-null object 2 Cease Date 822 non-null object 3 DETE Start Date 822 non-null object 4 Role Start Date 822 non-null object 5 Position 817 non-null object 6 Classification 455 non-null object 7 Region 822 non-null object 8 Business Unit 126 non-null object 9 Employment Status 817 non-null object 10 Career move to public sector 822 non-null bool 11 Career move to private sector 822 non-null bool 12 Interpersonal conflicts 822 non-null bool 13 Job dissatisfaction 822 non-null bool 14 Dissatisfaction with the department 822 non-null bool 15 Physical work environment 822 non-null bool 16 Lack of recognition 822 non-null bool 17 Lack of job security 822 non-null bool 18 Work location 822 non-null bool 19 Employment conditions 822 non-null bool 20 Maternity/family 822 non-null bool 21 Relocation 822 non-null bool 22 Study/Travel 822 non-null bool 23 Ill Health 822 non-null bool 24 Traumatic incident 822 non-null bool 25 Work life balance 822 non-null bool 26 Workload 822 non-null bool 27 None of the above 822 non-null bool 28 Professional Development 808 non-null object 29 Opportunities for promotion 735 non-null object 30 Staff morale 816 non-null object 31 Workplace issue 788 non-null object 32 Physical environment 817 non-null object 33 Worklife balance 815 non-null object 34 Stress and pressure support 810 non-null object 35 Performance of supervisor 813 non-null object 36 Peer support 812 non-null object 37 Initiative 813 non-null object 38 Skills 811 non-null object 39 Coach 767 non-null object 40 Career Aspirations 746 non-null object 41 Feedback 792 non-null object 42 Further PD 768 non-null object 43 Communication 814 non-null object 44 My say 812 non-null object 45 Information 816 non-null object 46 Kept informed 813 non-null object 47 Wellness programs 766 non-null object 48 Health & Safety 793 non-null object 49 Gender 798 non-null object 50 Age 811 non-null object 51 Aboriginal 16 non-null object 52 Torres Strait 3 non-null object 53 South Sea 7 non-null object 54 Disability 23 non-null object 55 NESB 32 non-null object dtypes: bool(18), int64(1), object(37) memory usage: 258.6+ KB
df_tafe.info()
<class 'pandas.core.frame.DataFrame'> RangeIndex: 702 entries, 0 to 701 Data columns (total 72 columns): # Column Non-Null Count Dtype --- ------ -------------- ----- 0 Record ID 702 non-null float64 1 Institute 702 non-null object 2 WorkArea 702 non-null object 3 CESSATION YEAR 695 non-null float64 4 Reason for ceasing employment 701 non-null object 5 Contributing Factors. Career Move - Public Sector 437 non-null object 6 Contributing Factors. Career Move - Private Sector 437 non-null object 7 Contributing Factors. Career Move - Self-employment 437 non-null object 8 Contributing Factors. Ill Health 437 non-null object 9 Contributing Factors. Maternity/Family 437 non-null object 10 Contributing Factors. Dissatisfaction 437 non-null object 11 Contributing Factors. Job Dissatisfaction 437 non-null object 12 Contributing Factors. Interpersonal Conflict 437 non-null object 13 Contributing Factors. Study 437 non-null object 14 Contributing Factors. Travel 437 non-null object 15 Contributing Factors. Other 437 non-null object 16 Contributing Factors. NONE 437 non-null object 17 Main Factor. Which of these was the main factor for leaving? 113 non-null object 18 InstituteViews. Topic:1. I feel the senior leadership had a clear vision and direction 608 non-null object 19 InstituteViews. Topic:2. I was given access to skills training to help me do my job better 613 non-null object 20 InstituteViews. Topic:3. I was given adequate opportunities for personal development 610 non-null object 21 InstituteViews. Topic:4. I was given adequate opportunities for promotion within %Institute]Q25LBL% 608 non-null object 22 InstituteViews. Topic:5. I felt the salary for the job was right for the responsibilities I had 615 non-null object 23 InstituteViews. Topic:6. The organisation recognised when staff did good work 607 non-null object 24 InstituteViews. Topic:7. Management was generally supportive of me 614 non-null object 25 InstituteViews. Topic:8. Management was generally supportive of my team 608 non-null object 26 InstituteViews. Topic:9. I was kept informed of the changes in the organisation which would affect me 610 non-null object 27 InstituteViews. Topic:10. Staff morale was positive within the Institute 602 non-null object 28 InstituteViews. Topic:11. If I had a workplace issue it was dealt with quickly 601 non-null object 29 InstituteViews. Topic:12. If I had a workplace issue it was dealt with efficiently 597 non-null object 30 InstituteViews. Topic:13. If I had a workplace issue it was dealt with discreetly 601 non-null object 31 WorkUnitViews. Topic:14. I was satisfied with the quality of the management and supervision within my work unit 609 non-null object 32 WorkUnitViews. Topic:15. I worked well with my colleagues 605 non-null object 33 WorkUnitViews. Topic:16. My job was challenging and interesting 607 non-null object 34 WorkUnitViews. Topic:17. I was encouraged to use my initiative in the course of my work 610 non-null object 35 WorkUnitViews. Topic:18. I had sufficient contact with other people in my job 613 non-null object 36 WorkUnitViews. Topic:19. I was given adequate support and co-operation by my peers to enable me to do my job 609 non-null object 37 WorkUnitViews. Topic:20. I was able to use the full range of my skills in my job 609 non-null object 38 WorkUnitViews. Topic:21. I was able to use the full range of my abilities in my job. ; Category:Level of Agreement; Question:YOUR VIEWS ABOUT YOUR WORK UNIT] 608 non-null object 39 WorkUnitViews. Topic:22. I was able to use the full range of my knowledge in my job 608 non-null object 40 WorkUnitViews. Topic:23. My job provided sufficient variety 611 non-null object 41 WorkUnitViews. Topic:24. I was able to cope with the level of stress and pressure in my job 610 non-null object 42 WorkUnitViews. Topic:25. My job allowed me to balance the demands of work and family to my satisfaction 611 non-null object 43 WorkUnitViews. Topic:26. My supervisor gave me adequate personal recognition and feedback on my performance 606 non-null object 44 WorkUnitViews. Topic:27. My working environment was satisfactory e.g. sufficient space, good lighting, suitable seating and working area 610 non-null object 45 WorkUnitViews. Topic:28. I was given the opportunity to mentor and coach others in order for me to pass on my skills and knowledge prior to my cessation date 609 non-null object 46 WorkUnitViews. Topic:29. There was adequate communication between staff in my unit 603 non-null object 47 WorkUnitViews. Topic:30. Staff morale was positive within my work unit 606 non-null object 48 Induction. Did you undertake Workplace Induction? 619 non-null object 49 InductionInfo. Topic:Did you undertake a Corporate Induction? 432 non-null object 50 InductionInfo. Topic:Did you undertake a Institute Induction? 483 non-null object 51 InductionInfo. Topic: Did you undertake Team Induction? 440 non-null object 52 InductionInfo. Face to Face Topic:Did you undertake a Corporate Induction; Category:How it was conducted? 555 non-null object 53 InductionInfo. On-line Topic:Did you undertake a Corporate Induction; Category:How it was conducted? 555 non-null object 54 InductionInfo. Induction Manual Topic:Did you undertake a Corporate Induction? 555 non-null object 55 InductionInfo. Face to Face Topic:Did you undertake a Institute Induction? 530 non-null object 56 InductionInfo. On-line Topic:Did you undertake a Institute Induction? 555 non-null object 57 InductionInfo. Induction Manual Topic:Did you undertake a Institute Induction? 553 non-null object 58 InductionInfo. Face to Face Topic: Did you undertake Team Induction; Category? 555 non-null object 59 InductionInfo. On-line Topic: Did you undertake Team Induction?process you undertook and how it was conducted.] 555 non-null object 60 InductionInfo. Induction Manual Topic: Did you undertake Team Induction? 555 non-null object 61 Workplace. Topic:Did you and your Manager develop a Performance and Professional Development Plan (PPDP)? 608 non-null object 62 Workplace. Topic:Does your workplace promote a work culture free from all forms of unlawful discrimination? 594 non-null object 63 Workplace. Topic:Does your workplace promote and practice the principles of employment equity? 587 non-null object 64 Workplace. Topic:Does your workplace value the diversity of its employees? 586 non-null object 65 Workplace. Topic:Would you recommend the Institute as an employer to others? 581 non-null object 66 Gender. What is your Gender? 596 non-null object 67 CurrentAge. Current Age 596 non-null object 68 Employment Type. Employment Type 596 non-null object 69 Classification. Classification 596 non-null object 70 LengthofServiceOverall. Overall Length of Service at Institute (in years) 596 non-null object 71 LengthofServiceCurrent. Length of Service at current workplace (in years) 596 non-null object dtypes: float64(2), object(70) memory usage: 395.0+ KB
df_dete.isna().sum()
ID 0 SeparationType 0 Cease Date 0 DETE Start Date 0 Role Start Date 0 Position 5 Classification 367 Region 0 Business Unit 696 Employment Status 5 Career move to public sector 0 Career move to private sector 0 Interpersonal conflicts 0 Job dissatisfaction 0 Dissatisfaction with the department 0 Physical work environment 0 Lack of recognition 0 Lack of job security 0 Work location 0 Employment conditions 0 Maternity/family 0 Relocation 0 Study/Travel 0 Ill Health 0 Traumatic incident 0 Work life balance 0 Workload 0 None of the above 0 Professional Development 14 Opportunities for promotion 87 Staff morale 6 Workplace issue 34 Physical environment 5 Worklife balance 7 Stress and pressure support 12 Performance of supervisor 9 Peer support 10 Initiative 9 Skills 11 Coach 55 Career Aspirations 76 Feedback 30 Further PD 54 Communication 8 My say 10 Information 6 Kept informed 9 Wellness programs 56 Health & Safety 29 Gender 24 Age 11 Aboriginal 806 Torres Strait 819 South Sea 815 Disability 799 NESB 790 dtype: int64
df_tafe.isna().sum()
Record ID 0 Institute 0 WorkArea 0 CESSATION YEAR 7 Reason for ceasing employment 1 ... CurrentAge. Current Age 106 Employment Type. Employment Type 106 Classification. Classification 106 LengthofServiceOverall. Overall Length of Service at Institute (in years) 106 LengthofServiceCurrent. Length of Service at current workplace (in years) 106 Length: 72, dtype: int64
The data in Dete Survey contains several missing values,but instead given as - 'Not Stated', This infact is NaN. Thus to rectify this, the dataset is read into pandas
again with the na_values
parameter set to 'Not Stated'. This will convert every occurence of 'Not Stated' to NaN value.
df_dete = pd.read_csv('dete_survey.csv',na_values='Not Stated')
After analyzing the column names in the Dete Survey dataset the columns Professional Development to Health & Safety (28:49) are not required for the analysis. This data is general employee survey data regarding the employee's engagement with the company on the Likert scale. Since the purpose is to find dissatisfied employees and which employee is likely to report dissatisafaction, the engagement of employee with the institute is not relevant for now.
df_dete.iloc[:,28:49].head(5)
df_dete.drop(columns=df_dete.columns[28:49],axis=1,inplace=True)
On similar lines, the Tafe Survey dataset contains the columns Main Factor. Which of these was the main factor for leaving? to Workplace. Topic:Would you recommend the Institute as an employer to others? (17:66) are irrelevant to the analysis, as this data is of employee's engagement with the institute on the Likert scale.
df_tafe.iloc[:,17:66].head(5)
df_tafe.drop(columns=df_tafe.columns[17:66],axis=1,inplace=True)
For the ease of analysis and eventually combining both datasets for inference later, the columns names are cleaned made uniform across both the datasets (the columns common between the two).
columns = df_dete.columns.str.replace(" ","_").str.lower().str.strip()
df_dete.columns = columns
new_name = {
'Record ID': 'id',
'CESSATION YEAR': 'cease_date',
'Reason for ceasing employment': 'separationtype',
'Gender. What is your Gender?': 'gender',
'CurrentAge. Current Age': 'age',
'Employment Type. Employment Type': 'employment_status',
'Classification. Classification': 'position',
'LengthofServiceOverall. Overall Length of Service at Institute (in years)': 'institute_service',
'LengthofServiceCurrent. Length of Service at current workplace (in years)': 'role_service'
}
df_tafe.rename(new_name,inplace=True,axis=1)
The separationtype column in both the datasets holds the reason why an employee left the institute. For the given analysis, the label 'Resignation' only is relevant, since we are interested in employees who resigned due to dissatisfaction. The datasets are reduced to only employees who have resigned.
df_dete.separationtype.value_counts()
Age Retirement 285 Resignation-Other reasons 150 Resignation-Other employer 91 Resignation-Move overseas/interstate 70 Voluntary Early Retirement (VER) 67 Ill Health Retirement 61 Other 49 Contract Expired 34 Termination 15 Name: separationtype, dtype: int64
df_tafe.separationtype.value_counts()
Resignation 340 Contract Expired 127 Retrenchment/ Redundancy 104 Retirement 82 Transfer 25 Termination 23 Name: separationtype, dtype: int64
df_dete = df_dete[df_dete.separationtype.str.contains('Resignation')].copy()
df_tafe = df_tafe[df_tafe.separationtype == 'Resignation'].copy()
The cease_date column indicates the last date the employee worked for or basically resignation date. These columns are in both the sets and hence have to be made uniform.
All NaN values are removed and only the year is maintained rather than the exact date or month.
df_tafe.cease_date.value_counts()
2011.0 116 2012.0 94 2010.0 68 2013.0 55 2009.0 2 Name: cease_date, dtype: int64
df_dete.cease_date.value_counts(dropna=False)
2012 126 2013 74 01/2014 22 12/2013 17 06/2013 14 09/2013 11 NaN 11 11/2013 9 07/2013 9 10/2013 6 08/2013 4 05/2013 2 05/2012 2 2010 1 07/2012 1 09/2010 1 07/2006 1 Name: cease_date, dtype: int64
df_dete = df_dete[~df_dete.cease_date.isna()]
def clean_date(row):
if '/' in row:
return row.split('/')[1]
else:
return row
df_dete.cease_date = df_dete.cease_date.apply(clean_date).astype(float)
df_dete.cease_date.value_counts()
2013.0 146 2012.0 129 2014.0 22 2010.0 2 2006.0 1 Name: cease_date, dtype: int64
The end_date and start date columns give the period the employee has worked for the insitute. The assumption can be made - The start dates shouldn't be greater than current date and the start date shouldn't be previous to 1970 given that people are usually employed in their 20s.
Future dates are obviously out of question and for dates previous to 1970 means that given the person was employed in 20s the current age of the person would be 70+, which is usually the retirement age. For these reasons, the lower limit has been set to 1970.
A box plot is a good way to catch outiers if any and to view general distribution of the data.
plt.figure(figsize=(12,8))
sns.set_style('white')
sns.boxplot(df_dete.dete_start_date)
plt.xlabel('Start Year')
plt.title("Start Year Distribution")
plt.gca().spines['left'].set_visible(False)
plt.gca().spines['top'].set_visible(False)
plt.gca().spines['right'].set_visible(False)
The distribution contains one entry with the year 1963 which is previous to the stipulated lower bound. Hence the removal of the outlier.
df_dete = df_dete[~(df_dete.dete_start_date == 1963.0)]
The Tafe Survey dataset's column institute_service describes the service years of employees. This column does not exist for the Dete Survery dataset.
To deduce this column, 'cease_date' and 'dete_start_date' can be used. The subtraction of the two columns results in the length of service of an employee. The rows with null values are dropped for convenience.
df_tafe.institute_service.value_counts(dropna=False)
Less than 1 year 73 1-2 64 3-4 63 NaN 50 5-6 33 11-20 26 7-10 21 More than 20 years 10 Name: institute_service, dtype: int64
df_dete = df_dete[~df_dete.dete_start_date.isna()]
df_dete['institute_service'] = abs(df_dete.cease_date - df_dete.dete_start_date)
The purpose of the project is to understand which employees are dissatisfied. For this we have the following column relevant to us :
In the Tafe Survey dataset -
Contributing Factors. Dissatisfaction
Contributing Factors. Job Dissatisfaction
In the Dete Survey dataset -
job_dissatisfaction
dissatisfaction_with_the_department
physical_work_environment
lack_of_recognition
lack_of_job_security
work_location
employment_conditions
work_life_balance
workload
From the above columns, we can infer that each column talks about dissatisafaction of an employee. Any one of them being true indicates the employee has resigned (already filtered) due to dissatisfaction of some sorts. A new column is created in both the datasets indicating afore-mentioned employee.
df_tafe[['Contributing Factors. Dissatisfaction'
,'Contributing Factors. Job Dissatisfaction']]
Contributing Factors. Dissatisfaction | Contributing Factors. Job Dissatisfaction | |
---|---|---|
3 | - | - |
4 | - | - |
5 | - | - |
6 | - | - |
7 | - | - |
... | ... | ... |
696 | - | - |
697 | - | - |
698 | - | - |
699 | - | - |
701 | - | - |
340 rows × 2 columns
The '-' value in these columns simply stands for not answered which can be concluded as False
. Similarly, if the question was answered it indicates that there was dissatisfaction related to the employement. Hence for ease of compiling the data and making the afore-mentioned column, these columns will be cleaned and converted to boolean.
df_dete[[
'job_dissatisfaction',
'dissatisfaction_with_the_department',
'physical_work_environment',
'lack_of_recognition',
'lack_of_job_security',
'work_location',
'employment_conditions',
'work_life_balance',
'workload'
]]
job_dissatisfaction | dissatisfaction_with_the_department | physical_work_environment | lack_of_recognition | lack_of_job_security | work_location | employment_conditions | work_life_balance | workload | |
---|---|---|---|---|---|---|---|---|---|
3 | False | False | False | False | False | False | False | False | False |
5 | False | False | False | False | False | False | True | False | False |
8 | False | False | False | False | False | False | False | False | False |
9 | True | True | False | False | False | False | False | False | False |
11 | False | False | False | False | False | False | False | False | False |
... | ... | ... | ... | ... | ... | ... | ... | ... | ... |
807 | False | True | False | False | False | False | False | True | False |
808 | False | False | False | False | False | False | False | False | False |
815 | False | False | False | False | False | False | False | False | False |
816 | False | False | False | False | False | False | False | False | False |
819 | False | False | False | False | False | False | False | True | False |
272 rows × 9 columns
print(df_tafe['Contributing Factors. Dissatisfaction'].value_counts(dropna=False))
def clean_factors(row):
if row == '-':
return False
elif pd.isnull(row):
return np.NaN
else:
return True
df_tafe['Contributing Factors. Dissatisfaction'] = df_tafe['Contributing Factors. Dissatisfaction'].apply(clean_factors)
- 277 Contributing Factors. Dissatisfaction 55 NaN 8 Name: Contributing Factors. Dissatisfaction, dtype: int64
print(df_tafe['Contributing Factors. Job Dissatisfaction'].value_counts(dropna=False))
df_tafe['Contributing Factors. Job Dissatisfaction'] = df_tafe['Contributing Factors. Job Dissatisfaction'].apply(clean_factors)
- 270 Job Dissatisfaction 62 NaN 8 Name: Contributing Factors. Job Dissatisfaction, dtype: int64
Using the DataFrame.any()
function, the columns are compiled along the rows i.e. if any value along a row for these columns is True
, then the dissatisfied column takes a True
value.
df_dete['dissatisfied'] = df_dete[[
'job_dissatisfaction',
'dissatisfaction_with_the_department',
'physical_work_environment',
'lack_of_recognition',
'lack_of_job_security',
'work_location',
'employment_conditions',
'work_life_balance',
'workload'
]].any(axis=1,skipna=False)
df_tafe['dissatisfied'] = df_tafe[[
'Contributing Factors. Dissatisfaction',
'Contributing Factors. Job Dissatisfaction'
]].any(axis=1,skipna=False)
The cleaning and identification of dissatisfied employees is concluded for both the datasets individually. For further analysis, the datasets need to be combined, to find a generalized trend in terms of dissatisfaction.
To differentiate between the rows of the two datasets - df_dete and df_tafe, a new column is created, identifying the institute.
df_dete['institute'] = 'DETE'
df_tafe['institute'] = 'TAFE'
The institute_service columns in both the datasets do not match. In the DETE dataset, these values are on the interval scale, where as for the TAFE column these are on an ordinal scale. For uniformity, the DETE dataset column is converted to an ordinal scale with the labels :
These labels are derived from the TAFE dataset.
bins = pd.IntervalIndex.from_tuples([
(-1,0),(1,2),(3,4),(5,6),(7,10),(11,20),(21,100)
],
closed='both'
)
tmp = pd.cut(
x=df_dete.institute_service,
bins=bins
)
def assign_labels(row):
if row == pd.Interval(0,1,closed='both'):
return 'Less than 1 year'
elif row == pd.Interval(1,2,closed='both'):
return '1-2'
elif row == pd.Interval(3,4,closed='both'):
return '3-4'
elif row == pd.Interval(5,6,closed='both'):
return '5-6'
elif row == pd.Interval(7,10,closed='both'):
return '7-10'
elif row == pd.Interval(11,20,closed='both'):
return '11-20'
else:
return 'More than 20 years'
df_dete.institute_service = tmp.apply(assign_labels)
The datasets are finally uniform in terms of the common column. Since the relevant columns for the analysis have been cleaned or derived, all other columns are irrelevant now and hence are removed before joining the datasets.
The relevant columns are:-
relevant_cols = ['institute_service','gender','age','employment_status',
'position','cease_date','dissatisfied','id',
'separationtype','institute']
df_tafe = df_tafe[relevant_cols]
df_dete = df_dete[relevant_cols]
df = pd.concat([df_dete,df_tafe])
df.head(5)
institute_service | gender | age | employment_status | position | cease_date | dissatisfied | id | separationtype | institute | |
---|---|---|---|---|---|---|---|---|---|---|
3 | 7-10 | Female | 36-40 | Permanent Full-time | Teacher | 2012.0 | False | 4.0 | Resignation-Other reasons | DETE |
5 | 11-20 | Female | 41-45 | Permanent Full-time | Guidance Officer | 2012.0 | True | 6.0 | Resignation-Other reasons | DETE |
8 | 3-4 | Female | 31-35 | Permanent Full-time | Teacher | 2012.0 | False | 9.0 | Resignation-Other reasons | DETE |
9 | 11-20 | Female | 46-50 | Permanent Part-time | Teacher Aide | 2012.0 | True | 10.0 | Resignation-Other employer | DETE |
11 | 3-4 | Male | 31-35 | Permanent Full-time | Teacher | 2012.0 | False | 12.0 | Resignation-Move overseas/interstate | DETE |
The datasets are concatenated to form one dataset with the columns afore mentioned. The institute_service column is uniform, but not exactly intuitive, hence the column is further binned and categorized into the following :
New: Less than 3 years at a company
Experienced: 3-6 years at a company
Established: 7-10 years at a company
Veteran: 11 or more years at a company
These categories are intuitive and easy to understand. The previous labels did show the limits and hence conveyed more data but in terms of analysis, that is very granular and a technical jargon.
NOTE : The service length (category) of an employee does not infer the age of the employee. The employee can be New (recently joined the institute) and yet be quite old in terms of age.
def service_catgs(row):
if row in ['Less than 1 year','1-2']:
return 'New'
elif row in ['3-4','5-6']:
return 'Experienced'
elif row == '7-10':
return 'Established'
elif pd.isnull(row):
return np.NaN
else:
return 'Veteran'
df['service_catg'] = df.institute_service.apply(service_catgs)
The questions to be answered via this analysis were:-
Are employees who only worked for the institutes for a short period of time resigning due to some kind of dissatisfaction? What about employees who have been there longer?
Are younger employees resigning due to some kind of dissatisfaction? What about older employees?
For the analysis we have coagulated the service lengths for each employee into a column service_catg. This column is now compared with the previously derived column dissatisfied. The end goal is to understand which serive category in general shows dissapointment in the employement and hence resigned.
print(df.dissatisfied.value_counts(dropna=False))
df.dissatisfied.fillna(False,inplace=True)
False 376 True 228 NaN 8 Name: dissatisfied, dtype: int64
catg_percent= df.pivot_table(
values='dissatisfied',
index='service_catg'
)
catg_percent.reset_index(inplace=True)
catg_percent = catg_percent.iloc[[2,1,0,3]]
catg_percent
service_catg | dissatisfied | |
---|---|---|
2 | New | 0.265896 |
1 | Experienced | 0.343023 |
0 | Established | 0.516129 |
3 | Veteran | 0.496774 |
The pivot_table
function is used with the default numpy.meam
as the aggregate function. This grouped the data by service_catg and aggregated the disstatisfied column. Since the dissatisfied is boolean, the mean is nothing but the proportion of True
values for that group.
plt.figure(figsize=(12,7))
sns.set_style('white')
splt = sns.barplot(
x='service_catg',
y='dissatisfied',
data=catg_percent,
color='skyblue'
)
plt.yticks([])
plt.xlabel("Service Categories")
plt.ylabel("Dissatisfaction percentage")
plt.title("Dissatisfaction in Service Categories")
for loc in ['left','right','top']:
plt.gca().spines[loc].set_visible(False)
for p in splt.patches:
splt.annotate(format(p.get_height(),'.3f'),
(p.get_x()+p.get_width()/2,p.get_height()+0.01),
ha='center',
va='center'
)
Plotting the resulting proportions obtained, the following conclusion can be drawn:-
The conclusion made above categorizes which employee is more likely to be dissatisfied and resign. The service lenghts are just one aspect of it. There are various aspects to an employee. One such aspect is the age, analogous to service categories. Using the age in a similar way as service category, the aim is to find categories that more likely to be dissatisfied and resign.
The age column is on the ordinal scale but the labels are intervals and less inuitive. To make the comparision easier, the age is converted to the labels given below :
Young: Aged 20 or younger to 30
Middle: Aged 31 to 45
Senior: Aged 46 to 55
Elder: Aged 56 or older
These catgories are intuitive and are easier to compare than the previous labels
df.age.value_counts(dropna=False)
51-55 69 NaN 52 41 45 45 41-45 44 46 50 39 36-40 36 46-50 34 21 25 33 36 40 32 26 30 32 31 35 32 26-30 31 56 or older 29 31-35 29 21-25 26 56-60 22 61 or older 17 20 or younger 10 Name: age, dtype: int64
df.age = df.age.str.replace(" ","-")
df.age = df.age.str.replace("56 or older","56-60")
def age_catg(row):
if row in ['20 or younger','21-25','26-30']:
return 'Young'
elif row in ['31-35','36-40','41-45']:
return 'Middle'
elif row in ['46-50','51-55']:
return 'Senior'
elif pd.isna(row):
return np.NaN
else:
return 'Elder'
df['age_catg'] = df.age.apply(age_catg)
print(df.age_catg.value_counts(dropna=False))
Middle 218 Senior 142 Young 132 Elder 68 NaN 52 Name: age_catg, dtype: int64
The age still contains NaN values. These values cannot be imputed from any available data.
Similar to service_catg, the pivot_table
function is used on the age and dissatisfied columns to retrieve the proportions of dissatisfied reignations amongst employees for each age category.
age_catg_percent = df.pivot_table(index='age_catg',values='dissatisfied')
age_catg_percent.reset_index(inplace=True)
age_catg_percent = age_catg_percent.iloc[[3,1,2,0]]
age_catg_percent
age_catg | dissatisfied | |
---|---|---|
3 | Young | 0.340909 |
1 | Middle | 0.376147 |
2 | Senior | 0.408451 |
0 | Elder | 0.426471 |
plt.figure(figsize=(12,7))
sns.set_style('white')
splt = sns.barplot(
x='age_catg',
y='dissatisfied',
data=age_catg_percent,
color='skyblue'
)
plt.yticks([])
plt.xlabel("Age Categories")
plt.ylabel("Dissatisfaction percentage")
plt.title("Dissatisfaction in Age Categories")
for loc in ['left','right','top']:
plt.gca().spines[loc].set_visible(False)
for p in splt.patches:
splt.annotate(format(p.get_height(),'.3f'),
(p.get_x()+p.get_width()/2,p.get_height()+0.01),
ha='center',
va='center'
)
#splt.annote("bar text",(loc_x,loc_y),ha(horizontal allign),va(vertical allign))
The following conclusions can be made from the resulting plot:-
Uptil now the analysis has focused on the mainly the temporal aspects of the employee. Since there are two institutes under analysis, the comparision between the two institutes in terms of having dissatiesfied employees can give a peak at how the institute engages with its employees.
Between the two institutions - DETE and TAFE, using pivot_table
function, proportion of dissatisfied employees is found.
df.institute.value_counts(dropna=False)
TAFE 340 DETE 272 Name: institute, dtype: int64
institute_catg = df.pivot_table(index='institute',values='dissatisfied')
institute_catg.reset_index(inplace=True)
institute_catg
institute | dissatisfied | |
---|---|---|
0 | DETE | 0.503676 |
1 | TAFE | 0.267647 |
plt.figure(figsize=(12,7))
sns.set_style('white')
splt = sns.barplot(
x='institute',
y='dissatisfied',
data=institute_catg,
color='skyblue'
)
plt.yticks([])
plt.xlabel("Institute")
plt.ylabel("Dissatisfaction percentage")
plt.title("Dissatisfaction in Institutions")
for loc in ['left','right','top']:
plt.gca().spines[loc].set_visible(False)
for p in splt.patches:
splt.annotate(format(p.get_height(),'.3f'),
(p.get_x()+p.get_width()/2,p.get_height()+0.01),
ha='center',
va='center'
)
#splt.annote("bar text",(loc_x,loc_y),ha(horizontal allign),va(vertical allign))
From the results, the plot concludes :-
The employee_status which describes the kind of employment can be analyzed for the dissatisafaction the categories are made simpler as -
Permanent: Permanent Full-time / Part-time
Temporary: Temporary Full-time / Part-time
Casual: Contract / Casual
The results of this analysis would convey, which of these category employees are likely to resign due to dissatisfaction.
df.employment_status.value_counts(dropna=False)
Permanent Full-time 244 Permanent Part-time 130 Temporary Full-time 120 NaN 50 Temporary Part-time 35 Contract/casual 29 Casual 4 Name: employment_status, dtype: int64
def emp_status(row):
if row in ['Permanent Full-time','Permanent Part-time']:
return 'Permanent'
elif row in ['Temporary Full-time','Temporary Part-time']:
return 'Temporary'
elif pd.isna(row):
return np.NaN
else:
return "Casual"
df['employment_catg'] = df.employment_status.apply(emp_status)
emp_status_catg = df.pivot_table(
index='employment_catg',
values='dissatisfied'
)
emp_status_catg.reset_index(inplace=True)
emp_status_catg
employment_catg | dissatisfied | |
---|---|---|
0 | Casual | 0.181818 |
1 | Permanent | 0.459893 |
2 | Temporary | 0.232258 |
plt.figure(figsize=(12,7))
sns.set_style('white')
splt = sns.barplot(
x='employment_catg',
y='dissatisfied',
data=emp_status_catg,
color='skyblue'
)
plt.yticks([])
plt.xticks(rotation=0)
plt.xlabel("Employment Type")
plt.ylabel("Dissatisfaction percentage")
plt.title("Dissatisfaction vs Employment Type")
for loc in ['left','right','top']:
plt.gca().spines[loc].set_visible(False)
for p in splt.patches:
splt.annotate(format(p.get_height(),'.3f'),
(p.get_x()+p.get_width()/2,p.get_height()+0.01),
ha='center',
va='center'
)
#splt.annote("bar text",(loc_x,loc_y),ha(horizontal allign),va(vertical allign))
The conclusion from the resulting plot:-
The next step in analysis, is only to get an idea of sorts whether dissatisfaction is the driving criterion for attrition in these institutes.
dissatisfied_employees = df.dissatisfied.value_counts().reset_index()
dissatisfied_employees['index'] = pd.Series(['Not Dissatisfied','Dissatisfied'])
dissatisfied_employees.set_index('index',inplace=True)
dissatisfied_employees
dissatisfied | |
---|---|
index | |
Not Dissatisfied | 384 |
Dissatisfied | 228 |
plt.figure(
figsize=(12,6),
FigureClass=Waffle,
rows=1,
columns=5,
values=dissatisfied_employees.to_dict()['dissatisfied'],
legend={'loc': 'upper left', 'bbox_to_anchor': (1.1, 1)},
icons='child',
font_size=50,
title={'label': 'Resignation due to dissatisfaction per 5 employees', 'loc': 'center'}
)
plt.show()
The waffle plot shows the number of resignations due to dissatisfaction per 5 people in the survey. 2 out of 5 people resign due to dissatisfaction. Conclusions drawn are :-
The analysis comes to an end here. The final conclusions drawn from the project are :-