As many of you probably know, being a data scientist can require a pretty large skill set . . .
<img src="DSMap.png", width = 1000, height = 800> Credit: Swami Chandrasekaran
To master all of that at a high level would probably take a lifetime! I'm sure many data scientists would love to be highly skilled in all of these areas if possible, but busy Ph.D. students like me (that are trying to graduate in an efficient manner!) don't have enough time to focus on all of these skills.
So which of these skills are most employers actually looking for?
On DataScienceCentral, I saw a link to this posting by Kumaran Ponnambalam where he examined this question with R by scraping job postings. I thought it was an interesting idea. I wanted to find out this information for myself, as updated as possible, so I could see any changing trends in the job market.
It would also be nice to see if different cities have different skills they like to emphasize (i.e. does the Silicon Valley market have different skills they prefer compared to New York City?)
Data like this isn't going to be easily accessible through a .csv or database. Sometimes, you are going to have to get it yourself. That may require web scraping, which automates the process of collecting data from websites. I've always thought this sounded very cool, but I didn't know how to do it.
Luckily, Greg Reda at Datascope Analytics had a great blog post about web scraping that helped me complete this project (see it here). He did a great job! I'm not going to go into as much detail about web scraping as he did in this notebook, so I would recommend going to his blog post if you want to learn the basics.
So, in this notebook, I am going to scrape job postings from Indeed.com for data science jobs and see which skills employers want the most (Python or R? Are they interested in Spark yet? How dominant are NoSQL databases? Are they using proprietary software like SAS or are companies preferring open source now?) To make it even better, I will create the program so that I can have a detailed breakdown by city.
The basic workflow of the program will be:
We will create two functions. The first will scape an individual job posting for the HTML, clean it up to get the words only, then output the final list of words. The second will manage which URLs to access via the job postings Indeed's website links to.
In this notebook, we will use the urllib2 library to connect to the websites, the BeautifulSoup library to collect all of the HTML, the re library for parsing the words and filtering out other markup based on regular expressions, and pandas to manage and plot the final results.
This function will be called every time we access a new job posting. Its input is a URL for a website, while the output will be a final set of words collected from that website.
In this notebook, I am going to import all of the libraries necessary first.
from bs4 import BeautifulSoup # For HTML parsing import urllib2 # Website connections import re # Regular expressions from time import sleep # To prevent overwhelming the server between connections from collections import Counter # Keep track of our term counts from nltk.corpus import stopwords # Filter out stopwords, such as 'the', 'or', 'and' import pandas as pd # For converting results to a dataframe and bar chart plots %matplotlib inline
Now create our first website parsing function.
def text_cleaner(website): ''' This function just cleans up the raw html so that I can look at it. Inputs: a URL to investigate Outputs: Cleaned text only ''' try: site = urllib2.urlopen(website).read() # Connect to the job posting except: return # Need this in case the website isn't there anymore or some other weird connection problem soup_obj = BeautifulSoup(site) # Get the html from the site if len(soup_obj) == 0: # In case the default parser lxml doesn't work, try another one soup_obj = BeautifulSoup(site, 'html5lib') for script in soup_obj(["script", "style"]): script.extract() # Remove these two elements from the BS4 object text = soup_obj.get_text() # Get the text from this lines = (line.strip() for line in text.splitlines()) # break into lines chunks = (phrase.strip() for line in lines for phrase in line.split(" ")) # break multi-headlines into a line each text = ''.join(chunk for chunk in chunks if chunk).encode('utf-8') # Get rid of all blank lines and ends of line # Now clean out all of the unicode junk (this line works great!!!) try: text = text.decode('unicode_escape').encode('ascii', 'ignore') # Need this as some websites aren't formatted except: # in a way that this works, can occasionally throw return # an exception text = re.sub("[^a-zA-Z+3]"," ", text) # Now get rid of any terms that aren't words (include 3 for d3.js) # Also include + for C++ text = re.sub(r"([a-z])([A-Z])", r"\1 \2", text) # Fix spacing issue from merged words text = text.lower().split() # Go to lower case and split them apart stop_words = set(stopwords.words("english")) # Filter out any stop words text = [w for w in text if not w in stop_words] text = list(set(text)) # Last, just get the set of these. Ignore counts (we are just looking at whether a term existed # or not on the website) return text
As you can see in the code above, a lot of cleaning for the raw html is necessary to get the final terms we are looking for. It extracts the relevant portions of the html, gets the text, removes blank lines and line endings, removes unicode, and filters with regular expressions to include only words. To see what the final result looks like, let's try calling this function on a sample job posting. The one I am using is a job posting for a Data Scientist at Indeed itself!
If you are reading the notebook interactively, the example job posting may have disappeared so you can try your own to see how the function works.
sample = text_cleaner('http://www.indeed.com/viewjob?jk=5505e59f8e5a32a4&q=%22data+scientist%22&tk=19ftfgsmj19ti0l3&from=web&advn=1855944161169178&sjdu=QwrRXKrqZ3CNX5W-O9jEvWC1RT2wMYkGnZrqGdrncbKqQ7uwTLXzT1_ME9WQ4M-7om7mrHAlvyJT8cA_14IV5w&pub=pub-indeed') sample[:20] # Just show the first 20 words
['gd', 'code', 'cj', 'indeed', 'competitive', 'detail', 'label', 'scientist', 'frequency', 'per', 'human', 'keywords', 'follow', 'alt', 'viewthroughconversion', 'decisions', 'find', 'looking', 'style', 'retirement']
Now that we can extract terms from the website with our text_cleaner function, let's build another function that will call this and automatically loop through all of the websites on Indeed for us.
This next function will allow us to search for "data scientist" jobs in a particular city (or nationally if we want to see everything!) and plot the final results in a bar chart so we can see which skills are most frequently desired.
This second function is fairly long, so I will try to explain how everything works through a lot of commentary. The basic idea is to look through Indeed's pages of job results and click on all of the job links, but only in the center of the page where all of the jobs are posted (not on the edges). See an example here. I just want the URLs in the "center" column of the website. You can get an idea of how Indeed organized the website by using a browser like Firefox or Chrome. Right click on the page to see the "Inspect Element" option in Firefox and the HTML will now be visible to you.
Let's now try running our new function on Seattle, Washington to see what results we get. Just as a note, all of these results were run on March 8, 2015 (with the exception of the national results that were run the next day).
seattle_info = skills_info(city = 'Seattle', state = 'WA')
There were 73 jobs found, Seattle Getting page 1 Getting page 2 Getting page 3 Getting page 4 Getting page 5 Getting page 6 Getting page 7 Done with collecting the job postings! There were 74 jobs successfully found.
Looking at the plot above, it seems Python is definitely the most commonly requested skill. Relational databases are used far more commonly than unstructured NoSQL databases. Spark doesn't seem to have caught on to very many jobs yet (only about 7%) but it is listed more frequently than Pig, which has been around much longer! It also seems R is starting to lose the infamous Python vs. R battle that data scientists like arguing about (in my opinion just use both for their strengths!)
What if we tried a different job market, such as Chicago? How do things change here?
chicago_info = skills_info(city = 'Chicago', state = 'IL')
There were 63 jobs found, Chicago Getting page 1 Getting page 2 Getting page 3 Getting page 4 Getting page 5 Getting page 6 Done with collecting the job postings! There were 68 jobs successfully found.
In the case of Chicago, it seems Hadoop is the top skill to have. My guess is some of the companies list Hadoop without really understanding that Hadoop is an entire framework and not a specific "skill." Python has now dropped to third place compared to Seattle. It also seems Spark appears in a greater percentage of job ads from Chicago than it did in Seattle, but Pig is requested more often.
What about the largest Data Scientist job markets like San Francisco and New York City?
silicon_val_info = skills_info(city = 'San Francisco', state = 'CA')
There were 299 jobs found, San Francisco Getting page 1 Getting page 2 Getting page 3 Getting page 4 Getting page 5 Getting page 6 Getting page 7 Getting page 8 Getting page 9 Getting page 10 Getting page 11 Getting page 12 Getting page 13 Getting page 14 Getting page 15 Getting page 16 Getting page 17 Getting page 18 Getting page 19 Getting page 20 Getting page 21 Getting page 22 Getting page 23 Getting page 24 Getting page 25 Getting page 26 Getting page 27 Getting page 28 Getting page 29 Done with collecting the job postings! There were 336 jobs successfully found.
Once again, Python is the top skill demanded, with R in second. In terms of the Hadoop framework, Hive was the most in demand. This makes sense given most companies are still dealing with structured data and Hive's strong similarity to SQL. Spark also beat Excel (thank goodness!) yet the percentage of job descriptions mentioning Spark was smaller in the Bay Area than in Chicago. A bit surprising since Spark originated at Berkeley, but this is just a snapshot in time after all.
nyc_info = skills_info(city = 'New York', state = 'NY')
There were 300 jobs found, New York Getting page 1 Getting page 2 Getting page 3 Getting page 4 Getting page 5 Getting page 6 Getting page 7 Getting page 8 Getting page 9 Getting page 10 Getting page 11 Getting page 12 Getting page 13 Getting page 14 Getting page 15 Getting page 16 Getting page 17 Getting page 18 Getting page 19 Getting page 20 Getting page 21 Getting page 22 Getting page 23 Getting page 24 Getting page 25 Getting page 26 Getting page 27 Getting page 28 Getting page 29 Getting page 30 Done with collecting the job postings! There were 335 jobs successfully found.
New York City and San Francisco had almost exactly the same number of job postings, which seems to show the job market is relatively balanced between the two coasts. R finally comes in first place in the Big Apple, with Python close behind. Demand for Spark seems pretty limited still, however. I think Spark has a bright future, so this may change by the end of the year.
Last, let's see the overall national trend. (I ran this separately because it takes a long time to run).
From a national standpoint, I notice a few interesting things:
Python is more in demand than R now for data science jobs. I think this will continue to be the trend as the Python data science community is further developed. Python is the "glue" that can hold almost every aspect of data science together. From data manipulation in pandas, machine learning with scikit-learn, web applications with Flash/Django, and an interface to Spark via PySpark, Python probably has you covered somewhere along the way. Is it always the best tool for the job? No, but it can usually get you there. R is still important in some areas where Python isn't quite as good yet (such as plotting or statistical libraries). With Microsoft buying Revolution Analytics recently as an example, R still has a role to play which is why it is in second place.
Data visualization doesn't seem to be in very high demand yet. D3 (or D3.js depending on the terminology the company used) is very low on the list. This surprised me because D3 seems to have a lot of power, yet it doesn't seem to have won many companies over as a skill they value. Tableau is still more popular, even though it isn't open source.
Even nationally, Spark is now (at least for this snapshot in time) a more in-demand skill than Excel. Perhaps companies are starting to realize that Excel isn't exactly the best tool for data science at scale (sorry Microsoft!) and is better suited for a Data Analyst role.
Designing our own web crawler, we were able to find out which skills companies are most interested in for data science jobs on Indeed. Trends vary somewhat between cities, but the top four skills were Python, R, SQL, and Hadoop pretty consistently. Java, SAS, and Hive would probably be considered the second tier.
Notice, however, that these skills are just for the software/programming languages that a data scientist should know. There are many others, such as machine learning, statistics, mathematics, and an understanding of what insights or products can be helpful to a business. These are also very important skills that are necessary to be a successful data scientist in my opinion, so don't forget about them!