In this project, we'll work with a data set of submissions to popular technology site Hacker News.
We are will analyze this data set, but note that it has been reduced from almost 300,000 rows to approximately 20,000 rows by removing all submissions that did not receive any comments, and then randomly sampling from the remaining submissions.
We're specifically interested in posts whose titles begin with either Ask HN
or Show HN
.
Ask HN
- post where users ask questionsShow HN
- post where users show a project, product, etc.We'll compare these two types of posts to determine the following:
After analyzing the data, we made list of top5 hourse you wil recieved most comments to your posts. The best hour is 15:00 EST.
Let's start by importing the libraries we need and reading the data set into a list of lists.
# Read in the data
from csv import reader
opened_file = open('hacker_news.csv')
read_file = reader(opened_file)
hn = list(read_file)
header = hn[0] # column names
hn = hn[1:] # data set
# Quick exploration of the data
print(header)
print('\n')
print(hn[:5])
['id', 'title', 'url', 'num_points', 'num_comments', 'author', 'created_at'] [['12224879', 'Interactive Dynamic Video', 'http://www.interactivedynamicvideo.com/', '386', '52', 'ne0phyte', '8/4/2016 11:52'], ['10975351', 'How to Use Open Source and Shut the Fuck Up at the Same Time', 'http://hueniverse.com/2016/01/26/how-to-use-open-source-and-shut-the-fuck-up-at-the-same-time/', '39', '10', 'josep2', '1/26/2016 19:30'], ['11964716', "Florida DJs May Face Felony for April Fools' Water Joke", 'http://www.thewire.com/entertainment/2013/04/florida-djs-april-fools-water-joke/63798/', '2', '1', 'vezycash', '6/23/2016 22:20'], ['11919867', 'Technology ventures: From Idea to Enterprise', 'https://www.amazon.com/Technology-Ventures-Enterprise-Thomas-Byers/dp/0073523429', '3', '1', 'hswarna', '6/17/2016 0:01'], ['10301696', 'Note by Note: The Making of Steinway L1037 (2007)', 'http://www.nytimes.com/2007/11/07/movies/07stein.html?_r=0', '8', '2', 'walterbell', '9/30/2015 4:12']]
Since we're only concerned with post titles beginning with Ask HN or Show HN, we'll create new lists of lists containing just the data for those titles.
To find the posts that begin with either Ask HN or Show HN, we'll use the string method startswith
and than add them to three diffrent lists.
ask_posts = []
show_posts = []
other_posts = []
for row in hn:
title = row[1]
if title.lower().startswith('ask hn'): # method lower. to make tittle lowercase
ask_posts.append(row)
elif title.lower().startswith('show hn'):
show_posts.append(row)
else:
other_posts.append(row)
print('Number of posts in ASK HN:', len(ask_posts))
print('\n')
print('Number of posts in SHOW HN:', len(show_posts))
print('\n')
print('Other posts:', len(other_posts))
Number of posts in ASK HN: 1744 Number of posts in SHOW HN: 1162 Other posts: 17194
Next, let's determine if ask posts or show posts receive more comments on average.
total_ask_comments = 0
for row in ask_posts:
num_comments = int(row[4])
total_ask_comments += num_comments
avg_ask_comments = total_ask_comments / len(ask_posts)
print('Average number of comments on aks posts:', round(avg_ask_comments, 2))
print('\n')
total_show_comments = 0
for row in show_posts:
num_comments = int(row[4])
total_show_comments += num_comments
avg_show_comments = total_show_comments / len(show_posts)
print('Average number of comments on show posts:', round(avg_show_comments, 2))
Average number of comments on aks posts: 14.04 Average number of comments on show posts: 10.32
Users on Hacker News on average more often are getting answers for their questions, than feedback for the show posts. It seems like this community is very eager to help each other.
Since ask posts are more likely to receive comments, we'll focus our remaining analysis just on these posts.
Next, we'll determine if ask posts created at a certain time are more likely to attract comments. We'll use the following steps to perform this analysis:
We will create an empty list where we will store data about date of creating a post and number of comments for it.
import datetime as dt
result_list = []
for row in ask_posts:
created_at = row[6]
num_comments = int(row[4])
result_list.append([created_at, num_comments])
print(result_list[:6])
[['8/16/2016 9:55', 6], ['11/22/2015 13:43', 29], ['5/2/2016 10:14', 1], ['8/2/2016 14:20', 3], ['10/15/2015 16:38', 17], ['9/26/2015 23:23', 1]]
# create a datetime object #
date_format = '%m/%d/%Y %H:%M'
for row in result_list:
hour = row[0]
dt_hour = dt.datetime.strptime(hour, date_format)
row[0] = dt_hour
Than we will make two dictionaries:
counts_by_hour = {}
comments_by_hour = {}
for row in result_list:
hour = row[0]
hour = hour.strftime('%H') # .strftime to select just the hour
num_comments = int(row[1])
if hour not in counts_by_hour:
counts_by_hour[hour] = 1
comments_by_hour[hour] = num_comments
else:
counts_by_hour[hour] += 1
comments_by_hour[hour] += num_comments
print(counts_by_hour)
print('\n')
print(comments_by_hour)
{'09': 45, '13': 85, '10': 59, '14': 107, '16': 108, '23': 68, '12': 73, '17': 100, '15': 116, '21': 109, '20': 80, '02': 58, '18': 109, '03': 54, '05': 46, '19': 110, '01': 60, '22': 71, '08': 48, '04': 47, '00': 55, '06': 44, '07': 34, '11': 58} {'09': 251, '13': 1253, '10': 793, '14': 1416, '16': 1814, '23': 543, '12': 687, '17': 1146, '15': 4477, '21': 1745, '20': 1722, '02': 1381, '18': 1439, '03': 421, '05': 464, '19': 1188, '01': 683, '22': 479, '08': 492, '04': 337, '00': 447, '06': 397, '07': 267, '11': 641}
After quickly examine data, we can see that users are most active between 14:00 and 21:00.
Next, we'll use these two dictionaries to calculate the average number of comments for posts created during each hour of the day
avg_by_hour = []
for hour in comments_by_hour and counts_by_hour:
avg_by_hour.append([hour, (comments_by_hour[hour] / counts_by_hour[hour])])
print(avg_by_hour)
[['09', 5.5777777777777775], ['13', 14.741176470588234], ['10', 13.440677966101696], ['14', 13.233644859813085], ['16', 16.796296296296298], ['23', 7.985294117647059], ['12', 9.41095890410959], ['17', 11.46], ['15', 38.5948275862069], ['21', 16.009174311926607], ['20', 21.525], ['02', 23.810344827586206], ['18', 13.20183486238532], ['03', 7.796296296296297], ['05', 10.08695652173913], ['19', 10.8], ['01', 11.383333333333333], ['22', 6.746478873239437], ['08', 10.25], ['04', 7.170212765957447], ['00', 8.127272727272727], ['06', 9.022727272727273], ['07', 7.852941176470588], ['11', 11.051724137931034]]
Although we now have the results we need, this format makes it hard to identify the hours with the highest values. Let's finish by sorting the list of lists and printing the five highest values in a format that's easier to read.
# creat a list that eqals avg_by_hour with swapped columns
swap_avg_by_hour = []
for hour in avg_by_hour:
swap_avg_by_hour.append([hour[1], hour[0]])
print(swap_avg_by_hour)
[[5.5777777777777775, '09'], [14.741176470588234, '13'], [13.440677966101696, '10'], [13.233644859813085, '14'], [16.796296296296298, '16'], [7.985294117647059, '23'], [9.41095890410959, '12'], [11.46, '17'], [38.5948275862069, '15'], [16.009174311926607, '21'], [21.525, '20'], [23.810344827586206, '02'], [13.20183486238532, '18'], [7.796296296296297, '03'], [10.08695652173913, '05'], [10.8, '19'], [11.383333333333333, '01'], [6.746478873239437, '22'], [10.25, '08'], [7.170212765957447, '04'], [8.127272727272727, '00'], [9.022727272727273, '06'], [7.852941176470588, '07'], [11.051724137931034, '11']]
sorted_swap = sorted(swap_avg_by_hour, reverse = True)
print('Top 5 Hours for Ask Postrs Comments')
print('\n')
for average, hour in sorted_swap[:5]:
dt_hour = dt.datetime.strptime(str(hour), '%H')
hour = dt_hour.strftime('%H:%M')
print('{}: {:.2f} average comments per post'.format(hour, average))
Top 5 Hours for Ask Postrs Comments 15:00: 38.59 average comments per post 02:00: 23.81 average comments per post 20:00: 21.52 average comments per post 16:00: 16.80 average comments per post 21:00: 16.01 average comments per post
In this project, we analyzed data from Hacker News to find the best hour to make your posts.
We created a list of top 5 hours to get more response for your post. The best hour is 15:00 EST, when you should get around 39 comments for you post.