In this project, we'll work with a dataset of submissions to popular technology site Hacker News. Hacker News is a site started by the startup incubator Y Combinator, where user-submitted stories (known as "posts") receive votes and comments, similar to reddit. You can find the data set here.
We're specifically interested in posts with titles that begin with either Ask HN
or Show HN
. Users submit Ask HN
posts to ask the Hacker News community a specific question. Likewise, users submit Show HN
posts to show the Hacker News community a project, product, or just something interesting.
We'll compare these two types of posts to determine the following:
Ask HN
or Show HN
receive more comments on average?Read the hacker_news.csv
file in as a list of lists.
hn
Display the first five rows of hn
from csv import reader
opened_file = open('hacker_news.csv')
read_file = reader(opened_file)
hn = list(read_file)
hn_header = hn[0] #the header row will be assigned to variable hn_header
hn = hn[1:] # all remaining rows, excluding the header, will be assigned to hn
print("Header row:")
print("\n")
print(hn_header)
print("\n")
print("First 5 rows without header row:")
print("\n")
print(hn[:4])
Header row: ['id', 'title', 'url', 'num_points', 'num_comments', 'author', 'created_at'] First 5 rows without header row: [['12224879', 'Interactive Dynamic Video', 'http://www.interactivedynamicvideo.com/', '386', '52', 'ne0phyte', '8/4/2016 11:52'], ['10975351', 'How to Use Open Source and Shut the Fuck Up at the Same Time', 'http://hueniverse.com/2016/01/26/how-to-use-open-source-and-shut-the-fuck-up-at-the-same-time/', '39', '10', 'josep2', '1/26/2016 19:30'], ['11964716', "Florida DJs May Face Felony for April Fools' Water Joke", 'http://www.thewire.com/entertainment/2013/04/florida-djs-april-fools-water-joke/63798/', '2', '1', 'vezycash', '6/23/2016 22:20'], ['11919867', 'Technology ventures: From Idea to Enterprise', 'https://www.amazon.com/Technology-Ventures-Enterprise-Thomas-Byers/dp/0073523429', '3', '1', 'hswarna', '6/17/2016 0:01']]
Since we're only concerned with post titles beginning with Ask HN or Show HN, we'll create new lists of lists containing just the data for those titles. To find the posts that begin with either Ask HN
or Show HN
, we'll use the string startswith
. Capitalization matters so we'll also use string method lower
.
Let's use these methods to separate posts beginning with Ask HN
and Show HN
(and case variations) into two different lists.
ask_posts = []
show_posts = []
other_posts = []
for row in hn:
title = row[1]
title = title.lower()
if title.startswith('ask hn'):
ask_posts.append(row)
elif title.startswith('show hn'):
show_posts.append(row)
else:
other_posts.append(row)
print("Ask HN: ",len(ask_posts))
print("\n")
print(ask_posts[:4])
print("\n")
print("Show HN: ",len(show_posts))
print("\n")
print(show_posts[:4])
print("\n")
print("Other: ",len(other_posts))
print("\n")
print(other_posts[:4])
Ask HN: 1744 [['12296411', 'Ask HN: How to improve my personal website?', '', '2', '6', 'ahmedbaracat', '8/16/2016 9:55'], ['10610020', 'Ask HN: Am I the only one outraged by Twitter shutting down share counts?', '', '28', '29', 'tkfx', '11/22/2015 13:43'], ['11610310', 'Ask HN: Aby recent changes to CSS that broke mobile?', '', '1', '1', 'polskibus', '5/2/2016 10:14'], ['12210105', 'Ask HN: Looking for Employee #3 How do I do it?', '', '1', '3', 'sph130', '8/2/2016 14:20']] Show HN: 1162 [['10627194', 'Show HN: Wio Link ESP8266 Based Web of Things Hardware Development Platform', 'https://iot.seeed.cc', '26', '22', 'kfihihc', '11/25/2015 14:03'], ['10646440', 'Show HN: Something pointless I made', 'http://dn.ht/picklecat/', '747', '102', 'dhotson', '11/29/2015 22:46'], ['11590768', 'Show HN: Shanhu.io, a programming playground powered by e8vm', 'https://shanhu.io', '1', '1', 'h8liu', '4/28/2016 18:05'], ['12178806', 'Show HN: Webscope Easy way for web developers to communicate with Clients', 'http://webscopeapp.com', '3', '3', 'fastbrick', '7/28/2016 7:11']] Other: 17194 [['12224879', 'Interactive Dynamic Video', 'http://www.interactivedynamicvideo.com/', '386', '52', 'ne0phyte', '8/4/2016 11:52'], ['10975351', 'How to Use Open Source and Shut the Fuck Up at the Same Time', 'http://hueniverse.com/2016/01/26/how-to-use-open-source-and-shut-the-fuck-up-at-the-same-time/', '39', '10', 'josep2', '1/26/2016 19:30'], ['11964716', "Florida DJs May Face Felony for April Fools' Water Joke", 'http://www.thewire.com/entertainment/2013/04/florida-djs-april-fools-water-joke/63798/', '2', '1', 'vezycash', '6/23/2016 22:20'], ['11919867', 'Technology ventures: From Idea to Enterprise', 'https://www.amazon.com/Technology-Ventures-Enterprise-Thomas-Byers/dp/0073523429', '3', '1', 'hswarna', '6/17/2016 0:01']]
Next, let's determine if ask posts or show posts receive more comments on average. We'll use a for loop
to iterate over the ask_posts
and show_posts
list created above.
# calculating total_ask_comments
total_ask_comments = 0
for row in ask_posts:
num_comments = int(row[4])
total_ask_comments += num_comments
avg_ask_comments = total_ask_comments / len(ask_posts)
print("Average ask comments: ", avg_ask_comments)
print("Length of ask posts list: ", len(ask_posts))
Average ask comments: 14.038417431192661 Length of ask posts list: 1744
# calculating total_show_comments
total_show_comments = 0
for row in show_posts:
num_comments = int(row[4])
total_show_comments += num_comments
avg_show_comments = total_show_comments / len(show_posts)
print("Average show comments: ", avg_show_comments)
print("Length of show posts list: ",len(show_posts))
Average show comments: 10.31669535283993 Length of show posts list: 1162
Do show posts or ask posts receive more comments on average? Ask HN
receives an average of 14.038417431192661
compared to Show HN
of 10.31669535283993
. We'll focus on Ask HN
posts moving forward since they receive a higher average of comments.
Next, we'll determine if ask posts created at a certain time are more likely to attract comments. We'll use the following steps to perform this analysis:
import datetime as dt
result_list = []
for row in ask_posts:
created_at = row[6]
num_comments = int(row[4])
result_list.append([created_at,num_comments])
counts_by_hour = {}
comments_by_hour = {}
for row in result_list:
hour_created = row[0]
hour_created = dt.datetime.strptime(hour_created,"%m/%d/%Y %H:%M")
hour_created = dt.datetime.strftime(hour_created, "%H")
if hour_created not in counts_by_hour:
counts_by_hour[hour_created] = 1
comments_by_hour[hour_created] = row[1]
else:
counts_by_hour[hour_created] += 1
comments_by_hour[hour_created] += row[1]
print("Counts by Hour:")
print(counts_by_hour)
print("\n")
print("Comments by Hour:")
print(comments_by_hour)
Counts by Hour: {'09': 45, '13': 85, '10': 59, '14': 107, '16': 108, '23': 68, '12': 73, '17': 100, '15': 116, '21': 109, '20': 80, '02': 58, '18': 109, '03': 54, '05': 46, '19': 110, '01': 60, '22': 71, '08': 48, '04': 47, '00': 55, '06': 44, '07': 34, '11': 58} Comments by Hour: {'09': 251, '13': 1253, '10': 793, '14': 1416, '16': 1814, '23': 543, '12': 687, '17': 1146, '15': 4477, '21': 1745, '20': 1722, '02': 1381, '18': 1439, '03': 421, '05': 464, '19': 1188, '01': 683, '22': 479, '08': 492, '04': 337, '00': 447, '06': 397, '07': 267, '11': 641}
We'll use the counts_by_hour
and comments_by_hour
dictionaries to calculate the average number of comments for post created during each hour of the day.
Create a list of lists containing the hours during which posts were created, and the average number of comments those posts were created.
avg_by_hour = []
for number in comments_by_hour:
avg_by_hour.append([number, comments_by_hour[number]/counts_by_hour[number]])
print("Average Comments by Hour:")
print("\n")
print(avg_by_hour)
Average Comments by Hour: [['09', 5.5777777777777775], ['13', 14.741176470588234], ['10', 13.440677966101696], ['14', 13.233644859813085], ['16', 16.796296296296298], ['23', 7.985294117647059], ['12', 9.41095890410959], ['17', 11.46], ['15', 38.5948275862069], ['21', 16.009174311926607], ['20', 21.525], ['02', 23.810344827586206], ['18', 13.20183486238532], ['03', 7.796296296296297], ['05', 10.08695652173913], ['19', 10.8], ['01', 11.383333333333333], ['22', 6.746478873239437], ['08', 10.25], ['04', 7.170212765957447], ['00', 8.127272727272727], ['06', 9.022727272727273], ['07', 7.852941176470588], ['11', 11.051724137931034]]
The list above makes it challenging to identify the hours with the highest values. We'll use the sorted()
function to sort a list of lists in descending order. The new list of lists will have a first element of average number of comments, and a second element of hour of day. This is essentially avg_by_hour
list but swapped.
swap_avg_by_hour = []
for row in avg_by_hour:
swap_avg_by_hour.append([row[1],row[0]])
print("Swapped:")
print(swap_avg_by_hour)
Swapped: [[5.5777777777777775, '09'], [14.741176470588234, '13'], [13.440677966101696, '10'], [13.233644859813085, '14'], [16.796296296296298, '16'], [7.985294117647059, '23'], [9.41095890410959, '12'], [11.46, '17'], [38.5948275862069, '15'], [16.009174311926607, '21'], [21.525, '20'], [23.810344827586206, '02'], [13.20183486238532, '18'], [7.796296296296297, '03'], [10.08695652173913, '05'], [10.8, '19'], [11.383333333333333, '01'], [6.746478873239437, '22'], [10.25, '08'], [7.170212765957447, '04'], [8.127272727272727, '00'], [9.022727272727273, '06'], [7.852941176470588, '07'], [11.051724137931034, '11']]
Since the first column of this list is the average number of comments, sorting the list will sort by the average number of comments.
sorted_swap = sorted(swap_avg_by_hour,reverse=True)
print(sorted_swap)
[[38.5948275862069, '15'], [23.810344827586206, '02'], [21.525, '20'], [16.796296296296298, '16'], [16.009174311926607, '21'], [14.741176470588234, '13'], [13.440677966101696, '10'], [13.233644859813085, '14'], [13.20183486238532, '18'], [11.46, '17'], [11.383333333333333, '01'], [11.051724137931034, '11'], [10.8, '19'], [10.25, '08'], [10.08695652173913, '05'], [9.41095890410959, '12'], [9.022727272727273, '06'], [8.127272727272727, '00'], [7.985294117647059, '23'], [7.852941176470588, '07'], [7.796296296296297, '03'], [7.170212765957447, '04'], [6.746478873239437, '22'], [5.5777777777777775, '09']]
Use the str.format()
method to print the hour and average in the following format: 15:00: 38.59 average comments per post
.
datetime.strptime()
constructor to return a datetime object, and then use the strftime()
method to specify the format of the time.{:.2f}
to indicate only two decimal places.print("Top 5 Hours for Ask Posts Comments:")
print(sorted_swap[:5])
Top 5 Hours for Ask Posts Comments: [[38.5948275862069, '15'], [23.810344827586206, '02'], [21.525, '20'], [16.796296296296298, '16'], [16.009174311926607, '21']]
template = "{}: {:.2f} average comments per post"
for row in sorted_swap[:5]:
hour = row[1]
avg_comments = row[0]
hour = dt.datetime.strptime(hour,"%H")
hour = dt.datetime.strftime(hour,"%H:%M")
output = template.format(hour,avg_comments)
print(output)
15:00: 38.59 average comments per post 02:00: 23.81 average comments per post 20:00: 21.52 average comments per post 16:00: 16.80 average comments per post 21:00: 16.01 average comments per post
During which hours should you create a post to have a higher chance of receiving comments? With an average of 38.59
comments per post, 3pm is ideal time. If you're a nightowl, then 2am is second best with an average of 23.81
comments per post.