In this project, we will work with a data set of submissions to popular technology site Hacker News.
We are specifically interested in posts whose titles begin with either Ask HN or Show HN. Users submit Ask HN posts to ask the Hacker News community a specific question.
Our goal is to compare these two types of posts to determine the following:
We will start by importing the libraries we need and reading the data set into a list of lists.
from csv import reader
opened_file = open('hacker_news.csv')
read_file = reader(opened_file)
hn = list(read_file)
print(hn[:5])
[['id', 'title', 'url', 'num_points', 'num_comments', 'author', 'created_at'], ['12224879', 'Interactive Dynamic Video', 'http://www.interactivedynamicvideo.com/', '386', '52', 'ne0phyte', '8/4/2016 11:52'], ['10975351', 'How to Use Open Source and Shut the Fuck Up at the Same Time', 'http://hueniverse.com/2016/01/26/how-to-use-open-source-and-shut-the-fuck-up-at-the-same-time/', '39', '10', 'josep2', '1/26/2016 19:30'], ['11964716', "Florida DJs May Face Felony for April Fools' Water Joke", 'http://www.thewire.com/entertainment/2013/04/florida-djs-april-fools-water-joke/63798/', '2', '1', 'vezycash', '6/23/2016 22:20'], ['11919867', 'Technology ventures: From Idea to Enterprise', 'https://www.amazon.com/Technology-Ventures-Enterprise-Thomas-Byers/dp/0073523429', '3', '1', 'hswarna', '6/17/2016 0:01']]
We can see that the imported list has a header, we shall proceed to remove the header.
headers = hn[0]
hn = hn[1:]
print(hn[:5])
[['12224879', 'Interactive Dynamic Video', 'http://www.interactivedynamicvideo.com/', '386', '52', 'ne0phyte', '8/4/2016 11:52'], ['10975351', 'How to Use Open Source and Shut the Fuck Up at the Same Time', 'http://hueniverse.com/2016/01/26/how-to-use-open-source-and-shut-the-fuck-up-at-the-same-time/', '39', '10', 'josep2', '1/26/2016 19:30'], ['11964716', "Florida DJs May Face Felony for April Fools' Water Joke", 'http://www.thewire.com/entertainment/2013/04/florida-djs-april-fools-water-joke/63798/', '2', '1', 'vezycash', '6/23/2016 22:20'], ['11919867', 'Technology ventures: From Idea to Enterprise', 'https://www.amazon.com/Technology-Ventures-Enterprise-Thomas-Byers/dp/0073523429', '3', '1', 'hswarna', '6/17/2016 0:01'], ['10301696', 'Note by Note: The Making of Steinway L1037 (2007)', 'http://www.nytimes.com/2007/11/07/movies/07stein.html?_r=0', '8', '2', 'walterbell', '9/30/2015 4:12']]
We are ready to filter our data. Since we're only concerned with post titles beginning with Ask HN or Show HN, we'll create new lists of lists containing just the data for those titles.
ask_posts = []
show_posts = []
other_posts = []
for row in hn:
title = row[1]
title = title.lower()
if title.startswith('ask hn'):
ask_posts.append(row)
elif title.startswith('show hn'):
show_posts.append(row)
else:
other_posts.append(row)
print(ask_posts[:5])
print(show_posts[:5])
print(other_posts[:5])
[['12296411', 'Ask HN: How to improve my personal website?', '', '2', '6', 'ahmedbaracat', '8/16/2016 9:55'], ['10610020', 'Ask HN: Am I the only one outraged by Twitter shutting down share counts?', '', '28', '29', 'tkfx', '11/22/2015 13:43'], ['11610310', 'Ask HN: Aby recent changes to CSS that broke mobile?', '', '1', '1', 'polskibus', '5/2/2016 10:14'], ['12210105', 'Ask HN: Looking for Employee #3 How do I do it?', '', '1', '3', 'sph130', '8/2/2016 14:20'], ['10394168', 'Ask HN: Someone offered to buy my browser extension from me. What now?', '', '28', '17', 'roykolak', '10/15/2015 16:38']] [['10627194', 'Show HN: Wio Link ESP8266 Based Web of Things Hardware Development Platform', 'https://iot.seeed.cc', '26', '22', 'kfihihc', '11/25/2015 14:03'], ['10646440', 'Show HN: Something pointless I made', 'http://dn.ht/picklecat/', '747', '102', 'dhotson', '11/29/2015 22:46'], ['11590768', 'Show HN: Shanhu.io, a programming playground powered by e8vm', 'https://shanhu.io', '1', '1', 'h8liu', '4/28/2016 18:05'], ['12178806', 'Show HN: Webscope Easy way for web developers to communicate with Clients', 'http://webscopeapp.com', '3', '3', 'fastbrick', '7/28/2016 7:11'], ['10872799', 'Show HN: GeoScreenshot Easily test Geo-IP based web pages', 'https://www.geoscreenshot.com/', '1', '9', 'kpsychwave', '1/9/2016 20:45']] [['12224879', 'Interactive Dynamic Video', 'http://www.interactivedynamicvideo.com/', '386', '52', 'ne0phyte', '8/4/2016 11:52'], ['10975351', 'How to Use Open Source and Shut the Fuck Up at the Same Time', 'http://hueniverse.com/2016/01/26/how-to-use-open-source-and-shut-the-fuck-up-at-the-same-time/', '39', '10', 'josep2', '1/26/2016 19:30'], ['11964716', "Florida DJs May Face Felony for April Fools' Water Joke", 'http://www.thewire.com/entertainment/2013/04/florida-djs-april-fools-water-joke/63798/', '2', '1', 'vezycash', '6/23/2016 22:20'], ['11919867', 'Technology ventures: From Idea to Enterprise', 'https://www.amazon.com/Technology-Ventures-Enterprise-Thomas-Byers/dp/0073523429', '3', '1', 'hswarna', '6/17/2016 0:01'], ['10301696', 'Note by Note: The Making of Steinway L1037 (2007)', 'http://www.nytimes.com/2007/11/07/movies/07stein.html?_r=0', '8', '2', 'walterbell', '9/30/2015 4:12']]
We will now determine if ask posts or show posts receive more comments on average.
total_ask_comments = 0
for row in ask_posts:
comments = int(row[4])
total_ask_comments += comments
avg_ask_comments = total_ask_comments/len(ask_posts)
print('Average ask comments:',avg_ask_comments)
Average ask comments: 14.038417431192661
total_show_comments = 0
for row in show_posts:
comments = int(row[4])
total_show_comments += comments
avg_show_comments = total_show_comments/len(show_posts)
print('Average show comments:',avg_show_comments)
Average show comments: 10.31669535283993
Form our analysis, ask post recived higher average comment. Since ask posts are more likely to receive comments, we will focus our remaining analysis on the ask posts.
Our mission here is to determin if ask posts created at a certain time are more likely to attract comments.
We'll use the following steps to perform this analysis:
#calculating the amount of ask posts and comments by hour created
import datetime as dt
result_list = []
for row in ask_posts:
created_time = row[6]
num_comment = int(row[4])
result_list.append([created_time, num_comment])
counts_by_hour = {}
comments_by_hour = {}
for row in result_list:
time_date = row[0]
dt_time_date = dt.datetime.strptime(time_date, '%m/%d/%Y %H:%M' )
dt_time = dt.datetime.strftime(dt_time_date, '%H')
if dt_time not in counts_by_hour:
counts_by_hour[dt_time] = 1
comments_by_hour[dt_time] = row[1]
else:
counts_by_hour[dt_time] += 1
comments_by_hour[dt_time] += row[1]
print(counts_by_hour)
print('\n')
print(comments_by_hour)
{'20': 80, '01': 60, '16': 108, '12': 73, '17': 100, '23': 68, '10': 59, '07': 34, '06': 44, '05': 46, '19': 110, '22': 71, '13': 85, '15': 116, '08': 48, '09': 45, '03': 54, '18': 109, '21': 109, '00': 55, '02': 58, '11': 58, '04': 47, '14': 107} {'20': 1722, '01': 683, '16': 1814, '12': 687, '17': 1146, '23': 543, '10': 793, '07': 267, '06': 397, '05': 464, '19': 1188, '22': 479, '13': 1253, '15': 4477, '08': 492, '09': 251, '03': 421, '18': 1439, '21': 1745, '00': 447, '02': 1381, '11': 641, '04': 337, '14': 1416}
We shall use counts_by_hour and comments_by_hour dictionaries to calculate the average number of comments for posts created during each hour of the day. To do his , we have to create a list of lists containing the hours during which posts were created and the average number of comments those posts received.
avg_by_hour = []
for key in counts_by_hour:
avg_by_hour.append([key,comments_by_hour[key]/counts_by_hour[key]])
print(avg_by_hour)
[['20', 21.525], ['01', 11.383333333333333], ['16', 16.796296296296298], ['12', 9.41095890410959], ['17', 11.46], ['23', 7.985294117647059], ['10', 13.440677966101696], ['07', 7.852941176470588], ['06', 9.022727272727273], ['05', 10.08695652173913], ['19', 10.8], ['22', 6.746478873239437], ['13', 14.741176470588234], ['15', 38.5948275862069], ['08', 10.25], ['09', 5.5777777777777775], ['03', 7.796296296296297], ['18', 13.20183486238532], ['21', 16.009174311926607], ['00', 8.127272727272727], ['02', 23.810344827586206], ['11', 11.051724137931034], ['04', 7.170212765957447], ['14', 13.233644859813085]]
swap_avg_by_hour = []
for row in avg_by_hour:
swap_avg_by_hour.append([row[1],row[0]])
print(swap_avg_by_hour)
[[21.525, '20'], [11.383333333333333, '01'], [16.796296296296298, '16'], [9.41095890410959, '12'], [11.46, '17'], [7.985294117647059, '23'], [13.440677966101696, '10'], [7.852941176470588, '07'], [9.022727272727273, '06'], [10.08695652173913, '05'], [10.8, '19'], [6.746478873239437, '22'], [14.741176470588234, '13'], [38.5948275862069, '15'], [10.25, '08'], [5.5777777777777775, '09'], [7.796296296296297, '03'], [13.20183486238532, '18'], [16.009174311926607, '21'], [8.127272727272727, '00'], [23.810344827586206, '02'], [11.051724137931034, '11'], [7.170212765957447, '04'], [13.233644859813085, '14']]
sorted_swap = sorted(swap_avg_by_hour ,reverse = True )
print("Top 5 Hours for Ask Posts Comments")
for row in sorted_swap[:5]:
dt_list_time = dt.datetime.strptime(row[1], '%H')
list_time = dt.datetime.strftime(dt_list_time, '%H:%M')
list_coments = row[0]
result = "{}: {:.2f} average comments per post".format(list_time, list_coments )
print(result)
Top 5 Hours for Ask Posts Comments 15:00: 38.59 average comments per post 02:00: 23.81 average comments per post 20:00: 21.52 average comments per post 16:00: 16.80 average comments per post 21:00: 16.01 average comments per post
According to the above result, post from 15:00 hours to 16:00 hours EST will recive more comments. I can further convert the time to the time zone I live in (GMT +1). The hour diffrence betwen EST and GMT +1 is 5 hours.
print("Top 5 Hours for Ask Posts Comments (Nigeria time GMT+1)")
for row in sorted_swap[:5]:
dt_list_time = dt.datetime.strptime(row[1], '%H') + dt.timedelta(hours=5)
list_time = dt.datetime.strftime(dt_list_time, '%H:%M')
list_coments = row[0]
result = "{}: {:.2f} average comments per post".format(list_time, list_coments )
print(result)
Top 5 Hours for Ask Posts Comments (Nigeria time GMT+1) 20:00: 38.59 average comments per post 07:00: 23.81 average comments per post 01:00: 21.52 average comments per post 21:00: 16.80 average comments per post 02:00: 16.01 average comments per post
Post from 20:00 hours to 21:00 hours GMT +1 will recive more comments.