In this project, we'll work with a data set of submissions to popular technology site Hacker News.
We're specifically interested in posts whose titles begin with either *Ask HN* or *Show HN*. Users submit Ask HN posts to ask the Hacker News community a specific question.
We'll compare these two types of posts to determine the following:
Resourse: https://www.kaggle.com/hacker-news/hacker-news-posts
This data set is Hacker News posts from the last 12 months (up to September 26 2016).
It includes the following columns:
Read the hacker_news.csv file in as a list of lists.
### Assign the result to the variable hn
from csv import reader
opened_file = open("hacker_news.csv")
read_file = reader(opened_file)
hn = list(read_file)
headers = hn[0]
hn = hn[1:]
### Display headers and the first five rows
print(headers)
hn[:5]
['id', 'title', 'url', 'num_points', 'num_comments', 'author', 'created_at']
[['12224879', 'Interactive Dynamic Video', 'http://www.interactivedynamicvideo.com/', '386', '52', 'ne0phyte', '8/4/2016 11:52'], ['10975351', 'How to Use Open Source and Shut the Fuck Up at the Same Time', 'http://hueniverse.com/2016/01/26/how-to-use-open-source-and-shut-the-fuck-up-at-the-same-time/', '39', '10', 'josep2', '1/26/2016 19:30'], ['11964716', "Florida DJs May Face Felony for April Fools' Water Joke", 'http://www.thewire.com/entertainment/2013/04/florida-djs-april-fools-water-joke/63798/', '2', '1', 'vezycash', '6/23/2016 22:20'], ['11919867', 'Technology ventures: From Idea to Enterprise', 'https://www.amazon.com/Technology-Ventures-Enterprise-Thomas-Byers/dp/0073523429', '3', '1', 'hswarna', '6/17/2016 0:01'], ['10301696', 'Note by Note: The Making of Steinway L1037 (2007)', 'http://www.nytimes.com/2007/11/07/movies/07stein.html?_r=0', '8', '2', 'walterbell', '9/30/2015 4:12']]
Since we're only concerned with post titles beginning with *Ask HN* or *Show HN*, we'll create new lists of lists containing just the data for those titles.
Let's use *str.startwith()* and *str.lower()* methods to separate posts beginning with Ask HN and Show HN (and case variations) into two different lists next.
Instructions:
ask_posts = list()
show_posts = list()
other_posts = list()
for row in hn:
title = row[1]
if (title.lower()).startswith('ask hn'):
ask_posts.append(row)
elif (title.lower()).startswith('show hn'):
show_posts.append(row)
else:
other_posts.append(row)
print('Number of posts: ', len(hn))
print('Number of posts in ask_posts: ', len(ask_posts))
print('Number of posts in show_posts: ', len(show_posts))
print('Number of posts in other_posts: ', len(other_posts))
Number of posts: 20100 Number of posts in ask_posts: 1744 Number of posts in show_posts: 1162 Number of posts in other_posts: 17194
Next, let's determine if ask posts or show posts receive more comments on average.
Instructions:
total_ask_comments = 0
for row in ask_posts:
total_ask_comments += int(row[4])
avg_ask_comments = total_ask_comments / len(ask_posts)
print('Average number of comments on ask posts:', avg_ask_comments)
total_show_comments = 0
for row in show_posts:
total_show_comments += int(row[4])
avg_show_comments = total_show_comments / len(show_posts)
print('Average number of comments on show posts:', avg_show_comments)
Average number of comments on ask posts: 14.038417431192661 Average number of comments on show posts: 10.31669535283993
The result shows that there are more comments on average in ask posts than that in show posts.
Next, we'll determine if ask posts created at a certain time are more likely to attract comments. We'll use the following steps to perform this analysis:
Instructions:
import datetime as dt
result_list = list()
for row in ask_posts:
result_list.append([row[6], int(row[4])]) # created_at, num_comments
counts_by_hour = {}
comments_by_hour = {}
for row in result_list:
# created_at: '9/25/2016 22:57'
dt_object = dt.datetime.strptime(row[0], "%m/%d/%Y %H:%M")
dt_hour = dt_object.strftime("%H")
if dt_hour not in counts_by_hour:
counts_by_hour[dt_hour] = 1
comments_by_hour[dt_hour] = row[1]
else:
counts_by_hour[dt_hour] += 1
comments_by_hour[dt_hour] += row[1]
print(counts_by_hour['22'])
print(comments_by_hour['22'])
71 479
Next, we'll use these two dictionaries to calculate the average number of comments for posts created during each hour of the day.
Instructions:
avg_by_hour = list()
for key in counts_by_hour:
avg_by_hour.append([key, comments_by_hour[key] / counts_by_hour[key]])
avg_by_hour
[['09', 5.5777777777777775], ['13', 14.741176470588234], ['10', 13.440677966101696], ['14', 13.233644859813085], ['16', 16.796296296296298], ['23', 7.985294117647059], ['12', 9.41095890410959], ['17', 11.46], ['15', 38.5948275862069], ['21', 16.009174311926607], ['20', 21.525], ['02', 23.810344827586206], ['18', 13.20183486238532], ['03', 7.796296296296297], ['05', 10.08695652173913], ['19', 10.8], ['01', 11.383333333333333], ['22', 6.746478873239437], ['08', 10.25], ['04', 7.170212765957447], ['00', 8.127272727272727], ['06', 9.022727272727273], ['07', 7.852941176470588], ['11', 11.051724137931034]]
Although we now have the results we need, this format makes it hard to identify the hours with the highest values. Let's finish by sorting the list of lists and printing the five highest values in a format that's easier to read.
Instructions:
swap_avg_by_hour = list()
for row in avg_by_hour:
swap_avg_by_hour.append([row[1], row[0]])
print(swap_avg_by_hour)
[[5.5777777777777775, '09'], [14.741176470588234, '13'], [13.440677966101696, '10'], [13.233644859813085, '14'], [16.796296296296298, '16'], [7.985294117647059, '23'], [9.41095890410959, '12'], [11.46, '17'], [38.5948275862069, '15'], [16.009174311926607, '21'], [21.525, '20'], [23.810344827586206, '02'], [13.20183486238532, '18'], [7.796296296296297, '03'], [10.08695652173913, '05'], [10.8, '19'], [11.383333333333333, '01'], [6.746478873239437, '22'], [10.25, '08'], [7.170212765957447, '04'], [8.127272727272727, '00'], [9.022727272727273, '06'], [7.852941176470588, '07'], [11.051724137931034, '11']]
sorted_swap = sorted(swap_avg_by_hour, reverse = True)
print('Top 5 Hours for Ask Posts Comments (Time zone: Eastern Time in the US)')
for row in sorted_swap[:5]:
# format the hours
str1 = dt.datetime.strptime(row[1], "%H").strftime("%H:%M")
output = "{hour}: {avg:.2f} average comments per post".format(hour = str1, avg = row[0])
print(output)
Top 5 Hours for Ask Posts Comments (Time zone: Eastern Time in the US) 15:00: 38.59 average comments per post 02:00: 23.81 average comments per post 20:00: 21.52 average comments per post 16:00: 16.80 average comments per post 21:00: 16.01 average comments per post
According to the result, if we create a post during 3 pm to 4 pm (ET), it might have a higher chance of receiving comments.
## Convert the times to Taipei Standard Time (GMT+8)
print('Top 5 Hours for Ask Posts Comments (Time zone: Taipei Standard Time (GMT+8))')
for row in sorted_swap[:5]:
# format the hours, ignore daylight saving time
str1 = (dt.datetime.strptime(row[1], "%H") + dt.timedelta(hours=12)).strftime("%H:%M")
output = "{hour}: {avg:.2f} average comments per post".format(hour = str1, avg = row[0])
print(output)
Top 5 Hours for Ask Posts Comments (Time zone: Taipei Standard Time (GMT+8)) 03:00: 38.59 average comments per post 14:00: 23.81 average comments per post 08:00: 21.52 average comments per post 04:00: 16.80 average comments per post 09:00: 16.01 average comments per post
According to the result, if we create a post during 3 am to 4 am (Taipei Standard Time), it might have a higher chance of receiving comments.