from csv import reader
###*Google play app data set*###
opened_file= open('AppleStore.csv')
read_file= reader(opened_file)
ios= list(read_file)
ios_header=ios[0]
ios=ios[1:]
###*App store dataset*###
opened_file = open('googleplaystore.csv')
read_file = reader(opened_file)
android= list(read_file)
android_header= android[0]
android= android[1:]
def explore_data(dataset, start,end, rows_and_columns =False):
dataset_slice =dataset[start:end]
for row in dataset_slice:
print(row)
print('\n')
# adds a new empty line after each row.
if rows_and_columns:
print('Number of columns:', len(dataset[0]))
print('Number of rows:', len(dataset))
print(android_header)
print('\n')
explore_data(android,0,3,True)
['App', 'Category', 'Rating', 'Reviews', 'Size', 'Installs', 'Type', 'Price', 'Content Rating', 'Genres', 'Last Updated', 'Current Ver', 'Android Ver'] ['Photo Editor & Candy Camera & Grid & ScrapBook', 'ART_AND_DESIGN', '4.1', '159', '19M', '10,000+', 'Free', '0', 'Everyone', 'Art & Design', 'January 7, 2018', '1.0.0', '4.0.3 and up'] ['Coloring book moana', 'ART_AND_DESIGN', '3.9', '967', '14M', '500,000+', 'Free', '0', 'Everyone', 'Art & Design;Pretend Play', 'January 15, 2018', '2.0.0', '4.0.3 and up'] ['U Launcher Lite – FREE Live Cool Themes, Hide Apps', 'ART_AND_DESIGN', '4.7', '87510', '8.7M', '5,000,000+', 'Free', '0', 'Everyone', 'Art & Design', 'August 1, 2018', '1.2.4', '4.0.3 and up'] Number of columns: 13 Number of rows: 10841
print(ios_header)
print('\n')
explore_data(ios,0,3,True)
['id', 'track_name', 'size_bytes', 'currency', 'price', 'rating_count_tot', 'rating_count_ver', 'user_rating', 'user_rating_ver', 'ver', 'cont_rating', 'prime_genre', 'sup_devices.num', 'ipadSc_urls.num', 'lang.num', 'vpp_lic'] ['284882215', 'Facebook', '389879808', 'USD', '0.0', '2974676', '212', '3.5', '3.5', '95.0', '4+', 'Social Networking', '37', '1', '29', '1'] ['389801252', 'Instagram', '113954816', 'USD', '0.0', '2161558', '1289', '4.5', '4.0', '10.23', '12+', 'Photo & Video', '37', '0', '29', '1'] ['529479190', 'Clash of Clans', '116476928', 'USD', '0.0', '2130805', '579', '4.5', '4.5', '9.24.12', '9+', 'Games', '38', '5', '18', '1'] Number of columns: 16 Number of rows: 7197
[Google Play apps documentation]https://www.kaggle.com/lava18/google-play-store-apps
[Mobile Apps store documentation]https://www.kaggle.com/ramamet4/app-store-apple-data-set-10k-apps
One of the discussions outlines an error in row 10472. i print tha row and compare it against the header with another row that has is correct
print(android[10472]) # incorrect row
print('\n')
print(android_header) # header
print('\n')
print(android[0])# correct row
['Life Made WI-Fi Touchscreen Photo Frame', '1.9', '19', '3.0M', '1,000+', 'Free', '0', 'Everyone', '', 'February 11, 2018', '1.0.19', '4.0 and up'] ['App', 'Category', 'Rating', 'Reviews', 'Size', 'Installs', 'Type', 'Price', 'Content Rating', 'Genres', 'Last Updated', 'Current Ver', 'Android Ver'] ['Photo Editor & Candy Camera & Grid & ScrapBook', 'ART_AND_DESIGN', '4.1', '159', '19M', '10,000+', 'Free', '0', 'Everyone', 'Art & Design', 'January 7, 2018', '1.0.0', '4.0.3 and up']
The row 10472 corresponds to the app Life Made Wifi Touch screen and we see that the rating is 19. this is above the naximum rating of 5. this problem is caused by a missing figure under the category column. As a consequence we will delete the row.
print(len(android))
del android[10472]
print(len(android))
10841 10840
A closer exploration of the google play data set reviels that some apps have got duplicate entries. For instance instagram has 4 entries.
for app in android:
name = app[0]
if name == 'Instagram':
print(app)
['Instagram', 'SOCIAL', '4.5', '66577313', 'Varies with device', '1,000,000,000+', 'Free', '0', 'Teen', 'Social', 'July 31, 2018', 'Varies with device', 'Varies with device'] ['Instagram', 'SOCIAL', '4.5', '66577446', 'Varies with device', '1,000,000,000+', 'Free', '0', 'Teen', 'Social', 'July 31, 2018', 'Varies with device', 'Varies with device'] ['Instagram', 'SOCIAL', '4.5', '66577313', 'Varies with device', '1,000,000,000+', 'Free', '0', 'Teen', 'Social', 'July 31, 2018', 'Varies with device', 'Varies with device'] ['Instagram', 'SOCIAL', '4.5', '66509917', 'Varies with device', '1,000,000,000+', 'Free', '0', 'Teen', 'Social', 'July 31, 2018', 'Varies with device', 'Varies with device']
duplicate_apps = []
unique_apps =[]
for app in android:
name=app[0]
if name in unique_apps:
duplicate_apps.append(name)
else:
unique_apps.append(name)
print('Number of duplicate apps:', len(duplicate_apps))
print('\n')
print('Examples of duplicate apps:', duplicate_apps [:15])
Number of duplicate apps: 1181 Examples of duplicate apps: ['Quick PDF Scanner + OCR FREE', 'Box', 'Google My Business', 'ZOOM Cloud Meetings', 'join.me - Simple Meetings', 'Box', 'Zenefits', 'Google Ads', 'Google My Business', 'Slack', 'FreshBooks Classic', 'Insightly CRM', 'QuickBooks Accounting: Invoicing & Expenses', 'HipChat - Chat Built for Teams', 'Xero Accounting Software']
We don't want to count certain apps more than once when we analyze data, so we need to remove the duplicate entries and keep only one entry per app. One thing we could do is remove the duplicate rows randomly, but we could probably find a better way.
If you examine the rows we printed two cells above for the Instagram bapp, the main difference happens on the fourth position of each row, which corresponds to the number of reviews. The different numbers show that the data was collected at different times. We can use this to build a criterion for keeping rows. We won't remove rows randomly, but rather we'll keep the rows that have the highest number of reviews because the higher the number of reviews, the more reliable the ratings.
To do that, we will:
We start by building a dictionary
reviews_max={}
for app in android:
name = app[0]
n_reviews = float(app[3])
if name in reviews_max and reviews_max[name]< n_reviews:
reviews_max[name] = n_reviews
elif name not in reviews_max:
reviews_max[name] = n_reviews
In a previous code cell, we found that there are 1,181 cases where an app occurs more than once, so the length of our dictionary (of unique apps) should be equal to the difference between the length of our data set and 1,181.
print('Expected length:', len(android) - 1181)
print('Actual length:', len(reviews_max))
Expected length: 9659 Actual length: 9659
Now, let's use the reviews_max dictionary to remove the duplicates. For the duplicate cases, we'll only keep the entries with the highest number of reviews. In the code cell below:
android_clean=[]
already_added=[]
for app in android:
name= app[0]
n_reviews= float(app[3])
if (reviews_max[name] == n_reviews) and (name not in already_added):
android_clean.append(app)
already_added.append(name)
we then explore the new data set
explore_data(android_clean, 0 , 3, True)
['Photo Editor & Candy Camera & Grid & ScrapBook', 'ART_AND_DESIGN', '4.1', '159', '19M', '10,000+', 'Free', '0', 'Everyone', 'Art & Design', 'January 7, 2018', '1.0.0', '4.0.3 and up'] ['U Launcher Lite – FREE Live Cool Themes, Hide Apps', 'ART_AND_DESIGN', '4.7', '87510', '8.7M', '5,000,000+', 'Free', '0', 'Everyone', 'Art & Design', 'August 1, 2018', '1.2.4', '4.0.3 and up'] ['Sketch - Draw & Paint', 'ART_AND_DESIGN', '4.5', '215644', '25M', '50,000,000+', 'Free', '0', 'Teen', 'Art & Design', 'June 8, 2018', 'Varies with device', '4.2 and up'] Number of columns: 13 Number of rows: 9659
We have 9659 rows, just as expected.
If you explore the data sets enough, you'll notice the names of some of the apps suggest they are not directed toward an English-speaking audience. Below, we see a couple of examples from both data sets:
print(ios[813][1])
print(ios[6731][1])
print(android_clean[4412][0])
print(android_clean[7940][0])
爱奇艺PPS -《欢乐颂2》电视剧热播 【脱出ゲーム】絶対に最後までプレイしないで 〜謎解き&ブロックパズル〜 中国語 AQリスニング لعبة تقدر تربح DZ
We're not interested in keeping these kind of apps, so we'll remove them. One way to go about this is to remove each app whose name contains a symbol that is not commonly used in English text — English text usually includes letters from the English alphabet, numbers composed of digits from 0 to 9, punctuation marks (., !, ?, ;, etc.), and other symbols (+, *, /, etc.).
All these characters that are specific to English texts are encoded using the ASCII standard. Each ASCII character has a corresponding number between 0 and 127 associated with it, and we can take advantage of that to build a function that checks an app name and tells us whether it contains non-ASCII characters.
We built this function below, and we use the built-in ord() function to find out the corresponding encoding number of each character.
def is_english(string): for character in string: if ord(character)>127: return False return True
print(is_english('Instagram')) print(is_english('爱奇艺PPS -《欢乐颂2》电视剧热播'))
The function seems to work fine, but some English app names use emojis or other symbols (™, — (em dash), – (en dash), etc.) that fall outside of the ASCII range. Because of this, we'll remove useful apps if we use the function in its current form.
print(is_english('Docs To Go™ Free Office Suite')) print(is_english('Instachat 😜'))
print(ord('™')) print(ord('😜'))
To minimize the impact of data loss, we'll only remove an app if its name has more than three non-ASCII characters:
def is_english(string):
non_ascii=0
for character in string :
if ord(character )>127:
non_ascii +=1
if non_ascii>3:
return False
else:
return True
print (is_english('Docs To Go™ Free Office Suite'))
print (is_english('Instachat 😜'))
True True
The function is still not perfect, and very few non-English apps might get past our filter, but this seems good enough at this point in our analysis — we shouldn't spend too much time on optimization at this point.
Below, we use the is_english() function to filter out the non-English apps for both data sets:
android_english=[]
ios_english=[]
for app in android_clean:
name=app[0]
if is_english(name):
android_english.append(app)
for app in ios:
name=app[1]
if is_english(name ):
ios_english.append(app)
explore_data(android_english,0,3,True)
print('\n')
explore_data(ios_english,0,3,True)
['Photo Editor & Candy Camera & Grid & ScrapBook', 'ART_AND_DESIGN', '4.1', '159', '19M', '10,000+', 'Free', '0', 'Everyone', 'Art & Design', 'January 7, 2018', '1.0.0', '4.0.3 and up'] ['U Launcher Lite – FREE Live Cool Themes, Hide Apps', 'ART_AND_DESIGN', '4.7', '87510', '8.7M', '5,000,000+', 'Free', '0', 'Everyone', 'Art & Design', 'August 1, 2018', '1.2.4', '4.0.3 and up'] ['Sketch - Draw & Paint', 'ART_AND_DESIGN', '4.5', '215644', '25M', '50,000,000+', 'Free', '0', 'Teen', 'Art & Design', 'June 8, 2018', 'Varies with device', '4.2 and up'] Number of columns: 13 Number of rows: 9614 ['284882215', 'Facebook', '389879808', 'USD', '0.0', '2974676', '212', '3.5', '3.5', '95.0', '4+', 'Social Networking', '37', '1', '29', '1'] ['389801252', 'Instagram', '113954816', 'USD', '0.0', '2161558', '1289', '4.5', '4.0', '10.23', '12+', 'Photo & Video', '37', '0', '29', '1'] ['529479190', 'Clash of Clans', '116476928', 'USD', '0.0', '2130805', '579', '4.5', '4.5', '9.24.12', '9+', 'Games', '38', '5', '18', '1'] Number of columns: 16 Number of rows: 6183
we see that we are left with 9614 Android apps and 683 IOSapps.
As we mentioned in the introduction, we only build apps that are free to download and install, and our main source of revenue consists of in-app ads. Our data sets contain both free and non-free apps, and we'll need to isolate only the free apps for our analysis. Below, we isolate the free apps for both our data sets.
android_final=[]
ios_final=[]
for app in android_english:
price =app[7]
if price =='0':
android_final.append(app)
for app in ios_english:
price=app[4]
if price=='0.0':
ios_final.append(app)
print(len(android_final))
print(len(ios_final))
8864 3222
we are left with 8864 Android apps and 3222 iOS apps for analsis
As we mentioned in the introduction, our aim is to determine the kinds of apps that are likely to attract more users because our revenue is highly influenced by the number of people using our apps.
To minimize risks and overhead, our validation strategy for an app idea is comprised of three steps:
Because our end goal is to add the app on both the App Store and Google Play, we need to find app profiles that are successful on both markets. For instance, a profile that might work well for both markets might be a productivity app that makes use of gamification.
Let's begin the analysis by getting a sense of the most common genres for each market. For this, we'll build a frequency table for the prime_genre column of the App Store data set, and the Genres and Category columns of the Google Play data set.
We'll build two functions we can use to analyze the frequency tables:
def freq_table(dataset,index):
table={}
total=0
for row in dataset:
total+=1
value=row[index]
if value in table:
table[value]+=1
else:
table[value]=1
table_percentages={}
for key in table:
percentage =(table[key]/total) * 100
table_percentages [key]=percentage
return table_percentages
def display_table(dataset, index):
table = freq_table(dataset, index)
table_display = []
for key in table:
key_val_as_tuple = (table[key], key)
table_display.append(key_val_as_tuple)
table_sorted = sorted(table_display, reverse = True)
for entry in table_sorted:
print(entry[1], ':', entry[0])
We start by examining the ffrequency table for the prime_genre column of the App store data set
display_table(ios_final, -5)
Social Networking : 3.2898820608317814
We can see that among the free English apps, more than a half (58.16%) are games. Entertainment apps are close to 8%, followed by photo and video apps, which are close to 5%. Only 3.66% of the apps are designed for education, followed by social networking apps which amount for 3.29% of the apps in our data set.
The general impression is that App Store (at least the part containing free English apps) is dominated by apps that are designed for fun (games, entertainment, photo and video, social networking, sports, music, etc.), while apps with practical purposes (education, shopping, utilities, productivity, lifestyle, etc.) are more rare. However, the fact that fun apps are the most numerous doesn't also imply that they also have the greatest number of users — the demand might not be the same as the offer.
Let's continue by examining the Genres and Category columns of the Google Play data set (two columns which seem to be related).
display_table (android_final,1)# Category
EDUCATION : 1.1620036101083033
The landscape seems significantly different on Google Play: there are not that many apps designed for fun, and it seems that a good number of apps are designed for practical purposes (family, tools, business, lifestyle, productivity, etc.). However, if we investigate this further, we can see that the family category which accounts for almost 19% of the apps means mostly games for kids.
display_table(android_final,-4)
Education;Creativity : 0.04512635379061372
One way to find out what genres are the most popular (have the most users) is to calculate the average number of installs for each app genre. For the Google Play data set, we can find this information in the Installs column, but for the App Store data set this information is missing. As a workaround, we'll take the total number of user ratings as a proxy, which we can find in the rating_count_tot app.
Below, we calculate the average number of user ratings per app genre on the App Store:
genre_ios=freq_table(ios_final,-5)
for genre in genres_ios:
total = 0
len_genre =0
for app in ios_final:
genre_app=app[-5]
if genre_app==genre:
n_ratings=float(app[5])
total +=n_ratings
len_genre+=1
avg_n_ratings=total/len_genre
print(genre,':',avg_n_ratings)
NameErrorTraceback (most recent call last) <ipython-input-20-5465c0c1a6fb> in <module>() 1 genre_ios=freq_table(ios_final,-5) 2 ----> 3 for genre in genres_ios: 4 total = 0 5 len_genre =0 NameError: name 'genres_ios' is not defined