"Large stock moves are almost always driven by a compelling new product or service, or by new management that brings new ideas, or by new industry conditions that affect an entire industry group in a positive way."
Using Apple as our subject, we wish to examine how a company's stock performances are affected by certain products. Ultimately, we want to open up and provide easily-accessible insights and analysis. We would like for the success of a product to be more presentable than just numbers.
urllib,urllib2 - scraping html content from URLs re - regular expressions matching bs4 - beautifulsoup for clean scraping of html
%gui wx
import urllib, urllib2
import re
import datetime
import calendar
from bs4 import BeautifulSoup
import random
import pylab as pl
from pylab import get_current_fig_manager as gcfm
import wx
import numpy as np
Used for standardizing and making more convenient the way that we deal with dates from various sources such as Yahoo Finance and a datetime object.
class Date:
def __init__(self, date):
months = ['January', 'February', 'March', 'April', 'May', 'June', 'July', 'August', 'September', 'October', 'November', 'December']
if re.search('[a-zA-Z]{3}\ [0-9]{1,2},\ [0-9]{4}',date):
splitDate = date.split(' ')
threeLetterMonths = map(lambda month: month[:3], months)
self.m = int(threeLetterMonths.index(splitDate[0])+1)
if ',' in splitDate[1]:
self.d = int(splitDate[1].replace(',',''))
self.y = int(splitDate[2])
self.date = `self.m` + '-' + `self.d` + '-' + `self.y`
else:
splitDate = date.split('-')
self.date = date
self.m = int(splitDate[0])
self.d = int(splitDate[1])
self.y = int(splitDate[2])
def __repr__(self):
# return str('Date(' + self.date + ')')
return str(self.date)
# returns (start date, end date)
def dateRange(self, daysPadding): # daysPadding is in days
import datetime
if self.d == 0: # handle the case the day is 0 (day wasn't indicated on wikipedia page)
self.d = 1
date = datetime.date(int(self.y), int(self.m), int(self.d))
difference = datetime.timedelta(days=daysPadding)
beginInterval = date - difference
endInterval = date + difference
earliestDate = datetime.date(1985, 9, 2)
# earliest borderline check
if beginInterval < earliestDate:
beginInterval = earliestDate
endInterval = beginInterval + difference + difference
beginDate = Date(str(beginInterval.month) + '-' + str(beginInterval.day) + '-' + str(beginInterval.year))
endDate = Date(str(endInterval.month) + '-' + str(endInterval.day) + '-' + str(endInterval.year))
#make sure none of dates returned are weekends
import calendar
if calendar.weekday(beginDate.y, beginDate.m, beginDate.d) == 6: # beginDate is Sunday
# print 'beginDate is Sunday'
difference = datetime.timedelta(days=2)
date = datetime.date(int(beginDate.y), int(beginDate.m), int(beginDate.d))
beginInterval = date - difference
elif calendar.weekday(beginDate.y, beginDate.m, beginDate.d) == 5: # beginDate is Saturday
# print 'beginDate is Saturday'
difference = datetime.timedelta(days=1)
date = datetime.date(int(beginDate.y), int(beginDate.m), int(beginDate.d))
beginInterval = date - difference
if calendar.weekday(endDate.y, endDate.m, endDate.d) == 6: # endDate is Sunday
# print 'endDate is Sunday'
difference = datetime.timedelta(days=1)
date = datetime.date(int(endDate.y), int(endDate.m), int(endDate.d))
endInterval = date + difference
elif calendar.weekday(endDate.y, endDate.m, endDate.d) == 5: # endDate is Saturday
# print 'endDate is Saturday'
difference = datetime.timedelta(days=2)
date = datetime.date(int(endDate.y), int(endDate.m), int(endDate.d))
endInterval = date + difference
# begin interval is a weekend:
beginDate = Date(str(beginInterval.month) + '-' + str(beginInterval.day) + '-' + str(beginInterval.year))
endDate = Date(str(endInterval.month) + '-' + str(endInterval.day) + '-' + str(endInterval.year))
# print self
# print beginDate
# print endDate
return (beginDate, endDate)
#returns an integer representation of the date that makes the date easy to sort
def numericDate(self):
year = str(self.y)
month = str(self.m)
day = str(self.d)
if len(year) == 1: year = '0'+ year
if len(month) == 1: month = '0'+ month
if len(day) == 1: day = '0'+ day
return int(year + month + day)
Scrape and parse the Wikipedia page for Apple products from 'Timeline of Apple Inc. Products'
def parse_yahoo_stock(line):
parts = line.split(',')
parts_dict = {}
# for product releases there may be a difference between diff('high', 'close')
# as opposed to more steady differences for a 'normal' non-release day
legends = ['Date', 'Open', 'High', 'Low', 'Close', 'Volume', 'Adj Close']
floats = [0, 1, 1, 1, 1, 0, 1]
for i in range(len(parts)):
if i == 5:
parts_dict[legends[i]] = int(parts[i])
if floats[i]:
parts_dict[legends[i]] = float(parts[i])
else:
parts_dict[legends[i]] = parts[i]
return parts_dict
def getProductReleasesForApple():
import urllib, urllib2
from bs4 import BeautifulSoup
import re
article = "Timeline of Apple Inc. products"
article = urllib.quote(article) # sanitize
opener = urllib2.build_opener()
opener.addheaders = [('User-agent', 'Mozilla/5.0')] # wikipedia blocks obvious bot attempts
resource = opener.open("http://en.wikipedia.org/wiki/"+article)
data = resource.read()
resource.close()
soup = BeautifulSoup(data)
# print soup.find('div',id="bodyContent")
bodyContent = soup.find('div',id="bodyContent")
wikitables = bodyContent.find_all('table',class_="wikitable")
products = {}
for wikitable in wikitables:
m = re.search('<b>[0-9]+</b>', str(wikitable))
year = m.group(0)
# if first == 0:
# first = 1
first = 1
trs = wikitable.find_all('tr')
year = ''
date = ''
productName = ''
family = ''
deathDate = ''
rowspan = 0
rowspanFix = 1
spanSub = 0
for tr in trs:
tr = BeautifulSoup(str(tr))
if len(tr.find_all('td')) != 0:
tds = tr.find_all('td')
count = 0
# if len(tds.find_all('b')) != 0:
if first == 1:
for td in tds:
if count == 0:
m = re.search('<b>[0-9]+</b>', str(td))
year = m.group(0)
year = year[3:-4]
elif count == 1:
# m = re.search('>[a-zA-Z0-9\ ]+<', str(td))
# date = m.group(0)
date = td.text
date = date + ' ' + year
elif count == 2:
productName = td.text
elif count == 3:
family = td.text
elif count == 4:
deathDate = td.text
count += 1
first = 0
months = ['January', 'February', 'March', 'April', 'May', 'June', 'July', 'August', 'September', 'October', 'November', 'December']
dateSplit = str(date).split(' ')
if ',' in dateSplit[1]: dateSplit[1] = dateSplit[1][:-1]
if len(dateSplit) == 3:
newD = str(months.index(dateSplit[0])+1)+'-'+dateSplit[1]+'-'+dateSplit[2]
else:
newD = str(months.index(dateSplit[0])+1)+'-'+'00'+'-'+dateSplit[1]
products[productName] = [Date(newD), family, deathDate]
else:
if len(tds) == 3:
for td in tds:
if count == 0:
productName = td.text
if count == 1:
family = td.text
if count == 2:
deathDate = td.text
count += 1
months = ['January', 'February', 'March', 'April', 'May', 'June', 'July', 'August', 'September', 'October', 'November', 'December']
dateSplit = str(date).split(' ')
if ',' in dateSplit[1]: dateSplit[1] = dateSplit[1][:-1]
if len(dateSplit) == 3:
newD = str(months.index(dateSplit[0])+1)+'-'+dateSplit[1]+'-'+dateSplit[2]
else:
newD = str(months.index(dateSplit[0])+1)+'-'+'00'+'-'+dateSplit[1]
products[str(productName)] = [Date(newD), str(family), str(deathDate)]
elif len(tds) == 4:
for td in tds:
if count == 0:
date = td.text
date = date + ' ' + year
if count == 1:
productName = td.text
if count == 2:
family = td.text
if count == 3:
deathDate = td.text
count += 1
months = ['January', 'February', 'March', 'April', 'May', 'June', 'July', 'August', 'September', 'October', 'November', 'December']
dateSplit = str(date).split(' ')
if ',' in dateSplit[1]: dateSplit[1] = dateSplit[1][:-1]
if len(dateSplit) == 3:
newD = str(months.index(dateSplit[0])+1)+'-'+dateSplit[1]+'-'+dateSplit[2]
else:
newD = str(months.index(dateSplit[0])+1)+'-'+'00'+'-'+dateSplit[1]
products[str(productName)] = [Date(newD), str(family), str(deathDate)]
elif len(tr.find_all('th')) != 0:
ths = tr.find_all('th')
return products
getProductReleasesForApple()
{'AirPort (802.11b, "Graphite")': [7-21-1999, 'AirPort', 'November 13, 2001'], 'AirPort Express (802.11g)': [6-7-2004, 'AirPort Express', 'March 17, 2008'], 'AirPort Express 802.11n (1st gen)': [3-17-2008, 'AirPort Express', 'June 11, 2012'], 'AirPort Express 802.11n (2nd gen)': [6-11-2012, 'AirPort Express', 'current'], 'AirPort Extreme 802.11n (3rd gen)': [3-3-2009, 'AirPort', 'October 20, 2009'], 'AirPort Extreme 802.11n (4th gen)': [10-20-2009, 'AirPort', 'June 21, 2011'], 'AirPort Extreme 802.11n (5th gen)': [6-21-2011, 'AirPort', 'current'], 'AirPort Utility': [10-12-2011, 'Software', 'current'], 'Aperture 2': [2-12-2008, 'Software', 'February 9, 2010'], u'Aperture 3': [2-9-2010, u'Software', u'current'], 'Apple 3.5" Drive': [9-1-1986, 'Drives', ''], 'Apple 5.25" Drive': [9-1-1986, 'Drives', ''], 'Apple Color Plotter': [6-00-1984, 'Printers', ''], 'Apple ColorMonitor IIc': [9-1-1985, 'Displays', ''], 'Apple ColorMonitor IIe': [9-1-1985, 'Displays', ''], 'Apple Design Powered Speakers': [3-22-1993, 'PowerCD', 'January 1, 1995'], u'Apple Dot Matrix Printer': [10-1-1982, u'Printers', u'December 1, 1983'], 'Apple DuoDisk 5.25': [5-1-1984, 'Drives', ''], 'Apple EarPods': [9-12-2012, 'iPod Accessories', 'current'], 'Apple FDHD SuperDrive': [8-1-1989, 'Drives', ''], 'Apple Hard Disk 20SC': [9-1-1986, 'Drives', ''], 'Apple Hi-Resolution Monochrome Display': [3-7-1989, 'Displays', ''], u'Apple I': [7-1-1976, u'Apple I', u'September 1, 1977'], u'Apple II': [4-1-1977, u'Apple II', u'June 1, 1979'], 'Apple II EuroPlus': [6-1-1979, 'Apple II', 'December 1, 1982'], 'Apple II J-Plus': [6-1-1979, 'Apple II', 'December 1, 1982'], u'Apple II Plus': [6-1-1979, u'Apple II', u'December 1, 1982'], 'Apple IIGS': [9-1-1986, 'Apple II', 'December 1, 1992'], 'Apple IIGS (1 MB, ROM 3)[3]': [10-1-1989, 'Apple II', 'December 1, 1992'], u'Apple III': [9-1-1980, u'Apple III', u'December 1, 1981'], 'Apple III Plus': [12-1-1983, 'Apple III', 'April 1, 1984'], 'Apple III Revised[1]': [12-1-1981, 'Apple III', 'December 1, 1983'], 'Apple IIc': [4-1-1984, 'Apple II', 'September 1, 1986'], 'Apple IIc Memory Expansion': [9-1-1986, 'Apple II', 'September 1, 1988'], 'Apple IIc Plus': [9-1-1988, 'Apple II', 'September 1, 1990'], u'Apple IIe': [1-1-1983, u'Apple II', u'March 1, 1985'], 'Apple IIe Card (requires Macintosh LC)': [3-1-1991, 'Apple II', 'May 1, 1995'], 'Apple IIe Enhanced': [3-1-1985, 'Apple II', 'January 1, 1987'], 'Apple IIe Platinum': [1-1-1987, 'Apple II', 'November 1, 1993'], 'Apple ImageWriter': [12-1-1983, 'Printers', 'December 1, 1985'], 'Apple ImageWriter II': [4-1-1985, 'Printers', '1996'], 'Apple ImageWriter Wide Carriage': [6-00-1984, 'Printers', ''], 'Apple Keyboard (short)': [3-3-2009, 'Apple Keyboard', 'October 20, 2009'], 'Apple Keyboard with Numeric Keypad': [8-7-2007, 'Apple Keyboard', 'current'], 'Apple LaserWriter': [1-1-1985, 'Printers', 'December 1, 1985'], u'Apple LaserWriter Family': [1-1-1988, u'Printers', u''], 'Apple LaserWriter Plus': [1-16-1986, 'Printers', ''], 'Apple Letter Quality Printer': [10-1-1982, 'Printers', 'December 1, 1983'], u'Apple Lisa 2': [1-1-1984, u'68000', u'January 1, 1985'], 'Apple Lisa[2]': [1-1-1983, '68000', 'January 1, 1984'], 'Apple LocalTalk Connector': [1-1-1985, 'Networking', ''], 'Apple Macintosh Portrait Display': [3-7-1989, 'Displays', ''], 'Apple Mighty Mouse (revised)': [8-7-2007, 'Apple Mouse', 'current'], 'Apple Modem 1200': [1-24-1984, 'Modems', ''], 'Apple Modem 2400': [7-00-1989, 'Modems', 'December 1992'], 'Apple Modem 300': [1-24-1984, 'Modems', ''], 'Apple Monochrome Monitor': [9-1-1986, 'Displays', ''], 'Apple Mouse IIc': [4-1-1984, 'Apple Mouse', 'December 1, 1985'], u'Apple Network Server 500': [2-15-1996, u'Network Server', u'April 1, 1997'], 'Apple Network Server 700/150': [2-15-1996, 'Network Server', 'April 1, 1997'], 'Apple Network Server 700/200': [10-16-1996, 'Network Server', 'April 1, 1997'], 'Apple PC 5.25" Drive': [8-00-1987, 'Drives', ''], 'Apple Personal Modem': [4-1-1985, 'Modems', ''], u'Apple ProFile': [9-1-1981, u'Drives', u''], 'Apple QuickTake 100': [2-2-1994, 'QuickTake', 'January 1, 1995'], 'Apple Remote Desktop 3': [4-11-2006, 'Software', 'current'], 'Apple Scanner': [8-00-1988, 'Scanner', ''], 'Apple Scribe Printer': [4-1-1984, 'Printers', 'December 1, 1985'], 'Apple SilenType': [6-1-1979, 'Printers', 'October 1, 1982'], u'Apple TV (1st gen)': [3-21-2007, u'Apple TV', u'September 1, 2010'], 'Apple TV (2nd gen)': [9-1-2010, 'Apple TV', 'March 7, 2012'], 'Apple TV (3rd gen)': [3-16-2012, 'Apple TV', 'current'], 'Apple Tape Backup 40SC': [8-00-1987, 'Drives', ''], 'Apple Two Page Monochrome Monitor': [3-7-1989, 'Displays', ''], 'Apple UniDisk 3.5': [9-1-1985, 'Drives', 'January 1, 1987'], 'Apple UniDisk 5.25': [6-00-1985, 'Drives', 'September 1, 1986'], 'Apple Wireless Keyboard (Aluminum)': [8-7-2007, 'Apple Keyboard', 'current'], 'Apple Writer 1.0': [6-1-1979, 'Software', 'January 1, 1992'], 'AppleCD SC': [3-00-1988, 'Drives', ''], 'AppleColor 100': [12-00-1984, 'Displays', ''], 'AppleColor Composite Monitor': [9-1-1986, 'Displays', ''], 'AppleColor Hi-Resolution RGB Monitor': [3-2-1987, 'Displays', 'December 1992'], 'AppleColor RGB Monitor': [9-1-1986, 'Displays', ''], 'AppleFax Modem': [8-00-1987, 'Modems', ''], 'AppleShare Server 1.0': [1-1-1987, 'Software', ''], 'Bell & Howell': [6-1-1979, 'Apple II', 'December 1, 1982'], 'Bell & Howell Disk II': [6-1-1979, 'Drives', 'December 1, 1982'], 'Bento 2.0': [10-14-2008, 'Software', 'September 29, 2009'], 'Cards': [10-12-2011, 'Software', 'current'], 'Centris / Quadra 660AV': [7-29-1993, 'Centris / Quadra', 'September 12, 1994'], 'Centris 610': [2-10-1993, 'Centris', 'October 21, 1993'], 'Centris 650': [2-10-1993, 'Centris', 'October 21, 1993'], 'Cinema Display (20")': [6-28-2004, 'Displays', 'February 2009'], 'Cinema Display (22")': [7-19-2000, 'Displays', 'January 28, 2003'], 'Cinema Display (23")': [6-28-2004, 'Displays', 'November 2008'], 'Cinema Display (30")': [6-28-2004, 'Displays', 'July 27, 2010'], u'Disk II': [6-1-1978, u'Drives', u'May 1, 1984'], 'Disk III': [9-1-1980, 'Drives', 'May 1, 1984'], 'Disk IIc': [4-1-1984, 'Drives', ''], 'FileMaker Pro 10': [1-6-2009, 'Software', 'March 9, 2010'], 'Final Cut Express 4': [11-15-2007, 'Software', 'July 23, 2009'], 'Final Cut Server': [9-18-2008, 'Software', 'July 23, 2009'], 'Final Cut Server 1.5': [7-23-2009, 'Software', 'June 21, 2011'], 'Final Cut Studio 2': [4-17-2008, 'Software', 'July 23, 2009'], 'Final Cut Studio 3': [7-23-2009, 'Software', 'current'], 'Find My Friends': [10-12-2011, 'Software', 'current'], 'ImageWriter LQ': [8-00-1987, 'Printers', 'December 1990'], 'In-Ear Headphones': [9-9-2008, 'iPod Accessories', 'current'], 'LED Cinema Display': [10-14-2008, 'Displays', 'current'], 'Logic Express 8': [5-29-2008, 'Software', 'July 23, 2009'], 'Logic Express 9': [7-23-2009, 'Software', 'December 13, 2011'], 'Logic Studio': [9-12-2007, 'Software', 'July 23, 2009'], 'Logic Studio 2': [7-23-2009, 'Software', 'current'], u'Mac Mini': [1-11-2005, u'Mac Mini', u'February 28, 2006'], 'Mac Mini (Early 2009)': [3-3-2009, 'Mac Mini', 'October 20, 2009'], 'Mac Mini (Late 2009)': [10-20-2009, 'Mac Mini', ''], 'Mac Mini (Late 2012)': [10-23-2012, 'Mac Mini', 'current'], 'Mac Mini (Mid 2007)': [8-7-2007, 'Mac Mini', 'March 3, 2009'], 'Mac Mini (Mid 2010)': [6-15-2010, 'Mac Mini', 'July 20, 2011'], 'Mac Mini (Mid 2011)': [7-20-2011, 'Mac Mini', 'October 23, 2012'], 'Mac Mini Core Duo': [2-28-2006, 'Mac Mini', 'August 7, 2007'], 'Mac Mini Core Solo': [2-28-2006, 'Mac Mini', 'September 6, 2006'], 'Mac OS X Leopard (10.5)': [10-27-2007, 'Software', 'August 28, 2009'], 'Mac OS X Lion (10.7)': [7-20-2011, 'Software', 'July 25, 2012'], 'Mac OS X Server (10.5)': [10-27-2007, 'Software', 'August 28, 2009'], 'Mac OS X Server (10.6)': [8-28-2009, 'Software', 'July 20, 2011'], 'Mac OS X Snow Leopard (10.6)': [8-28-2009, 'Software', 'July 20, 2011'], 'Mac Pro': [8-7-2006, 'Mac Pro', 'January 8, 2008'], 'Mac Pro (Early 2008)': [1-8-2008, 'Mac Pro', 'March 3, 2009'], 'Mac Pro (Early 2009)': [3-3-2009, 'Mac Pro', 'August 9, 2010'], 'Mac Pro (Mid 2010)': [8-9-2010, 'Mac Pro', 'June 11, 2012'], 'Mac Pro (Mid 2012)': [6-11-2012, 'Mac Pro', 'current'], 'MacBook': [5-16-2006, 'MacBook', 'February 26, 2008'], 'MacBook (Early 2008)': [2-26-2008, 'MacBook', 'October 14, 2008'], 'MacBook (Early 2009) (White)': [1-29-2009, 'MacBook', 'May 27, 2009'], 'MacBook (Late 2008) (Aluminum)': [10-14-2008, 'MacBook', 'June 8, 2009'], 'MacBook (Late 2008) (White)': [10-14-2008, 'MacBook', 'January 29, 2009'], 'MacBook (Late 2009)': [10-20-2009, 'MacBook', 'May 18, 2010'], 'MacBook (Mid 2009)': [5-27-2009, 'MacBook', 'October 20, 2009'], 'MacBook (Mid 2010)': [5-18-2010, 'MacBook', 'July 20, 2011'], 'MacBook Air (Early 2008)': [1-15-2008, 'MacBook Air', 'October 14, 2008'], 'MacBook Air (Late 2008)': [10-14-2008, 'MacBook Air', 'June 8, 2009'], 'MacBook Air (Late 2010)': [10-20-2010, 'MacBook Air', 'July 20, 2011'], 'MacBook Air (Mid 2009)': [6-8-2009, 'MacBook Air', 'October 20, 2010'], 'MacBook Air (Mid 2011)': [7-20-2011, 'MacBook Air', 'June 11, 2012'], 'MacBook Air (Mid 2012)': [6-11-2012, 'MacBook Air', 'current'], 'MacBook Air External SuperDrive': [1-15-2008, 'Drives', 'current'], 'MacBook Pro (15")': [2-14-2006, 'MacBook Pro', 'February 26, 2008'], 'MacBook Pro (17")': [4-24-2006, 'MacBook Pro', 'February 26, 2008'], 'MacBook Pro (Early 2008) (15")': [2-26-2008, 'MacBook Pro', 'October 14, 2008'], 'MacBook Pro (Early 2008) (17")': [2-26-2008, 'MacBook Pro', 'January 6, 2009'], u'MacBook Pro (Early 2009) (17")': [1-6-2009, u'MacBook Pro', u'June 8, 2009'], 'MacBook Pro (Early 2011)': [2-24-2011, 'MacBook Pro', 'October 24, 2011'], 'MacBook Pro (Late 2008) (15")': [10-14-2008, 'MacBook Pro', 'June 8, 2009'], 'MacBook Pro (Late 2011)': [10-24-2011, 'MacBook Pro', 'June 11, 2012'], 'MacBook Pro (Mid 2009)': [6-8-2009, 'MacBook Pro', 'April 13, 2010'], 'MacBook Pro (Mid 2010)': [4-13-2010, 'MacBook Pro', 'February 24, 2011'], 'MacBook Pro (Mid 2012)': [6-11-2012, 'MacBook Pro', 'current'], 'MacPaint 1.0': [1-24-1984, 'Software', ''], 'MacWrite 1.0': [1-24-1984, 'Software', ''], 'Macintosh (128K)': [1-24-1984, 'Compact', 'September 10, 1984'], 'Macintosh 128K (revised)': [9-10-1984, 'Compact', 'October 1, 1985'], 'Macintosh 512K': [9-10-1984, 'Compact', 'April 14, 1986'], 'Macintosh 512Ke': [4-14-1986, 'Compact', 'October 1, 1987'], 'Macintosh 800K External Drive': [1-16-1986, 'Drives', ''], 'Macintosh Classic': [10-15-1990, 'Compact', 'September 14, 1992'], 'Macintosh Classic II': [10-21-1991, 'Compact', 'September 13, 1993'], 'Macintosh Color Classic': [2-10-1993, 'Compact', 'May 16, 1994'], 'Macintosh Color Classic II': [10-10-1993, 'Compact', 'November 1, 1995'], 'Macintosh External Disk Drive (400K)': [1-24-1984, 'Drives', 'January 1, 1986'], 'Macintosh Hard Disk 20': [9-1-1985, 'Drives', ''], 'Macintosh II': [3-2-1987, 'Mac II', 'January 15, 1990'], 'Macintosh IIci': [9-20-1989, 'Mac II', 'February 20, 1993'], 'Macintosh IIcx': [3-7-1989, 'Mac II', 'March 11, 1991'], u'Macintosh IIfx': [3-19-1990, u'Mac II', u'April 15, 1992'], 'Macintosh IIsi': [10-15-1990, 'Mac II', 'March 15, 1993'], 'Macintosh IIvi': [10-19-1992, 'Mac II', 'February 10, 1993'], 'Macintosh IIvx': [10-19-1992, 'Mac II', 'October 10, 1993'], 'Macintosh IIx': [9-19-1988, 'Mac II', 'October 15, 1990'], 'Macintosh LC': [10-15-1990, 'LC', 'March 23, 1992'], 'Macintosh LC 520': [6-28-1993, 'LC', 'February 2, 1994'], u'Macintosh LC 550': [2-2-1994, u'LC', u'March 23, 1995'], 'Macintosh LC 575': [2-2-1994, 'LC', 'April 3, 1995'], 'Macintosh LC 580': [4-3-1995, 'LC', 'October 1, 1995'], u'Macintosh LC II': [3-23-1992, u'LC', u'March 15, 1993'], u'Macintosh LC III / III+': [2-10-1993, u'LC', u'February 14, 1994'], u'Macintosh Plus': [1-16-1986, u'Compact', u'October 15, 1990'], u'Macintosh Plus (Platinum)': [1-1-1987, u'Compact', u'October 15, 1990'], 'Macintosh Portable': [9-20-1989, 'Portable', 'February 11, 1991'], u'Macintosh Portable (backlit screen)': [2-11-1991, u'Portable', u'October 21, 1991'], 'Macintosh SE': [3-2-1987, 'Compact', 'August 1, 1989'], 'Macintosh SE FDHD': [8-1-1989, 'Compact', 'October 15, 1990'], u'Macintosh SE/30': [1-19-1989, u'Compact', u'October 21, 1991'], 'Macintosh Server G3': [3-2-1998, 'Macintosh Server', 'January 1, 1999'], 'Macintosh Server G3 (Blue & White)': [1-5-1999, 'Macintosh Server', 'August 31, 1999'], 'Macintosh Server G4': [8-31-1999, 'Macintosh Server', 'July 19, 2000'], 'Macintosh Server G4 MDD': [8-27-2002, 'Macintosh Server', 'January 28, 2003'], 'Macintosh TV': [10-21-1993, 'LC', 'February 1, 1995'], u'Macintosh XL': [1-1-1985, u'68000', u'April 1, 1985'], 'Magic Mouse': [10-20-2009, 'Apple Mouse', 'current'], 'Magic Trackpad': [7-27-2010, 'Trackpad', 'current'], 'MobileMe': [7-9-2008, 'Software', 'June 30, 2012'], 'Modem IIB (Novation CAT)': [9-1-1980, 'Modems', ''], 'Monitor II (various third party)': [9-1-1980, 'Displays', ''], 'Monitor III': [9-1-1980, 'Displays', ''], 'Newton Message Pad': [8-16-1993, 'Newton', 'March 1, 1994'], 'Nike+iPod': [7-13-2006, 'iPod Accessories', 'current'], 'OS X Mountain Lion (10.8)': [7-25-2012, 'Software', 'current'], 'Performa 5260 / 5300': [3-10-1996, 'Performa', 'April 1, 1997'], 'Performa 5400': [4-1-1996, 'Performa', 'February 17, 1997'], 'Performa 6360': [10-17-1996, 'Performa', 'October 1, 1997'], 'Performa 6400': [10-23-1996, 'Performa', 'May 1, 1997'], 'Pippin': [12-1-1994, 'Pippin', 'December 1, 1997'], 'Power Macintosh 4400': [11-15-1996, 'Power Macintosh', 'October 11, 1997'], u'Power Macintosh 5500': [2-17-1997, u'Power Macintosh', u'March 31, 1998'], 'Power Macintosh 6100': [3-14-1994, 'Power Macintosh', 'May 18, 1996'], u'Power Macintosh 6200 / 6300': [1-28-1995, u'Power Macintosh', u'October 17, 1996'], 'Power Macintosh 6500': [2-17-1997, 'Power Macintosh', 'March 14, 1998'], 'Power Macintosh 7100': [3-14-1994, 'Power Macintosh', 'January 6, 1996'], 'Power Macintosh 7200': [8-7-1995, 'Power Macintosh', 'April 1, 1996'], 'Power Macintosh 7300': [2-17-1997, 'Power Macintosh', 'November 10, 1997'], 'Power Macintosh 7500': [8-7-1995, 'Power Macintosh', 'February 17, 1997'], 'Power Macintosh 7600': [4-1-1996, 'Power Macintosh', 'October 1, 1997'], 'Power Macintosh 8100': [3-14-1994, 'Power Macintosh', 'August 14, 1996'], 'Power Macintosh 8500': [8-7-1995, 'Power Macintosh', 'February 17, 1997'], 'Power Macintosh 8600': [2-17-1997, 'Power Macintosh', 'February 17, 1998'], 'Power Macintosh 9500': [6-19-1995, 'Power Macintosh', 'February 17, 1997'], 'Power Macintosh 9600': [2-17-1997, 'Power Macintosh', 'March 17, 1998'], u'Power Macintosh G3 (Blue & White)': [1-5-1999, u'Power Macintosh', u'October 13, 1999'], u'Power Macintosh G3 AIO': [1-31-1998, u'Power Macintosh', u'October 17, 1998'], 'Power Macintosh G3 desktop': [11-10-1997, 'Power Macintosh', 'January 5, 1999'], 'Power Macintosh G3 minitower': [11-10-1997, 'Power Macintosh', 'January 5, 1999'], 'Power Macintosh G4 Cube': [7-19-2000, 'Power Macintosh', 'July 3, 2001'], 'Power Macintosh G4 Graphite': [10-13-1999, 'Power Macintosh', 'July 18, 2001'], 'Power Macintosh G4 MDD': [8-13-2002, 'Power Macintosh', 'June 9, 2004'], 'Power Macintosh G4 Quicksilver': [7-18-2001, 'Power Macintosh', 'August 13, 2002'], 'Power Macintosh G5': [6-23-2003, 'Power Macintosh', 'June 9, 2004'], 'Power Macintosh G5 FX': [6-9-2004, 'Power Macintosh', 'October 19, 2005'], 'Power Macintosh G5 dual core': [10-19-2005, 'Power Macintosh', 'August 7, 2006'], 'Power Macintosh/Performa 5200': [4-3-1995, 'Performa', 'October 1, 1996'], 'Power Macintosh/Performa 5300': [8-28-1995, 'Performa', 'March 1, 1997'], u'PowerBook ("Pismo")': [2-16-2000, u'PowerBook G3', u'January 9, 2001'], 'PowerBook 100': [10-21-1991, 'PowerBook', 'August 3, 1992'], 'PowerBook 140': [10-21-1991, 'PowerBook', 'August 3, 1992'], 'PowerBook 1400': [11-20-1996, 'PowerBook', 'May 6, 1998'], 'PowerBook 145': [8-3-1992, 'PowerBook', 'July 7, 1993'], 'PowerBook 145b': [6-7-1993, 'PowerBook', 'July 18, 1994'], 'PowerBook 150': [7-18-1994, 'PowerBook', 'October 14, 1995'], 'PowerBook 160': [10-19-1992, 'PowerBook', 'August 16, 1993'], 'PowerBook 165': [8-16-1993, 'PowerBook', 'July 18, 1994'], 'PowerBook 165c': [2-10-1993, 'PowerBook', 'December 13, 1993'], 'PowerBook 170': [10-21-1991, 'PowerBook', 'October 19, 1992'], 'PowerBook 180': [10-19-1992, 'PowerBook', 'May 16, 1994'], 'PowerBook 180c': [6-7-1993, 'PowerBook', 'March 14, 1994'], 'PowerBook 190': [8-28-1995, 'PowerBook', 'September 1, 1996'], 'PowerBook 2400c': [5-8-1997, 'PowerBook', 'March 14, 1998'], 'PowerBook 3400': [2-17-1997, 'PowerBook', 'March 14, 1998'], 'PowerBook 520/c': [5-16-1994, 'PowerBook 500', 'September 16, 1995'], 'PowerBook 5300': [8-28-1995, 'PowerBook', 'September 1, 1996'], 'PowerBook 540/c': [5-16-1994, 'PowerBook 500', 'August 16, 1995'], 'PowerBook 550': [5-16-1994, 'PowerBook 500', 'April 1, 1996'], 'PowerBook Duo 210': [10-19-1992, 'PowerBook Duo', 'October 21, 1993'], 'PowerBook Duo 230': [10-19-1992, 'PowerBook Duo', 'July 27, 1994'], 'PowerBook Duo 2300c': [8-28-1995, 'PowerBook Duo', 'February 1, 1997'], 'PowerBook Duo 250': [10-21-1993, 'PowerBook Duo', 'May 16, 1994'], 'PowerBook Duo 270c': [10-21-1993, 'PowerBook Duo', 'May 16, 1994'], 'PowerBook Duo 280': [5-16-1994, 'PowerBook Duo', 'November 14, 1994'], 'PowerBook Duo 280c': [5-16-1994, 'PowerBook Duo', 'January 1, 1996'], 'PowerBook G3': [11-10-1997, 'PowerBook G3', 'March 14, 1998'], 'PowerBook G3 ("Lombard")': [5-10-1999, 'PowerBook G3', 'February 16, 2000'], 'PowerBook G3 series': [5-6-1998, 'PowerBook G3', 'May 10, 1999'], u'PowerBook G4 Aluminum (12")': [1-7-2003, u'PowerBook G4', u'May 16, 2006'], 'PowerBook G4 Aluminum (15")': [9-16-2003, 'PowerBook G4', 'February 14, 2006'], 'PowerBook G4 Aluminum (17")': [1-7-2003, 'PowerBook G4', 'April 24, 2006'], u'PowerBook G4 Titanium': [1-00-2001, u'PowerBook G4', u'September 16, 2003'], 'PowerCD': [3-22-1993, 'PowerCD', 'January 1, 1995'], 'Printer IIA (Centronics 779)': [9-1-1980, 'Printers', ''], 'Quadra 605': [10-21-1993, 'Quadra', 'October 17, 1994'], 'Quadra 610': [10-21-1993, 'Quadra', 'July 18, 1994'], 'Quadra 630': [7-18-1994, 'Quadra', 'April 17, 1995'], 'Quadra 650': [10-21-1993, 'Quadra', 'September 12, 1994'], 'Quadra 700': [10-21-1991, 'Quadra', 'March 15, 1993'], 'Quadra 800': [2-10-1993, 'Quadra', 'March 14, 1994'], 'Quadra 840AV': [7-29-1993, 'Quadra', 'July 18, 1994'], 'Quadra 900': [10-21-1991, 'Quadra', 'May 18, 1992'], 'Quadra 950': [5-18-1992, 'Quadra', 'October 14, 1995'], u'Retina MacBook Pro': [2-13-2013, u'MacBook Pro', u'current'], 'Retina MacBook Pro (3rd gen) (13")': [10-23-2012, 'MacBook Pro', 'current'], 'Retina MacBook Pro (3rd gen) (15")': [6-11-2012, 'MacBook Pro', 'February 13, 2013'], 'Server G4 Quicksilver': [9-8-2001, 'Macintosh Server', 'May 14, 2002'], 'Shake 4': [6-20-2006, 'Software', 'July 30, 2009'], 'Thunderbolt Display': [7-20-2011, 'Displays', 'current'], 'Time Capsule (1st gen)': [2-29-2008, 'AirPort, drives', 'March 3, 2009'], 'Time Capsule (2nd gen) (1 TB)': [3-3-2009, 'AirPort, drives', 'March 31, 2010'], 'Time Capsule (2nd gen) (2 TB)': [7-30-2009, 'AirPort, drives', 'March 31, 2010'], 'Time Capsule (2nd gen) (500 GB)': [3-3-2009, 'AirPort, drives', 'July 30, 2009'], 'Time Capsule (3rd gen)': [3-31-2010, 'AirPort, drives', 'June 21, 2011'], 'Time Capsule (4th gen)': [6-21-2011, 'AirPort, drives', 'current'], 'Twentieth Anniversary Macintosh': [3-20-1997, 'Power Macintosh', 'March 14, 1998'], 'Workgroup Server 60': [7-26-1993, 'Workgroup Server', 'October 17, 1995'], 'Workgroup Server 6150': [4-26-1994, 'Workgroup Server', 'October 1, 1995'], 'Workgroup Server 7250': [2-26-1996, 'Workgroup Server', 'April 21, 1997'], 'Workgroup Server 7350': [4-21-1997, 'Workgroup Server', 'March 2, 1998'], 'Workgroup Server 80': [3-22-1993, 'Workgroup Server', 'October 17, 1995'], 'Workgroup Server 8150': [4-26-1994, 'Workgroup Server', 'February 26, 1996'], 'Workgroup Server 8550': [2-26-1996, 'Workgroup Server', 'April 21, 1997'], 'Workgroup Server 9150': [4-26-1994, 'Workgroup Server', 'February 26, 1996'], 'Workgroup Server 95': [3-22-1993, 'Workgroup Server', 'April 3, 1995'], 'Workgroup Server 9650': [4-21-1997, 'Workgroup Server', 'March 2, 1998'], 'Xsan 2': [2-19-2008, 'Software', 'current'], 'Xserve': [5-14-2002, 'Xserve', 'February 10, 2003'], 'Xserve (2009)': [4-7-2009, 'Xserve', 'January 31, 2011'], u'Xserve (Early 2008)': [1-8-2008, u'Xserve', u'April 7, 2009'], 'Xserve (Intel)': [8-7-2006, 'Xserve', 'January 8, 2008'], 'Xserve Cluster Node': [2-10-2003, 'Xserve', 'January 6, 2004'], 'Xserve Cluster Node G5': [1-6-2004, 'Xserve', 'August 7, 2006'], u'Xserve G5': [1-6-2004, u'Xserve', u'August 7, 2006'], 'Xserve slot loading': [2-10-2003, 'Xserve', 'January 6, 2004'], 'eMac': [4-29-2002, 'eMac', 'July 5, 2006'], 'eMate 300': [3-7-1997, 'Newton', 'February 27, 1998'], 'iBook': [7-21-1999, 'iBook', 'September 13, 2000'], 'iBook (14")': [1-7-2002, 'iBook', 'October 22, 2003'], 'iBook (FireWire)': [9-13-2000, 'iBook', 'May 1, 2001'], 'iBook (white)': [5-1-2001, 'iBook', 'October 2003'], 'iBook G4 (12" / 14")': [10-22-2003, 'iBook', 'May 16, 2006'], u'iBooks Author': [1-19-2012, u'Software', u'current'], 'iCloud': [10-12-2011, 'Software', 'current'], "iLife '09": [1-27-2009, 'Software', 'October 20, 2010'], "iLife '11": [10-20-2010, 'Software', 'current'], 'iMac (21.5") (Late 2012)': [11-30-2012, 'iMac', 'current'], 'iMac (27") (Late 2012)': [12-00-2012, 'iMac', 'current'], u'iMac (Early 2006)': [1-10-2006, u'iMac', u'September 6, 2006'], 'iMac (Early 2008)': [4-28-2008, 'iMac', 'March 3, 2009'], 'iMac (Early 2009)': [3-3-2009, 'iMac', 'October 20, 2009'], 'iMac (Late 2009)': [10-20-2009, 'iMac', 'July 27, 2010'], 'iMac (Mid 2006)': [9-6-2006, 'iMac', 'August 7, 2007'], 'iMac (Mid 2007)': [8-7-2007, 'iMac', 'April 28, 2008'], 'iMac (Mid 2011)': [5-3-2011, 'iMac', 'November 30, 2012'], 'iMac (slot loading)': [10-5-1999, 'iMac', 'January 7, 2002'], 'iMac G3': [8-15-1998, 'iMac', 'May 10, 1999'], u'iMac G4 15"': [1-7-2002, u'iMac', u'August 31, 2004'], 'iMac G4 17"': [7-17-2002, 'iMac', 'August 31, 2004'], 'iMac G4 20"': [11-18-2003, 'iMac', 'August 31, 2004'], 'iMac G5 17"': [8-31-2004, 'iMac', 'January 10, 2006'], 'iMac G5 20"': [8-31-2004, 'iMac', 'March 20, 2006'], 'iOS 5': [10-12-2011, 'Software', 'September 19, 2012'], 'iOS 6': [9-19-2012, 'Software', 'current'], 'iPad (3rd gen)': [3-16-2012, 'iPad', 'October 23, 2012'], 'iPad (4th gen) (Wi-Fi + Cellular)': [11-16-2012, 'iPad', 'current'], 'iPad (4th gen) (Wi-Fi)': [11-2-2012, 'iPad', 'current'], 'iPad (Wi-Fi + 3G)': [4-30-2010, 'iPad', 'March 2, 2011'], 'iPad (Wi-Fi)': [4-3-2010, 'iPad', 'March 2, 2011'], 'iPad 2 (16 GB)': [3-11-2011, 'iPad', 'current'], 'iPad 2 (32 & 64 GB)': [3-11-2011, 'iPad', 'March 7, 2012'], 'iPad Mini (Wi-Fi + Cellular)': [11-16-2012, 'iPad', 'current'], 'iPad Mini (Wi-Fi)': [11-2-2012, 'iPad', 'current'], 'iPhone (1st generation)': [6-29-2007, 'iPhone', 'July 11, 2008'], 'iPhone 3G (16 GB)': [7-11-2008, 'iPhone', 'June 19, 2009'], 'iPhone 3G (8 GB)': [7-11-2008, 'iPhone', 'June 24, 2010'], 'iPhone 3GS (16 & 32 GB)': [6-19-2009, 'iPhone', 'June 24, 2010'], 'iPhone 3GS (8 GB)': [6-24-2010, 'iPhone', 'September 12, 2012'], 'iPhone 4 (8 GB)': [10-14-2011, 'iPhone', 'current'], u'iPhone 4 (CDMA) (16 & 32 GB)': [2-10-2011, u'iPhone', u'October 4, 2011'], 'iPhone 4 (GSM) (16 & 32 GB)': [6-24-2010, 'iPhone', 'October 4, 2011'], 'iPhone 4S (16 GB)': [10-14-2011, 'iPhone', 'current'], 'iPhone 4S (32 & 64 GB)': [10-14-2011, 'iPhone', 'September 12, 2012'], 'iPhone 5': [9-21-2012, 'iPhone', 'current'], 'iPhone Micro USB Adapter': [10-14-2011, 'iPhone', 'current'], 'iPod (2nd gen)': [7-17-2002, 'iPod Classic', 'April 28, 2003'], 'iPod (3rd gen)': [4-00-28, 'iPod Classic', 'July 19, 2004'], 'iPod (4th gen)': [7-00-19, 'iPod Classic', 'October 12, 2005'], 'iPod (5th gen)': [10-12-2005, 'iPod Classic', 'September 5, 2007'], 'iPod (st gen)': [10-23-2001, 'iPod Classic', 'July 17, 2002'], 'iPod Classic (6th gen)': [9-5-2007, 'iPod Classic', 'September 9, 2008'], 'iPod Classic (6th gen) (120 GB)': [9-9-2008, 'iPod Classic', 'September 9, 2009'], 'iPod Classic (6th gen) (160 GB)': [9-9-2009, 'iPod Classic', 'current'], 'iPod Mini (1st gen)': [1-6-2004, 'iPod Mini', 'February 23, 2005'], 'iPod Mini (2nd gen)': [2-23-2005, 'iPod Mini', 'September 7, 2005'], 'iPod Nano (3rd gen)': [9-5-2007, 'iPod Nano', 'September 6, 2008'], 'iPod Nano (4th gen)': [9-9-2008, 'iPod Nano', 'September 9, 2009'], 'iPod Nano (5th gen)': [9-9-2009, 'iPod Nano', 'September 1, 2010'], 'iPod Nano (6th gen)': [9-1-2010, 'iPod Nano', 'September 12, 2012'], 'iPod Nano (7th gen)': [10-11-2012, 'iPod Nano', 'current'], 'iPod Photo': [9-26-2004, 'iPod Classic', 'June 28, 2005'], 'iPod Radio Remote': [1-10-2006, 'iPod Accessories', 'June 25, 2009'], 'iPod Shuffle (1st gen)': [1-11-2005, 'iPod Shuffle', 'September 12, 2006'], 'iPod Shuffle (2nd gen)': [9-00-12, 'iPod Shuffle', 'March 11, 2009'], 'iPod Shuffle (3rd gen) (2 GB)': [9-9-2009, 'iPod Shuffle', 'September 1, 2010'], 'iPod Shuffle (3rd gen) (4 GB)': [3-11-2009, 'iPod Shuffle', 'September 1, 2010'], 'iPod Shuffle (4th gen)': [9-1-2010, 'iPod Shuffle', 'current'], 'iPod Touch (1st gen) (32 GB)': [2-27-2008, 'iPod Touch', 'September 9, 2008'], 'iPod Touch (1st gen) (8 & 16 GB)': [9-5-2007, 'iPod Touch', 'September 9, 2008'], 'iPod Touch (2nd gen) (16 & 32 GB)': [9-9-2008, 'iPod Touch', 'September 9, 2009'], 'iPod Touch (2nd gen) (8 GB)': [9-9-2008, 'iPod Touch', 'September 1, 2010'], 'iPod Touch (3rd gen)': [9-9-2009, 'iPod Touch', 'September 1, 2010'], 'iPod Touch (4th gen) (16 GB)': [9-12-2012, 'iPod Touch', 'current'], 'iPod Touch (4th gen) (32 GB)': [9-1-2010, 'iPod Touch', 'current'], 'iPod Touch (4th gen) (8 & 64 GB)': [9-1-2010, 'iPod Touch', 'September 12, 2012'], 'iPod Touch (5th gen)': [10-11-2012, 'iPod Touch', 'current'], 'iPod+HP': [1-8-2004, 'iPod Classic', 'July 29, 2005'], "iWork '09": [1-6-2009, 'Software', 'current']}
Print first ten of each category we just scraped.
timeline = getProductReleasesForApple()
sortedTimeline = sorted(timeline.items(),key=lambda tup: tup[1][0].numericDate())
productName = []
family = []
releaseDate = []
discontinueDate = []
for item in sortedTimeline:
productName.append(item[0])
releaseDate.append(item[1][0])
family.append(item[1][1])
discontinueDate.append(item[1][2])
print productName[:10]
print
# print releaseDate[:10]
# print
print family[:10]
['iPod Shuffle (2nd gen)', 'iPod (4th gen)', 'iPod (3rd gen)', u'Apple I', u'Apple II', u'Disk II', 'Apple Writer 1.0', 'Apple SilenType', 'Apple II EuroPlus', 'Apple II J-Plus'] ['iPod Shuffle', 'iPod Classic', 'iPod Classic', u'Apple I', u'Apple II', u'Drives', 'Software', 'Printers', 'Apple II', 'Apple II']
We've defined a separate query class for handling the different kinds of queries. We have looking up a single product, a family of products, and a more general time range.
class Query:
#queryTypes: "timerange, family, product"
def __init__(self, queryType, arg, daysPadding=5):
self.daysPadding = daysPadding
self.symbol = "AAPL"
self.queryType = queryType
self.arg = arg
# set up the dataFrame
if queryType=="timerange":
self.startDate = arg[0].dateRange(daysPadding)[0]
self.endDate = arg[1].dateRange(daysPadding)[1]
# self.dateBoundary()
criterion = timelineDataFrame['Release Date'].map(lambda date: (date.numericDate() > self.startDate.numericDate()) and (date.numericDate() < self.endDate.numericDate()))
self.dataFrame = timelineDataFrame[criterion]
elif queryType=="family":
criterion = timelineDataFrame['Family'] == arg
self.dataFrame = timelineDataFrame[criterion]
for date in self.dataFrame['Release Date']:
dateRange = date.dateRange(daysPadding)
self.startDate = self.dataFrame['Release Date'][0].dateRange(daysPadding)[0]
self.endDate = self.dataFrame['Release Date'][-1].dateRange(daysPadding)[1]
# self.dateBoundary()
elif queryType=="product":
self.dataFrame = timelineDataFrame[timelineDataFrame.index == arg]
dateRange = self.dataFrame['Release Date'][0].dateRange(daysPadding)
self.startDate = dateRange[0]
self.endDate = dateRange[1]
# self.dateBoundary()
else:
print "invalid queryType"
self.setStockData()
# print self.dataFrame
def plotIndividualStockDifferences(self):
import matplotlib.pyplot as plt
plot = mouseHoverPlot(self.dataFrame['Individual Stock Difference'], self.dataFrame, 'Individual Stock Difference')
pl.ylabel('Stock Difference over %d day(s) in dollars' % self.daysPadding)
pl.xlabel('Product Names')
pl.show()
def plotSlopeChanges(self):
import matplotlib.pyplot as plt
plot = mouseHoverPlot(self.dataFrame['Stock Slope Change'], self.dataFrame, 'Stock Slope Change')
pl.ylabel('Stock slope changes over %d day(s) in dollars' % self.daysPadding)
pl.xlabel('Product Names')
pl.show()
def getIndividualStock(self, releaseDate):
dateRange = releaseDate.dateRange(self.daysPadding)
startIndividualDate = dateRange[0]
endIndividualDate = dateRange[1]
interval = 'd'
url = "http://ichart.yahoo.com/table.csv?s=%s&a=%i&b=%i&c=%i&d=%i&e=%i&f=%i&g=%s&ignore=.csv" \
% ( self.symbol, startIndividualDate.m-1, startIndividualDate.d, startIndividualDate.y, endIndividualDate.m-1, endIndividualDate.d, endIndividualDate.y, interval)
from time import sleep
u = urllib.urlopen(url)
ulines = u.read().split("\n")
start = ulines[-2]
end = ulines[1]
difference = parse_yahoo_stock(end)['Close'] - parse_yahoo_stock(start)['Close']
# print 'getIndividualStock works'
if difference > 0:
sign = '+'
else:
sign = '-'
#print sign + str(difference)
return difference
def getRangeStockData(self):
interval = 'd'
url = "http://ichart.yahoo.com/table.csv?s=%s&a=%i&b=%i&c=%i&d=%i&e=%i&f=%i&g=%s&ignore=.csv" \
% ( self.symbol, self.startDate.m-1, self.startDate.d, self.startDate.y, self.endDate.m-1, self.endDate.d, self.endDate.y, interval)
from time import sleep
u = urllib.urlopen(url)
ulines = u.read().split("\n")
start = ulines[-2]
end = ulines[1]
difference = parse_yahoo_stock(end)['Close'] - parse_yahoo_stock(start)['Close']
# print 'rangestockdata works'
if difference > 0:
sign = '+'
else:
sign = '-'
#print sign + str(difference)
return difference
def getStockSlope(self, releaseDate):
interval = 'd'
startDate = releaseDate.dateRange(self.daysPadding)[0]
endDate = releaseDate.dateRange(self.daysPadding)[1]
# print self.dataFrame
url = "http://ichart.yahoo.com/table.csv?s=%s&a=%i&b=%i&c=%i&d=%i&e=%i&f=%i&g=%s&ignore=.csv" \
% ( self.symbol, startDate.m-1, startDate.d, startDate.y, releaseDate.m-1, releaseDate.d, releaseDate.y, interval)
from time import sleep
u = urllib.urlopen(url)
ulines = u.read().split("\n")
start = ulines[-2]
end = ulines[1]
# print end
# print self.startDate
# print releaseDate
# print self.endDate
# print 'about to start stockSlopes'
# print 'startDate: ', startDate
# print 'releaseDate: ', releaseDate
# print 'endDate: ', endDate
# print start,end
# print parse_yahoo_stock(end)
leadingDifference = parse_yahoo_stock(end)['Close'] - parse_yahoo_stock(start)['Close']
# print 'getstockslope part 1 works'
leadingSlope = leadingDifference / self.daysPadding
url = "http://ichart.yahoo.com/table.csv?s=%s&a=%i&b=%i&c=%i&d=%i&e=%i&f=%i&g=%s&ignore=.csv" \
% ( self.symbol, releaseDate.m-1, releaseDate.d, releaseDate.y, endDate.m-1, endDate.d, endDate.y, interval)
# from time import sleep
u = urllib.urlopen(url)
ulines = u.read().split("\n")
start = ulines[-2]
end = ulines[1]
leavingDifference = parse_yahoo_stock(end)['Close'] - parse_yahoo_stock(start)['Close']
# print 'getstockslope part 2 works'
leavingSlope = leavingDifference / self.daysPadding
# print start,end
slopeDifference = leavingSlope - leadingSlope
return slopeDifference
def setStockData(self): # should set both RangeStock and IndividualStock
rangeStockImpact = self.getRangeStockData()
# print 'set rangeStockData'
self.dataFrame['Range Stock Difference'] = rangeStockImpact
indivStocks = []
stockSlopes = []
for date in self.dataFrame['Release Date']:
indivStocks.append(self.getIndividualStock(date))
stockSlopes.append(self.getStockSlope(date))
self.dataFrame['Individual Stock Difference'] = indivStocks
# print 'set individual stock'
self.dataFrame['Stock Slope Change'] = stockSlopes
# print 'set stock slopes'
#returns row of most influential product
def getMostInfluencial(self):
maxIndex = self.dataFrame['Individual Stock Difference'].argmax()
maxProduct = self.dataFrame.index[maxIndex]
maxProductRow = self.dataFrame[self.dataFrame.index == maxProduct]
return maxProductRow
Special bit we needed for enabling mouse over hover on the plot.
import matplotlib as plt
plt.use('WXAgg')
plt.interactive(False)
class mouseHoverPlot(object):
def __init__(self, dataY, dataFrame, plotType):
import pylab as pl
from pylab import get_current_fig_manager as gcfm
import wx
import numpy as np
import random
self.plotType = plotType
self.dataFrame = dataFrame
self.figure = pl.figure()
self.axis = self.figure.add_subplot(111)
# create a long tooltip with newline to get around wx bug (in v2.6.3.3)
# where newlines aren't recognized on subsequent self.tooltip.SetTip() calls
self.tooltip = wx.ToolTip(tip='tip with a long %s line and a newline\n' % (' '*100))
gcfm().canvas.SetToolTip(self.tooltip)
self.tooltip.Enable(False)
self.tooltip.SetDelay(0)
self.figure.canvas.mpl_connect('motion_notify_event', self._onMotion)
self.dataX = range(len(dataY))
self.dataY = dataY
self.xTicks = dataFrame.index
pl.xticks(self.dataX, self.xTicks)
self.axis.plot(self.dataX, self.dataY, linestyle='-', marker='o', markersize=15, label='myplot')
def _onMotion(self, event):
collisionFound = False
if event.xdata != None and event.ydata != None: # mouse is inside the axes
for i in xrange(len(self.dataX)):
radius = .2
if abs(event.xdata - self.dataX[i]) < radius and abs(event.ydata - self.dataY[i]) < radius:
def productName(xPos):
return self.xTicks[int(round(xPos))]
# print self.dataFrame[self.dataFrame.index == productName(event.xdata)]['Release Date'][0]
top = tip='Product: %s\nRelease Date: %s\nStock Price Difference: $%.2f' % (productName(event.xdata),self.dataFrame[self.dataFrame.index == productName(event.xdata)]['Release Date'][0], self.dataFrame[self.dataFrame.index == productName(event.xdata)][self.plotType][0] )
self.tooltip.SetTip(tip)
self.tooltip.Enable(True)
collisionFound = True
break
if not collisionFound:
self.tooltip.Enable(False)
c:\Python27\lib\site-packages\matplotlib\__init__.py:908: UserWarning: This call to matplotlib.use() has no effect because the the backend has already been chosen; matplotlib.use() must be called *before* pylab, matplotlib.pyplot, or matplotlib.backends is imported for the first time. if warn: warnings.warn(_use_error_msg)
Generating Options for Interacting
from IPython.core.display import HTML
selectform = ""
selectform += "<p>Single Products</p>"
selectform += "<select id='singleProducts'>"
# singleProducts = ['iMac (27") (Late 2012)', 'iMac (21.5") (Late 2012)', 'iPad (4th gen) (Wi-Fi + Cellular)', 'iPad Mini (Wi-Fi + Cellular)']
selectform += "<option value=\"0\">----Select a Single Product----</option>"
counter = 1
for singleProduct in productName:
selectform += "<option value=\"%s\">%s</option>" % (counter, singleProduct)
counter += 1
selectform += "</select>"
selectform += "<p>Families</p>"
selectform += "<select id='families'>"
# families = ['iMac', 'iPad', 'MacBook Pro', 'Mac Mini']
selectform += "<option value=\"0\">---Select a Family of Products---</option>"
counter = 1
for fam in list(set(family)):
selectform += "<option value=\"%s\">%s</option>" % (counter, fam)
counter += 1
selectform += "</select>"
"var selected=$('select#singleProducts').find(':selected').text();py1='selected = "'+selected+'"';IPython.notebook.kernel.execute(py1);"
#selectform += "<input type=\"button\" value=\"Compute\" onclick=\"var selected=$('select#singleProducts').find(':selected').text();var py1='selected = "'+selected+'"';IPython.notebook.kernel.execute(py1);\">"
selectform += "<p>Time Range</p>"
selectform += "<input type='text' id='timeRanges'/>"
script=""
HTML(selectform+"<script>"+script+"</script>")
Single Products
Families
Time Range
argument = ""
if qtype == 'singleProduct':
print 'Single Product'
print singleProduct
qtype = 'product'
argument = singleProduct
elif qtype == 'family':
print 'Family'
print family
qtype = 'family'
argument = family
elif qtype == 'timeRange':
print 'Time Range'
print timeRange
qtype = 'timerange'
argument = timeRange
Family Mac Pro
import pandas
timelineDataFrame = pandas.DataFrame({'Product Name': productName, 'Release Date':releaseDate,
'Family': family, 'Date Discontinued': discontinueDate,
'Individual Stock Difference': 0, 'Range Stock Difference': 0,
'Stock Slope Change': 0}).set_index('Product Name')
def removeOldItems(borderDate):
criterion = timelineDataFrame['Release Date'].map(lambda date: date.numericDate() > borderDate.numericDate())
return timelineDataFrame[criterion]
earliestDate = Date("9-2-1985")
timelineDataFrame = removeOldItems(earliestDate)
timeline = getProductReleasesForApple()
sortedTimeline = sorted(timeline.items(),key=lambda tup: tup[1][0].numericDate())
productName = []
family = []
releaseDate = []
discontinueDate = []
for item in sortedTimeline:
productName.append(item[0])
releaseDate.append(item[1][0])
family.append(item[1][1])
discontinueDate.append(item[1][2])
import pandas
timelineDataFrame = pandas.DataFrame({'Product Name': productName, 'Release Date':releaseDate,
'Family': family, 'Date Discontinued': discontinueDate,
'Individual Stock Difference': 0, 'Range Stock Difference': 0,
'Stock Slope Change': 0}).set_index('Product Name')
def removeOldItems(borderDate):
criterion = timelineDataFrame['Release Date'].map(lambda date: date.numericDate() > borderDate.numericDate())
return timelineDataFrame[criterion]
earliestDate = Date("9-2-1985")
timelineDataFrame = removeOldItems(earliestDate)
# print timelineDataFrame[:20]
# sampleQuery = Query("timerange", (Date("1-1-1911"), Date("3-6-1992")))
# sampleQuery.plotSlopeChanges()
print qtype
print argument
sampleFamilyQuery = Query(qtype, argument)
sampleFamilyQuery.plotIndividualStockDifferences()
sampleFamilyQuery.plotSlopeChanges()
family Mac Pro
If we were to continue this proect beyond the course, we would like to extend this kind of analysis to more than just Apple Inc. We chose Apple because of the well organized timeline of products page that made collection of data for Apple products easy and throrough. In order for us to have a more generalized means of getting company product information for numerous companies, we would have to change our approach and look at sources of information with broader scopes. We considered press releases and review sites as potential viable candidates because of timely releases and a somewhat uniform page format that would make aquiring product data systematic, instead of having to look at products on a company by company basis.
Aside from increasing the scope of companies we looked at, we would also like to increase the quality of our stock analysis. As computer science majors with no formal background in financial analysis, we chose this project because we were interested in the idea, not because we were particularly adept at this kind of analysis. As a result, we utilized very simple analysis methods to gauge the success of a particular product or product family that left many questions unanswered. We would like to use more advanced and comprehensive methods of analysis after doing a bit more research to get a more accurate picture of product's success.
Eddie Lee was in charge of data aquisition. This included scraping product data from the "Timeline of Apple Inc. products" Wikipedia page and accessing the Yahoo! Finance API so that we could get closing prices for days we were looking into. He also implmented the dropdown menus in this Python notebooks so that the creation of a query could be fast and easy.
Eugene Kim focused on data analysis. He organized all data that was scraped into a DataFrame, added stock data for products in the DataFrame, and implemented interactive plots from matplotlib.