-----20 Movies Script Visualizations will be done------This could be done for the 1000+ movie script I segmented--- I just randomly chose 20 movie scripts from the segmented movies....

The aim of this project is to provide detailed insights into different movies analyzed focusing on the characters, their dialogues, scene locations, emotional and sentiment analysis of the whole movie and the individual characters, character's interaction with one another and finally gender distribution in the each movie analyzed.¶

To execute this project, the following objectives were done:¶

Web scraping of the movie scripts (Over 1000+ movies were scraped from imsdb website)
Movies segmentation into Scenes --> Scene Location, Scene Action/Description, Scene Dialogues, Scene Characters (All the movies scraped were segmented except those that do not follow the "Screenplay format i.e. INT / EXT)"
Character extraction and appearances plot ---> Here, characters were plotted based on how many times they appeared and spoke in each scene and across the movie.
Character Interaction Mapping --> We mapped out the connection between all the characters in the movie and also the interaction between the Top 10 characters in the movie.
Here, we looked at the Most mentioned character based on the Scene dialogues and also the characters each character mention the most in their conversation.
Similar to Number 5., Here looked at who a specific character talks with the most in the Movie.
Emotional and Sentiment Analysis across the whole movie and for each individual character, However for this project we limited it to only the Top 10 characters. ---> This gives us the character's emotion when he/she appears in the movie.
Additional Scene Informations --> Exact Scene Locations, Scenes with dialogs and no dialogs, Scenes that occurred during the Day or in the Night, Scenes location based on Outdoor or Indoor appearances.
Gender Distribution in the movie

*(python Code) Modules for this project: imsbd_moviescript_scraper_AND_Scene_Segmentation.py, dialogue_appearance.py, characters_extract.py, xter_interaction.py, characters_mt.py, emotions.py, movie_info.py, gend_distribution_plot.py*

Tools: Python libraries

In [ ]:

In [1]:

#Import all the necessary python modules needed for this analysis

from characters_extract import extract_characters
from dialogue_appearance import scene_dialogues
from xter_interaction import interaction
from emotions import emotions_sentiments
from characters_mt import character_mentions
from gend_distribution_plot import gender
from movie_info import scene_info_plots

In [2]:

import glob
import random
import secrets
import re

import cufflinks as cf
import networkx as net
import itertools
# plotly
from plotly.subplots import make_subplots
import plotly.graph_objects as go
import pandas as pd
import chart_studio.plotly as py
import plotly.graph_objs as go
from plotly import tools
from plotly.offline import init_notebook_mode, iplot
init_notebook_mode(connected=True)
import plotly.express as px

1. Load in the Segmented Movie scripts stored in .pkl format¶

In [3]:

films = []
for f in glob.glob('Films/*'):
    film_name = re.sub(r'.pkl|Films\\', '', f)
    films.append(film_name)

In [4]:

#Number of Films we segmented into scenes, scene_actions, characters, and characters dialogue
print('Number of films available for Analysis:  ', len(films), ' Movies')

Number of films available for Analysis:   1037  Movies

In [5]:

#Random 10 films from the 1000 films scraped from the internet
films_list = random.sample(films, 10)

In [6]:

#Randomly select film to analyze
film = secrets.choice(films_list)
print(film)

Social-Network,-The

In [7]:

##load the scenes, dialogues, characters into dataframe

df_film = pd.read_pickle('Films/' + film + '.pkl')
df_film_dialogue = pd.read_pickle('Dialogues/' + film + '.pkl')
df_film_characters = pd.read_pickle('Characters/' + film + '.pkl')

In [8]:

#Randomly generate 10 scenes from the movie script
df_film.sample(10)

Out[8]:

	Scene_Names	Scene_action	Scene_Characters	Scene_Dialogue	Contents
115	INT. CLASSROOM NIGHT	where'60 or so STUDENTS are in a semicircle, f...	[EDUARDO, EDUARDO, MARK, EDUARDO, MARK, EDUARD...	[Mark? MARY, Yeah., What's goin' on?, They hav...	where'60 or so STUDENTS are in a semicircle, ...
88	INT. MARK'S DORM ROOM NIGHT		[EDUARDO]	[the lack of hardware we had to deal with, the...	EDUARDO the lack of hardware we had to deal w...
71	INT. EDUARDO'S DORM ROOM NIGHT	EDUARDO's studying at his desk but this time w...	None	None	EDUARDO's studying at his desk but this time ...
82	INT. SECOND DEPOSITION ROOM DAY		[GAGE, MARK, GAGE, MARK, GAGE, MARK, GAGE, SY,...	[During the time when you say you had this ide...	GAGE During the time when you say you had thi...
109	INT. FIRST DEPOSITION ROOM DAY		[EDUARDO, MARK, EDUARDO, SY, ARTICLE, MARK, DU...	[I had gotten into the Phoenix. I had been acc...	EDUARDO I had gotten into the Phoenix. I had ...
59	INT. SECOND DEPOSITION ROOM DAY	EDUARDO's in different.clothes and being quest...	[GAGE, EDUARDO, GAGE, EDUARDO, GAGE, EDUARDO, ...	[We recognize that you are a plaintiff in one ...	EDUARDO's in different.clothes and being ques...
16	INT. ANOTHER DORM ROOM NIGHT	THREE MALE STUDENTS AT A COMPUTER	[ALL]	[On the right. ]	THREE MALE STUDENTS AT A COMPUTER ALL On the ...
55	INT. FIRST DEPOSITION ROOM DAY		[GRETCHEN, EDUARDO, GRETCHEN, EDUARDO]	[And you said?, I said Let's do it., Okay. Did...	GRETCHEN And you said? EDUARDO I said Let's d...
147	INT. BEDROOM CONTINUOUS	leaving the door open.	[SEAN, SEAN, SEAN, POLICEMAN, SEAN, SEAN, POLI...	[It's the cops. And they all spring into actio...	leaving the door open. SEAN It's the cops. An...
140	INT. CONFERENCE ROOM CONTINUOUS		[EDUARDO, EDUARDO, LAWYER, EDUARDO, LAWYER]	[At first I thought he was joking, giving me m...	EDUARDO At first I thought he was joking, gi...

In [9]:

#check how many scenes the movie script has
df_film.shape

Out[9]:

(151, 5)

In [10]:

#Randomly select characters and their corresponding dialogues
df_film_dialogue.sample(10)

Out[10]:

	characters	Character_dialogue
1456	MARK	Okay. MARK heads into the office building. SEA...
258	EDUARDO	It's alright.
337	GRETCHEN	What's that?
1173	MARK	Hang on. MARK's scratching something out on a ...
591	GAGE	It's not important that you be sure why I am a...
877	AMY	YEAH
410	WE	TNT. MARK'S DORM ROOM NIGHT Every available w...
1454	MARK	You sure aboutthis?
164	ALL	On the left.
33	MARK	Hm.

2. Character appearance plots¶

In [11]:

ext = extract_characters(df_film, df_film_dialogue, df_film_characters, film)

In [12]:

movie_characters = ext.extract_character_plot()

In [13]:

dia = scene_dialogues(df_film, film)

In [14]:

df_xter_app = dia.character_appearances(movie_characters)

Lets check the appearance of the top 3 characters across the Movie¶

In [15]:

#Movie characters....some names may not be characters....
print('Movie Characters:  \n', movie_characters)

Movie Characters:  
 ['MARK', 'EDUARDO', 'SEAN', 'CAMERON', 'DIVYA', 'ERICA', 'TYLER', 'GRETCHEN', 'GAGE', 'AMY', 'JENNY', 'SUMMERS', 'MARYLIN', 'SY', 'DUSTIN', 'GIRL', 'LAWYER', 'ASHLEIGH', 'POLICEMAN', 'COX', 'MARX', 'ALL', 'MARY', 'ADMINISTRATOR', 'SENIOR', 'MR. WINKLEVOSS', 'CHRIS', 'SPEAKER', 'PRINCE ALBERT', 'PAUL YOUNG', 'GUY', 'STUDENTS', 'PROFESSOR', 'BOB', 'SECRETARY', 'KENWRIGHT', 'MART', 'STUART', 'WE', 'COMMON PEOPLE', 'CAN', 'GRAD STUDENT', 'MARE', 'AND WE', 'SOPHOMORE', 'ALICE', 'ANNE', 'WAITRESS', 'CANDIDATE', 'INTERN', 'ERIC', 'SHARON', 'WOMAN', 'THIEL']

In [16]:

df_1st_count, df_1st_dialogue = dia.xter_count_perscene(movie_characters[0])

In [17]:

dia.scene_dialogue_plot(df_1st_count)

In [18]:

df_2nd_count, df_2nd_dialogue = dia.xter_count_perscene(movie_characters[1])
dia.scene_dialogue_plot(df_2nd_count)

In [19]:

df_third_count, df_third_dialogue = dia.xter_count_perscene(movie_characters[2])
dia.scene_dialogue_plot(df_third_count)

In [20]:

df_2_count, df_2_dialogue = dia.xter_count_perscene(movie_characters[:2])
dia.scene_dialogue_plot(df_2_count)

In [21]:

df_3_count, df_3_dialogue = dia.xter_count_perscene(movie_characters[1:3])
dia.scene_dialogue_plot(df_3_count)

3. Character Interaction Mapping¶

In [22]:

interact = interaction(df_film, film)

In [23]:

graph_list = interact.character_interaction()

Note the Net.MultiGraph code below will throw an error, BUT this does not stop it from displaying the Character Network Multigraph figure,¶

To run it, remove the hashtags i.e. comment it out¶

In [29]:

# G = net.MultiGraph()
# for scene in graph_list:
#     nodes = list(itertools.combinations(scene,2))
#     for pair in nodes:
#         G.add_edges_from([pair])
# page_ranked_nodes = net.pagerank_numpy(G,0.95)
# net.enumerate_all_cliques(G)
# between_nodes = net.betweenness_centrality(G, normalized=True, endpoints=True)

In [26]:

interact.character_interaction_plot(G, page_ranked_nodes)

Let break it down to the Top 10 Characters¶

In [27]:

#Remember to Re-run the above multigraph code aafter running this code line
graph_list = interact.top10_character_interaction(movie_characters[:10])

In [30]:

interact.character_interaction_plot(G, page_ranked_nodes)

4. Characters mentioned the most in Scene Dialogues (Most-talked about characters in the movie.......) and who each character mention the most in their specific dialogue¶

In [31]:

xtr = character_mentions(df_film, movie_characters, film)

In [32]:

xter_mentions = xtr.most_mentioned()

Lets the check the appearance of the 5 most mentioned characters in the Movie¶

In [33]:

xtr.top_xters_mentions(xter_mentions, 5)

Lets check who specific characters mention the most in their dialogues in the movie¶

In [34]:

print(movie_characters)

['MARK', 'EDUARDO', 'SEAN', 'CAMERON', 'DIVYA', 'ERICA', 'TYLER', 'GRETCHEN', 'GAGE', 'AMY', 'JENNY', 'SUMMERS', 'MARYLIN', 'SY', 'DUSTIN', 'GIRL', 'LAWYER', 'ASHLEIGH', 'POLICEMAN', 'COX', 'MARX', 'ALL', 'MARY', 'ADMINISTRATOR', 'SENIOR', 'MR. WINKLEVOSS', 'CHRIS', 'SPEAKER', 'PRINCE ALBERT', 'PAUL YOUNG', 'GUY', 'STUDENTS', 'PROFESSOR', 'BOB', 'SECRETARY', 'KENWRIGHT', 'MART', 'STUART', 'WE', 'COMMON PEOPLE', 'CAN', 'GRAD STUDENT', 'MARE', 'AND WE', 'SOPHOMORE', 'ALICE', 'ANNE', 'WAITRESS', 'CANDIDATE', 'INTERN', 'ERIC', 'SHARON', 'WOMAN', 'THIEL']

In [36]:

df_mk = xtr.talk_about_xters(df_film_dialogue, 'MARK')

In [37]:

df_ed = xtr.talk_about_xters(df_film_dialogue, 'EDUARDO')

In [38]:

df_sean = xtr.talk_about_xters(df_film_dialogue, 'SEAN')

Lets check who specific characters talk with the most in the movie¶

In [39]:

df_mk = xtr.most_talked_with('MARK')

In [40]:

df_ed = xtr.most_talked_with(movie_characters[1])

We could continue to check this for every character but I will pause here....¶

5. Emotional Arcs and Sentiment Analysis across the Movie and for each characters¶

In [41]:

etn = emotions_sentiments(df_film, film)

In [42]:

df_film_sentiment = etn.film_sentiment('darkslategray')

In [43]:

df_film_emotion = etn.film_emotional_arc()

Lets check the emotional content for the Top 10 characters in the Movie¶

In [44]:

df_top10_emotions =  etn.emotional_content_plot(df_film_dialogue, movie_characters, 11)

Lets check emotional arcs for specific characters¶

Emotions at scenes where the character is not present is set to ZERO¶

In [45]:

df_mark_emotions = etn.emotional_arc_xter_plot(df_film_emotion, 'MARK')

In [46]:

df_ed_emotions = etn.emotional_arc_xter_plot(df_film_emotion, 'EDUARDO')

In [47]:

df_sean_emotions = etn.emotional_arc_xter_plot(df_film_emotion, 'SEAN')

As usual, this can be done for all characters in the movie, but we will pause here....¶

4. Movie additional information: Exact Scene location, Indoor/Outdoor Scenes percent, Time of Day occurences in the movie....¶

In [48]:

info = scene_info_plots(df_film, film)

In [49]:

info.extract_scene_locations()

In [50]:

info.pie_plots()

5. Gender Distribution in the Movie¶

In [51]:

gd = gender(movie_characters, film)

In [52]:

df_gender = gd.gender_types(px.colors.sequential.Viridis)

[nltk_data] Downloading package names to C:\Users\Adeboye
[nltk_data]     Adeniyi\AppData\Roaming\nltk_data...
[nltk_data]   Package names is already up-to-date!

This should give a screen writer a broad overview of the underlying insights that could be discovered from an existing screen play or movie script.¶

Thanks for visualizing this extensive film script analysis...¶

Remember all codes for this analysis are already in python modules....¶