Textual data in spite of being highly unstructured, can be classified into two major types of documents.
Sentiment Analysis is also popularly known as opinion analysis or opinion mining. The key idea is to use techniques from text analytics, NLP, machine learning and linguistics to extract important information or data points from unstructured text. This in turn can help us derive the sentiment from text data
Here we will be looking at building supervised sentiment analysis classification models thanks to the advantage of labeled data! The dataset we will be working with is the IMDB Large Movie Review Dataset having 50000 reviews classified into positive and negative sentiment. I have provided a compressed version of the dataset in this repository itself for your benefit!
Do remember that the focus here is not sentiment analysis but text classification by leveraging universal sentence embeddings.
We will leverage the following sentence encoders here for demonstration from TensorFlow Hub:
Developed by Dipanjan (DJ) Sarkar
!pip install tensorflow-hub
Collecting tensorflow-hub
Downloading https://files.pythonhosted.org/packages/5f/22/64f246ef80e64b1a13b2f463cefa44f397a51c49a303294f5f3d04ac39ac/tensorflow_hub-0.1.1-py2.py3-none-any.whl (52kB)
100% |################################| 61kB 8.5MB/s ta 0:00:011
Requirement already satisfied: numpy>=1.12.0 in /usr/local/lib/python3.6/dist-packages (from tensorflow-hub) (1.14.3)
Requirement already satisfied: six>=1.10.0 in /usr/local/lib/python3.6/dist-packages (from tensorflow-hub) (1.11.0)
Requirement already satisfied: protobuf>=3.4.0 in /usr/local/lib/python3.6/dist-packages (from tensorflow-hub) (3.5.2.post1)
Requirement already satisfied: setuptools in /usr/local/lib/python3.6/dist-packages (from protobuf>=3.4.0->tensorflow-hub) (39.1.0)
Installing collected packages: tensorflow-hub
Successfully installed tensorflow-hub-0.1.1
You are using pip version 10.0.1, however version 18.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
import tensorflow as tf
import tensorflow_hub as hub
import numpy as np
import pandas as pd
tf.test.is_gpu_available()
True
tf.test.gpu_device_name()
'/device:GPU:0'
dataset = pd.read_csv('movie_reviews.csv.bz2', compression='bz2')
dataset.info()
<class 'pandas.core.frame.DataFrame'> RangeIndex: 50000 entries, 0 to 49999 Data columns (total 2 columns): review 50000 non-null object sentiment 50000 non-null object dtypes: object(2) memory usage: 781.3+ KB
dataset['sentiment'] = [1 if sentiment == 'positive' else 0 for sentiment in dataset['sentiment'].values]
dataset.head()
review | sentiment | |
---|---|---|
0 | One of the other reviewers has mentioned that ... | 1 |
1 | A wonderful little production. <br /><br />The... | 1 |
2 | I thought this was a wonderful way to spend ti... | 1 |
3 | Basically there's a family where a little boy ... | 0 |
4 | Petter Mattei's "Love in the Time of Money" is... | 1 |
reviews = dataset['review'].values
sentiments = dataset['sentiment'].values
train_reviews = reviews[:30000]
train_sentiments = sentiments[:30000]
val_reviews = reviews[30000:35000]
val_sentiments = sentiments[30000:35000]
test_reviews = reviews[35000:]
test_sentiments = sentiments[35000:]
train_reviews.shape, val_reviews.shape, test_reviews.shape
((30000,), (5000,), (15000,))
!pip install contractions
!pip install beautifulsoup4
Requirement already satisfied: contractions in /usr/local/lib/python3.6/dist-packages (0.0.17) You are using pip version 10.0.1, however version 18.1 is available. You should consider upgrading via the 'pip install --upgrade pip' command. Requirement already satisfied: beautifulsoup4 in /usr/local/lib/python3.6/dist-packages (4.6.3) You are using pip version 10.0.1, however version 18.1 is available. You should consider upgrading via the 'pip install --upgrade pip' command.
import contractions
from bs4 import BeautifulSoup
import unicodedata
import re
def strip_html_tags(text):
soup = BeautifulSoup(text, "html.parser")
[s.extract() for s in soup(['iframe', 'script'])]
stripped_text = soup.get_text()
stripped_text = re.sub(r'[\r|\n|\r\n]+', '\n', stripped_text)
return stripped_text
def remove_accented_chars(text):
text = unicodedata.normalize('NFKD', text).encode('ascii', 'ignore').decode('utf-8', 'ignore')
return text
def expand_contractions(text):
return contractions.fix(text)
def remove_special_characters(text, remove_digits=False):
pattern = r'[^a-zA-Z0-9\s]' if not remove_digits else r'[^a-zA-Z\s]'
text = re.sub(pattern, '', text)
return text
def pre_process_document(document):
# strip HTML
document = strip_html_tags(document)
# lower case
document = document.lower()
# remove extra newlines (often might be present in really noisy text)
document = document.translate(document.maketrans("\n\t\r", " "))
# remove accented characters
document = remove_accented_chars(document)
# expand contractions
document = expand_contractions(document)
# remove special characters and\or digits
# insert spaces between special characters to isolate them
special_char_pattern = re.compile(r'([{.(-)!}])')
document = special_char_pattern.sub(" \\1 ", document)
document = remove_special_characters(document, remove_digits=True)
# remove extra whitespace
document = re.sub(' +', ' ', document)
document = document.strip()
return document
pre_process_corpus = np.vectorize(pre_process_document)
train_reviews = pre_process_corpus(train_reviews)
val_reviews = pre_process_corpus(val_reviews)
test_reviews = pre_process_corpus(test_reviews)
# Training input on the whole training set with no limit on training epochs.
train_input_fn = tf.estimator.inputs.numpy_input_fn(
{'sentence': train_reviews}, train_sentiments,
batch_size=256, num_epochs=None, shuffle=True)
# Prediction on the whole training set.
predict_train_input_fn = tf.estimator.inputs.numpy_input_fn(
{'sentence': train_reviews}, train_sentiments, shuffle=False)
# Prediction on the whole validation set.
predict_val_input_fn = tf.estimator.inputs.numpy_input_fn(
{'sentence': val_reviews}, val_sentiments, shuffle=False)
# Prediction on the test set.
predict_test_input_fn = tf.estimator.inputs.numpy_input_fn(
{'sentence': test_reviews}, test_sentiments, shuffle=False)
embedding_feature = hub.text_embedding_column(
key='sentence',
module_spec="https://tfhub.dev/google/universal-sentence-encoder/2",
trainable=False)
INFO:tensorflow:Using /tmp/tfhub_modules to cache modules.
dnn = tf.estimator.DNNClassifier(
hidden_units=[512, 128],
feature_columns=[embedding_feature],
n_classes=2,
activation_fn=tf.nn.relu,
dropout=0.1,
optimizer=tf.train.AdagradOptimizer(learning_rate=0.005))
INFO:tensorflow:Using default config. WARNING:tensorflow:Using temporary folder as model directory: /tmp/tmpn9bphscn INFO:tensorflow:Using config: {'_model_dir': '/tmp/tmpn9bphscn', '_tf_random_seed': None, '_save_summary_steps': 100, '_save_checkpoints_steps': None, '_save_checkpoints_secs': 600, '_session_config': None, '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 100, '_train_distribute': None, '_service': None, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x7f009de6d9b0>, '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1}
256*1500 / 30000
12.8
tf.logging.set_verbosity(tf.logging.ERROR)
import time
TOTAL_STEPS = 1500
STEP_SIZE = 100
for step in range(0, TOTAL_STEPS+1, STEP_SIZE):
print()
print('-'*100)
print('Training for step =', step)
start_time = time.time()
dnn.train(input_fn=train_input_fn, steps=STEP_SIZE)
elapsed_time = time.time() - start_time
print('Train Time (s):', elapsed_time)
print('Eval Metrics (Train):', dnn.evaluate(input_fn=predict_train_input_fn))
print('Eval Metrics (Validation):', dnn.evaluate(input_fn=predict_val_input_fn))
---------------------------------------------------------------------------------------------------- Training for step = 0 Train Time (s): 78.62789511680603 Eval Metrics (Train): {'accuracy': 0.84863335, 'accuracy_baseline': 0.5005, 'auc': 0.9279859, 'auc_precision_recall': 0.92819566, 'average_loss': 0.34581015, 'label/mean': 0.5005, 'loss': 44.145977, 'precision': 0.86890674, 'prediction/mean': 0.47957155, 'recall': 0.8215118, 'global_step': 100} Eval Metrics (Validation): {'accuracy': 0.8454, 'accuracy_baseline': 0.505, 'auc': 0.92413086, 'auc_precision_recall': 0.9200026, 'average_loss': 0.35258815, 'label/mean': 0.495, 'loss': 44.073517, 'precision': 0.8522351, 'prediction/mean': 0.48447067, 'recall': 0.8319192, 'global_step': 100} ---------------------------------------------------------------------------------------------------- Training for step = 100 Train Time (s): 76.1651611328125 Eval Metrics (Train): {'accuracy': 0.85436666, 'accuracy_baseline': 0.5005, 'auc': 0.9321357, 'auc_precision_recall': 0.93224275, 'average_loss': 0.3330773, 'label/mean': 0.5005, 'loss': 42.520508, 'precision': 0.8501513, 'prediction/mean': 0.5098621, 'recall': 0.86073923, 'global_step': 200} Eval Metrics (Validation): {'accuracy': 0.8494, 'accuracy_baseline': 0.505, 'auc': 0.92772096, 'auc_precision_recall': 0.92323804, 'average_loss': 0.34418356, 'label/mean': 0.495, 'loss': 43.022945, 'precision': 0.83501947, 'prediction/mean': 0.5149463, 'recall': 0.86707073, 'global_step': 200} ---------------------------------------------------------------------------------------------------- Training for step = 200 Train Time (s): 76.43350577354431 Eval Metrics (Train): {'accuracy': 0.8561, 'accuracy_baseline': 0.5005, 'auc': 0.93400496, 'auc_precision_recall': 0.93414706, 'average_loss': 0.32816008, 'label/mean': 0.5005, 'loss': 41.892776, 'precision': 0.85856014, 'prediction/mean': 0.49918926, 'recall': 0.85301363, 'global_step': 300} Eval Metrics (Validation): {'accuracy': 0.8508, 'accuracy_baseline': 0.505, 'auc': 0.9290328, 'auc_precision_recall': 0.9250712, 'average_loss': 0.33986613, 'label/mean': 0.495, 'loss': 42.48327, 'precision': 0.84319174, 'prediction/mean': 0.50402075, 'recall': 0.85818183, 'global_step': 300} ---------------------------------------------------------------------------------------------------- Training for step = 300 Train Time (s): 78.08082580566406 Eval Metrics (Train): {'accuracy': 0.8584, 'accuracy_baseline': 0.5005, 'auc': 0.93560475, 'auc_precision_recall': 0.9356024, 'average_loss': 0.3264787, 'label/mean': 0.5005, 'loss': 41.678127, 'precision': 0.84423554, 'prediction/mean': 0.522159, 'recall': 0.8793207, 'global_step': 400} Eval Metrics (Validation): {'accuracy': 0.8494, 'accuracy_baseline': 0.505, 'auc': 0.929911, 'auc_precision_recall': 0.9256854, 'average_loss': 0.34194976, 'label/mean': 0.495, 'loss': 42.74372, 'precision': 0.82564294, 'prediction/mean': 0.5267772, 'recall': 0.8820202, 'global_step': 400} ---------------------------------------------------------------------------------------------------- Training for step = 400 Train Time (s): 78.66700315475464 Eval Metrics (Train): {'accuracy': 0.8597, 'accuracy_baseline': 0.5005, 'auc': 0.9370333, 'auc_precision_recall': 0.93680507, 'average_loss': 0.32163766, 'label/mean': 0.5005, 'loss': 41.060127, 'precision': 0.8540629, 'prediction/mean': 0.51153994, 'recall': 0.86799866, 'global_step': 500} Eval Metrics (Validation): {'accuracy': 0.851, 'accuracy_baseline': 0.505, 'auc': 0.93050015, 'auc_precision_recall': 0.92593473, 'average_loss': 0.33805788, 'label/mean': 0.495, 'loss': 42.257233, 'precision': 0.83579195, 'prediction/mean': 0.5157132, 'recall': 0.869899, 'global_step': 500} ---------------------------------------------------------------------------------------------------- Training for step = 500 Train Time (s): 81.03562021255493 Eval Metrics (Train): {'accuracy': 0.8628, 'accuracy_baseline': 0.5005, 'auc': 0.9386675, 'auc_precision_recall': 0.93854874, 'average_loss': 0.31690273, 'label/mean': 0.5005, 'loss': 40.45567, 'precision': 0.86361516, 'prediction/mean': 0.5004919, 'recall': 0.86200464, 'global_step': 600} Eval Metrics (Validation): {'accuracy': 0.8562, 'accuracy_baseline': 0.505, 'auc': 0.9315411, 'auc_precision_recall': 0.92733943, 'average_loss': 0.3339314, 'label/mean': 0.495, 'loss': 41.741425, 'precision': 0.8481364, 'prediction/mean': 0.5051916, 'recall': 0.86424243, 'global_step': 600} ---------------------------------------------------------------------------------------------------- Training for step = 600 Train Time (s): 82.44215893745422 Eval Metrics (Train): {'accuracy': 0.86523336, 'accuracy_baseline': 0.5005, 'auc': 0.94012344, 'auc_precision_recall': 0.93995917, 'average_loss': 0.31317216, 'label/mean': 0.5005, 'loss': 39.979427, 'precision': 0.86268675, 'prediction/mean': 0.50560397, 'recall': 0.8690643, 'global_step': 700} Eval Metrics (Validation): {'accuracy': 0.8572, 'accuracy_baseline': 0.505, 'auc': 0.9326249, 'auc_precision_recall': 0.9284968, 'average_loss': 0.33208466, 'label/mean': 0.495, 'loss': 41.510582, 'precision': 0.84651715, 'prediction/mean': 0.51005256, 'recall': 0.8690909, 'global_step': 700} ---------------------------------------------------------------------------------------------------- Training for step = 700 Train Time (s): 81.71217703819275 Eval Metrics (Train): {'accuracy': 0.8632333, 'accuracy_baseline': 0.5005, 'auc': 0.94086003, 'auc_precision_recall': 0.9406356, 'average_loss': 0.31374687, 'label/mean': 0.5005, 'loss': 40.05279, 'precision': 0.88191235, 'prediction/mean': 0.4772521, 'recall': 0.8390942, 'global_step': 800} Eval Metrics (Validation): {'accuracy': 0.854, 'accuracy_baseline': 0.505, 'auc': 0.9327511, 'auc_precision_recall': 0.9289065, 'average_loss': 0.33162636, 'label/mean': 0.495, 'loss': 41.453293, 'precision': 0.86128366, 'prediction/mean': 0.48188075, 'recall': 0.84040403, 'global_step': 800} ---------------------------------------------------------------------------------------------------- Training for step = 800 Train Time (s): 83.28980422019958 Eval Metrics (Train): {'accuracy': 0.8681667, 'accuracy_baseline': 0.5005, 'auc': 0.94270587, 'auc_precision_recall': 0.94251335, 'average_loss': 0.30656302, 'label/mean': 0.5005, 'loss': 39.135704, 'precision': 0.86419916, 'prediction/mean': 0.5073946, 'recall': 0.8739261, 'global_step': 900} Eval Metrics (Validation): {'accuracy': 0.8586, 'accuracy_baseline': 0.505, 'auc': 0.933738, 'auc_precision_recall': 0.9295292, 'average_loss': 0.32951483, 'label/mean': 0.495, 'loss': 41.189354, 'precision': 0.84612375, 'prediction/mean': 0.5115125, 'recall': 0.87313133, 'global_step': 900} ---------------------------------------------------------------------------------------------------- Training for step = 900 Train Time (s): 82.47694706916809 Eval Metrics (Train): {'accuracy': 0.86903334, 'accuracy_baseline': 0.5005, 'auc': 0.9443012, 'auc_precision_recall': 0.94409984, 'average_loss': 0.3057844, 'label/mean': 0.5005, 'loss': 39.03631, 'precision': 0.8514456, 'prediction/mean': 0.5249473, 'recall': 0.8943723, 'global_step': 1000} Eval Metrics (Validation): {'accuracy': 0.8566, 'accuracy_baseline': 0.505, 'auc': 0.9347619, 'auc_precision_recall': 0.93128407, 'average_loss': 0.33120963, 'label/mean': 0.495, 'loss': 41.401203, 'precision': 0.829955, 'prediction/mean': 0.5288649, 'recall': 0.8933333, 'global_step': 1000} ---------------------------------------------------------------------------------------------------- Training for step = 1000 Train Time (s): 84.39996337890625 Eval Metrics (Train): {'accuracy': 0.8716, 'accuracy_baseline': 0.5005, 'auc': 0.9453286, 'auc_precision_recall': 0.94506085, 'average_loss': 0.30098185, 'label/mean': 0.5005, 'loss': 38.423214, 'precision': 0.8595169, 'prediction/mean': 0.51834613, 'recall': 0.8887113, 'global_step': 1100} Eval Metrics (Validation): {'accuracy': 0.859, 'accuracy_baseline': 0.505, 'auc': 0.9349775, 'auc_precision_recall': 0.93106484, 'average_loss': 0.3288155, 'label/mean': 0.495, 'loss': 41.101936, 'precision': 0.83727133, 'prediction/mean': 0.5222167, 'recall': 0.8876768, 'global_step': 1100} ---------------------------------------------------------------------------------------------------- Training for step = 1100 Train Time (s): 84.88116145133972 Eval Metrics (Train): {'accuracy': 0.87226665, 'accuracy_baseline': 0.5005, 'auc': 0.94644105, 'auc_precision_recall': 0.94621253, 'average_loss': 0.2976978, 'label/mean': 0.5005, 'loss': 38.00398, 'precision': 0.88602, 'prediction/mean': 0.4845446, 'recall': 0.85474527, 'global_step': 1200} Eval Metrics (Validation): {'accuracy': 0.8612, 'accuracy_baseline': 0.505, 'auc': 0.9357711, 'auc_precision_recall': 0.93195754, 'average_loss': 0.32375482, 'label/mean': 0.495, 'loss': 40.469353, 'precision': 0.86272913, 'prediction/mean': 0.48821172, 'recall': 0.8557576, 'global_step': 1200} ---------------------------------------------------------------------------------------------------- Training for step = 1200 Train Time (s): 85.49790549278259 Eval Metrics (Train): {'accuracy': 0.8763, 'accuracy_baseline': 0.5005, 'auc': 0.94796634, 'auc_precision_recall': 0.9477355, 'average_loss': 0.29294312, 'label/mean': 0.5005, 'loss': 37.396996, 'precision': 0.87395793, 'prediction/mean': 0.5043318, 'recall': 0.8797203, 'global_step': 1300} Eval Metrics (Validation): {'accuracy': 0.861, 'accuracy_baseline': 0.505, 'auc': 0.9365514, 'auc_precision_recall': 0.9326272, 'average_loss': 0.32206511, 'label/mean': 0.495, 'loss': 40.25814, 'precision': 0.8526149, 'prediction/mean': 0.50739485, 'recall': 0.869495, 'global_step': 1300} ---------------------------------------------------------------------------------------------------- Training for step = 1300 Train Time (s): 85.8064513206482 Eval Metrics (Train): {'accuracy': 0.87743336, 'accuracy_baseline': 0.5005, 'auc': 0.9490614, 'auc_precision_recall': 0.94886625, 'average_loss': 0.29016578, 'label/mean': 0.5005, 'loss': 37.042442, 'precision': 0.87052286, 'prediction/mean': 0.5104429, 'recall': 0.8870463, 'global_step': 1400} Eval Metrics (Validation): {'accuracy': 0.8622, 'accuracy_baseline': 0.505, 'auc': 0.9367896, 'auc_precision_recall': 0.9333618, 'average_loss': 0.3225043, 'label/mean': 0.495, 'loss': 40.31304, 'precision': 0.8474708, 'prediction/mean': 0.514252, 'recall': 0.88, 'global_step': 1400} ---------------------------------------------------------------------------------------------------- Training for step = 1400 Train Time (s): 85.99037742614746 Eval Metrics (Train): {'accuracy': 0.8783, 'accuracy_baseline': 0.5005, 'auc': 0.9500882, 'auc_precision_recall': 0.94986326, 'average_loss': 0.28882334, 'label/mean': 0.5005, 'loss': 36.871063, 'precision': 0.865308, 'prediction/mean': 0.5196238, 'recall': 0.8963703, 'global_step': 1500} Eval Metrics (Validation): {'accuracy': 0.8626, 'accuracy_baseline': 0.505, 'auc': 0.93708724, 'auc_precision_recall': 0.9336051, 'average_loss': 0.32389137, 'label/mean': 0.495, 'loss': 40.486423, 'precision': 0.84044176, 'prediction/mean': 0.5226699, 'recall': 0.8917172, 'global_step': 1500} ---------------------------------------------------------------------------------------------------- Training for step = 1500 Train Time (s): 86.91469407081604 Eval Metrics (Train): {'accuracy': 0.8802, 'accuracy_baseline': 0.5005, 'auc': 0.95115364, 'auc_precision_recall': 0.950775, 'average_loss': 0.2844779, 'label/mean': 0.5005, 'loss': 36.316326, 'precision': 0.8735527, 'prediction/mean': 0.51057553, 'recall': 0.8893773, 'global_step': 1600} Eval Metrics (Validation): {'accuracy': 0.8626, 'accuracy_baseline': 0.505, 'auc': 0.9373224, 'auc_precision_recall': 0.9336302, 'average_loss': 0.32108024, 'label/mean': 0.495, 'loss': 40.135033, 'precision': 0.8478599, 'prediction/mean': 0.5134171, 'recall': 0.88040406, 'global_step': 1600}
dnn.evaluate(input_fn=predict_train_input_fn)
{'accuracy': 0.8802, 'accuracy_baseline': 0.5005, 'auc': 0.95115364, 'auc_precision_recall': 0.950775, 'average_loss': 0.2844779, 'label/mean': 0.5005, 'loss': 36.316326, 'precision': 0.8735527, 'prediction/mean': 0.51057553, 'recall': 0.8893773, 'global_step': 1600}
dnn.evaluate(input_fn=predict_test_input_fn)
{'accuracy': 0.8663333, 'accuracy_baseline': 0.5006667, 'auc': 0.9406502, 'auc_precision_recall': 0.93988097, 'average_loss': 0.31214723, 'label/mean': 0.5006667, 'loss': 39.679733, 'precision': 0.8597569, 'prediction/mean': 0.5120608, 'recall': 0.8758988, 'global_step': 1600}
import time
TOTAL_STEPS = 1500
STEP_SIZE = 500
my_checkpointing_config = tf.estimator.RunConfig(
keep_checkpoint_max = 2, # Retain the 2 most recent checkpoints.
)
def train_and_evaluate_with_sentence_encoder(hub_module, train_module=False, path=''):
embedding_feature = hub.text_embedding_column(
key='sentence', module_spec=hub_module, trainable=train_module)
print()
print('='*100)
print('Training with', hub_module)
print('Trainable is:', train_module)
print('='*100)
dnn = tf.estimator.DNNClassifier(
hidden_units=[512, 128],
feature_columns=[embedding_feature],
n_classes=2,
activation_fn=tf.nn.relu,
dropout=0.1,
optimizer=tf.train.AdagradOptimizer(learning_rate=0.005),
model_dir=path,
config=my_checkpointing_config)
for step in range(0, TOTAL_STEPS+1, STEP_SIZE):
print('-'*100)
print('Training for step =', step)
start_time = time.time()
dnn.train(input_fn=train_input_fn, steps=STEP_SIZE)
elapsed_time = time.time() - start_time
print('Train Time (s):', elapsed_time)
print('Eval Metrics (Train):', dnn.evaluate(input_fn=predict_train_input_fn))
print('Eval Metrics (Validation):', dnn.evaluate(input_fn=predict_val_input_fn))
train_eval_result = dnn.evaluate(input_fn=predict_train_input_fn)
test_eval_result = dnn.evaluate(input_fn=predict_test_input_fn)
return {
"Model Dir": dnn.model_dir,
"Training Accuracy": train_eval_result["accuracy"],
"Test Accuracy": test_eval_result["accuracy"],
"Training AUC": train_eval_result["auc"],
"Test AUC": test_eval_result["auc"],
"Training Precision": train_eval_result["precision"],
"Test Precision": test_eval_result["precision"],
"Training Recall": train_eval_result["recall"],
"Test Recall": test_eval_result["recall"]
}
tf.logging.set_verbosity(tf.logging.ERROR)
results = {}
results["nnlm-en-dim128"] = train_and_evaluate_with_sentence_encoder(
"https://tfhub.dev/google/nnlm-en-dim128/1", path='/storage/models/nnlm-en-dim128_f/')
results["nnlm-en-dim128-with-training"] = train_and_evaluate_with_sentence_encoder(
"https://tfhub.dev/google/nnlm-en-dim128/1", train_module=True, path='/storage/models/nnlm-en-dim128_t/')
results["use-512"] = train_and_evaluate_with_sentence_encoder(
"https://tfhub.dev/google/universal-sentence-encoder/2", path='/storage/models/use-512_f/')
results["use-512-with-training"] = train_and_evaluate_with_sentence_encoder(
"https://tfhub.dev/google/universal-sentence-encoder/2", train_module=True, path='/storage/models/use-512_t/')
==================================================================================================== Training with https://tfhub.dev/google/nnlm-en-dim128/1 Trainable is: False ==================================================================================================== ---------------------------------------------------------------------------------------------------- Training for step = 0 Train Time (s): 30.525171756744385 Eval Metrics (Train): {'accuracy': 0.8480667, 'accuracy_baseline': 0.5005, 'auc': 0.9287864, 'auc_precision_recall': 0.9287345, 'average_loss': 0.34465897, 'label/mean': 0.5005, 'loss': 43.99902, 'precision': 0.8288572, 'prediction/mean': 0.5302467, 'recall': 0.8776557, 'global_step': 2500} Eval Metrics (Validation): {'accuracy': 0.8288, 'accuracy_baseline': 0.505, 'auc': 0.91452694, 'auc_precision_recall': 0.9113482, 'average_loss': 0.37722248, 'label/mean': 0.495, 'loss': 47.15281, 'precision': 0.7999259, 'prediction/mean': 0.53336626, 'recall': 0.8723232, 'global_step': 2500} ---------------------------------------------------------------------------------------------------- Training for step = 500 Train Time (s): 27.883334159851074 Eval Metrics (Train): {'accuracy': 0.8558, 'accuracy_baseline': 0.5005, 'auc': 0.93249583, 'auc_precision_recall': 0.93237436, 'average_loss': 0.33365154, 'label/mean': 0.5005, 'loss': 42.59381, 'precision': 0.87215376, 'prediction/mean': 0.48501322, 'recall': 0.8341658, 'global_step': 3000} Eval Metrics (Validation): {'accuracy': 0.8356, 'accuracy_baseline': 0.505, 'auc': 0.91525424, 'auc_precision_recall': 0.9126499, 'average_loss': 0.36975703, 'label/mean': 0.495, 'loss': 46.219627, 'precision': 0.8397041, 'prediction/mean': 0.48800987, 'recall': 0.82545453, 'global_step': 3000} ---------------------------------------------------------------------------------------------------- Training for step = 1000 Train Time (s): 28.585613012313843 Eval Metrics (Train): {'accuracy': 0.8552667, 'accuracy_baseline': 0.5005, 'auc': 0.9354528, 'auc_precision_recall': 0.9354019, 'average_loss': 0.33256087, 'label/mean': 0.5005, 'loss': 42.45458, 'precision': 0.89311236, 'prediction/mean': 0.4612543, 'recall': 0.80745924, 'global_step': 3500} Eval Metrics (Validation): {'accuracy': 0.8348, 'accuracy_baseline': 0.505, 'auc': 0.91587234, 'auc_precision_recall': 0.9131782, 'average_loss': 0.3731109, 'label/mean': 0.495, 'loss': 46.638863, 'precision': 0.8573905, 'prediction/mean': 0.4640141, 'recall': 0.7991919, 'global_step': 3500} ---------------------------------------------------------------------------------------------------- Training for step = 1500 Train Time (s): 28.242169618606567 Eval Metrics (Train): {'accuracy': 0.8616, 'accuracy_baseline': 0.5005, 'auc': 0.9385461, 'auc_precision_recall': 0.9383698, 'average_loss': 0.32105166, 'label/mean': 0.5005, 'loss': 40.985317, 'precision': 0.8443543, 'prediction/mean': 0.52729076, 'recall': 0.8869797, 'global_step': 4000} Eval Metrics (Validation): {'accuracy': 0.828, 'accuracy_baseline': 0.505, 'auc': 0.91572505, 'auc_precision_recall': 0.91319984, 'average_loss': 0.37505153, 'label/mean': 0.495, 'loss': 46.88144, 'precision': 0.80322945, 'prediction/mean': 0.53075755, 'recall': 0.86424243, 'global_step': 4000} ==================================================================================================== Training with https://tfhub.dev/google/nnlm-en-dim128/1 Trainable is: True ==================================================================================================== ---------------------------------------------------------------------------------------------------- Training for step = 0 Train Time (s): 45.97756814956665 Eval Metrics (Train): {'accuracy': 0.9997, 'accuracy_baseline': 0.5005, 'auc': 0.9998141, 'auc_precision_recall': 0.99985945, 'average_loss': 0.0038648716, 'label/mean': 0.5005, 'loss': 0.49338785, 'precision': 0.99980015, 'prediction/mean': 0.5008926, 'recall': 0.9996004, 'global_step': 2500} Eval Metrics (Validation): {'accuracy': 0.877, 'accuracy_baseline': 0.505, 'auc': 0.9225529, 'auc_precision_recall': 0.9297111, 'average_loss': 0.67985016, 'label/mean': 0.495, 'loss': 84.98127, 'precision': 0.86671925, 'prediction/mean': 0.50768346, 'recall': 0.88808084, 'global_step': 2500} ---------------------------------------------------------------------------------------------------- Training for step = 500 Train Time (s): 44.74027681350708 Eval Metrics (Train): {'accuracy': 0.99986666, 'accuracy_baseline': 0.5005, 'auc': 0.9999234, 'auc_precision_recall': 0.9999373, 'average_loss': 0.002121097, 'label/mean': 0.5005, 'loss': 0.27077833, 'precision': 0.9998668, 'prediction/mean': 0.5002479, 'recall': 0.9998668, 'global_step': 3000} Eval Metrics (Validation): {'accuracy': 0.8764, 'accuracy_baseline': 0.505, 'auc': 0.9195744, 'auc_precision_recall': 0.9288729, 'average_loss': 0.74133915, 'label/mean': 0.495, 'loss': 92.6674, 'precision': 0.8742443, 'prediction/mean': 0.4980622, 'recall': 0.87636364, 'global_step': 3000} ---------------------------------------------------------------------------------------------------- Training for step = 1000 Train Time (s): 45.068076372146606 Eval Metrics (Train): {'accuracy': 0.9999667, 'accuracy_baseline': 0.5005, 'auc': 1.0, 'auc_precision_recall': 1.0, 'average_loss': 0.0009565035, 'label/mean': 0.5005, 'loss': 0.12210683, 'precision': 1.0, 'prediction/mean': 0.5007308, 'recall': 0.9999334, 'global_step': 3500} Eval Metrics (Validation): {'accuracy': 0.8748, 'accuracy_baseline': 0.505, 'auc': 0.9156478, 'auc_precision_recall': 0.9250907, 'average_loss': 0.8001606, 'label/mean': 0.495, 'loss': 100.02007, 'precision': 0.86584884, 'prediction/mean': 0.50656, 'recall': 0.8840404, 'global_step': 3500} ---------------------------------------------------------------------------------------------------- Training for step = 1500 Train Time (s): 44.654765605926514 Eval Metrics (Train): {'accuracy': 1.0, 'accuracy_baseline': 0.5005, 'auc': 1.0, 'auc_precision_recall': 1.0, 'average_loss': 0.00060436723, 'label/mean': 0.5005, 'loss': 0.077153265, 'precision': 1.0, 'prediction/mean': 0.50068456, 'recall': 1.0, 'global_step': 4000} Eval Metrics (Validation): {'accuracy': 0.875, 'accuracy_baseline': 0.505, 'auc': 0.91479605, 'auc_precision_recall': 0.9244194, 'average_loss': 0.8459253, 'label/mean': 0.495, 'loss': 105.74066, 'precision': 0.8661916, 'prediction/mean': 0.5066238, 'recall': 0.8840404, 'global_step': 4000} ==================================================================================================== Training with https://tfhub.dev/google/universal-sentence-encoder/2 Trainable is: False ==================================================================================================== ---------------------------------------------------------------------------------------------------- Training for step = 0 Train Time (s): 261.7671597003937 Eval Metrics (Train): {'accuracy': 0.8591, 'accuracy_baseline': 0.5005, 'auc': 0.9373971, 'auc_precision_recall': 0.93691623, 'average_loss': 0.3231426, 'label/mean': 0.5005, 'loss': 41.252243, 'precision': 0.8820655, 'prediction/mean': 0.47581005, 'recall': 0.8293706, 'global_step': 501} Eval Metrics (Validation): {'accuracy': 0.8522, 'accuracy_baseline': 0.505, 'auc': 0.93081224, 'auc_precision_recall': 0.9264202, 'average_loss': 0.33680823, 'label/mean': 0.495, 'loss': 42.10103, 'precision': 0.8631799, 'prediction/mean': 0.47982788, 'recall': 0.8335354, 'global_step': 501} ---------------------------------------------------------------------------------------------------- Training for step = 500 Train Time (s): 259.56947469711304 Eval Metrics (Train): {'accuracy': 0.8696333, 'accuracy_baseline': 0.5005, 'auc': 0.9444929, 'auc_precision_recall': 0.94410896, 'average_loss': 0.30284137, 'label/mean': 0.5005, 'loss': 38.660603, 'precision': 0.8605663, 'prediction/mean': 0.5134329, 'recall': 0.88251746, 'global_step': 1001} Eval Metrics (Validation): {'accuracy': 0.8608, 'accuracy_baseline': 0.505, 'auc': 0.93478435, 'auc_precision_recall': 0.93100405, 'average_loss': 0.32802072, 'label/mean': 0.495, 'loss': 41.00259, 'precision': 0.8446339, 'prediction/mean': 0.5172887, 'recall': 0.88080806, 'global_step': 1001} ---------------------------------------------------------------------------------------------------- Training for step = 1000 Train Time (s): 258.70958161354065 Eval Metrics (Train): {'accuracy': 0.8782333, 'accuracy_baseline': 0.5005, 'auc': 0.9505533, 'auc_precision_recall': 0.94987434, 'average_loss': 0.28575605, 'label/mean': 0.5005, 'loss': 36.479496, 'precision': 0.8756281, 'prediction/mean': 0.5043332, 'recall': 0.8819847, 'global_step': 1501} Eval Metrics (Validation): {'accuracy': 0.8616, 'accuracy_baseline': 0.505, 'auc': 0.9369738, 'auc_precision_recall': 0.9333807, 'average_loss': 0.32096896, 'label/mean': 0.495, 'loss': 40.121117, 'precision': 0.8505702, 'prediction/mean': 0.5080113, 'recall': 0.8739394, 'global_step': 1501} ---------------------------------------------------------------------------------------------------- Training for step = 1500 Train Time (s): 258.4421606063843 Eval Metrics (Train): {'accuracy': 0.88733333, 'accuracy_baseline': 0.5005, 'auc': 0.9558296, 'auc_precision_recall': 0.95508415, 'average_loss': 0.2716801, 'label/mean': 0.5005, 'loss': 34.682564, 'precision': 0.8979955, 'prediction/mean': 0.4882649, 'recall': 0.8741925, 'global_step': 2001} Eval Metrics (Validation): {'accuracy': 0.864, 'accuracy_baseline': 0.505, 'auc': 0.938815, 'auc_precision_recall': 0.9357392, 'average_loss': 0.31562653, 'label/mean': 0.495, 'loss': 39.453316, 'precision': 0.864393, 'prediction/mean': 0.49126464, 'recall': 0.860202, 'global_step': 2001} ==================================================================================================== Training with https://tfhub.dev/google/universal-sentence-encoder/2 Trainable is: True ==================================================================================================== ---------------------------------------------------------------------------------------------------- Training for step = 0 Train Time (s): 313.1993100643158 Eval Metrics (Train): {'accuracy': 0.99916667, 'accuracy_baseline': 0.5005, 'auc': 0.9996535, 'auc_precision_recall': 0.9996587, 'average_loss': 0.0054427227, 'label/mean': 0.5005, 'loss': 0.69481564, 'precision': 0.9989349, 'prediction/mean': 0.50010633, 'recall': 0.9994006, 'global_step': 500} Eval Metrics (Validation): {'accuracy': 0.9056, 'accuracy_baseline': 0.505, 'auc': 0.95068294, 'auc_precision_recall': 0.95441175, 'average_loss': 0.40755096, 'label/mean': 0.495, 'loss': 50.94387, 'precision': 0.9020474, 'prediction/mean': 0.4965181, 'recall': 0.9078788, 'global_step': 500} ---------------------------------------------------------------------------------------------------- Training for step = 500 Train Time (s): 306.61662435531616 Eval Metrics (Train): {'accuracy': 0.9999667, 'accuracy_baseline': 0.5005, 'auc': 0.9999665, 'auc_precision_recall': 0.9999665, 'average_loss': 0.00048276514, 'label/mean': 0.5005, 'loss': 0.061629593, 'precision': 0.9999334, 'prediction/mean': 0.50048435, 'recall': 1.0, 'global_step': 1000} Eval Metrics (Validation): {'accuracy': 0.9024, 'accuracy_baseline': 0.505, 'auc': 0.93500155, 'auc_precision_recall': 0.94411886, 'average_loss': 0.5513662, 'label/mean': 0.495, 'loss': 68.92078, 'precision': 0.892843, 'prediction/mean': 0.50568086, 'recall': 0.91232324, 'global_step': 1000} ---------------------------------------------------------------------------------------------------- Training for step = 1000 Train Time (s): 306.62883472442627 Eval Metrics (Train): {'accuracy': 0.9999667, 'accuracy_baseline': 0.5005, 'auc': 0.9999666, 'auc_precision_recall': 0.9999666, 'average_loss': 0.00030994884, 'label/mean': 0.5005, 'loss': 0.039567936, 'precision': 0.9999334, 'prediction/mean': 0.50050503, 'recall': 1.0, 'global_step': 1500} Eval Metrics (Validation): {'accuracy': 0.9018, 'accuracy_baseline': 0.505, 'auc': 0.9302231, 'auc_precision_recall': 0.9409413, 'average_loss': 0.61150163, 'label/mean': 0.495, 'loss': 76.4377, 'precision': 0.8905512, 'prediction/mean': 0.50745887, 'recall': 0.9139394, 'global_step': 1500} ---------------------------------------------------------------------------------------------------- Training for step = 1500 Train Time (s): 305.9913341999054 Eval Metrics (Train): {'accuracy': 1.0, 'accuracy_baseline': 0.5005, 'auc': 1.0, 'auc_precision_recall': 1.0, 'average_loss': 5.714076e-05, 'label/mean': 0.5005, 'loss': 0.0072945654, 'precision': 1.0, 'prediction/mean': 0.5004708, 'recall': 1.0, 'global_step': 2000} Eval Metrics (Validation): {'accuracy': 0.9032, 'accuracy_baseline': 0.505, 'auc': 0.929281, 'auc_precision_recall': 0.9409471, 'average_loss': 0.6407001, 'label/mean': 0.495, 'loss': 80.08751, 'precision': 0.8986784, 'prediction/mean': 0.4996146, 'recall': 0.9066667, 'global_step': 2000}
results_df = pd.DataFrame.from_dict(results, orient="index")
results_df
Model Dir | Training Accuracy | Test Accuracy | Training AUC | Test AUC | Training Precision | Test Precision | Training Recall | Test Recall | |
---|---|---|---|---|---|---|---|---|---|
nnlm-en-dim128 | /storage/models/nnlm-en-dim128_f/ | 0.861600 | 0.836133 | 0.938546 | 0.918221 | 0.844354 | 0.822770 | 0.886980 | 0.857390 |
nnlm-en-dim128-with-training | /storage/models/nnlm-en-dim128_t/ | 1.000000 | 0.878467 | 1.000000 | 0.919655 | 1.000000 | 0.875975 | 1.000000 | 0.882157 |
use-512 | /storage/models/use-512_f/ | 0.887333 | 0.867067 | 0.955830 | 0.942319 | 0.897995 | 0.876776 | 0.874192 | 0.854594 |
use-512-with-training | /storage/models/use-512_t/ | 1.000000 | 0.904533 | 1.000000 | 0.930401 | 1.000000 | 0.904660 | 1.000000 | 0.904660 |
best_model_dir = results_df[results_df['Test Accuracy'] == results_df['Test Accuracy'].max()]['Model Dir'].values[0]
best_model_dir
'/storage/models/use-512_t/'
embedding_feature = hub.text_embedding_column(
key='sentence', module_spec="https://tfhub.dev/google/universal-sentence-encoder/2", trainable=True)
dnn = tf.estimator.DNNClassifier(
hidden_units=[512, 128],
feature_columns=[embedding_feature],
n_classes=2,
activation_fn=tf.nn.relu,
dropout=0.1,
optimizer=tf.train.AdagradOptimizer(learning_rate=0.005),
model_dir=best_model_dir)
dnn
<tensorflow.python.estimator.canned.dnn.DNNClassifier at 0x7f46486c3278>
def get_predictions(estimator, input_fn):
return [x["class_ids"][0] for x in estimator.predict(input_fn=input_fn)]
predictions = get_predictions(estimator=dnn, input_fn=predict_test_input_fn)
predictions[:10]
[0, 1, 0, 1, 1, 0, 1, 1, 1, 1]
!pip install seaborn
Requirement already satisfied: seaborn in /usr/local/lib/python3.6/dist-packages (0.9.0)
Requirement already satisfied: scipy>=0.14.0 in /usr/local/lib/python3.6/dist-packages (from seaborn) (1.0.1)
Requirement already satisfied: pandas>=0.15.2 in /usr/local/lib/python3.6/dist-packages (from seaborn) (0.22.0)
Requirement already satisfied: numpy>=1.9.3 in /usr/local/lib/python3.6/dist-packages (from seaborn) (1.14.3)
Requirement already satisfied: matplotlib>=1.4.3 in /usr/local/lib/python3.6/dist-packages (from seaborn) (2.2.2)
Requirement already satisfied: pytz>=2011k in /usr/local/lib/python3.6/dist-packages (from pandas>=0.15.2->seaborn) (2018.4)
Requirement already satisfied: python-dateutil>=2 in /usr/local/lib/python3.6/dist-packages (from pandas>=0.15.2->seaborn) (2.7.2)
Requirement already satisfied: six>=1.10 in /usr/local/lib/python3.6/dist-packages (from matplotlib>=1.4.3->seaborn) (1.11.0)
Requirement already satisfied: kiwisolver>=1.0.1 in /usr/local/lib/python3.6/dist-packages (from matplotlib>=1.4.3->seaborn) (1.0.1)
Requirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.1 in /usr/local/lib/python3.6/dist-packages (from matplotlib>=1.4.3->seaborn) (2.2.0)
Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.6/dist-packages (from matplotlib>=1.4.3->seaborn) (0.10.0)
Requirement already satisfied: setuptools in /usr/local/lib/python3.6/dist-packages (from kiwisolver>=1.0.1->matplotlib>=1.4.3->seaborn) (39.1.0)
You are using pip version 10.0.1, however version 18.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
import seaborn as sns
import matplotlib.pyplot as plt
%matplotlib inline
with tf.Session() as session:
cm = tf.confusion_matrix(test_sentiments, predictions).eval()
LABELS = ['negative', 'positive']
sns.heatmap(cm, annot=True, xticklabels=LABELS, yticklabels=LABELS, fmt='g')
xl = plt.xlabel("Predicted")
yl = plt.ylabel("Actuals")
from sklearn.metrics import classification_report
print(classification_report(y_true=test_sentiments, y_pred=predictions, target_names=LABELS))
precision recall f1-score support negative 0.90 0.90 0.90 7490 positive 0.90 0.90 0.90 7510 avg / total 0.90 0.90 0.90 15000