BentoML makes moving trained ML models to production easy:
BentoML is a framework for serving, managing, and deploying machine learning models. It is aiming to bridge the gap between Data Science and DevOps, and enable teams to deliver prediction services in a fast, repeatable, and scalable way.
This tutorial provides an end-to-end guide using BentoML with AWS SageMaker -- a machine learning model training platform. It demonstrates the workflow of integrating BentoML with SageMaker, including: setting up a SageMaker notebook instance, model training, creating an S3 bucket, uploading the BentoService bundle into S3, and deploying the BentoML packaged model to SageMaker as an API endpoint using the BentoML CLI tool.
For demonstration, this tutorial uses the IMDB movie review sentiment dataset with BERT and Tensorflow 2.0.(please note: the following model is a modification of the original version: https://github.com/kpe/bert-for-tf2/blob/master/examples/gpu_movie_reviews.ipynb)
For model training in SageMaker, we will start by creating a notebook instance. After logging into the AWS management console -- type SageMaker to launch the service. From the SageMaker dashboard, select Notebook instances. Go ahead enter a notebook name and select the instance type
Next,under Permissions and encryption, select Create a new role or choosing an existing role. This allows both the notebook instance and user to access and upload data to Amazon S3. Then, select Any S3 bucket, which allows your SageMaker to access all S3 buckets in your account.
After the notebook instance is created, the status will change from pending to InService. Select Open Jupyter under Actions, and choose Conda_python 3 under New tab to launch the Jupyter notebook within SageMaker.
Note: SageMaker also provides a local model through pip install SageMaker.
Finally to prepare for the model training, let's import some libraries -- Boto3 and SageMaker and set up the IAM role. Boto3 is the AWS SDK for Python, which makes it easier to integrate our model with AWS services such as Amazon S3
import boto3, sagemaker
from sagemaker import get_execution_role
# Define IAM role
role = get_execution_role()
prefix = 'sagemaker/bert-moviereview-bento'
my_region = boto3.session.Session().region_name # set the region of the instance
In this step, we will create an S3 bucket named movie-review-dataset to store the dataset. Users could click on the bucket name and upload the dataset directly into S3. Alternatively, for cost-efficiency, users could train the model locally using the SageMaker local mode
bucket_name = 'movie-review-dataset'
s3 = boto3.resource('s3')
s3.create_bucket(Bucket=bucket_name)
s3.Bucket(name='movie-review-dataset')
The second step of this tutorial is model training. We will be using the IMDB movie review dataset to create a sentiment analysis model which contains 25K positive and negative movie reviews each. First, let's install the bert-for-tf2 package.
!pip install -q bentoml bert-for-tf2>=0.14.5
Collecting bert-for-tf2 Downloading https://files.pythonhosted.org/packages/35/5c/6439134ecd17b33fe0396fb0b7d6ce3c5a120c42a4516ba0e9a2d6e43b25/bert-for-tf2-0.14.4.tar.gz (40kB) |████████████████████████████████| 40kB 2.3MB/s Collecting py-params>=0.9.6 Downloading https://files.pythonhosted.org/packages/a4/bf/c1c70d5315a8677310ea10a41cfc41c5970d9b37c31f9c90d4ab98021fd1/py-params-0.9.7.tar.gz Collecting params-flow>=0.8.0 Downloading https://files.pythonhosted.org/packages/a9/95/ff49f5ebd501f142a6f0aaf42bcfd1c192dc54909d1d9eb84ab031d46056/params-flow-0.8.2.tar.gz Requirement already satisfied: numpy in /usr/local/lib/python3.6/dist-packages (from params-flow>=0.8.0->bert-for-tf2) (1.18.5) Requirement already satisfied: tqdm in /usr/local/lib/python3.6/dist-packages (from params-flow>=0.8.0->bert-for-tf2) (4.41.1) Building wheels for collected packages: bert-for-tf2, py-params, params-flow Building wheel for bert-for-tf2 (setup.py) ... done Created wheel for bert-for-tf2: filename=bert_for_tf2-0.14.4-cp36-none-any.whl size=30114 sha256=5f3374b29261a7a31e5ddeb2d661d5c326dd68e720d3897502670561b6fd2f74 Stored in directory: /root/.cache/pip/wheels/cf/3f/4d/79d7735015a5f523648df90d871ce8e89a7df8185f7703eeab Building wheel for py-params (setup.py) ... done Created wheel for py-params: filename=py_params-0.9.7-cp36-none-any.whl size=7302 sha256=3454cb699e6561be3b0787bd7b0d6e9af9a4ed711b6d11138b3ea7b11f96bf20 Stored in directory: /root/.cache/pip/wheels/67/f5/19/b461849a50aefdf4bab47c4756596e82ee2118b8278e5a1980 Building wheel for params-flow (setup.py) ... done Created wheel for params-flow: filename=params_flow-0.8.2-cp36-none-any.whl size=19473 sha256=a43985d8a6541a645abc4514701aab9390d0a9d6c5840291c172d6e82a83f6a5 Stored in directory: /root/.cache/pip/wheels/08/c8/7f/81c86b9ff2b86e2c477e3914175be03e679e596067dc630c06 Successfully built bert-for-tf2 py-params params-flow Installing collected packages: py-params, params-flow, bert-for-tf2 Successfully installed bert-for-tf2-0.14.4 params-flow-0.8.2 py-params-0.9.7
import bert
from bert import BertModelLayer
from bert.loader import StockBertConfig, map_stock_config_to_params, load_stock_weights
from bert.tokenization.bert_tokenization import FullTokenizer
import os
import re
import sys
import math
import datetime
import pandas as pd
import numpy as np
import tensorflow as tf
from tensorflow import keras
print ('Tensorflow: ', tf.__version__)
print ('Python: ', sys.version)
Tensorflow: 2.2.0 Python: 3.7.6 (default, Jan 8 2020, 13:42:34) [Clang 4.0.1 (tags/RELEASE_401/final)]
Here, we will download, extracts and import the IMDB large movie review dataset.
from tensorflow import keras
import os
import re
# load all files from the directory into a dataframe
def load_directory_data(directory):
data ={}
data['sentence'] = []
data['sentiment'] = []
for file_path in os.listdir(directory):
with tf.io.gfile.GFile(os.path.join(directory, file_path), 'r') as f:
data['sentence'].append(f.read())
data['sentiment'].append(re.match('\d+_(\d+)\.txt',file_path).group(1))
return pd.DataFrame.from_dict(data)
# combine positive and negative reviews into a dataframe; add a polarity column
def load_dataset(directory):
pos_df = load_directory_data(os.path.join(directory, 'pos'))
neg_df = load_directory_data(os.path.join(directory,'neg'))
pos_df['polarity'] = 1
neg_df['polarity'] = 0
return pd.concat([pos_df, neg_df]).sample(frac=1).reset_index(drop=True)
# download dataset from link
def download_and_load_datasets(force_download=False):
dataset = tf.keras.utils.get_file(
fname = 'acImbd.tar.gz',
origin = 'http://ai.stanford.edu/~amaas/data/sentiment/aclImdb_v1.tar.gz',
extract = True)
train_df = load_dataset(os.path.join(os.path.dirname(dataset),"aclImdb",'train'))
test_df = load_dataset(os.path.join(os.path.dirname(dataset),"aclImdb",'test'))
return train_df, test_df
Let's use the MovieReviewData class below, to prepare/encode the data for feeding into our BERT model, by:
class MovieReviewData:
DATA_COLUMN = 'sentence'
LABEL_COLUMN = 'polarity'
def __init__(self, tokenizer: FullTokenizer, sample_size =None, max_seq_len =1024):
self.tokenizer = tokenizer
self.sample_size = sample_size
self.max_seq_len = 0
train, test = download_and_load_datasets()
train,test =map(lambda df: df.reindex(df[MovieReviewData.DATA_COLUMN].str.len().sort_values().index),[train,test])
if sample_size is not None:
train, test = train.head(sample_size), test.head(sample_size)
((self.train_x, self.train_y),
(self.test_x, self.test_y)) = map(self._prepare, [train,test])
print('max_seq_len', self.max_seq_len)
self.max_seq_len = min(self.max_seq_len,max_seq_len)
((self.train_x, self.train_x_token_types),
(self.test_x, self.test_x_token_types)) = map(self._pad,[self.train_x , self.test_x])
def _prepare(self,df):
x,y =[],[]
with tqdm(total =df.shape[0], unit_scale=True) as pbar:
for ndx, row in df.iterrows():
text, label = row[MovieReviewData.DATA_COLUMN], row[MovieReviewData.LABEL_COLUMN]
tokens = self.tokenizer.tokenize(text)
tokens = ['[CLS]'] + tokens + ['[SEP]']
token_ids = self.tokenizer.convert_tokens_to_ids(tokens)
self.max_seq_len = max(self.max_seq_len , len(token_ids))
x.append(token_ids)
y.append(int(label))
pbar.update()
return np.array(x),np.array(y)
def _pad(self,ids):
x,t = [],[]
token_type_ids = [0] * self.max_seq_len
for input_ids in ids:
input_ids = input_ids[:min(len(input_ids),self.max_seq_len - 2)]
input_ids = input_ids + [0] * (self.max_seq_len - len(input_ids))
x.append(np.array(input_ids))
t.append(token_type_ids)
return np.array(x), np.array(t)
This tutorial uses the pre-trained BERT model -- BERT-Base,Uncased, which could be downloaded here. https://github.com/google-research/bert. Users could save the BERT weights in the S3 bucket created early or use the local SageMaker enviroment and save the weights locally.
assert_path = 'asset'
bert_model_name = 'uncased_L-12_H-768_A-12'
bert_ckpt_dir = os.path.join(assert_path , bert_model_name)
bert_ckpt_file = os.path.join(bert_ckpt_dir, 'bert_model.ckpt')
bert_config_file = os.path.join(bert_ckpt_dir, 'bert_config.json')
After getting both the IMDB movie review data and the BERT weights ready, we will use the S3 bucket to store these data by directly uploading them.
In this step, we are ready to fetch the data using the BERT tokenizer. We will take the first 128 tokens by setting the max_seq_len = 128 and a sample size of 2500 each for train and test data due to transformer memory concerns.
%%time
tokenizer = FullTokenizer(vocab_file =os.path.join(bert_ckpt_dir, 'vocab.txt'))
data = MovieReviewData(tokenizer, sample_size = 10*128*2, max_seq_len =128) # sample_size:5000
Downloading data from http://ai.stanford.edu/~amaas/data/sentiment/aclImdb_v1.tar.gz 84131840/84125825 [==============================] - 3s 0us/step
100%|██████████| 2.56k/2.56k [00:02<00:00, 870it/s] 100%|██████████| 2.56k/2.56k [00:02<00:00, 874it/s]
max_seq_len 178 CPU times: user 23.5 s, sys: 8.87 s, total: 32.4 s Wall time: 34.9 s
print('train_x',data.train_x.shape)
print('train_x_token_types',data.train_x_token_types.shape)
print('train_y',data.train_y.shape)
print('test_x',data.test_x.shape)
print('max_seq_len',data.max_seq_len)
train_x (2560, 128) train_x_token_types (2560, 128) train_y (2560,) test_x (2560, 128) max_seq_len 128
In this tutorial, we will also be using Adapter BERT, which requires us to frezee the original BERT layers first. In short, adapter BERT is a more parameter efficient way for fine-tuning. Instead of using an entire new model for every task, adapter BERT adds only a few trainable parameters per task while achieving near state-of-the-art performance. For more information on adapter BERT, visit here: https://arxiv.org/abs/1902.00751
def flatten_layers(root_layer):
if isinstance(root_layer, keras.layers.Layer):
yield root_layer
for layer in root_layer._layers:
for sub_layer in flatten_layers(layer):
yield sub_layer
def freeze_bert_layers(l_bert):
"""
Freezes all but LayerNorm and adapter layers - see arXiv:1902.00751.
"""
for layer in flatten_layers(l_bert):
if layer.name in ["LayerNorm", "adapter-down", "adapter-up"]:
layer.trainable = True
elif len(layer._layers) == 0:
layer.trainable = False
l_bert.embeddings_layer.trainable = False
def create_learning_rate_scheduler(max_learn_rate=5e-5,
end_learn_rate=1e-7,
warmup_epoch_count=10,
total_epoch_count=90):
def lr_scheduler(epoch):
if epoch < warmup_epoch_count:
res = (max_learn_rate/warmup_epoch_count) * (epoch + 1)
else:
res = max_learn_rate*math.exp(
math.log(end_learn_rate/max_learn_rate)*(epoch-warmup_epoch_count+1)/(total_epoch_count-warmup_epoch_count+1))
return float(res)
learning_rate_scheduler = tf.keras.callbacks.LearningRateScheduler(lr_scheduler, verbose=1)
return learning_rate_scheduler
Now, we are ready to create and train our model
def create_model(max_seq_len, adapter_size =64):
#create the bert layer
with tf.io.gfile.GFile(bert_config_file, 'r') as reader:
bc = StockBertConfig.from_json_string(reader.read())
bert_params = map_stock_config_to_params(bc)
bert_params.adapter_size = adapter_size
bert = BertModelLayer.from_params(bert_params, name = 'bert')
input_ids = keras.layers.Input(shape=(max_seq_len,), dtype='int32', name = 'input_ids')
output = bert(input_ids)
print('bert shape',output.shape)
cls_out = keras.layers.Lambda(lambda seq: seq[:,0,:])(output)
cls_out = keras.layers.Dropout(0.5)(cls_out)
logits = keras.layers.Dense(units=768, activation='tanh')(cls_out)
logits = keras.layers.Dropout(0.5)(logits)
logits = keras.layers.Dense(units=2, activation = 'softmax')(logits)
model = keras.Model(inputs = input_ids, outputs = logits)
model.build(input_shape = (None, max_seq_len))
load_stock_weights(bert, bert_ckpt_file)
#freeze weights if adapter-BERT is used
if adapter_size is not None:
freeze_bert_layers(bert)
model.compile(optimizer = keras.optimizers.Adam(),
loss = keras.losses.SparseCategoricalCrossentropy(from_logits=True),
metrics = [keras.metrics.SparseCategoricalAccuracy(name = 'acc')])
model.summary()
return model
adapter_size = None
model = create_model(data.max_seq_len, adapter_size = adapter_size)
bert shape (None, 128, 768) Done loading 196 BERT weights from: asset/uncased_L-12_H-768_A-12/bert_model.ckpt into <bert.model.BertModelLayer object at 0x7febec1387b8> (prefix:bert). Count of weights not found in the checkpoint was: [0]. Count of weights with mismatched shape: [0] Unused weights from checkpoint: bert/embeddings/token_type_embeddings bert/pooler/dense/bias bert/pooler/dense/kernel cls/predictions/output_bias cls/predictions/transform/LayerNorm/beta cls/predictions/transform/LayerNorm/gamma cls/predictions/transform/dense/bias cls/predictions/transform/dense/kernel cls/seq_relationship/output_bias cls/seq_relationship/output_weights Model: "model" _________________________________________________________________ Layer (type) Output Shape Param # ================================================================= input_ids (InputLayer) [(None, 128)] 0 _________________________________________________________________ bert (BertModelLayer) (None, 128, 768) 108890112 _________________________________________________________________ lambda (Lambda) (None, 768) 0 _________________________________________________________________ dropout (Dropout) (None, 768) 0 _________________________________________________________________ dense (Dense) (None, 768) 590592 _________________________________________________________________ dropout_1 (Dropout) (None, 768) 0 _________________________________________________________________ dense_1 (Dense) (None, 2) 1538 ================================================================= Total params: 109,482,242 Trainable params: 109,482,242 Non-trainable params: 0 _________________________________________________________________
%%time
import datetime
log_dir = ".log/movie_reviews/" + datetime.datetime.now().strftime("%Y%m%d-%H%M%s")
tensorboard_callback = keras.callbacks.TensorBoard(log_dir=log_dir)
total_epoch_count = 20
callbacks = [
create_learning_rate_scheduler(max_learn_rate=1e-5,
end_learn_rate=1e-7,
warmup_epoch_count=20,
total_epoch_count=total_epoch_count),
keras.callbacks.EarlyStopping(patience=20, restore_best_weights=True),
tensorboard_callback]
model.fit(x=data.train_x, y=data.train_y,
validation_split = 0.1,
batch_size = 12,
shuffle=True,
epochs = total_epoch_count,
callbacks=callbacks
)
Epoch 00001: LearningRateScheduler reducing learning rate to 5.000000000000001e-07. Epoch 1/20 192/192 [==============================] - 78s 404ms/step - loss: 0.7004 - acc: 0.5425 - val_loss: 0.6933 - val_acc: 0.5469 - lr: 5.0000e-07 Epoch 00002: LearningRateScheduler reducing learning rate to 1.0000000000000002e-06. Epoch 2/20 192/192 [==============================] - 75s 390ms/step - loss: 0.6884 - acc: 0.5621 - val_loss: 0.6683 - val_acc: 0.5625 - lr: 1.0000e-06 Epoch 00003: LearningRateScheduler reducing learning rate to 1.5000000000000002e-06. Epoch 3/20 192/192 [==============================] - 74s 385ms/step - loss: 0.6484 - acc: 0.6176 - val_loss: 0.5679 - val_acc: 0.7734 - lr: 1.5000e-06 Epoch 00004: LearningRateScheduler reducing learning rate to 2.0000000000000003e-06. Epoch 4/20 192/192 [==============================] - 76s 394ms/step - loss: 0.5064 - acc: 0.8125 - val_loss: 0.4224 - val_acc: 0.8906 - lr: 2.0000e-06 Epoch 00005: LearningRateScheduler reducing learning rate to 2.5000000000000006e-06. Epoch 5/20 192/192 [==============================] - 75s 388ms/step - loss: 0.4216 - acc: 0.8867 - val_loss: 0.4148 - val_acc: 0.9023 - lr: 2.5000e-06 Epoch 00006: LearningRateScheduler reducing learning rate to 3.0000000000000005e-06. Epoch 6/20 192/192 [==============================] - 75s 392ms/step - loss: 0.3891 - acc: 0.9249 - val_loss: 0.4110 - val_acc: 0.8945 - lr: 3.0000e-06 Epoch 00007: LearningRateScheduler reducing learning rate to 3.5000000000000004e-06. Epoch 7/20 192/192 [==============================] - 74s 386ms/step - loss: 0.3753 - acc: 0.9375 - val_loss: 0.4270 - val_acc: 0.8789 - lr: 3.5000e-06 Epoch 00008: LearningRateScheduler reducing learning rate to 4.000000000000001e-06. Epoch 8/20 192/192 [==============================] - 74s 384ms/step - loss: 0.3653 - acc: 0.9470 - val_loss: 0.4175 - val_acc: 0.8945 - lr: 4.0000e-06 Epoch 00009: LearningRateScheduler reducing learning rate to 4.500000000000001e-06. Epoch 9/20 192/192 [==============================] - 73s 383ms/step - loss: 0.3508 - acc: 0.9648 - val_loss: 0.4278 - val_acc: 0.8711 - lr: 4.5000e-06 Epoch 00010: LearningRateScheduler reducing learning rate to 5.000000000000001e-06. Epoch 10/20 192/192 [==============================] - 74s 385ms/step - loss: 0.3454 - acc: 0.9679 - val_loss: 0.4338 - val_acc: 0.8711 - lr: 5.0000e-06 Epoch 00011: LearningRateScheduler reducing learning rate to 5.500000000000001e-06. Epoch 11/20 192/192 [==============================] - 74s 383ms/step - loss: 0.3424 - acc: 0.9701 - val_loss: 0.4405 - val_acc: 0.8750 - lr: 5.5000e-06 Epoch 00012: LearningRateScheduler reducing learning rate to 6.000000000000001e-06. Epoch 12/20 192/192 [==============================] - 74s 385ms/step - loss: 0.3411 - acc: 0.9722 - val_loss: 0.4238 - val_acc: 0.8906 - lr: 6.0000e-06 Epoch 00013: LearningRateScheduler reducing learning rate to 6.500000000000001e-06. Epoch 13/20 192/192 [==============================] - 74s 384ms/step - loss: 0.3394 - acc: 0.9744 - val_loss: 0.4260 - val_acc: 0.8828 - lr: 6.5000e-06 Epoch 00014: LearningRateScheduler reducing learning rate to 7.000000000000001e-06. Epoch 14/20 192/192 [==============================] - 74s 383ms/step - loss: 0.3378 - acc: 0.9757 - val_loss: 0.4239 - val_acc: 0.8867 - lr: 7.0000e-06 Epoch 00015: LearningRateScheduler reducing learning rate to 7.500000000000001e-06. Epoch 15/20 192/192 [==============================] - 74s 383ms/step - loss: 0.3431 - acc: 0.9701 - val_loss: 0.4377 - val_acc: 0.8672 - lr: 7.5000e-06 Epoch 00016: LearningRateScheduler reducing learning rate to 8.000000000000001e-06. Epoch 16/20 192/192 [==============================] - 73s 381ms/step - loss: 0.3381 - acc: 0.9744 - val_loss: 0.4238 - val_acc: 0.8828 - lr: 8.0000e-06 Epoch 00017: LearningRateScheduler reducing learning rate to 8.500000000000002e-06. Epoch 17/20 192/192 [==============================] - 74s 383ms/step - loss: 0.3471 - acc: 0.9661 - val_loss: 0.4323 - val_acc: 0.8789 - lr: 8.5000e-06 Epoch 00018: LearningRateScheduler reducing learning rate to 9.000000000000002e-06. Epoch 18/20 192/192 [==============================] - 73s 382ms/step - loss: 0.3387 - acc: 0.9748 - val_loss: 0.4262 - val_acc: 0.8867 - lr: 9.0000e-06 Epoch 00019: LearningRateScheduler reducing learning rate to 9.500000000000002e-06. Epoch 19/20 192/192 [==============================] - 73s 382ms/step - loss: 0.3360 - acc: 0.9761 - val_loss: 0.4158 - val_acc: 0.8945 - lr: 9.5000e-06 Epoch 00020: LearningRateScheduler reducing learning rate to 1.0000000000000003e-05. Epoch 20/20 192/192 [==============================] - 74s 384ms/step - loss: 0.3378 - acc: 0.9748 - val_loss: 0.4104 - val_acc: 0.8984 - lr: 1.0000e-05 CPU times: user 29min 44s, sys: 2min 22s, total: 32min 6s Wall time: 25min 2s
#save model weights
model.save_weights('./movie_review.h5', overwrite=True)
%%time
_, train_acc = model.evaluate(data.train_x, data.train_y)
_, test_acc = model.evaluate(data.test_x, data.test_y)
print('train acc', train_acc)
print('test acc', test_acc)
80/80 [==============================] - 12s 155ms/step - loss: 0.3383 - acc: 0.9742 80/80 [==============================] - 12s 155ms/step - loss: 0.3960 - acc: 0.9152 train acc 0.9742187261581421 test acc 0.915234386920929 CPU times: user 17.5 s, sys: 9.31 s, total: 26.8 s Wall time: 27 s
For evaluation, let's load the previously saved model weights into a new model instance
model = create_model(data.max_seq_len, adapter_size=None)
model.load_weights('./movie_review.h5')
bert shape (None, 128, 768) Done loading 196 BERT weights from: asset/uncased_L-12_H-768_A-12/bert_model.ckpt into <bert.model.BertModelLayer object at 0x7f16a19737b8> (prefix:bert_3). Count of weights not found in the checkpoint was: [0]. Count of weights with mismatched shape: [0] Unused weights from checkpoint: bert/embeddings/token_type_embeddings bert/pooler/dense/bias bert/pooler/dense/kernel cls/predictions/output_bias cls/predictions/transform/LayerNorm/beta cls/predictions/transform/LayerNorm/gamma cls/predictions/transform/dense/bias cls/predictions/transform/dense/kernel cls/seq_relationship/output_bias cls/seq_relationship/output_weights Model: "model_3" _________________________________________________________________ Layer (type) Output Shape Param # ================================================================= input_ids (InputLayer) [(None, 128)] 0 _________________________________________________________________ bert (BertModelLayer) (None, 128, 768) 108890112 _________________________________________________________________ lambda_3 (Lambda) (None, 768) 0 _________________________________________________________________ dropout_6 (Dropout) (None, 768) 0 _________________________________________________________________ dense_6 (Dense) (None, 768) 590592 _________________________________________________________________ dropout_7 (Dropout) (None, 768) 0 _________________________________________________________________ dense_7 (Dense) (None, 2) 1538 ================================================================= Total params: 109,482,242 Trainable params: 109,482,242 Non-trainable params: 0 _________________________________________________________________
Our model achieved a 91.5% accuracy on test data
%%time
_, test_acc = model.evaluate(data.test_x, data.test_y)
print('test acc',test_acc)
80/80 [==============================] - 12s 155ms/step - loss: 0.3960 - acc: 0.9152 test acc 0.915234386920929 CPU times: user 9.57 s, sys: 4.67 s, total: 14.2 s Wall time: 14.3 s
In this section, we will demonstrate on using BentoML to build production-ready API endpoints and deploy it to AWS SageMaker. The core steps are as follow:
Note: for AWS SageMaker deployment, you will need the following prerequisites: 1) Install and configure the AWS CLI 2) Install Docker
for more information, please click here: https://docs.bentoml.org/en/latest/deployment/aws_sagemaker.html
First, let's create a prediction service file using BentoML. The three main BentoML concepts are:
Note: BentoML supports a variety of major ML frameworks and input data format. For more details, please check available model artifacts here
https://docs.bentoml.org/en/latest/api/artifacts.html and adapters here https://docs.bentoml.org/en/latest/api/adapters.html
For defining the BentoML service environment and trouble-shooting, you would also use auto_pip_dependencies= True
or pass the BentoML generated requirement.txt through @bentoml.env(requirements_tex_file ='./requirements.txt')
%%writefile bentoml_service.py
import tensorflow as tf
import numpy as np
import pandas as pd
import bentoml
from bentoml.frameworks.tensorflow import TensorflowSavedModelArtifact
from bentoml.service.artifacts.common import PickleArtifact
from bentoml.adapters import DataframeInput
CLASSES = ['negative','positive']
max_seq_len = 128
try:
tf.config.set_visible_devices([],'GPU')
except:
pass
#define bentoml service environment
@bentoml.env(pip_packages=['tensorflow','bert','bert-for-tf2','numpy==1.18.1','pandas==1.0.1'])
#define model artifacts
@bentoml.artifacts([TensorflowSavedModelArtifact('model'), PickleArtifact('tokenizer')])
class Service(bentoml.BentoService):
def tokenize(self, inputs: pd.DataFrame):
tokenizer = self.artifacts.tokenizer
if isinstance(inputs, pd.DataFrame):
inputs = inputs.to_numpy()[:, 0].tolist()
else:
inputs = inputs.tolist()
pred_tokens = map(tokenizer.tokenize, inputs)
pred_tokens = map(lambda tok: ["[CLS]"] + tok + ["[SEP]"], pred_tokens)
pred_token_ids = list(map(tokenizer.convert_tokens_to_ids, pred_tokens))
pred_token_ids = map(lambda tids: tids + [0] * (max_seq_len - len(tids)), pred_token_ids)
pred_token_ids = tf.constant(list(pred_token_ids), dtype=tf.int32)
return pred_token_ids
# choose dataframe input adapter
@bentoml.api(input=DataframeInput() batch=True)
def predict(self, inputs):
model = self.artifacts.model
pred_token_ids = self.tokenize(inputs)
res = model(pred_token_ids).numpy().argmax(axis =-1)
return [CLASSES[i] for i in res]
Overwriting bentoml_service.py
The following few lines of codes demonstrate the simplicity and time-saving benefits of using BentoML. Here, we first create a BentoService instance and then use the BentoService pack method to bundle our trained movie review model together. Finally, we use the BentoService save method to save this BentoService bundle, which is now ready for inference. This process eliminates the needs for reproducing the same prediction service for testing and production environment - making it easier for data science teams to deploy their models.
By default, the BentoService bundle is saved under ~/bentoml/repository/
directory. Users could also modify the model repository through BentoML's standalone component YataiService
, for more information, please visit here: https://docs.bentoml.org/en/latest/concepts.html#model-management
from bentoml_service import Service
#create a service instance for the movie review model
bento_svc = Service()
# pack model artifacts
bento_svc.pack('model',model)
bento_svc.pack('tokenizer',tokenizer)
#save the prediction service for model serving
saved_path = bento_svc.save()
[2020-06-25 19:56:15,489] WARNING - BentoML local changes detected - Local BentoML repository including all code changes will be packaged together with saved bundle created, under './bundled_pip_dependencies' directory. For using a modified version of BentoML for production deployment, it is recommended to set the 'core/bentoml_deploy_version' config to a http location or your BentoML for on github, e.g.: 'bentoml_deploy_version = git+https://github.com/{username}/bentoml.git@{branch}' [2020-06-25 19:56:15,927] WARNING - bert package does not exist in the current python session WARNING:tensorflow:From /Users/amy/opt/anaconda3/lib/python3.7/site-packages/tensorflow/python/ops/resource_variable_ops.py:1817: calling BaseResourceVariable.__init__ (from tensorflow.python.ops.resource_variable_ops) with constraint is deprecated and will be removed in a future version. Instructions for updating: If using Keras pass *_constraint arguments to layers. INFO:tensorflow:Assets written to: /private/var/folders/vn/bytl5x0n3vgg1vmg7n6qkqtc0000gn/T/bentoml-temp-35n_doz7/Service/artifacts/model_saved_model/assets [2020-06-25 19:57:01,302] INFO - Detect BentoML installed in development model, copying local BentoML module file to target saved bundle path running sdist running egg_info writing BentoML.egg-info/PKG-INFO writing dependency_links to BentoML.egg-info/dependency_links.txt writing entry points to BentoML.egg-info/entry_points.txt writing requirements to BentoML.egg-info/requires.txt writing top-level names to BentoML.egg-info/top_level.txt reading manifest file 'BentoML.egg-info/SOURCES.txt' reading manifest template 'MANIFEST.in'
warning: no previously-included files matching '*~' found anywhere in distribution warning: no previously-included files matching '*.pyo' found anywhere in distribution warning: no previously-included files matching '.git' found anywhere in distribution warning: no previously-included files matching '.ipynb_checkpoints' found anywhere in distribution warning: no previously-included files matching '__pycache__' found anywhere in distribution warning: no directories found matching 'bentoml/yatai/web/dist' warning: no files found matching 'bentoml/yatai/deployment/sagemaker/sagemaker_nginx.conf' no previously-included directories found matching 'e2e_tests' no previously-included directories found matching 'tests' no previously-included directories found matching 'benchmark'
writing manifest file 'BentoML.egg-info/SOURCES.txt' running check creating BentoML-0.8.1+0.g5b6bd29.dirty creating BentoML-0.8.1+0.g5b6bd29.dirty/BentoML.egg-info creating BentoML-0.8.1+0.g5b6bd29.dirty/bentoml creating BentoML-0.8.1+0.g5b6bd29.dirty/bentoml/adapters creating BentoML-0.8.1+0.g5b6bd29.dirty/bentoml/artifact creating BentoML-0.8.1+0.g5b6bd29.dirty/bentoml/cli creating BentoML-0.8.1+0.g5b6bd29.dirty/bentoml/clipper creating BentoML-0.8.1+0.g5b6bd29.dirty/bentoml/configuration creating BentoML-0.8.1+0.g5b6bd29.dirty/bentoml/configuration/__pycache__ creating BentoML-0.8.1+0.g5b6bd29.dirty/bentoml/handlers creating BentoML-0.8.1+0.g5b6bd29.dirty/bentoml/marshal creating BentoML-0.8.1+0.g5b6bd29.dirty/bentoml/saved_bundle creating BentoML-0.8.1+0.g5b6bd29.dirty/bentoml/server creating BentoML-0.8.1+0.g5b6bd29.dirty/bentoml/server/static creating BentoML-0.8.1+0.g5b6bd29.dirty/bentoml/utils creating BentoML-0.8.1+0.g5b6bd29.dirty/bentoml/yatai creating BentoML-0.8.1+0.g5b6bd29.dirty/bentoml/yatai/client creating BentoML-0.8.1+0.g5b6bd29.dirty/bentoml/yatai/deployment creating BentoML-0.8.1+0.g5b6bd29.dirty/bentoml/yatai/deployment/aws_lambda creating BentoML-0.8.1+0.g5b6bd29.dirty/bentoml/yatai/deployment/sagemaker creating BentoML-0.8.1+0.g5b6bd29.dirty/bentoml/yatai/migrations creating BentoML-0.8.1+0.g5b6bd29.dirty/bentoml/yatai/migrations/__pycache__ creating BentoML-0.8.1+0.g5b6bd29.dirty/bentoml/yatai/migrations/versions creating BentoML-0.8.1+0.g5b6bd29.dirty/bentoml/yatai/migrations/versions/__pycache__ creating BentoML-0.8.1+0.g5b6bd29.dirty/bentoml/yatai/proto creating BentoML-0.8.1+0.g5b6bd29.dirty/bentoml/yatai/repository creating BentoML-0.8.1+0.g5b6bd29.dirty/bentoml/yatai/validator copying files to BentoML-0.8.1+0.g5b6bd29.dirty... copying LICENSE -> BentoML-0.8.1+0.g5b6bd29.dirty copying MANIFEST.in -> BentoML-0.8.1+0.g5b6bd29.dirty copying README.md -> BentoML-0.8.1+0.g5b6bd29.dirty copying pyproject.toml -> BentoML-0.8.1+0.g5b6bd29.dirty copying setup.cfg -> BentoML-0.8.1+0.g5b6bd29.dirty copying setup.py -> BentoML-0.8.1+0.g5b6bd29.dirty copying versioneer.py -> BentoML-0.8.1+0.g5b6bd29.dirty copying BentoML.egg-info/PKG-INFO -> BentoML-0.8.1+0.g5b6bd29.dirty/BentoML.egg-info copying BentoML.egg-info/SOURCES.txt -> BentoML-0.8.1+0.g5b6bd29.dirty/BentoML.egg-info copying BentoML.egg-info/dependency_links.txt -> BentoML-0.8.1+0.g5b6bd29.dirty/BentoML.egg-info copying BentoML.egg-info/entry_points.txt -> BentoML-0.8.1+0.g5b6bd29.dirty/BentoML.egg-info copying BentoML.egg-info/requires.txt -> BentoML-0.8.1+0.g5b6bd29.dirty/BentoML.egg-info copying BentoML.egg-info/top_level.txt -> BentoML-0.8.1+0.g5b6bd29.dirty/BentoML.egg-info copying bentoml/__init__.py -> BentoML-0.8.1+0.g5b6bd29.dirty/bentoml copying bentoml/_version.py -> BentoML-0.8.1+0.g5b6bd29.dirty/bentoml copying bentoml/exceptions.py -> BentoML-0.8.1+0.g5b6bd29.dirty/bentoml copying bentoml/service.py -> BentoML-0.8.1+0.g5b6bd29.dirty/bentoml copying bentoml/service_env.py -> BentoML-0.8.1+0.g5b6bd29.dirty/bentoml copying bentoml/adapters/__init__.py -> BentoML-0.8.1+0.g5b6bd29.dirty/bentoml/adapters copying bentoml/adapters/base_input.py -> BentoML-0.8.1+0.g5b6bd29.dirty/bentoml/adapters copying bentoml/adapters/base_output.py -> BentoML-0.8.1+0.g5b6bd29.dirty/bentoml/adapters copying bentoml/adapters/clipper_input.py -> BentoML-0.8.1+0.g5b6bd29.dirty/bentoml/adapters copying bentoml/adapters/dataframe_input.py -> BentoML-0.8.1+0.g5b6bd29.dirty/bentoml/adapters copying bentoml/adapters/dataframe_output.py -> BentoML-0.8.1+0.g5b6bd29.dirty/bentoml/adapters copying bentoml/adapters/default_output.py -> BentoML-0.8.1+0.g5b6bd29.dirty/bentoml/adapters copying bentoml/adapters/fastai_image_input.py -> BentoML-0.8.1+0.g5b6bd29.dirty/bentoml/adapters copying bentoml/adapters/image_input.py -> BentoML-0.8.1+0.g5b6bd29.dirty/bentoml/adapters copying bentoml/adapters/json_input.py -> BentoML-0.8.1+0.g5b6bd29.dirty/bentoml/adapters copying bentoml/adapters/json_output.py -> BentoML-0.8.1+0.g5b6bd29.dirty/bentoml/adapters copying bentoml/adapters/legacy_image_input.py -> BentoML-0.8.1+0.g5b6bd29.dirty/bentoml/adapters copying bentoml/adapters/pytorch_tensor_input.py -> BentoML-0.8.1+0.g5b6bd29.dirty/bentoml/adapters copying bentoml/adapters/tensorflow_tensor_input.py -> BentoML-0.8.1+0.g5b6bd29.dirty/bentoml/adapters copying bentoml/adapters/tensorflow_tensor_output.py -> BentoML-0.8.1+0.g5b6bd29.dirty/bentoml/adapters copying bentoml/adapters/utils.py -> BentoML-0.8.1+0.g5b6bd29.dirty/bentoml/adapters copying bentoml/artifact/__init__.py -> BentoML-0.8.1+0.g5b6bd29.dirty/bentoml/artifact copying bentoml/artifact/artifact.py -> BentoML-0.8.1+0.g5b6bd29.dirty/bentoml/artifact copying bentoml/artifact/fastai2_model_artifact.py -> BentoML-0.8.1+0.g5b6bd29.dirty/bentoml/artifact copying bentoml/artifact/fastai_model_artifact.py -> BentoML-0.8.1+0.g5b6bd29.dirty/bentoml/artifact copying bentoml/artifact/fasttext_model_artifact.py -> BentoML-0.8.1+0.g5b6bd29.dirty/bentoml/artifact copying bentoml/artifact/h2o_model_artifact.py -> BentoML-0.8.1+0.g5b6bd29.dirty/bentoml/artifact copying bentoml/artifact/json_artifact.py -> BentoML-0.8.1+0.g5b6bd29.dirty/bentoml/artifact copying bentoml/artifact/keras_model_artifact.py -> BentoML-0.8.1+0.g5b6bd29.dirty/bentoml/artifact copying bentoml/artifact/lightgbm_model_artifact.py -> BentoML-0.8.1+0.g5b6bd29.dirty/bentoml/artifact copying bentoml/artifact/onnx_model_artifact.py -> BentoML-0.8.1+0.g5b6bd29.dirty/bentoml/artifact copying bentoml/artifact/pickle_artifact.py -> BentoML-0.8.1+0.g5b6bd29.dirty/bentoml/artifact copying bentoml/artifact/pytorch_model_artifact.py -> BentoML-0.8.1+0.g5b6bd29.dirty/bentoml/artifact copying bentoml/artifact/sklearn_model_artifact.py -> BentoML-0.8.1+0.g5b6bd29.dirty/bentoml/artifact copying bentoml/artifact/spacy_model_artifact.py -> BentoML-0.8.1+0.g5b6bd29.dirty/bentoml/artifact copying bentoml/artifact/text_file_artifact.py -> BentoML-0.8.1+0.g5b6bd29.dirty/bentoml/artifact copying bentoml/artifact/tf_savedmodel_artifact.py -> BentoML-0.8.1+0.g5b6bd29.dirty/bentoml/artifact copying bentoml/artifact/xgboost_model_artifact.py -> BentoML-0.8.1+0.g5b6bd29.dirty/bentoml/artifact copying bentoml/cli/__init__.py -> BentoML-0.8.1+0.g5b6bd29.dirty/bentoml/cli copying bentoml/cli/aws_lambda.py -> BentoML-0.8.1+0.g5b6bd29.dirty/bentoml/cli copying bentoml/cli/aws_sagemaker.py -> BentoML-0.8.1+0.g5b6bd29.dirty/bentoml/cli copying bentoml/cli/bento.py -> BentoML-0.8.1+0.g5b6bd29.dirty/bentoml/cli copying bentoml/cli/click_utils.py -> BentoML-0.8.1+0.g5b6bd29.dirty/bentoml/cli copying bentoml/cli/config.py -> BentoML-0.8.1+0.g5b6bd29.dirty/bentoml/cli copying bentoml/cli/deployment.py -> BentoML-0.8.1+0.g5b6bd29.dirty/bentoml/cli copying bentoml/cli/utils.py -> BentoML-0.8.1+0.g5b6bd29.dirty/bentoml/cli copying bentoml/cli/yatai_service.py -> BentoML-0.8.1+0.g5b6bd29.dirty/bentoml/cli copying bentoml/clipper/__init__.py -> BentoML-0.8.1+0.g5b6bd29.dirty/bentoml/clipper copying bentoml/configuration/__init__.py -> BentoML-0.8.1+0.g5b6bd29.dirty/bentoml/configuration copying bentoml/configuration/configparser.py -> BentoML-0.8.1+0.g5b6bd29.dirty/bentoml/configuration copying bentoml/configuration/default_bentoml.cfg -> BentoML-0.8.1+0.g5b6bd29.dirty/bentoml/configuration copying bentoml/configuration/__pycache__/__init__.cpython-37.pyc -> BentoML-0.8.1+0.g5b6bd29.dirty/bentoml/configuration/__pycache__ copying bentoml/configuration/__pycache__/configparser.cpython-37.pyc -> BentoML-0.8.1+0.g5b6bd29.dirty/bentoml/configuration/__pycache__ copying bentoml/handlers/__init__.py -> BentoML-0.8.1+0.g5b6bd29.dirty/bentoml/handlers copying bentoml/marshal/__init__.py -> BentoML-0.8.1+0.g5b6bd29.dirty/bentoml/marshal copying bentoml/marshal/dispatcher.py -> BentoML-0.8.1+0.g5b6bd29.dirty/bentoml/marshal copying bentoml/marshal/marshal.py -> BentoML-0.8.1+0.g5b6bd29.dirty/bentoml/marshal copying bentoml/marshal/utils.py -> BentoML-0.8.1+0.g5b6bd29.dirty/bentoml/marshal copying bentoml/saved_bundle/__init__.py -> BentoML-0.8.1+0.g5b6bd29.dirty/bentoml/saved_bundle copying bentoml/saved_bundle/bentoml-init.sh -> BentoML-0.8.1+0.g5b6bd29.dirty/bentoml/saved_bundle copying bentoml/saved_bundle/bundler.py -> BentoML-0.8.1+0.g5b6bd29.dirty/bentoml/saved_bundle copying bentoml/saved_bundle/config.py -> BentoML-0.8.1+0.g5b6bd29.dirty/bentoml/saved_bundle copying bentoml/saved_bundle/docker-entrypoint.sh -> BentoML-0.8.1+0.g5b6bd29.dirty/bentoml/saved_bundle copying bentoml/saved_bundle/loader.py -> BentoML-0.8.1+0.g5b6bd29.dirty/bentoml/saved_bundle copying bentoml/saved_bundle/pip_pkg.py -> BentoML-0.8.1+0.g5b6bd29.dirty/bentoml/saved_bundle copying bentoml/saved_bundle/py_module_utils.py -> BentoML-0.8.1+0.g5b6bd29.dirty/bentoml/saved_bundle copying bentoml/saved_bundle/templates.py -> BentoML-0.8.1+0.g5b6bd29.dirty/bentoml/saved_bundle copying bentoml/server/__init__.py -> BentoML-0.8.1+0.g5b6bd29.dirty/bentoml/server copying bentoml/server/bento_api_server.py -> BentoML-0.8.1+0.g5b6bd29.dirty/bentoml/server copying bentoml/server/gunicorn_config.py -> BentoML-0.8.1+0.g5b6bd29.dirty/bentoml/server copying bentoml/server/gunicorn_server.py -> BentoML-0.8.1+0.g5b6bd29.dirty/bentoml/server copying bentoml/server/instruments.py -> BentoML-0.8.1+0.g5b6bd29.dirty/bentoml/server copying bentoml/server/marshal_server.py -> BentoML-0.8.1+0.g5b6bd29.dirty/bentoml/server copying bentoml/server/open_api.py -> BentoML-0.8.1+0.g5b6bd29.dirty/bentoml/server copying bentoml/server/trace.py -> BentoML-0.8.1+0.g5b6bd29.dirty/bentoml/server copying bentoml/server/utils.py -> BentoML-0.8.1+0.g5b6bd29.dirty/bentoml/server copying bentoml/server/static/swagger-ui-bundle.js -> BentoML-0.8.1+0.g5b6bd29.dirty/bentoml/server/static copying bentoml/server/static/swagger-ui.css -> BentoML-0.8.1+0.g5b6bd29.dirty/bentoml/server/static copying bentoml/utils/__init__.py -> BentoML-0.8.1+0.g5b6bd29.dirty/bentoml/utils copying bentoml/utils/alg.py -> BentoML-0.8.1+0.g5b6bd29.dirty/bentoml/utils copying bentoml/utils/benchmark.py -> BentoML-0.8.1+0.g5b6bd29.dirty/bentoml/utils copying bentoml/utils/cloudpickle.py -> BentoML-0.8.1+0.g5b6bd29.dirty/bentoml/utils copying bentoml/utils/hybridmethod.py -> BentoML-0.8.1+0.g5b6bd29.dirty/bentoml/utils copying bentoml/utils/log.py -> BentoML-0.8.1+0.g5b6bd29.dirty/bentoml/utils copying bentoml/utils/s3.py -> BentoML-0.8.1+0.g5b6bd29.dirty/bentoml/utils copying bentoml/utils/tempdir.py -> BentoML-0.8.1+0.g5b6bd29.dirty/bentoml/utils copying bentoml/utils/usage_stats.py -> BentoML-0.8.1+0.g5b6bd29.dirty/bentoml/utils copying bentoml/yatai/__init__.py -> BentoML-0.8.1+0.g5b6bd29.dirty/bentoml/yatai copying bentoml/yatai/alembic.ini -> BentoML-0.8.1+0.g5b6bd29.dirty/bentoml/yatai copying bentoml/yatai/db.py -> BentoML-0.8.1+0.g5b6bd29.dirty/bentoml/yatai copying bentoml/yatai/deployment_utils.py -> BentoML-0.8.1+0.g5b6bd29.dirty/bentoml/yatai copying bentoml/yatai/status.py -> BentoML-0.8.1+0.g5b6bd29.dirty/bentoml/yatai copying bentoml/yatai/utils.py -> BentoML-0.8.1+0.g5b6bd29.dirty/bentoml/yatai copying bentoml/yatai/yatai_service.py -> BentoML-0.8.1+0.g5b6bd29.dirty/bentoml/yatai copying bentoml/yatai/yatai_service_impl.py -> BentoML-0.8.1+0.g5b6bd29.dirty/bentoml/yatai copying bentoml/yatai/client/__init__.py -> BentoML-0.8.1+0.g5b6bd29.dirty/bentoml/yatai/client copying bentoml/yatai/client/bento_repository_api.py -> BentoML-0.8.1+0.g5b6bd29.dirty/bentoml/yatai/client copying bentoml/yatai/client/deployment_api.py -> BentoML-0.8.1+0.g5b6bd29.dirty/bentoml/yatai/client copying bentoml/yatai/deployment/__init__.py -> BentoML-0.8.1+0.g5b6bd29.dirty/bentoml/yatai/deployment copying bentoml/yatai/deployment/operator.py -> BentoML-0.8.1+0.g5b6bd29.dirty/bentoml/yatai/deployment copying bentoml/yatai/deployment/store.py -> BentoML-0.8.1+0.g5b6bd29.dirty/bentoml/yatai/deployment copying bentoml/yatai/deployment/utils.py -> BentoML-0.8.1+0.g5b6bd29.dirty/bentoml/yatai/deployment copying bentoml/yatai/deployment/aws_lambda/__init__.py -> BentoML-0.8.1+0.g5b6bd29.dirty/bentoml/yatai/deployment/aws_lambda copying bentoml/yatai/deployment/aws_lambda/download_extra_resources.py -> BentoML-0.8.1+0.g5b6bd29.dirty/bentoml/yatai/deployment/aws_lambda copying bentoml/yatai/deployment/aws_lambda/lambda_app.py -> BentoML-0.8.1+0.g5b6bd29.dirty/bentoml/yatai/deployment/aws_lambda copying bentoml/yatai/deployment/aws_lambda/operator.py -> BentoML-0.8.1+0.g5b6bd29.dirty/bentoml/yatai/deployment/aws_lambda copying bentoml/yatai/deployment/aws_lambda/utils.py -> BentoML-0.8.1+0.g5b6bd29.dirty/bentoml/yatai/deployment/aws_lambda copying bentoml/yatai/deployment/sagemaker/__init__.py -> BentoML-0.8.1+0.g5b6bd29.dirty/bentoml/yatai/deployment/sagemaker copying bentoml/yatai/deployment/sagemaker/model_server.py -> BentoML-0.8.1+0.g5b6bd29.dirty/bentoml/yatai/deployment/sagemaker copying bentoml/yatai/deployment/sagemaker/operator.py -> BentoML-0.8.1+0.g5b6bd29.dirty/bentoml/yatai/deployment/sagemaker copying bentoml/yatai/deployment/sagemaker/wsgi.py -> BentoML-0.8.1+0.g5b6bd29.dirty/bentoml/yatai/deployment/sagemaker copying bentoml/yatai/migrations/README -> BentoML-0.8.1+0.g5b6bd29.dirty/bentoml/yatai/migrations copying bentoml/yatai/migrations/__init__.py -> BentoML-0.8.1+0.g5b6bd29.dirty/bentoml/yatai/migrations copying bentoml/yatai/migrations/env.py -> BentoML-0.8.1+0.g5b6bd29.dirty/bentoml/yatai/migrations copying bentoml/yatai/migrations/script.py.mako -> BentoML-0.8.1+0.g5b6bd29.dirty/bentoml/yatai/migrations copying bentoml/yatai/migrations/__pycache__/env.cpython-37.pyc -> BentoML-0.8.1+0.g5b6bd29.dirty/bentoml/yatai/migrations/__pycache__ copying bentoml/yatai/migrations/versions/__init__.py -> BentoML-0.8.1+0.g5b6bd29.dirty/bentoml/yatai/migrations/versions copying bentoml/yatai/migrations/versions/a6b00ae45279_add_last_updated_at_for_deployments.py -> BentoML-0.8.1+0.g5b6bd29.dirty/bentoml/yatai/migrations/versions copying bentoml/yatai/migrations/versions/__pycache__/a6b00ae45279_add_last_updated_at_for_deployments.cpython-37.pyc -> BentoML-0.8.1+0.g5b6bd29.dirty/bentoml/yatai/migrations/versions/__pycache__ copying bentoml/yatai/proto/__init__.py -> BentoML-0.8.1+0.g5b6bd29.dirty/bentoml/yatai/proto copying bentoml/yatai/proto/deployment_pb2.py -> BentoML-0.8.1+0.g5b6bd29.dirty/bentoml/yatai/proto copying bentoml/yatai/proto/repository_pb2.py -> BentoML-0.8.1+0.g5b6bd29.dirty/bentoml/yatai/proto copying bentoml/yatai/proto/status_pb2.py -> BentoML-0.8.1+0.g5b6bd29.dirty/bentoml/yatai/proto copying bentoml/yatai/proto/yatai_service_pb2.py -> BentoML-0.8.1+0.g5b6bd29.dirty/bentoml/yatai/proto copying bentoml/yatai/proto/yatai_service_pb2_grpc.py -> BentoML-0.8.1+0.g5b6bd29.dirty/bentoml/yatai/proto copying bentoml/yatai/repository/__init__.py -> BentoML-0.8.1+0.g5b6bd29.dirty/bentoml/yatai/repository copying bentoml/yatai/repository/base_repository.py -> BentoML-0.8.1+0.g5b6bd29.dirty/bentoml/yatai/repository copying bentoml/yatai/repository/local_repository.py -> BentoML-0.8.1+0.g5b6bd29.dirty/bentoml/yatai/repository copying bentoml/yatai/repository/metadata_store.py -> BentoML-0.8.1+0.g5b6bd29.dirty/bentoml/yatai/repository copying bentoml/yatai/repository/repository.py -> BentoML-0.8.1+0.g5b6bd29.dirty/bentoml/yatai/repository copying bentoml/yatai/repository/s3_repository.py -> BentoML-0.8.1+0.g5b6bd29.dirty/bentoml/yatai/repository copying bentoml/yatai/validator/__init__.py -> BentoML-0.8.1+0.g5b6bd29.dirty/bentoml/yatai/validator copying bentoml/yatai/validator/deployment_pb_validator.py -> BentoML-0.8.1+0.g5b6bd29.dirty/bentoml/yatai/validator Writing BentoML-0.8.1+0.g5b6bd29.dirty/setup.cfg UPDATING BentoML-0.8.1+0.g5b6bd29.dirty/bentoml/_version.py set BentoML-0.8.1+0.g5b6bd29.dirty/bentoml/_version.py to '0.8.1+0.g5b6bd29.dirty' Creating tar archive removing 'BentoML-0.8.1+0.g5b6bd29.dirty' (and everything under it) [2020-06-25 19:57:03,142] WARNING - Saved BentoService bundle version mismatch: loading BentoService bundle create with BentoML version 0.8.1, but loading from BentoML version 0.8.1+0.g5b6bd29.dirty [2020-06-25 19:57:03,958] INFO - BentoService bundle 'Service:20200625195616_62D0DB' saved to: /Users/amy/bentoml/repository/Service/20200625195616_62D0DB
As mentioned earlier, BentoML also provides ways to change the model repository - allowing data science teams to share the BentoService bundle easily for better collaborations. One way is by uploading it to the cloud services such as AWS S3. Using the same scripts as above and passing the S3 bucket URL into .save()
, it will deploy the BentoService bundle directly into the S3 movie-review-dataset bucket we created earlier.
from bentoml_service import Service
#create a service instance for the movie review model
bento_svc = Service()
# pack model artifacts
bento_svc.pack('model',model)
bento_svc.pack('tokenizer',tokenizer)
#save the prediction service to aws S3
saved_path = bento_svc.save(''s3://movie-review-dataset/'')
Using the BentoML CLI, we can see a list of BentoService generated here
!bentoml list
BENTO_SERVICE AGE APIS ARTIFACTS Service:20200625195616_62D0DB 29.09 seconds predict<DataframeInput:DefaultOutput> model<TensorflowSavedModelArtifact>, tokenizer<PickleArtifact> Service:20200622153915_614FE2 3 days and 4 hours predict<DataframeInput:DefaultOutput> model<TensorflowSavedModelArtifact>, tokenizer<PickleArtifact> Service:20200622113634_A6EFDD 3 days and 8 hours predict<DataframeInput:DefaultOutput> model<TensorflowSavedModelArtifact>, tokenizer<PickleArtifact> IrisClassifier:20200615204826_CAA9DD 1 week and 2 days predict<DataframeInput:DefaultOutput> model<SklearnModelArtifact> IrisClassifier:20200615194906_60F775 1 week and 3 days predict<DataframeInput:DefaultOutput> model<SklearnModelArtifact>
Before deploying the model to AWS SageMaker, we could test it locally first using the BentoML CLI. By using bentoml serve
, it provides a near real-time prediction via API endpoints.
!bentoml serve Service:20200702134432_033DAB
[2020-06-26 13:43:43,372] WARNING - BentoML local changes detected - Local BentoML repository including all code changes will be packaged together with saved bundle created, under './bundled_pip_dependencies' directory. For using a modified version of BentoML for production deployment, it is recommended to set the 'core/bentoml_deploy_version' config to a http location or your BentoML for on github, e.g.: 'bentoml_deploy_version = git+https://github.com/{username}/bentoml.git@{branch}' [2020-06-26 13:43:43,399] WARNING - Saved BentoService bundle version mismatch: loading BentoService bundle create with BentoML version 0.8.1, but loading from BentoML version 0.8.1+0.g5b6bd29.dirty [2020-06-26 13:43:48,611] WARNING - bert package does not exist in the current python session 2020-06-26 13:43:49.603289: I tensorflow/core/platform/cpu_feature_guard.cc:143] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 AVX512F FMA 2020-06-26 13:43:49.634653: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7f8e1d8f9410 initialized for platform Host (this does not guarantee that XLA will be used). Devices: 2020-06-26 13:43:49.634673: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version * Serving Flask app "Service" (lazy loading) * Environment: production WARNING: This is a development server. Do not use it in a production deployment. Use a production WSGI server instead. * Debug mode: off * Running on http://127.0.0.1:5000/ (Press CTRL+C to quit) 127.0.0.1 - - [26/Jun/2020 13:44:08] "GET / HTTP/1.1" 200 - 127.0.0.1 - - [26/Jun/2020 13:44:09] "GET /static/swagger-ui.css HTTP/1.1" 200 - 127.0.0.1 - - [26/Jun/2020 13:44:09] "GET /static/swagger-ui-bundle.js HTTP/1.1" 304 - 127.0.0.1 - - [26/Jun/2020 13:44:09] "GET /docs.json HTTP/1.1" 200 - 127.0.0.1 - - [26/Jun/2020 13:44:39] "POST /predict HTTP/1.1" 200 -
Alternatively, we could also use bentoml run
for local testing. BentoML provides many other model serving methods, such as: adaptive micro-batching, edge serving,and programmatic access. Please visit here: https://docs.bentoml.org/en/latest/concepts.html#model-serving
!bentoml run Service:20200702134432_033DAB predict --input '["the acting was a bit lacking."]'
[2020-06-25 19:59:59,724] WARNING - BentoML local changes detected - Local BentoML repository including all code changes will be packaged together with saved bundle created, under './bundled_pip_dependencies' directory. For using a modified version of BentoML for production deployment, it is recommended to set the 'core/bentoml_deploy_version' config to a http location or your BentoML for on github, e.g.: 'bentoml_deploy_version = git+https://github.com/{username}/bentoml.git@{branch}' [2020-06-25 19:59:59,747] WARNING - Saved BentoService bundle version mismatch: loading BentoService bundle create with BentoML version 0.8.1, but loading from BentoML version 0.8.1+0.g5b6bd29.dirty [2020-06-25 20:00:03,274] WARNING - bert package does not exist in the current python session 2020-06-25 20:00:04.447236: I tensorflow/core/platform/cpu_feature_guard.cc:143] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 AVX512F FMA 2020-06-25 20:00:04.460749: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7faeed381050 initialized for platform Host (this does not guarantee that XLA will be used). Devices: 2020-06-25 20:00:04.460780: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version ['negative']
Finally, we are ready to deploy our BentoML packaged model to AWS SageMaker. We need to pass the deployment name, the BentoService name and the API name. Depending on the size of the BentoService generated, the deployment for this tutorial took about 30mins.
!bentoml sagemaker deploy sagemaker-moviereview-deployment -b Service:20200702134432_033DAB --api-name predict
Deploying Sagemaker deployment /[2020-06-25 20:16:14,382] INFO - Step 1/9 : FROM bentoml/model-server:0.8.1 [2020-06-25 20:16:14,383] INFO - [2020-06-25 20:16:14,383] INFO - ---> e326316eaf10 [2020-06-25 20:16:14,383] INFO - Step 2/9 : ENV PORT 8080 [2020-06-25 20:16:14,384] INFO - [2020-06-25 20:16:14,384] INFO - ---> Using cache [2020-06-25 20:16:14,384] INFO - ---> 1330be3b89a8 [2020-06-25 20:16:14,385] INFO - Step 3/9 : EXPOSE $PORT [2020-06-25 20:16:14,385] INFO - [2020-06-25 20:16:14,385] INFO - ---> Using cache [2020-06-25 20:16:14,385] INFO - ---> 77d1dc3ea8a0 [2020-06-25 20:16:14,385] INFO - Step 4/9 : RUN apt-get update --fix-missing && apt-get install -y nginx && apt-get clean [2020-06-25 20:16:14,385] INFO - [2020-06-25 20:16:14,385] INFO - ---> Using cache [2020-06-25 20:16:14,385] INFO - ---> d0803d90cbb0 [2020-06-25 20:16:14,385] INFO - Step 5/9 : RUN pip install gevent==1.4 [2020-06-25 20:16:14,385] INFO - [2020-06-25 20:16:14,385] INFO - ---> Using cache [2020-06-25 20:16:14,386] INFO - ---> 11ba6eb13310 [2020-06-25 20:16:14,386] INFO - Step 6/9 : COPY . /bento [2020-06-25 20:16:14,386] INFO - |[2020-06-25 20:16:19,267] INFO - ---> eed150960591 [2020-06-25 20:16:19,268] INFO - Step 7/9 : WORKDIR /bento [2020-06-25 20:16:19,268] INFO - [2020-06-25 20:16:19,334] INFO - ---> Running in 7da011206c5a \[2020-06-25 20:16:19,446] INFO - ---> e8a953a94d41 [2020-06-25 20:16:19,447] INFO - Step 8/9 : RUN if [ -f /bento/bentoml-init.sh ]; then bash -c /bento/bentoml-init.sh; fi [2020-06-25 20:16:19,447] INFO - -[2020-06-25 20:16:19,494] INFO - ---> Running in 79635a5e5e5b /[2020-06-25 20:16:19,998] INFO - +++ dirname /bento/bentoml-init.sh [2020-06-25 20:16:20,001] INFO - ++ cd /bento ++ pwd -P [2020-06-25 20:16:20,001] INFO - + SAVED_BUNDLE_PATH=/bento + cd /bento [2020-06-25 20:16:20,001] INFO - + '[' -f ./setup.sh ']' [2020-06-25 20:16:20,002] INFO - + command -v conda [2020-06-25 20:16:20,002] INFO - + conda env update -n base -f ./environment.yml /[2020-06-25 20:16:22,426] INFO - Collecting package metadata (repodata.json): ...working... \[2020-06-25 20:16:25,861] INFO - done Solving environment: ...working... /[2020-06-25 20:16:26,533] INFO - done [2020-06-25 20:16:26,564] INFO - Downloading and Extracting Packages openssl-1.1.1g | 2.5 MB | | 0% openssl-1.1.1g | 2.5 MB | | 1% openssl-1.1.1g | 2.5 MB | ####9 | 50% openssl-1.1.1g | 2.5 MB | ########## | 100% [2020-06-25 20:16:26,959] INFO - certifi-2020.6.20 | 156 KB | | 0% certifi-2020.6.20 | 156 KB | ########## | 100% [2020-06-25 20:16:27,014] INFO - pip-20.1.1 | 1.7 MB | | 0% pip-20.1.1 | 1.7 MB | ######5 | 66% pip-20.1.1 | 1.7 MB | ########## | 100% [2020-06-25 20:16:27,254] INFO - Preparing transaction: ...working... /[2020-06-25 20:16:27,291] INFO - done [2020-06-25 20:16:27,292] INFO - Verifying transaction: ...working... |[2020-06-25 20:16:27,440] INFO - done [2020-06-25 20:16:27,440] INFO - Executing transaction: ...working... /[2020-06-25 20:16:28,950] INFO - done |[2020-06-25 20:16:29,087] INFO - # # To activate this environment, use # # $ conda activate base # # To deactivate an active environment, use # # $ conda deactivate [2020-06-25 20:16:29,087] INFO - ==> WARNING: A newer version of conda exists. <== current version: 4.8.2 latest version: 4.8.3 Please update conda by running $ conda update -n base -c defaults conda |[2020-06-25 20:16:29,406] INFO - + pip install -r ./requirements.txt /[2020-06-25 20:16:30,183] INFO - Collecting tensorflow \[2020-06-25 20:16:30,321] INFO - Downloading tensorflow-2.2.0-cp37-cp37m-manylinux2010_x86_64.whl (516.2 MB) /[2020-06-25 20:17:05,813] INFO - Collecting pandas==1.0.1 |[2020-06-25 20:17:05,884] INFO - Downloading pandas-1.0.1-cp37-cp37m-manylinux1_x86_64.whl (10.1 MB) |[2020-06-25 20:17:07,137] INFO - Requirement already satisfied: bentoml==0.8.1 in /opt/conda/lib/python3.7/site-packages (from -r ./requirements.txt (line 3)) (0.8.1) -[2020-06-25 20:17:07,332] INFO - Collecting bert [2020-06-25 20:17:07,356] INFO - Downloading bert-2.2.0.tar.gz (3.5 kB) /[2020-06-25 20:17:07,847] INFO - Collecting bert-for-tf2 [2020-06-25 20:17:07,872] INFO - Downloading bert-for-tf2-0.14.4.tar.gz (40 kB) -[2020-06-25 20:17:08,516] INFO - Collecting numpy==1.18.1 -[2020-06-25 20:17:08,935] INFO - Downloading numpy-1.18.1-cp37-cp37m-manylinux1_x86_64.whl (20.1 MB) /[2020-06-25 20:17:10,325] INFO - Requirement already satisfied: six>=1.12.0 in /opt/conda/lib/python3.7/site-packages (from tensorflow->-r ./requirements.txt (line 1)) (1.14.0) \[2020-06-25 20:17:10,459] INFO - Collecting keras-preprocessing>=1.1.0 [2020-06-25 20:17:10,485] INFO - Downloading Keras_Preprocessing-1.1.2-py2.py3-none-any.whl (42 kB) /[2020-06-25 20:17:10,752] INFO - Collecting scipy==1.4.1; python_version >= "3" |[2020-06-25 20:17:10,793] INFO - Downloading scipy-1.4.1-cp37-cp37m-manylinux1_x86_64.whl (26.1 MB) /[2020-06-25 20:17:12,781] INFO - Collecting gast==0.3.3 |[2020-06-25 20:17:12,804] INFO - Downloading gast-0.3.3-py2.py3-none-any.whl (9.7 kB) [2020-06-25 20:17:12,832] INFO - Requirement already satisfied: grpcio>=1.8.6 in /opt/conda/lib/python3.7/site-packages (from tensorflow->-r ./requirements.txt (line 1)) (1.27.2) \[2020-06-25 20:17:12,900] INFO - Collecting tensorboard<2.3.0,>=2.2.0 [2020-06-25 20:17:12,932] INFO - Downloading tensorboard-2.2.2-py3-none-any.whl (3.0 MB) |[2020-06-25 20:17:13,248] INFO - Collecting absl-py>=0.7.0 [2020-06-25 20:17:13,276] INFO - Downloading absl-py-0.9.0.tar.gz (104 kB) /[2020-06-25 20:17:13,531] INFO - Requirement already satisfied: protobuf>=3.8.0 in /opt/conda/lib/python3.7/site-packages (from tensorflow->-r ./requirements.txt (line 1)) (3.12.2) [2020-06-25 20:17:13,592] INFO - Collecting wrapt>=1.11.1 |[2020-06-25 20:17:13,613] INFO - Downloading wrapt-1.12.1.tar.gz (27 kB) /[2020-06-25 20:17:13,907] INFO - Collecting google-pasta>=0.1.8 [2020-06-25 20:17:13,930] INFO - Downloading google_pasta-0.2.0-py3-none-any.whl (57 kB) |[2020-06-25 20:17:14,012] INFO - Collecting termcolor>=1.1.0 [2020-06-25 20:17:14,035] INFO - Downloading termcolor-1.1.0.tar.gz (3.9 kB) /[2020-06-25 20:17:14,319] INFO - Collecting astunparse==1.6.3 [2020-06-25 20:17:14,350] INFO - Downloading astunparse-1.6.3-py2.py3-none-any.whl (12 kB) |[2020-06-25 20:17:14,478] INFO - Collecting h5py<2.11.0,>=2.10.0 \[2020-06-25 20:17:14,511] INFO - Downloading h5py-2.10.0-cp37-cp37m-manylinux1_x86_64.whl (2.9 MB) /[2020-06-25 20:17:14,771] INFO - Collecting tensorflow-estimator<2.3.0,>=2.2.0 |[2020-06-25 20:17:14,827] INFO - Downloading tensorflow_estimator-2.2.0-py2.py3-none-any.whl (454 kB) [2020-06-25 20:17:14,898] INFO - Requirement already satisfied: wheel>=0.26; python_version >= "3" in /opt/conda/lib/python3.7/site-packages (from tensorflow->-r ./requirements.txt (line 1)) (0.34.2) \[2020-06-25 20:17:14,962] INFO - Collecting opt-einsum>=2.3.2 [2020-06-25 20:17:14,982] INFO - Downloading opt_einsum-3.2.1-py3-none-any.whl (63 kB) -[2020-06-25 20:17:15,030] INFO - Requirement already satisfied: python-dateutil>=2.6.1 in /opt/conda/lib/python3.7/site-packages (from pandas==1.0.1->-r ./requirements.txt (line 2)) (2.8.0) /[2020-06-25 20:17:15,196] INFO - Collecting pytz>=2017.2 |[2020-06-25 20:17:15,220] INFO - Downloading pytz-2020.1-py2.py3-none-any.whl (510 kB) \[2020-06-25 20:17:15,355] INFO - Requirement already satisfied: psutil in /opt/conda/lib/python3.7/site-packages (from bentoml==0.8.1->-r ./requirements.txt (line 3)) (5.7.0) [2020-06-25 20:17:15,358] INFO - Requirement already satisfied: configparser in /opt/conda/lib/python3.7/site-packages (from bentoml==0.8.1->-r ./requirements.txt (line 3)) (5.0.0) [2020-06-25 20:17:15,367] INFO - Requirement already satisfied: py-zipkin in /opt/conda/lib/python3.7/site-packages (from bentoml==0.8.1->-r ./requirements.txt (line 3)) (0.20.0) [2020-06-25 20:17:15,374] INFO - Requirement already satisfied: aiohttp in /opt/conda/lib/python3.7/site-packages (from bentoml==0.8.1->-r ./requirements.txt (line 3)) (3.6.2) [2020-06-25 20:17:15,388] INFO - Requirement already satisfied: gunicorn in /opt/conda/lib/python3.7/site-packages (from bentoml==0.8.1->-r ./requirements.txt (line 3)) (20.0.4) [2020-06-25 20:17:15,397] INFO - Requirement already satisfied: flask in /opt/conda/lib/python3.7/site-packages (from bentoml==0.8.1->-r ./requirements.txt (line 3)) (1.1.2) [2020-06-25 20:17:15,410] INFO - Requirement already satisfied: sqlalchemy>=1.3.0 in /opt/conda/lib/python3.7/site-packages (from bentoml==0.8.1->-r ./requirements.txt (line 3)) (1.3.17) [2020-06-25 20:17:15,423] INFO - Requirement already satisfied: sqlalchemy-utils in /opt/conda/lib/python3.7/site-packages (from bentoml==0.8.1->-r ./requirements.txt (line 3)) (0.36.6) -[2020-06-25 20:17:15,484] INFO - Requirement already satisfied: humanfriendly in /opt/conda/lib/python3.7/site-packages (from bentoml==0.8.1->-r ./requirements.txt (line 3)) (8.2) [2020-06-25 20:17:15,489] INFO - Requirement already satisfied: packaging in /opt/conda/lib/python3.7/site-packages (from bentoml==0.8.1->-r ./requirements.txt (line 3)) (20.4) [2020-06-25 20:17:15,493] INFO - Requirement already satisfied: click>=7.0 in /opt/conda/lib/python3.7/site-packages (from bentoml==0.8.1->-r ./requirements.txt (line 3)) (7.1.2) [2020-06-25 20:17:15,496] INFO - Requirement already satisfied: docker in /opt/conda/lib/python3.7/site-packages (from bentoml==0.8.1->-r ./requirements.txt (line 3)) (4.2.1) [2020-06-25 20:17:15,509] INFO - Requirement already satisfied: alembic in /opt/conda/lib/python3.7/site-packages (from bentoml==0.8.1->-r ./requirements.txt (line 3)) (1.4.2) [2020-06-25 20:17:15,515] INFO - Requirement already satisfied: prometheus-client in /opt/conda/lib/python3.7/site-packages (from bentoml==0.8.1->-r ./requirements.txt (line 3)) (0.8.0) [2020-06-25 20:17:15,519] INFO - Requirement already satisfied: ruamel.yaml>=0.15.0 in /opt/conda/lib/python3.7/site-packages (from bentoml==0.8.1->-r ./requirements.txt (line 3)) (0.16.10) /[2020-06-25 20:17:15,529] INFO - Requirement already satisfied: tabulate in /opt/conda/lib/python3.7/site-packages (from bentoml==0.8.1->-r ./requirements.txt (line 3)) (0.8.7) [2020-06-25 20:17:15,533] INFO - Requirement already satisfied: cerberus in /opt/conda/lib/python3.7/site-packages (from bentoml==0.8.1->-r ./requirements.txt (line 3)) (1.3.2) [2020-06-25 20:17:15,537] INFO - Requirement already satisfied: python-json-logger in /opt/conda/lib/python3.7/site-packages (from bentoml==0.8.1->-r ./requirements.txt (line 3)) (0.1.11) [2020-06-25 20:17:15,540] INFO - Requirement already satisfied: certifi in /opt/conda/lib/python3.7/site-packages (from bentoml==0.8.1->-r ./requirements.txt (line 3)) (2020.6.20) [2020-06-25 20:17:15,542] INFO - Requirement already satisfied: requests in /opt/conda/lib/python3.7/site-packages (from bentoml==0.8.1->-r ./requirements.txt (line 3)) (2.22.0) [2020-06-25 20:17:15,557] INFO - Requirement already satisfied: boto3 in /opt/conda/lib/python3.7/site-packages (from bentoml==0.8.1->-r ./requirements.txt (line 3)) (1.14.3) [2020-06-25 20:17:15,614] INFO - Collecting erlastic |[2020-06-25 20:17:15,706] INFO - Downloading erlastic-2.0.0.tar.gz (6.8 kB) |[2020-06-25 20:17:16,037] INFO - Collecting py-params>=0.9.6 [2020-06-25 20:17:16,071] INFO - Downloading py-params-0.9.7.tar.gz (6.8 kB) /[2020-06-25 20:17:16,406] INFO - Collecting params-flow>=0.8.0 [2020-06-25 20:17:16,428] INFO - Downloading params-flow-0.8.2.tar.gz (22 kB) -[2020-06-25 20:17:16,670] INFO - Requirement already satisfied: setuptools>=41.0.0 in /opt/conda/lib/python3.7/site-packages (from tensorboard<2.3.0,>=2.2.0->tensorflow->-r ./requirements.txt (line 1)) (45.2.0.post20200210) [2020-06-25 20:17:16,678] INFO - Requirement already satisfied: werkzeug>=0.11.15 in /opt/conda/lib/python3.7/site-packages (from tensorboard<2.3.0,>=2.2.0->tensorflow->-r ./requirements.txt (line 1)) (1.0.1) /[2020-06-25 20:17:16,747] INFO - Collecting markdown>=2.6.8 [2020-06-25 20:17:16,769] INFO - Downloading Markdown-3.2.2-py3-none-any.whl (88 kB) |[2020-06-25 20:17:16,865] INFO - Collecting google-auth-oauthlib<0.5,>=0.4.1 [2020-06-25 20:17:16,895] INFO - Downloading google_auth_oauthlib-0.4.1-py2.py3-none-any.whl (18 kB) \[2020-06-25 20:17:16,969] INFO - Collecting tensorboard-plugin-wit>=1.6.0 [2020-06-25 20:17:17,014] INFO - Downloading tensorboard_plugin_wit-1.6.0.post3-py3-none-any.whl (777 kB) /[2020-06-25 20:17:17,184] INFO - Collecting google-auth<2,>=1.6.3 [2020-06-25 20:17:17,202] INFO - Downloading google_auth-1.18.0-py2.py3-none-any.whl (90 kB) |[2020-06-25 20:17:17,247] INFO - Requirement already satisfied: thriftpy2>=0.4.0 in /opt/conda/lib/python3.7/site-packages (from py-zipkin->bentoml==0.8.1->-r ./requirements.txt (line 3)) (0.4.11) [2020-06-25 20:17:17,262] INFO - Requirement already satisfied: attrs>=17.3.0 in /opt/conda/lib/python3.7/site-packages (from aiohttp->bentoml==0.8.1->-r ./requirements.txt (line 3)) (19.3.0) [2020-06-25 20:17:17,283] INFO - Requirement already satisfied: chardet<4.0,>=2.0 in /opt/conda/lib/python3.7/site-packages (from aiohttp->bentoml==0.8.1->-r ./requirements.txt (line 3)) (3.0.4) [2020-06-25 20:17:17,285] INFO - Requirement already satisfied: async-timeout<4.0,>=3.0 in /opt/conda/lib/python3.7/site-packages (from aiohttp->bentoml==0.8.1->-r ./requirements.txt (line 3)) (3.0.1) [2020-06-25 20:17:17,289] INFO - Requirement already satisfied: multidict<5.0,>=4.5 in /opt/conda/lib/python3.7/site-packages (from aiohttp->bentoml==0.8.1->-r ./requirements.txt (line 3)) (4.7.6) [2020-06-25 20:17:17,292] INFO - Requirement already satisfied: yarl<2.0,>=1.0 in /opt/conda/lib/python3.7/site-packages (from aiohttp->bentoml==0.8.1->-r ./requirements.txt (line 3)) (1.4.2) [2020-06-25 20:17:17,297] INFO - Requirement already satisfied: itsdangerous>=0.24 in /opt/conda/lib/python3.7/site-packages (from flask->bentoml==0.8.1->-r ./requirements.txt (line 3)) (1.1.0) [2020-06-25 20:17:17,300] INFO - Requirement already satisfied: Jinja2>=2.10.1 in /opt/conda/lib/python3.7/site-packages (from flask->bentoml==0.8.1->-r ./requirements.txt (line 3)) (2.11.2) [2020-06-25 20:17:17,304] INFO - Requirement already satisfied: pyparsing>=2.0.2 in /opt/conda/lib/python3.7/site-packages (from packaging->bentoml==0.8.1->-r ./requirements.txt (line 3)) (2.4.7) [2020-06-25 20:17:17,307] INFO - Requirement already satisfied: websocket-client>=0.32.0 in /opt/conda/lib/python3.7/site-packages (from docker->bentoml==0.8.1->-r ./requirements.txt (line 3)) (0.57.0) [2020-06-25 20:17:17,310] INFO - Requirement already satisfied: python-editor>=0.3 in /opt/conda/lib/python3.7/site-packages (from alembic->bentoml==0.8.1->-r ./requirements.txt (line 3)) (1.0.4) [2020-06-25 20:17:17,313] INFO - Requirement already satisfied: Mako in /opt/conda/lib/python3.7/site-packages (from alembic->bentoml==0.8.1->-r ./requirements.txt (line 3)) (1.1.3) [2020-06-25 20:17:17,318] INFO - Requirement already satisfied: ruamel.yaml.clib>=0.1.2; platform_python_implementation == "CPython" and python_version < "3.9" in /opt/conda/lib/python3.7/site-packages (from ruamel.yaml>=0.15.0->bentoml==0.8.1->-r ./requirements.txt (line 3)) (0.2.0) [2020-06-25 20:17:17,321] INFO - Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /opt/conda/lib/python3.7/site-packages (from requests->bentoml==0.8.1->-r ./requirements.txt (line 3)) (1.25.8) [2020-06-25 20:17:17,334] INFO - Requirement already satisfied: idna<2.9,>=2.5 in /opt/conda/lib/python3.7/site-packages (from requests->bentoml==0.8.1->-r ./requirements.txt (line 3)) (2.8) [2020-06-25 20:17:17,337] INFO - Requirement already satisfied: jmespath<1.0.0,>=0.7.1 in /opt/conda/lib/python3.7/site-packages (from boto3->bentoml==0.8.1->-r ./requirements.txt (line 3)) (0.10.0) [2020-06-25 20:17:17,340] INFO - Requirement already satisfied: s3transfer<0.4.0,>=0.3.0 in /opt/conda/lib/python3.7/site-packages (from boto3->bentoml==0.8.1->-r ./requirements.txt (line 3)) (0.3.3) \[2020-06-25 20:17:17,345] INFO - Requirement already satisfied: botocore<1.18.0,>=1.17.3 in /opt/conda/lib/python3.7/site-packages (from boto3->bentoml==0.8.1->-r ./requirements.txt (line 3)) (1.17.3) [2020-06-25 20:17:17,352] INFO - Requirement already satisfied: tqdm in /opt/conda/lib/python3.7/site-packages (from params-flow>=0.8.0->bert-for-tf2->-r ./requirements.txt (line 5)) (4.42.1) [2020-06-25 20:17:17,436] INFO - Collecting importlib-metadata; python_version < "3.8" -[2020-06-25 20:17:17,458] INFO - Downloading importlib_metadata-1.6.1-py2.py3-none-any.whl (31 kB) [2020-06-25 20:17:17,539] INFO - Collecting requests-oauthlib>=0.7.0 /[2020-06-25 20:17:17,565] INFO - Downloading requests_oauthlib-1.3.0-py2.py3-none-any.whl (23 kB) |[2020-06-25 20:17:17,681] INFO - Collecting pyasn1-modules>=0.2.1 [2020-06-25 20:17:17,745] INFO - Downloading pyasn1_modules-0.2.8-py2.py3-none-any.whl (155 kB) -[2020-06-25 20:17:17,860] INFO - Collecting rsa<5,>=3.1.4; python_version >= "3" [2020-06-25 20:17:17,884] INFO - Downloading rsa-4.6-py3-none-any.whl (47 kB) /[2020-06-25 20:17:18,003] INFO - Collecting cachetools<5.0,>=2.0.0 [2020-06-25 20:17:18,027] INFO - Downloading cachetools-4.1.0-py3-none-any.whl (10 kB) |[2020-06-25 20:17:18,053] INFO - Requirement already satisfied: ply<4.0,>=3.4 in /opt/conda/lib/python3.7/site-packages (from thriftpy2>=0.4.0->py-zipkin->bentoml==0.8.1->-r ./requirements.txt (line 3)) (3.11) [2020-06-25 20:17:18,055] INFO - Requirement already satisfied: MarkupSafe>=0.23 in /opt/conda/lib/python3.7/site-packages (from Jinja2>=2.10.1->flask->bentoml==0.8.1->-r ./requirements.txt (line 3)) (1.1.1) [2020-06-25 20:17:18,058] INFO - Requirement already satisfied: docutils<0.16,>=0.10 in /opt/conda/lib/python3.7/site-packages (from botocore<1.18.0,>=1.17.3->boto3->bentoml==0.8.1->-r ./requirements.txt (line 3)) (0.15.2) [2020-06-25 20:17:18,128] INFO - Collecting zipp>=0.5 \[2020-06-25 20:17:18,159] INFO - Downloading zipp-3.1.0-py3-none-any.whl (4.9 kB) [2020-06-25 20:17:18,238] INFO - Collecting oauthlib>=3.0.0 -[2020-06-25 20:17:18,260] INFO - Downloading oauthlib-3.1.0-py2.py3-none-any.whl (147 kB) /[2020-06-25 20:17:18,419] INFO - Collecting pyasn1<0.5.0,>=0.4.6 [2020-06-25 20:17:18,452] INFO - Downloading pyasn1-0.4.8-py2.py3-none-any.whl (77 kB) |[2020-06-25 20:17:18,496] INFO - Building wheels for collected packages: bert, bert-for-tf2, absl-py, wrapt, termcolor, erlastic, py-params, params-flow [2020-06-25 20:17:18,497] INFO - Building wheel for bert (setup.py): started /[2020-06-25 20:17:18,782] INFO - Building wheel for bert (setup.py): finished with status 'done' [2020-06-25 20:17:18,782] INFO - Created wheel for bert: filename=bert-2.2.0-py3-none-any.whl size=3754 sha256=b8b4e6a6f6d69b4b5e25386f993e6470e5a788a4b0f97f5e9c5e69c52936bbbf Stored in directory: /root/.cache/pip/wheels/bb/31/1b/c05f362e347429b7436954d1a2280fe464731e8f569123a848 [2020-06-25 20:17:18,784] INFO - Building wheel for bert-for-tf2 (setup.py): started \[2020-06-25 20:17:19,042] INFO - Building wheel for bert-for-tf2 (setup.py): finished with status 'done' [2020-06-25 20:17:19,043] INFO - Created wheel for bert-for-tf2: filename=bert_for_tf2-0.14.4-py3-none-any.whl size=30114 sha256=1a9162d3268767c08656b63f77c1220cc883f50e65812221832719e057614d77 [2020-06-25 20:17:19,043] INFO - Stored in directory: /root/.cache/pip/wheels/6c/c9/9c/363182ea34a736dae336eeaf0dd4a7eec3c6a5afe32373e1fe [2020-06-25 20:17:19,045] INFO - Building wheel for absl-py (setup.py): started |[2020-06-25 20:17:19,322] INFO - Building wheel for absl-py (setup.py): finished with status 'done' [2020-06-25 20:17:19,323] INFO - Created wheel for absl-py: filename=absl_py-0.9.0-py3-none-any.whl size=121931 sha256=9957e0ab2b07822a87d0635f10e3dfde4edac694ca467668ccea34ee59376640 Stored in directory: /root/.cache/pip/wheels/cc/af/1a/498a24d0730ef484019e007bb9e8cef3ac00311a672c049a3e [2020-06-25 20:17:19,325] INFO - Building wheel for wrapt (setup.py): started /[2020-06-25 20:17:20,475] INFO - Building wheel for wrapt (setup.py): finished with status 'done' [2020-06-25 20:17:20,475] INFO - Created wheel for wrapt: filename=wrapt-1.12.1-cp37-cp37m-linux_x86_64.whl size=76414 sha256=acb25193c9fbbbd0a1fff5323a8778374e56d823a538df15f8fbc9682dd207a5 Stored in directory: /root/.cache/pip/wheels/62/76/4c/aa25851149f3f6d9785f6c869387ad82b3fd37582fa8147ac6 |[2020-06-25 20:17:20,477] INFO - Building wheel for termcolor (setup.py): started -[2020-06-25 20:17:20,716] INFO - Building wheel for termcolor (setup.py): finished with status 'done' [2020-06-25 20:17:20,716] INFO - Created wheel for termcolor: filename=termcolor-1.1.0-py3-none-any.whl size=4830 sha256=9ac668e50da75dd5898a386a98c08492225259d75718a5c10b8ddc2f6d399493 Stored in directory: /root/.cache/pip/wheels/3f/e3/ec/8a8336ff196023622fbcb36de0c5a5c218cbb24111d1d4c7f2 [2020-06-25 20:17:20,718] INFO - Building wheel for erlastic (setup.py): started |[2020-06-25 20:17:20,961] INFO - Building wheel for erlastic (setup.py): finished with status 'done' [2020-06-25 20:17:20,961] INFO - Created wheel for erlastic: filename=erlastic-2.0.0-py3-none-any.whl size=6787 sha256=42c7314b2539a2d1457f5ea5107b78d0be65323306df829b43e6474aa44aa40d Stored in directory: /root/.cache/pip/wheels/94/f1/b4/0b98b1e94775da6a0b1130e342d22af05cd269e1172c19f40f [2020-06-25 20:17:20,963] INFO - Building wheel for py-params (setup.py): started /[2020-06-25 20:17:21,204] INFO - Building wheel for py-params (setup.py): finished with status 'done' [2020-06-25 20:17:21,204] INFO - Created wheel for py-params: filename=py_params-0.9.7-py3-none-any.whl size=7302 sha256=71d7116a42f4756d897dba4861b4290a95705a3851749a79088bc05bd54fdab4 [2020-06-25 20:17:21,205] INFO - Stored in directory: /root/.cache/pip/wheels/47/3d/2d/bbffcfd6b9f4b8b5cbf07e7520ac2676192fe9431240c13ee8 [2020-06-25 20:17:21,206] INFO - Building wheel for params-flow (setup.py): started \[2020-06-25 20:17:21,468] INFO - Building wheel for params-flow (setup.py): finished with status 'done' [2020-06-25 20:17:21,468] INFO - Created wheel for params-flow: filename=params_flow-0.8.2-py3-none-any.whl size=19473 sha256=ce73a6a454b9477c74e5c7fafd4d6337e0eb50294f060bcb9aa7608eac515b47 [2020-06-25 20:17:21,469] INFO - Stored in directory: /root/.cache/pip/wheels/0e/fc/d2/a44fff33af0f233d7def6e7de413006d57c10e10ad736fe8f5 [2020-06-25 20:17:21,469] INFO - Successfully built bert bert-for-tf2 absl-py wrapt termcolor erlastic py-params params-flow |[2020-06-25 20:17:21,727] INFO - Installing collected packages: numpy, keras-preprocessing, scipy, gast, absl-py, zipp, importlib-metadata, markdown, oauthlib, requests-oauthlib, pyasn1, pyasn1-modules, rsa, cachetools, google-auth, google-auth-oauthlib, tensorboard-plugin-wit, tensorboard, wrapt, google-pasta, termcolor, astunparse, h5py, tensorflow-estimator, opt-einsum, tensorflow, pytz, pandas, erlastic, bert, py-params, params-flow, bert-for-tf2 Attempting uninstall: numpy [2020-06-25 20:17:21,728] INFO - Found existing installation: numpy 1.18.5 \[2020-06-25 20:17:21,880] INFO - Uninstalling numpy-1.18.5: \[2020-06-25 20:17:22,608] INFO - Successfully uninstalled numpy-1.18.5 |[2020-06-25 20:17:58,526] INFO - Successfully installed absl-py-0.9.0 astunparse-1.6.3 bert-2.2.0 bert-for-tf2-0.14.4 cachetools-4.1.0 erlastic-2.0.0 gast-0.3.3 google-auth-1.18.0 google-auth-oauthlib-0.4.1 google-pasta-0.2.0 h5py-2.10.0 importlib-metadata-1.6.1 keras-preprocessing-1.1.2 markdown-3.2.2 numpy-1.18.1 oauthlib-3.1.0 opt-einsum-3.2.1 pandas-1.0.1 params-flow-0.8.2 py-params-0.9.7 pyasn1-0.4.8 pyasn1-modules-0.2.8 pytz-2020.1 requests-oauthlib-1.3.0 rsa-4.6 scipy-1.4.1 tensorboard-2.2.2 tensorboard-plugin-wit-1.6.0.post3 tensorflow-2.2.0 tensorflow-estimator-2.2.0 termcolor-1.1.0 wrapt-1.12.1 zipp-3.1.0 |[2020-06-25 20:17:59,376] INFO - + for filename in ./bundled_pip_dependencies/*.tar.gz + '[' -e ./bundled_pip_dependencies/BentoML-0.8.1+0.g5b6bd29.dirty.tar.gz ']' [2020-06-25 20:17:59,376] INFO - + pip install -U ./bundled_pip_dependencies/BentoML-0.8.1+0.g5b6bd29.dirty.tar.gz |[2020-06-25 20:18:00,145] INFO - Processing ./bundled_pip_dependencies/BentoML-0.8.1+0.g5b6bd29.dirty.tar.gz \[2020-06-25 20:18:00,211] INFO - Installing build dependencies: started /[2020-06-25 20:18:02,497] INFO - Installing build dependencies: finished with status 'done' Getting requirements to build wheel: started /[2020-06-25 20:18:02,857] INFO - Getting requirements to build wheel: finished with status 'done' [2020-06-25 20:18:02,860] INFO - Preparing wheel metadata: started /[2020-06-25 20:18:03,290] INFO - Preparing wheel metadata: finished with status 'done' \[2020-06-25 20:18:03,441] INFO - Requirement already satisfied, skipping upgrade: prometheus-client in /opt/conda/lib/python3.7/site-packages (from BentoML==0.8.1+0.g5b6bd29.dirty) (0.8.0) [2020-06-25 20:18:03,446] INFO - Requirement already satisfied, skipping upgrade: aiohttp in /opt/conda/lib/python3.7/site-packages (from BentoML==0.8.1+0.g5b6bd29.dirty) (3.6.2) [2020-06-25 20:18:03,458] INFO - Requirement already satisfied, skipping upgrade: numpy in /opt/conda/lib/python3.7/site-packages (from BentoML==0.8.1+0.g5b6bd29.dirty) (1.18.1) [2020-06-25 20:18:03,461] INFO - Requirement already satisfied, skipping upgrade: boto3 in /opt/conda/lib/python3.7/site-packages (from BentoML==0.8.1+0.g5b6bd29.dirty) (1.14.3) [2020-06-25 20:18:03,466] INFO - Requirement already satisfied, skipping upgrade: packaging in /opt/conda/lib/python3.7/site-packages (from BentoML==0.8.1+0.g5b6bd29.dirty) (20.4) [2020-06-25 20:18:03,470] INFO - Requirement already satisfied, skipping upgrade: sqlalchemy-utils in /opt/conda/lib/python3.7/site-packages (from BentoML==0.8.1+0.g5b6bd29.dirty) (0.36.6) [2020-06-25 20:18:03,528] INFO - Requirement already satisfied, skipping upgrade: sqlalchemy>=1.3.0 in /opt/conda/lib/python3.7/site-packages (from BentoML==0.8.1+0.g5b6bd29.dirty) (1.3.17) -[2020-06-25 20:18:03,538] INFO - Requirement already satisfied, skipping upgrade: requests in /opt/conda/lib/python3.7/site-packages (from BentoML==0.8.1+0.g5b6bd29.dirty) (2.22.0) [2020-06-25 20:18:03,552] INFO - Requirement already satisfied, skipping upgrade: python-dateutil<2.8.1,>=2.1 in /opt/conda/lib/python3.7/site-packages (from BentoML==0.8.1+0.g5b6bd29.dirty) (2.8.0) [2020-06-25 20:18:03,555] INFO - Requirement already satisfied, skipping upgrade: ruamel.yaml>=0.15.0 in /opt/conda/lib/python3.7/site-packages (from BentoML==0.8.1+0.g5b6bd29.dirty) (0.16.10) [2020-06-25 20:18:03,565] INFO - Requirement already satisfied, skipping upgrade: protobuf>=3.6.0 in /opt/conda/lib/python3.7/site-packages (from BentoML==0.8.1+0.g5b6bd29.dirty) (3.12.2) [2020-06-25 20:18:03,568] INFO - Requirement already satisfied, skipping upgrade: psutil in /opt/conda/lib/python3.7/site-packages (from BentoML==0.8.1+0.g5b6bd29.dirty) (5.7.0) [2020-06-25 20:18:03,571] INFO - Requirement already satisfied, skipping upgrade: docker in /opt/conda/lib/python3.7/site-packages (from BentoML==0.8.1+0.g5b6bd29.dirty) (4.2.1) [2020-06-25 20:18:03,583] INFO - Requirement already satisfied, skipping upgrade: gunicorn in /opt/conda/lib/python3.7/site-packages (from BentoML==0.8.1+0.g5b6bd29.dirty) (20.0.4) [2020-06-25 20:18:03,589] INFO - Requirement already satisfied, skipping upgrade: certifi in /opt/conda/lib/python3.7/site-packages (from BentoML==0.8.1+0.g5b6bd29.dirty) (2020.6.20) [2020-06-25 20:18:03,591] INFO - Requirement already satisfied, skipping upgrade: py-zipkin in /opt/conda/lib/python3.7/site-packages (from BentoML==0.8.1+0.g5b6bd29.dirty) (0.20.0) [2020-06-25 20:18:03,596] INFO - Requirement already satisfied, skipping upgrade: alembic in /opt/conda/lib/python3.7/site-packages (from BentoML==0.8.1+0.g5b6bd29.dirty) (1.4.2) [2020-06-25 20:18:03,602] INFO - Requirement already satisfied, skipping upgrade: configparser in /opt/conda/lib/python3.7/site-packages (from BentoML==0.8.1+0.g5b6bd29.dirty) (5.0.0) [2020-06-25 20:18:03,611] INFO - Requirement already satisfied, skipping upgrade: click>=7.0 in /opt/conda/lib/python3.7/site-packages (from BentoML==0.8.1+0.g5b6bd29.dirty) (7.1.2) [2020-06-25 20:18:03,613] INFO - Requirement already satisfied, skipping upgrade: grpcio<=1.27.2 in /opt/conda/lib/python3.7/site-packages (from BentoML==0.8.1+0.g5b6bd29.dirty) (1.27.2) [2020-06-25 20:18:03,617] INFO - Requirement already satisfied, skipping upgrade: humanfriendly in /opt/conda/lib/python3.7/site-packages (from BentoML==0.8.1+0.g5b6bd29.dirty) (8.2) [2020-06-25 20:18:03,621] INFO - Requirement already satisfied, skipping upgrade: tabulate in /opt/conda/lib/python3.7/site-packages (from BentoML==0.8.1+0.g5b6bd29.dirty) (0.8.7) [2020-06-25 20:18:03,625] INFO - Requirement already satisfied, skipping upgrade: python-json-logger in /opt/conda/lib/python3.7/site-packages (from BentoML==0.8.1+0.g5b6bd29.dirty) (0.1.11) [2020-06-25 20:18:03,627] INFO - Requirement already satisfied, skipping upgrade: cerberus in /opt/conda/lib/python3.7/site-packages (from BentoML==0.8.1+0.g5b6bd29.dirty) (1.3.2) [2020-06-25 20:18:03,629] INFO - Requirement already satisfied, skipping upgrade: flask in /opt/conda/lib/python3.7/site-packages (from BentoML==0.8.1+0.g5b6bd29.dirty) (1.1.2) /[2020-06-25 20:18:03,644] INFO - Requirement already satisfied, skipping upgrade: async-timeout<4.0,>=3.0 in /opt/conda/lib/python3.7/site-packages (from aiohttp->BentoML==0.8.1+0.g5b6bd29.dirty) (3.0.1) [2020-06-25 20:18:03,646] INFO - Requirement already satisfied, skipping upgrade: multidict<5.0,>=4.5 in /opt/conda/lib/python3.7/site-packages (from aiohttp->BentoML==0.8.1+0.g5b6bd29.dirty) (4.7.6) [2020-06-25 20:18:03,648] INFO - Requirement already satisfied, skipping upgrade: attrs>=17.3.0 in /opt/conda/lib/python3.7/site-packages (from aiohttp->BentoML==0.8.1+0.g5b6bd29.dirty) (19.3.0) [2020-06-25 20:18:03,672] INFO - Requirement already satisfied, skipping upgrade: chardet<4.0,>=2.0 in /opt/conda/lib/python3.7/site-packages (from aiohttp->BentoML==0.8.1+0.g5b6bd29.dirty) (3.0.4) [2020-06-25 20:18:03,676] INFO - Requirement already satisfied, skipping upgrade: yarl<2.0,>=1.0 in /opt/conda/lib/python3.7/site-packages (from aiohttp->BentoML==0.8.1+0.g5b6bd29.dirty) (1.4.2) [2020-06-25 20:18:03,680] INFO - Requirement already satisfied, skipping upgrade: botocore<1.18.0,>=1.17.3 in /opt/conda/lib/python3.7/site-packages (from boto3->BentoML==0.8.1+0.g5b6bd29.dirty) (1.17.3) [2020-06-25 20:18:03,690] INFO - Requirement already satisfied, skipping upgrade: jmespath<1.0.0,>=0.7.1 in /opt/conda/lib/python3.7/site-packages (from boto3->BentoML==0.8.1+0.g5b6bd29.dirty) (0.10.0) [2020-06-25 20:18:03,694] INFO - Requirement already satisfied, skipping upgrade: s3transfer<0.4.0,>=0.3.0 in /opt/conda/lib/python3.7/site-packages (from boto3->BentoML==0.8.1+0.g5b6bd29.dirty) (0.3.3) [2020-06-25 20:18:03,699] INFO - Requirement already satisfied, skipping upgrade: pyparsing>=2.0.2 in /opt/conda/lib/python3.7/site-packages (from packaging->BentoML==0.8.1+0.g5b6bd29.dirty) (2.4.7) [2020-06-25 20:18:03,702] INFO - Requirement already satisfied, skipping upgrade: six in /opt/conda/lib/python3.7/site-packages (from packaging->BentoML==0.8.1+0.g5b6bd29.dirty) (1.14.0) [2020-06-25 20:18:03,705] INFO - Requirement already satisfied, skipping upgrade: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /opt/conda/lib/python3.7/site-packages (from requests->BentoML==0.8.1+0.g5b6bd29.dirty) (1.25.8) [2020-06-25 20:18:03,717] INFO - Requirement already satisfied, skipping upgrade: idna<2.9,>=2.5 in /opt/conda/lib/python3.7/site-packages (from requests->BentoML==0.8.1+0.g5b6bd29.dirty) (2.8) [2020-06-25 20:18:03,722] INFO - Requirement already satisfied, skipping upgrade: ruamel.yaml.clib>=0.1.2; platform_python_implementation == "CPython" and python_version < "3.9" in /opt/conda/lib/python3.7/site-packages (from ruamel.yaml>=0.15.0->BentoML==0.8.1+0.g5b6bd29.dirty) (0.2.0) [2020-06-25 20:18:03,725] INFO - Requirement already satisfied, skipping upgrade: setuptools in /opt/conda/lib/python3.7/site-packages (from protobuf>=3.6.0->BentoML==0.8.1+0.g5b6bd29.dirty) (45.2.0.post20200210) [2020-06-25 20:18:03,734] INFO - Requirement already satisfied, skipping upgrade: websocket-client>=0.32.0 in /opt/conda/lib/python3.7/site-packages (from docker->BentoML==0.8.1+0.g5b6bd29.dirty) (0.57.0) [2020-06-25 20:18:03,739] INFO - Requirement already satisfied, skipping upgrade: thriftpy2>=0.4.0 in /opt/conda/lib/python3.7/site-packages (from py-zipkin->BentoML==0.8.1+0.g5b6bd29.dirty) (0.4.11) |[2020-06-25 20:18:03,750] INFO - Requirement already satisfied, skipping upgrade: python-editor>=0.3 in /opt/conda/lib/python3.7/site-packages (from alembic->BentoML==0.8.1+0.g5b6bd29.dirty) (1.0.4) [2020-06-25 20:18:03,753] INFO - Requirement already satisfied, skipping upgrade: Mako in /opt/conda/lib/python3.7/site-packages (from alembic->BentoML==0.8.1+0.g5b6bd29.dirty) (1.1.3) [2020-06-25 20:18:03,760] INFO - Requirement already satisfied, skipping upgrade: itsdangerous>=0.24 in /opt/conda/lib/python3.7/site-packages (from flask->BentoML==0.8.1+0.g5b6bd29.dirty) (1.1.0) [2020-06-25 20:18:03,762] INFO - Requirement already satisfied, skipping upgrade: Werkzeug>=0.15 in /opt/conda/lib/python3.7/site-packages (from flask->BentoML==0.8.1+0.g5b6bd29.dirty) (1.0.1) [2020-06-25 20:18:03,772] INFO - Requirement already satisfied, skipping upgrade: Jinja2>=2.10.1 in /opt/conda/lib/python3.7/site-packages (from flask->BentoML==0.8.1+0.g5b6bd29.dirty) (2.11.2) [2020-06-25 20:18:03,777] INFO - Requirement already satisfied, skipping upgrade: docutils<0.16,>=0.10 in /opt/conda/lib/python3.7/site-packages (from botocore<1.18.0,>=1.17.3->boto3->BentoML==0.8.1+0.g5b6bd29.dirty) (0.15.2) [2020-06-25 20:18:03,780] INFO - Requirement already satisfied, skipping upgrade: ply<4.0,>=3.4 in /opt/conda/lib/python3.7/site-packages (from thriftpy2>=0.4.0->py-zipkin->BentoML==0.8.1+0.g5b6bd29.dirty) (3.11) [2020-06-25 20:18:03,781] INFO - Requirement already satisfied, skipping upgrade: MarkupSafe>=0.9.2 in /opt/conda/lib/python3.7/site-packages (from Mako->alembic->BentoML==0.8.1+0.g5b6bd29.dirty) (1.1.1) [2020-06-25 20:18:03,783] INFO - Building wheels for collected packages: BentoML [2020-06-25 20:18:03,785] INFO - Building wheel for BentoML (PEP 517): started /[2020-06-25 20:18:04,468] INFO - Building wheel for BentoML (PEP 517): finished with status 'done' [2020-06-25 20:18:04,471] INFO - Created wheel for BentoML: filename=BentoML-0.8.1+0.g5b6bd29.dirty-py3-none-any.whl size=574333 sha256=a1b82c0eecdfdf6feb653dfb81708ca24a92dca50f600c728a115c559a76cd5f Stored in directory: /root/.cache/pip/wheels/df/cb/03/c2ad61c37a532e5a3b1c22de0a7c2f2db6b11a50fd0ebac298 [2020-06-25 20:18:04,472] INFO - Successfully built BentoML /[2020-06-25 20:18:04,870] INFO - Installing collected packages: BentoML Attempting uninstall: BentoML [2020-06-25 20:18:04,871] INFO - Found existing installation: BentoML 0.8.1 |[2020-06-25 20:18:05,013] INFO - Uninstalling BentoML-0.8.1: -[2020-06-25 20:18:05,667] INFO - Successfully uninstalled BentoML-0.8.1 \[2020-06-25 20:18:05,959] INFO - Successfully installed BentoML-0.8.1+0.g5b6bd29.dirty |[2020-06-25 20:18:33,843] INFO - ---> f63d2c918a65 [2020-06-25 20:18:33,845] INFO - Step 9/9 : ENV PATH="/bento:$PATH" [2020-06-25 20:18:33,845] INFO - \[2020-06-25 20:18:33,914] INFO - ---> Running in 9f03401a44b6 -[2020-06-25 20:18:34,071] INFO - ---> 1e52bd886529 /[2020-06-25 20:18:34,080] INFO - Successfully built 1e52bd886529 [2020-06-25 20:18:34,085] INFO - Successfully tagged 899399195124.dkr.ecr.us-east-1.amazonaws.com/service-sagemaker:20200625195616_62D0DB \[2020-06-25 20:53:09,669] INFO - ApplyDeployment (bert-moviereview-sagemaker, namespace dev) succeeded Successfully created AWS Sagemaker deployment bert-moviereview-sagemaker { "namespace": "dev", "name": "bert-moviereview-sagemaker", "spec": { "bentoName": "Service", "bentoVersion": "20200625195616_62D0DB", "operator": "AWS_SAGEMAKER", "sagemakerOperatorConfig": { "region": "us-east-1", "instanceType": "ml.m4.xlarge", "instanceCount": 1, "apiName": "predict", "timeout": 60 } }, "state": { "state": "RUNNING", "infoJson": { "EndpointName": "dev-bert-moviereview-sagemaker", "EndpointArn": "arn:aws:sagemaker:us-east-1:899399195124:endpoint/dev-bert-moviereview-sagemaker", "EndpointConfigName": "dev-bert-moviereview-sagemaker-Service-20200625195616-62D0DB", "ProductionVariants": [ { "VariantName": "dev-bert-moviereview-sagemaker-Service-20200625195616-62D0DB", "DeployedImages": [ { "SpecifiedImage": "899399195124.dkr.ecr.us-east-1.amazonaws.com/service-sagemaker:20200625195616_62D0DB", "ResolvedImage": "899399195124.dkr.ecr.us-east-1.amazonaws.com/service-sagemaker@sha256:c064de18b75b18da26f5b8743491e13542a179915d5ea36ce4b8e971c6611062", "ResolutionTime": "2020-06-25 20:53:14.176000-04:00" } ], "CurrentWeight": 1.0, "DesiredWeight": 1.0, "CurrentInstanceCount": 1, "DesiredInstanceCount": 1 } ], "EndpointStatus": "InService", "CreationTime": "2020-06-25 20:53:09.599000-04:00", "LastModifiedTime": "2020-06-25 20:59:33.149000-04:00", "ResponseMetadata": { "RequestId": "202c6fcf-048c-45e8-ab11-3dcc5771072b", "HTTPStatusCode": 200, "HTTPHeaders": { "x-amzn-requestid": "202c6fcf-048c-45e8-ab11-3dcc5771072b", "content-type": "application/x-amz-json-1.1", "content-length": "831", "date": "Fri, 26 Jun 2020 00:59:34 GMT" }, "RetryAttempts": 0 } }, "timestamp": "2020-06-26T00:59:34.850115Z" }, "createdAt": "2020-06-26T00:15:56.839917Z", "lastUpdatedAt": "2020-06-26T00:15:56.839947Z" }
Here we could confirm that the API endpoints status is InService
!aws sagemaker describe-endpoint --endpoint-name dev-sagemaker-moviereview-deployment
{ "EndpointName": "dev-bert-moviereview-sagemaker", "EndpointArn": "arn:aws:sagemaker:us-east-1:899399195124:endpoint/dev-bert-moviereview-sagemaker", "EndpointConfigName": "dev-bert-moviereview-sagemaker-Service-20200625195616-62D0DB", "ProductionVariants": [ { "VariantName": "dev-bert-moviereview-sagemaker-Service-20200625195616-62D0DB", "DeployedImages": [ { "SpecifiedImage": "899399195124.dkr.ecr.us-east-1.amazonaws.com/service-sagemaker:20200625195616_62D0DB", "ResolvedImage": "899399195124.dkr.ecr.us-east-1.amazonaws.com/service-sagemaker@sha256:c064de18b75b18da26f5b8743491e13542a179915d5ea36ce4b8e971c6611062", "ResolutionTime": "2020-06-25T20:53:14.176000-04:00" } ], "CurrentWeight": 1.0, "DesiredWeight": 1.0, "CurrentInstanceCount": 1, "DesiredInstanceCount": 1 } ], "EndpointStatus": "InService", "CreationTime": "2020-06-25T20:53:09.599000-04:00", "LastModifiedTime": "2020-06-25T20:59:33.149000-04:00" }
Now, we are ready to test the SageMaker API endpoints by creating a small script using the boto3 SDK. Alternatively, users could also use the AWS CLI to test the endpoint. Please visit https://awscli.amazonaws.com/v2/documentation/api/latest/reference/sagemaker-runtime/invoke-endpoint.html
import boto3
import json
endpoint = 'dev-sagemaker-moviereview-deployment'
runtime = boto3.Session().client('sagemaker-runtime')
movie_example = '["The acting was a bit lacking."]'
response = runtime.invoke_endpoint(EndpointName=endpoint, ContentType='application/json', Body=movie_example)
# Unpack response
result = json.loads(response['Body'].read().decode())
print(result)
['negative']
Lastly, do not forget to terminate the AWS resources used in this tutorial. Users could also clean up used resources by logging into the SageMaker console. For more information, please see here: https://docs.aws.amazon.com/sagemaker/latest/dg/ex1-cleanup.html
bucket_to_delete = boto3.resource('s3').Bucket('movie-review-dataset')
bucket_to_delete.objects.all().delete()
[]
sagemaker.Session().delete_endpoint('dev-sagemaker-moviereview-deployment')