BentoML Example: Keras Toxic Comment Classification

BentoML makes moving trained ML models to production easy:

  • Package models trained with any ML framework and reproduce them for model serving in production
  • Deploy anywhere for online API serving or offline batch serving
  • High-Performance API model server with adaptive micro-batching support
  • Central hub for managing models and deployment process via Web UI and APIs
  • Modular and flexible design making it adaptable to your infrastrcuture

BentoML is a framework for serving, managing, and deploying machine learning models. It is aiming to bridge the gap between Data Science and DevOps, and enable teams to deliver prediction services in a fast, repeatable, and scalable way.

Before reading this example project, be sure to check out the Getting started guide to learn about the basic concepts in BentoML.

This notebook demonstrates how to use BentoML to turn a Keras model into a docker image containing a REST API server serving this model, how to use your ML service built with BentoML as a CLI tool, and how to distribute it a pypi package.

This notebook is built based on: https://www.kaggle.com/sarvajna/keras-sequential-model-lb-0-052

Impression

In [1]:
%reload_ext autoreload
%autoreload 2
%matplotlib inline
In [2]:
!pip install -q bentoml keras==2.3.1 kaggle tensorflow==1.14.0 scikit-learn>=0.23.0
Requirement already satisfied: bentoml in /home/bentoml/BentoML (0.8.3+47.g7af5a8f)
Requirement already satisfied: aiohttp in /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages (from bentoml) (3.6.2)
Requirement already satisfied: alembic in /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages (from bentoml) (1.4.2)
Requirement already satisfied: boto3 in /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages (from bentoml) (1.14.30)
Requirement already satisfied: cerberus in /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages (from bentoml) (1.3.2)
Requirement already satisfied: certifi in /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages (from bentoml) (2020.6.20)
Requirement already satisfied: click>=7.0 in /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages (from bentoml) (7.1.2)
Requirement already satisfied: configparser in /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages (from bentoml) (5.0.0)
Requirement already satisfied: docker in /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages (from bentoml) (4.2.0)
Requirement already satisfied: flask in /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages (from bentoml) (1.1.2)
Requirement already satisfied: grpcio<=1.27.2 in /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages (from bentoml) (1.27.2)
Requirement already satisfied: gunicorn in /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages (from bentoml) (20.0.4)
Requirement already satisfied: humanfriendly in /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages (from bentoml) (8.2)
Requirement already satisfied: numpy in /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages (from bentoml) (1.16.6)
Requirement already satisfied: packaging in /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages (from bentoml) (20.4)
Requirement already satisfied: prometheus_client in /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages (from bentoml) (0.7.1)
Requirement already satisfied: protobuf>=3.6.0 in /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages (from bentoml) (3.11.4)
Requirement already satisfied: psutil in /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages (from bentoml) (5.7.2)
Requirement already satisfied: py_zipkin in /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages (from bentoml) (0.20.0)
Requirement already satisfied: python-dateutil<2.8.1,>=2.1 in /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages (from bentoml) (2.8.0)
Requirement already satisfied: python-json-logger in /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages (from bentoml) (0.1.11)
Requirement already satisfied: requests in /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages (from bentoml) (2.23.0)
Requirement already satisfied: ruamel.yaml>=0.15.0 in /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages (from bentoml) (0.16.10)
Requirement already satisfied: sqlalchemy-utils in /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages (from bentoml) (0.36.8)
Requirement already satisfied: sqlalchemy>=1.3.0 in /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages (from bentoml) (1.3.13)
Requirement already satisfied: tabulate in /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages (from bentoml) (0.8.7)
Requirement already satisfied: multidict in /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages (from bentoml) (4.7.6)
Requirement already satisfied: contextvars in /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages (from bentoml) (2.4)
Requirement already satisfied: attrs>=17.3.0 in /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages (from aiohttp->bentoml) (19.3.0)
Requirement already satisfied: chardet<4.0,>=2.0 in /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages (from aiohttp->bentoml) (3.0.4)
Requirement already satisfied: typing-extensions>=3.6.5; python_version < "3.7" in /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages (from aiohttp->bentoml) (3.7.4.2)
Requirement already satisfied: async-timeout<4.0,>=3.0 in /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages (from aiohttp->bentoml) (3.0.1)
Requirement already satisfied: yarl<2.0,>=1.0 in /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages (from aiohttp->bentoml) (1.5.0)
Requirement already satisfied: idna-ssl>=1.0; python_version < "3.7" in /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages (from aiohttp->bentoml) (1.1.0)
Requirement already satisfied: python-editor>=0.3 in /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages (from alembic->bentoml) (1.0.4)
Requirement already satisfied: Mako in /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages (from alembic->bentoml) (1.1.2)
Requirement already satisfied: s3transfer<0.4.0,>=0.3.0 in /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages (from boto3->bentoml) (0.3.3)
Requirement already satisfied: jmespath<1.0.0,>=0.7.1 in /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages (from boto3->bentoml) (0.10.0)
Requirement already satisfied: botocore<1.18.0,>=1.17.30 in /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages (from boto3->bentoml) (1.17.30)
Requirement already satisfied: setuptools in /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages (from cerberus->bentoml) (46.2.0.post20200511)
Requirement already satisfied: six>=1.4.0 in /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages (from docker->bentoml) (1.14.0)
Requirement already satisfied: websocket-client>=0.32.0 in /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages (from docker->bentoml) (0.57.0)
Requirement already satisfied: Jinja2>=2.10.1 in /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages (from flask->bentoml) (2.11.2)
Requirement already satisfied: Werkzeug>=0.15 in /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages (from flask->bentoml) (1.0.1)
Requirement already satisfied: itsdangerous>=0.24 in /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages (from flask->bentoml) (1.1.0)
Requirement already satisfied: pyparsing>=2.0.2 in /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages (from packaging->bentoml) (2.4.7)
Requirement already satisfied: thriftpy2>=0.4.0 in /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages (from py_zipkin->bentoml) (0.4.11)
Requirement already satisfied: idna<3,>=2.5 in /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages (from requests->bentoml) (2.9)
Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages (from requests->bentoml) (1.24.3)
Requirement already satisfied: ruamel.yaml.clib>=0.1.2; platform_python_implementation == "CPython" and python_version < "3.9" in /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages (from ruamel.yaml>=0.15.0->bentoml) (0.2.0)
Requirement already satisfied: immutables>=0.9 in /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages (from contextvars->bentoml) (0.14)
Requirement already satisfied: MarkupSafe>=0.9.2 in /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages (from Mako->alembic->bentoml) (1.1.1)
Requirement already satisfied: docutils<0.16,>=0.10 in /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages (from botocore<1.18.0,>=1.17.30->boto3->bentoml) (0.15.2)
Requirement already satisfied: ply<4.0,>=3.4 in /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages (from thriftpy2>=0.4.0->py_zipkin->bentoml) (3.11)
Requirement already satisfied: keras==2.3.1 in /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages (2.3.1)
Requirement already satisfied: kaggle in /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages (1.5.6)
Requirement already satisfied: tensorflow==1.14.0 in /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages (1.14.0)
Requirement already satisfied: scikit-learn in /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages (0.23.1)
Requirement already satisfied: numpy>=1.9.1 in /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages (from keras==2.3.1) (1.16.6)
Requirement already satisfied: six>=1.9.0 in /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages (from keras==2.3.1) (1.14.0)
Requirement already satisfied: h5py in /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages (from keras==2.3.1) (2.10.0)
Requirement already satisfied: scipy>=0.14 in /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages (from keras==2.3.1) (1.4.1)
Requirement already satisfied: keras-preprocessing>=1.0.5 in /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages (from keras==2.3.1) (1.1.0)
Requirement already satisfied: keras-applications>=1.0.6 in /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages (from keras==2.3.1) (1.0.8)
Requirement already satisfied: pyyaml in /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages (from keras==2.3.1) (5.3.1)
Requirement already satisfied: python-slugify in /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages (from kaggle) (4.0.1)
Requirement already satisfied: urllib3<1.25,>=1.21.1 in /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages (from kaggle) (1.24.3)
Requirement already satisfied: python-dateutil in /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages (from kaggle) (2.8.0)
Requirement already satisfied: requests in /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages (from kaggle) (2.23.0)
Requirement already satisfied: tqdm in /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages (from kaggle) (4.48.2)
Requirement already satisfied: certifi in /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages (from kaggle) (2020.6.20)
Requirement already satisfied: wheel>=0.26 in /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages (from tensorflow==1.14.0) (0.34.2)
Requirement already satisfied: wrapt>=1.11.1 in /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages (from tensorflow==1.14.0) (1.12.1)
Requirement already satisfied: protobuf>=3.6.1 in /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages (from tensorflow==1.14.0) (3.11.4)
Requirement already satisfied: grpcio>=1.8.6 in /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages (from tensorflow==1.14.0) (1.27.2)
Requirement already satisfied: google-pasta>=0.1.6 in /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages (from tensorflow==1.14.0) (0.2.0)
Requirement already satisfied: astor>=0.6.0 in /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages (from tensorflow==1.14.0) (0.8.0)
Requirement already satisfied: tensorboard<1.15.0,>=1.14.0 in /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages (from tensorflow==1.14.0) (1.14.0)
Requirement already satisfied: tensorflow-estimator<1.15.0rc0,>=1.14.0rc0 in /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages (from tensorflow==1.14.0) (1.14.0)
Requirement already satisfied: gast>=0.2.0 in /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages (from tensorflow==1.14.0) (0.3.3)
Requirement already satisfied: termcolor>=1.1.0 in /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages (from tensorflow==1.14.0) (1.1.0)
Requirement already satisfied: absl-py>=0.7.0 in /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages (from tensorflow==1.14.0) (0.9.0)
Requirement already satisfied: threadpoolctl>=2.0.0 in /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages (from scikit-learn) (2.1.0)
Requirement already satisfied: joblib>=0.11 in /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages (from scikit-learn) (0.16.0)
Requirement already satisfied: text-unidecode>=1.3 in /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages (from python-slugify->kaggle) (1.3)
Requirement already satisfied: chardet<4,>=3.0.2 in /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages (from requests->kaggle) (3.0.4)
Requirement already satisfied: idna<3,>=2.5 in /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages (from requests->kaggle) (2.9)
Requirement already satisfied: setuptools in /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages (from protobuf>=3.6.1->tensorflow==1.14.0) (46.2.0.post20200511)
Requirement already satisfied: markdown>=2.6.8 in /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages (from tensorboard<1.15.0,>=1.14.0->tensorflow==1.14.0) (3.1.1)
Requirement already satisfied: werkzeug>=0.11.15 in /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages (from tensorboard<1.15.0,>=1.14.0->tensorflow==1.14.0) (1.0.1)
In [2]:
import bentoml
import numpy as np
import pandas as pd
from keras.preprocessing import text, sequence
from keras.models import Sequential
from keras.layers import Dense, Dropout, Activation
from keras.layers import Embedding
from keras.layers import Conv1D, GlobalMaxPooling1D
from sklearn.model_selection import train_test_split
In [3]:
list_of_classes = ["toxic", "severe_toxic", "obscene", "threat", "insult", "identity_hate"]
max_features = 20000
max_text_length = 400
embedding_dims = 50
filters = 250
kernel_size = 3
hidden_dims = 250
batch_size = 32
epochs = 2

Prepare Dataset

Please Download data with Kaggle at https://www.kaggle.com/sarvajna/keras-sequential-model-lb-0-052/data

If you are running this notebook in Google Colab, fill in your kaggle credential below and download the training dataset from Kaggle:

In [4]:
%%bash

export KAGGLE_USERNAME=
export KAGGLE_KEY=

if [ ! -f ./train.csv.zip ]; then
    kaggle competitions download -c jigsaw-toxic-comment-classification-challenge
    unzip jigsaw-toxic-comment-classification-challenge.zip
    unzip train.csv.zip
    unzip sample_submission.csv.zip
    unzip test.csv.zip
    unzip test_labels.csv.zip
fi
In [5]:
train_df = pd.read_csv('./train.csv')

print(train_df.head())
                 id                                       comment_text  toxic  \
0  0000997932d777bf  Explanation\nWhy the edits made under my usern...      0   
1  000103f0d9cfb60f  D'aww! He matches this background colour I'm s...      0   
2  000113f07ec002fd  Hey man, I'm really not trying to edit war. It...      0   
3  0001b41b1c6bb37e  "\nMore\nI can't make any real suggestions on ...      0   
4  0001d958c54c6e35  You, sir, are my hero. Any chance you remember...      0   

   severe_toxic  obscene  threat  insult  identity_hate  
0             0        0       0       0              0  
1             0        0       0       0              0  
2             0        0       0       0              0  
3             0        0       0       0              0  
4             0        0       0       0              0  
In [6]:
x = train_df['comment_text'].values
print(x)
["Explanation\nWhy the edits made under my username Hardcore Metallica Fan were reverted? They weren't vandalisms, just closure on some GAs after I voted at New York Dolls FAC. And please don't remove the template from the talk page since I'm retired now.89.205.38.27"
 "D'aww! He matches this background colour I'm seemingly stuck with. Thanks.  (talk) 21:51, January 11, 2016 (UTC)"
 "Hey man, I'm really not trying to edit war. It's just that this guy is constantly removing relevant information and talking to me through edits instead of my talk page. He seems to care more about the formatting than the actual info."
 ...
 'Spitzer \n\nUmm, theres no actual article for prostitution ring.  - Crunch Captain.'
 'And it looks like it was actually you who put on the speedy to have the first version deleted now that I look at it.'
 '"\nAnd ... I really don\'t think you understand.  I came here and my idea was bad right away.  What kind of community goes ""you have bad ideas"" go away, instead of helping rewrite them.   "']
In [7]:
y = train_df[list_of_classes].values
print(y)
[[0 0 0 0 0 0]
 [0 0 0 0 0 0]
 [0 0 0 0 0 0]
 ...
 [0 0 0 0 0 0]
 [0 0 0 0 0 0]
 [0 0 0 0 0 0]]
In [8]:
x_tokenizer = text.Tokenizer(num_words=max_features)
print(x_tokenizer)
x_tokenizer.fit_on_texts(list(x))
print(x_tokenizer)
x_tokenized = x_tokenizer.texts_to_sequences(x) #list of lists(containing numbers), so basically a list of sequences, not a numpy array
#pad_sequences:transform a list of num_samples sequences (lists of scalars) into a 2D Numpy array of shape 
x_train_val = sequence.pad_sequences(x_tokenized, maxlen=max_text_length)
<keras_preprocessing.text.Tokenizer object at 0x7f645796fb38>
<keras_preprocessing.text.Tokenizer object at 0x7f645796fb38>
In [9]:
x_train, x_val, y_train, y_val = train_test_split(x_train_val, y, test_size=0.1, random_state=1)
In [10]:
print('Build model...')
model = Sequential()

# we start off with an efficient embedding layer which maps
# our vocab indices into embedding_dims dimensions
model.add(Embedding(max_features,
                    embedding_dims,
                    input_length=max_text_length))
model.add(Dropout(0.2))

# we add a Convolution1D, which will learn filters
# word group filters of size filter_length:
model.add(Conv1D(filters,
                 kernel_size,
                 padding='valid',
                 activation='relu',
                 strides=1))
# we use max pooling:
model.add(GlobalMaxPooling1D())

# We add a vanilla hidden layer:
model.add(Dense(hidden_dims))
model.add(Dropout(0.2))
model.add(Activation('relu'))

# We project onto 6 output layers, and squash it with a sigmoid:
model.add(Dense(6))
model.add(Activation('sigmoid'))

model.compile(loss='binary_crossentropy',
              optimizer='adam',
              metrics=['accuracy'])

model.summary()
Build model...
WARNING:tensorflow:From /opt/conda/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages/tensorflow/python/ops/nn_impl.py:180: add_dispatch_support.<locals>.wrapper (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.where in 2.0, which has the same broadcast rule as np.where
Model: "sequential_1"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
embedding_1 (Embedding)      (None, 400, 50)           1000000   
_________________________________________________________________
dropout_1 (Dropout)          (None, 400, 50)           0         
_________________________________________________________________
conv1d_1 (Conv1D)            (None, 398, 250)          37750     
_________________________________________________________________
global_max_pooling1d_1 (Glob (None, 250)               0         
_________________________________________________________________
dense_1 (Dense)              (None, 250)               62750     
_________________________________________________________________
dropout_2 (Dropout)          (None, 250)               0         
_________________________________________________________________
activation_1 (Activation)    (None, 250)               0         
_________________________________________________________________
dense_2 (Dense)              (None, 6)                 1506      
_________________________________________________________________
activation_2 (Activation)    (None, 6)                 0         
=================================================================
Total params: 1,102,006
Trainable params: 1,102,006
Non-trainable params: 0
_________________________________________________________________
In [11]:
model.fit(x_train, y_train,
          batch_size=batch_size,
          epochs=epochs,
validation_data=(x_val, y_val))
WARNING:tensorflow:From /opt/conda/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py:422: The name tf.global_variables is deprecated. Please use tf.compat.v1.global_variables instead.

Train on 143613 samples, validate on 15958 samples
Epoch 1/2
143613/143613 [==============================] - 32s 226us/step - loss: 0.0654 - accuracy: 0.9783 - val_loss: 0.0523 - val_accuracy: 0.9806
Epoch 2/2
143613/143613 [==============================] - 32s 219us/step - loss: 0.0463 - accuracy: 0.9827 - val_loss: 0.0490 - val_accuracy: 0.9813
Out[11]:
<keras.callbacks.callbacks.History at 0x7f645796fcf8>
In [12]:
test_df = pd.read_csv('./test.csv')
In [13]:
x_test = test_df['comment_text'].values
In [14]:
x_test_tokenized = x_tokenizer.texts_to_sequences(x_test)
x_testing = sequence.pad_sequences(x_test_tokenized, maxlen=max_text_length)
In [15]:
y_testing = model.predict(x_testing, verbose=1)
153164/153164 [==============================] - 7s 46us/step
In [16]:
sample_submission = pd.read_csv("./sample_submission.csv")
sample_submission[list_of_classes] = y_testing
sample_submission.to_csv("toxic_comment_classification.csv", index=False)

Create BentoService for model serving

In [17]:
%%writefile toxic_comment_classifier.py

from typing import List

from bentoml import api, artifacts, env, BentoService
from bentoml.frameworks.keras import KerasModelArtifact
from bentoml.service.artifacts.common import PickleArtifact
from bentoml.adapters import DataframeInput, JsonOutput

from keras.preprocessing import text, sequence
import numpy as np
import pandas as pd

list_of_classes = ["toxic", "severe_toxic", "obscene", "threat", "insult", "identity_hate"]
max_text_length = 400

@env(pip_packages=['tensorflow==1.14.0', 'keras==2.3.1', 'pandas', 'numpy'])
@artifacts([PickleArtifact('x_tokenizer'), KerasModelArtifact('model')])
class ToxicCommentClassification(BentoService):
    
    def tokenize_df(self, df):
        comments = df['comment_text'].values
        tokenized = self.artifacts.x_tokenizer.texts_to_sequences(comments)        
        input_data = sequence.pad_sequences(tokenized, maxlen=max_text_length)
        return input_data
    
    @api(input=DataframeInput(), output=JsonOutput(), batch=True)
    def predict(self, df: pd.DataFrame) -> List[str]:
        input_data = self.tokenize_df(df)
        prediction = self.artifacts.model.predict(input_data)
        result = []
        for i in prediction:
            result.append(list_of_classes[np.argmax(i)])
        return result
Overwriting toxic_comment_classifier.py

Save BentoService to file archive

In [18]:
# 1) import the custom BentoService defined above
from toxic_comment_classifier import ToxicCommentClassification

# 2) `pack` it with required artifacts
svc = ToxicCommentClassification()
svc.pack('x_tokenizer', x_tokenizer)
svc.pack('model', model)

# 3) save your BentoSerivce
saved_path = svc.save()
[2020-09-23 13:14:00,469] WARNING - Using BentoML installed in `editable` model, the local BentoML repository including all code changes will be packaged together with saved bundle created, under the './bundled_pip_dependencies' directory of the saved bundle.
[2020-09-23 13:14:00,495] WARNING - pip package requirement pandas already exist
[2020-09-23 13:14:00,496] WARNING - pip package requirement tensorflow already exist
WARNING:tensorflow:From /home/bentoml/BentoML/bentoml/frameworks/keras.py:123: The name tf.keras.backend.get_session is deprecated. Please use tf.compat.v1.keras.backend.get_session instead.

[2020-09-23 13:14:05,439] INFO - Detected non-PyPI-released BentoML installed, copying local BentoML modulefiles to target saved bundle path..
/opt/conda/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages/setuptools/dist.py:452: UserWarning: Normalizing '0.9.0.pre+7.g8af1c8b' to '0.9.0rc0+7.g8af1c8b'
  warnings.warn(tmpl.format(**locals()))
warning: no previously-included files matching '*~' found anywhere in distribution
warning: no previously-included files matching '*.pyo' found anywhere in distribution
warning: no previously-included files matching '.git' found anywhere in distribution
warning: no previously-included files matching '.ipynb_checkpoints' found anywhere in distribution
warning: no previously-included files matching '__pycache__' found anywhere in distribution
warning: no directories found matching 'bentoml/yatai/web/dist'
no previously-included directories found matching 'e2e_tests'
no previously-included directories found matching 'tests'
no previously-included directories found matching 'benchmark'
UPDATING BentoML-0.9.0rc0+7.g8af1c8b/bentoml/_version.py
set BentoML-0.9.0rc0+7.g8af1c8b/bentoml/_version.py to '0.9.0.pre+7.g8af1c8b'
[2020-09-23 13:14:06,420] INFO - BentoService bundle 'ToxicCommentClassification:20200923131400_B5BD19' saved to: /home/bentoml/bentoml/repository/ToxicCommentClassification/20200923131400_B5BD19

REST API Model Serving

To start a REST API model server with the BentoService saved above, use the bentoml serve command:

In [2]:
!bentoml serve ToxicCommentClassification:latest
[2020-09-23 13:16:17,781] INFO - Getting latest version ToxicCommentClassification:20200923131400_B5BD19
[2020-09-23 13:16:17,782] INFO - Starting BentoML API server in development mode..
[2020-09-23 13:16:18,964] WARNING - Using BentoML installed in `editable` model, the local BentoML repository including all code changes will be packaged together with saved bundle created, under the './bundled_pip_dependencies' directory of the saved bundle.
[2020-09-23 13:16:19,000] WARNING - Saved BentoService bundle version mismatch: loading BentoService bundle create with BentoML version 0.9.0.pre, but loading from BentoML version 0.9.0.pre+7.g8af1c8b
Using TensorFlow backend.
/opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:516: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint8 = np.dtype([("qint8", np.int8, 1)])
/opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:517: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_quint8 = np.dtype([("quint8", np.uint8, 1)])
/opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:518: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint16 = np.dtype([("qint16", np.int16, 1)])
/opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:519: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_quint16 = np.dtype([("quint16", np.uint16, 1)])
/opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:520: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint32 = np.dtype([("qint32", np.int32, 1)])
/opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:525: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  np_resource = np.dtype([("resource", np.ubyte, 1)])
/opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:541: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint8 = np.dtype([("qint8", np.int8, 1)])
/opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:542: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_quint8 = np.dtype([("quint8", np.uint8, 1)])
/opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:543: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint16 = np.dtype([("qint16", np.int16, 1)])
/opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:544: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_quint16 = np.dtype([("quint16", np.uint16, 1)])
/opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:545: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint32 = np.dtype([("qint32", np.int32, 1)])
/opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:550: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  np_resource = np.dtype([("resource", np.ubyte, 1)])
2020-09-23 13:16:20.256156: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcuda.so.1
2020-09-23 13:16:20.270836: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-09-23 13:16:20.271284: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 0 with properties: 
name: GeForce GTX 1060 major: 6 minor: 1 memoryClockRate(GHz): 1.6705
pciBusID: 0000:01:00.0
2020-09-23 13:16:20.271458: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.1
2020-09-23 13:16:20.272658: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10
2020-09-23 13:16:20.273814: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcufft.so.10
2020-09-23 13:16:20.274036: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcurand.so.10
2020-09-23 13:16:20.275219: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusolver.so.10
2020-09-23 13:16:20.275789: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusparse.so.10
2020-09-23 13:16:20.278736: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudnn.so.7
2020-09-23 13:16:20.278860: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-09-23 13:16:20.279267: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-09-23 13:16:20.279598: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1763] Adding visible gpu devices: 0
2020-09-23 13:16:20.279854: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.1 SSE4.2 AVX AVX2 FMA
2020-09-23 13:16:20.525314: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-09-23 13:16:20.525775: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x564d1c5f0a40 executing computations on platform CUDA. Devices:
2020-09-23 13:16:20.525812: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (0): GeForce GTX 1060, Compute Capability 6.1
2020-09-23 13:16:20.526033: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-09-23 13:16:20.526406: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 0 with properties: 
name: GeForce GTX 1060 major: 6 minor: 1 memoryClockRate(GHz): 1.6705
pciBusID: 0000:01:00.0
2020-09-23 13:16:20.526484: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.1
2020-09-23 13:16:20.526516: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10
2020-09-23 13:16:20.526531: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcufft.so.10
2020-09-23 13:16:20.526546: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcurand.so.10
2020-09-23 13:16:20.526560: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusolver.so.10
2020-09-23 13:16:20.526574: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusparse.so.10
2020-09-23 13:16:20.526588: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudnn.so.7
2020-09-23 13:16:20.526652: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-09-23 13:16:20.527115: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-09-23 13:16:20.527441: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1763] Adding visible gpu devices: 0
2020-09-23 13:16:20.527486: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.1
2020-09-23 13:16:20.527943: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1181] Device interconnect StreamExecutor with strength 1 edge matrix:
2020-09-23 13:16:20.527957: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1187]      0 
2020-09-23 13:16:20.527978: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 0:   N 
2020-09-23 13:16:20.528080: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-09-23 13:16:20.528473: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-09-23 13:16:20.528887: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 5683 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1060, pci bus id: 0000:01:00.0, compute capability: 6.1)
2020-09-23 13:16:20.552900: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2699905000 Hz
2020-09-23 13:16:20.553219: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x564d1fb9f0b0 executing computations on platform Host. Devices:
2020-09-23 13:16:20.553239: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (0): <undefined>, <undefined>
WARNING:tensorflow:From /home/bentoml/BentoML/bentoml/frameworks/keras.py:137: The name tf.keras.backend.set_session is deprecated. Please use tf.compat.v1.keras.backend.set_session instead.

WARNING:tensorflow:From /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages/tensorflow/python/ops/nn_impl.py:180: add_dispatch_support.<locals>.wrapper (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.where in 2.0, which has the same broadcast rule as np.where
WARNING:tensorflow:From /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py:422: The name tf.global_variables is deprecated. Please use tf.compat.v1.global_variables instead.

WARNING:tensorflow:From /home/bentoml/BentoML/bentoml/frameworks/keras.py:123: The name tf.keras.backend.get_session is deprecated. Please use tf.compat.v1.keras.backend.get_session instead.

[2020-09-23 13:16:21,729] WARNING - pip package requirement pandas already exist
[2020-09-23 13:16:21,729] WARNING - pip package requirement tensorflow already exist
[2020-09-23 13:16:21,730] WARNING - pip package requirement keras already exist
 * Serving Flask app "ToxicCommentClassification" (lazy loading)
 * Environment: production
   WARNING: This is a development server. Do not use it in a production deployment.
   Use a production WSGI server instead.
 * Debug mode: off
 * Running on http://127.0.0.1:5000/ (Press CTRL+C to quit)
2020-09-23 13:16:26.830789: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10
2020-09-23 13:16:26.972844: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudnn.so.7
[2020-09-23 13:16:27,521] INFO - {'service_name': 'ToxicCommentClassification', 'service_version': '20200923131400_B5BD19', 'api': 'predict', 'task': {'data': {}, 'task_id': '785e29f5-1998-4c07-a7a6-3271df9602f8', 'batch': 1, 'http_headers': (('Host', '127.0.0.1:5000'), ('User-Agent', 'curl/7.72.0'), ('Content-Length', '34'), ('Accept', '*/*'), ('Content-Type', 'application/json'))}, 'result': {'data': '["toxic"]', 'http_status': 200, 'http_headers': (('Content-Type', 'application/json'),)}, 'request_id': '785e29f5-1998-4c07-a7a6-3271df9602f8'}
127.0.0.1 - - [23/Sep/2020 13:16:27] "POST /predict HTTP/1.1" 200 -
[2020-09-23 13:16:28,907] INFO - {'service_name': 'ToxicCommentClassification', 'service_version': '20200923131400_B5BD19', 'api': 'predict', 'task': {'data': {}, 'task_id': 'b0152664-cded-4503-8863-dbf7718fe24a', 'batch': 1, 'http_headers': (('Host', '127.0.0.1:5000'), ('User-Agent', 'curl/7.72.0'), ('Content-Length', '34'), ('Accept', '*/*'), ('Content-Type', 'application/json'))}, 'result': {'data': '["toxic"]', 'http_status': 200, 'http_headers': (('Content-Type', 'application/json'),)}, 'request_id': 'b0152664-cded-4503-8863-dbf7718fe24a'}
127.0.0.1 - - [23/Sep/2020 13:16:28] "POST /predict HTTP/1.1" 200 -
[2020-09-23 13:16:30,344] INFO - {'service_name': 'ToxicCommentClassification', 'service_version': '20200923131400_B5BD19', 'api': 'predict', 'task': {'data': {}, 'task_id': 'a512827b-d48f-4842-aa31-29aadc3b88fc', 'batch': 1, 'http_headers': (('Host', '127.0.0.1:5000'), ('User-Agent', 'curl/7.72.0'), ('Content-Length', '34'), ('Accept', '*/*'), ('Content-Type', 'application/json'))}, 'result': {'data': '["toxic"]', 'http_status': 200, 'http_headers': (('Content-Type', 'application/json'),)}, 'request_id': 'a512827b-d48f-4842-aa31-29aadc3b88fc'}
127.0.0.1 - - [23/Sep/2020 13:16:30] "POST /predict HTTP/1.1" 200 -
^C

If you are running this notebook from Google Colab, you can start the dev server with --run-with-ngrok option, to gain acccess to the API endpoint via a public endpoint managed by ngrok:

In [ ]:
!bentoml serve ToxicCommentClassification:latest --run-with-ngrok

Open http://127.0.0.1:5000 to see more information about the REST APIs server in your browser.

Send prediction requeset to the REST API server

Run the following curl command to send the request to REST API server and get a prediction result:

curl -i \
    --request POST \
    --header "Content-Type: application/json" \
    --data '[{"comment_text": "bad terrible"}]' \
    localhost:5000/predict

Containerize model server with Docker

One common way of distributing this model API server for production deployment, is via Docker containers. And BentoML provides a convenient way to do that.

Note that docker is not available in Google Colab. You will need to download and run this notebook locally to try out this containerization with docker feature.

If you already have docker configured, simply run the follow command to product a docker container serving the IrisClassifier prediction service created above:

In [3]:
!bentoml containerize ToxicCommentClassification:latest -t toxiccommentclassification:latest
[2020-09-23 13:16:56,633] INFO - Getting latest version ToxicCommentClassification:20200923131400_B5BD19
Found Bento: /home/bentoml/bentoml/repository/ToxicCommentClassification/20200923131400_B5BD19
[2020-09-23 13:16:56,653] WARNING - Using BentoML installed in `editable` model, the local BentoML repository including all code changes will be packaged together with saved bundle created, under the './bundled_pip_dependencies' directory of the saved bundle.
[2020-09-23 13:16:56,669] WARNING - Saved BentoService bundle version mismatch: loading BentoService bundle create with BentoML version 0.9.0.pre, but loading from BentoML version 0.9.0.pre+7.g8af1c8b
Building Docker image toxiccommentclassification:latest from ToxicCommentClassification:latest 
/Step 1/15 : FROM bentoml/model-server:0.9.0.pre-py36
 ---> 4aac43d10e50
Step 2/15 : ARG EXTRA_PIP_INSTALL_ARGS=
 ---> Using cache
 ---> 790054f5ad85
Step 3/15 : ENV EXTRA_PIP_INSTALL_ARGS $EXTRA_PIP_INSTALL_ARGS
 ---> Using cache
 ---> 85b0a1b40542
Step 4/15 : COPY environment.yml requirements.txt setup.sh* bentoml-init.sh python_version* /bento/
\ ---> 1dd626947a9b
Step 5/15 : WORKDIR /bento
 ---> Running in 5a7d835de25b
- ---> 20bfe3bb7cfd
Step 6/15 : RUN chmod +x /bento/bentoml-init.sh
 ---> Running in 1cb9a7f52ced
\ ---> 0d47b6c3ae6f
Step 7/15 : RUN if [ -f /bento/bentoml-init.sh ]; then bash -c /bento/bentoml-init.sh; fi
 ---> Running in 1dc6c2bfac5c
\+++ dirname /bento/bentoml-init.sh

++ cd /bento
++ pwd -P

+ SAVED_BUNDLE_PATH=/bento
+ cd /bento
+ '[' -f ./setup.sh ']'
+ '[' -f ./python_version ']'

++ cat ./python_version

+ PY_VERSION_SAVED=3.6.10
+ DESIRED_PY_VERSION=3.6

++ python -c 'import sys; print(f"{sys.version_info.major}.{sys.version_info.minor}")'

+ CURRENT_PY_VERSION=3.6
+ [[ 3.6 == \3\.\6 ]]
+ echo 'Python Version in docker base image 3.6 matches requirement python=3.6. Skipping.'

Python Version in docker base image 3.6 matches requirement python=3.6. Skipping.
+ command -v conda
+ echo 'Updating conda base environment with environment.yml'
+ conda env update -n base -f ./environment.yml

Updating conda base environment with environment.yml
\Collecting package metadata (repodata.json): ...working... 
|done
Solving environment: ...working... 
/done
|
Downloading and Extracting Packages
certifi-2020.6.20    | 151 KB    |            |   0% 
certifi-2020.6.20    | 151 KB    | #          |  11% 
certifi-2020.6.20    | 151 KB    | ####2      |  42% 
certifi-2020.6.20    | 151 KB    | #####3     |  53% 
certifi-2020.6.20    | 151 KB    | ######3    |  64% 
certifi-2020.6.20    | 151 KB    | #######4   |  74% 
certifi-2020.6.20    | 151 KB    | ########## | 100% 

pip-20.2.3           | 1.1 MB    |            |   0% 
pip-20.2.3           | 1.1 MB    | 1          |   1% 
pip-20.2.3           | 1.1 MB    | #7         |  18% 
pip-20.2.3           | 1.1 MB    | ##7        |  28% 
pip-20.2.3           | 1.1 MB    | #######    |  70% 
pip-20.2.3           | 1.1 MB    | ########6  |  86% 
pip-20.2.3           | 1.1 MB    | ########## | 100% 
pip-20.2.3           | 1.1 MB    | ########## | 100% 

python_abi-3.6       | 4 KB      |            |   0% 
python_abi-3.6       | 4 KB      | ########## | 100% 
python_abi-3.6       | 4 KB      | ########## | 100% 

ca-certificates-2020 | 145 KB    |            |   0% 
ca-certificates-2020 | 145 KB    | #1         |  11% 
ca-certificates-2020 | 145 KB    | ##2        |  22% 
ca-certificates-2020 | 145 KB    | ########## | 100% 

openssl-1.1.1h       | 2.1 MB    |            |   0% 
openssl-1.1.1h       | 2.1 MB    |            |   1% 
openssl-1.1.1h       | 2.1 MB    | 2          |   2% 
openssl-1.1.1h       | 2.1 MB    | 7          |   7% 
openssl-1.1.1h       | 2.1 MB    | 9          |  10% 
openssl-1.1.1h       | 2.1 MB    | #1         |  11% 
openssl-1.1.1h       | 2.1 MB    | ##9        |  29% 
openssl-1.1.1h       | 2.1 MB    | ###6       |  36% 
openssl-1.1.1h       | 2.1 MB    | #####2     |  52% 
openssl-1.1.1h       | 2.1 MB    | #####8     |  59% 
openssl-1.1.1h       | 2.1 MB    | ######5    |  66% 
openssl-1.1.1h       | 2.1 MB    | #######5   |  76% 
openssl-1.1.1h       | 2.1 MB    | ########## | 100% 
openssl-1.1.1h       | 2.1 MB    | ########## | 100% 

Preparing transaction: ...working... 
/done
Verifying transaction: ...working... 
|done
Executing transaction: ...working... 
/done
|#
# To activate this environment, use
#
#     $ conda activate base
#
# To deactivate an active environment, use
#
#     $ conda deactivate
|+ pip install -r ./requirements.txt --no-cache-dir

/Requirement already satisfied: bentoml==0.9.0.pre in /opt/conda/lib/python3.6/site-packages (from -r ./requirements.txt (line 1)) (0.9.0rc0)
-Collecting tensorflow==1.14.0
-  Downloading tensorflow-1.14.0-cp36-cp36m-manylinux1_x86_64.whl (109.2 MB)
\Collecting keras==2.3.1
-  Downloading Keras-2.3.1-py2.py3-none-any.whl (377 kB)
/Collecting pandas==1.0.3
|  Downloading pandas-1.0.3-cp36-cp36m-manylinux1_x86_64.whl (10.0 MB)
\Requirement already satisfied: numpy==1.19.2 in /opt/conda/lib/python3.6/site-packages (from -r ./requirements.txt (line 5)) (1.19.2)
Requirement already satisfied: configparser in /opt/conda/lib/python3.6/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (5.0.0)
Requirement already satisfied: click>=7.0 in /opt/conda/lib/python3.6/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (7.1.2)
Requirement already satisfied: aiohttp in /opt/conda/lib/python3.6/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (3.6.2)
Requirement already satisfied: certifi in /opt/conda/lib/python3.6/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (2020.6.20)
Requirement already satisfied: dataclasses; python_version < "3.7" in /opt/conda/lib/python3.6/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (0.7)
Requirement already satisfied: requests in /opt/conda/lib/python3.6/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (2.24.0)
Requirement already satisfied: multidict in /opt/conda/lib/python3.6/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (4.7.6)
Requirement already satisfied: sqlalchemy>=1.3.0 in /opt/conda/lib/python3.6/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (1.3.19)
Requirement already satisfied: prometheus-client in /opt/conda/lib/python3.6/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (0.8.0)
Requirement already satisfied: docker in /opt/conda/lib/python3.6/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (4.3.1)
Requirement already satisfied: boto3 in /opt/conda/lib/python3.6/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (1.15.2)
-Requirement already satisfied: sqlalchemy-utils<0.36.8 in /opt/conda/lib/python3.6/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (0.36.7)
Requirement already satisfied: flask in /opt/conda/lib/python3.6/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (1.1.2)
Requirement already satisfied: alembic in /opt/conda/lib/python3.6/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (1.4.3)
Requirement already satisfied: python-dateutil<3.0.0,>=2.7.3 in /opt/conda/lib/python3.6/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (2.8.1)
Requirement already satisfied: humanfriendly in /opt/conda/lib/python3.6/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (8.2)
Requirement already satisfied: protobuf>=3.6.0 in /opt/conda/lib/python3.6/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (3.13.0)
Requirement already satisfied: python-json-logger in /opt/conda/lib/python3.6/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (0.1.11)
Requirement already satisfied: contextvars; python_version < "3.7" in /opt/conda/lib/python3.6/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (2.4)
Requirement already satisfied: tabulate in /opt/conda/lib/python3.6/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (0.8.7)
/Requirement already satisfied: cerberus in /opt/conda/lib/python3.6/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (1.3.2)
Requirement already satisfied: psutil in /opt/conda/lib/python3.6/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (5.7.2)
Requirement already satisfied: py-zipkin in /opt/conda/lib/python3.6/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (0.20.0)
Requirement already satisfied: grpcio<=1.27.2 in /opt/conda/lib/python3.6/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (1.27.2)
Requirement already satisfied: ruamel.yaml>=0.15.0 in /opt/conda/lib/python3.6/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (0.15.87)
Requirement already satisfied: gunicorn in /opt/conda/lib/python3.6/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (20.0.4)
Requirement already satisfied: packaging in /opt/conda/lib/python3.6/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (20.4)
|Collecting gast>=0.2.0
\  Downloading gast-0.4.0-py3-none-any.whl (9.8 kB)
-Requirement already satisfied: wheel>=0.26 in /opt/conda/lib/python3.6/site-packages (from tensorflow==1.14.0->-r ./requirements.txt (line 2)) (0.35.1)
Requirement already satisfied: six>=1.10.0 in /opt/conda/lib/python3.6/site-packages (from tensorflow==1.14.0->-r ./requirements.txt (line 2)) (1.15.0)
/Collecting keras-applications>=1.0.6
|  Downloading Keras_Applications-1.0.8-py3-none-any.whl (50 kB)
-Collecting absl-py>=0.7.0
/  Downloading absl_py-0.10.0-py3-none-any.whl (127 kB)
|Collecting termcolor>=1.1.0
\  Downloading termcolor-1.1.0.tar.gz (3.9 kB)
-Collecting wrapt>=1.11.1
  Downloading wrapt-1.12.1.tar.gz (27 kB)
/Collecting keras-preprocessing>=1.0.5
|  Downloading Keras_Preprocessing-1.1.2-py2.py3-none-any.whl (42 kB)
-Collecting tensorboard<1.15.0,>=1.14.0
/  Downloading tensorboard-1.14.0-py3-none-any.whl (3.1 MB)
-Collecting google-pasta>=0.1.6
  Downloading google_pasta-0.2.0-py3-none-any.whl (57 kB)
|Collecting astor>=0.6.0
\  Downloading astor-0.8.1-py2.py3-none-any.whl (27 kB)
/Collecting tensorflow-estimator<1.15.0rc0,>=1.14.0rc0
|  Downloading tensorflow_estimator-1.14.0-py2.py3-none-any.whl (488 kB)
|Collecting scipy>=0.14
\  Downloading scipy-1.5.2-cp36-cp36m-manylinux1_x86_64.whl (25.9 MB)
-Collecting h5py
/  Downloading h5py-2.10.0-cp36-cp36m-manylinux1_x86_64.whl (2.9 MB)
/Collecting pyyaml
  Downloading PyYAML-5.3.1.tar.gz (269 kB)
-Collecting pytz>=2017.2
/  Downloading pytz-2020.1-py2.py3-none-any.whl (510 kB)
|Requirement already satisfied: attrs>=17.3.0 in /opt/conda/lib/python3.6/site-packages (from aiohttp->bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (20.2.0)
Requirement already satisfied: async-timeout<4.0,>=3.0 in /opt/conda/lib/python3.6/site-packages (from aiohttp->bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (3.0.1)
Requirement already satisfied: idna-ssl>=1.0; python_version < "3.7" in /opt/conda/lib/python3.6/site-packages (from aiohttp->bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (1.1.0)
Requirement already satisfied: yarl<2.0,>=1.0 in /opt/conda/lib/python3.6/site-packages (from aiohttp->bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (1.5.1)
Requirement already satisfied: typing-extensions>=3.6.5; python_version < "3.7" in /opt/conda/lib/python3.6/site-packages (from aiohttp->bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (3.7.4.3)
Requirement already satisfied: chardet<4.0,>=2.0 in /opt/conda/lib/python3.6/site-packages (from aiohttp->bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (3.0.4)
Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /opt/conda/lib/python3.6/site-packages (from requests->bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (1.25.10)
Requirement already satisfied: idna<3,>=2.5 in /opt/conda/lib/python3.6/site-packages (from requests->bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (2.10)
Requirement already satisfied: websocket-client>=0.32.0 in /opt/conda/lib/python3.6/site-packages (from docker->bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (0.57.0)
Requirement already satisfied: s3transfer<0.4.0,>=0.3.0 in /opt/conda/lib/python3.6/site-packages (from boto3->bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (0.3.3)
Requirement already satisfied: jmespath<1.0.0,>=0.7.1 in /opt/conda/lib/python3.6/site-packages (from boto3->bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (0.10.0)
Requirement already satisfied: botocore<1.19.0,>=1.18.2 in /opt/conda/lib/python3.6/site-packages (from boto3->bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (1.18.2)
\Requirement already satisfied: itsdangerous>=0.24 in /opt/conda/lib/python3.6/site-packages (from flask->bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (1.1.0)
Requirement already satisfied: Werkzeug>=0.15 in /opt/conda/lib/python3.6/site-packages (from flask->bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (1.0.1)
Requirement already satisfied: Jinja2>=2.10.1 in /opt/conda/lib/python3.6/site-packages (from flask->bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (2.11.2)
Requirement already satisfied: python-editor>=0.3 in /opt/conda/lib/python3.6/site-packages (from alembic->bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (1.0.4)
Requirement already satisfied: Mako in /opt/conda/lib/python3.6/site-packages (from alembic->bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (1.1.3)
Requirement already satisfied: setuptools in /opt/conda/lib/python3.6/site-packages (from protobuf>=3.6.0->bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (49.6.0.post20200814)
Requirement already satisfied: immutables>=0.9 in /opt/conda/lib/python3.6/site-packages (from contextvars; python_version < "3.7"->bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (0.14)
Requirement already satisfied: thriftpy2>=0.4.0 in /opt/conda/lib/python3.6/site-packages (from py-zipkin->bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (0.4.11)
Requirement already satisfied: pyparsing>=2.0.2 in /opt/conda/lib/python3.6/site-packages (from packaging->bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (2.4.7)
/Collecting markdown>=2.6.8
|  Downloading Markdown-3.2.2-py3-none-any.whl (88 kB)
Requirement already satisfied: MarkupSafe>=0.23 in /opt/conda/lib/python3.6/site-packages (from Jinja2>=2.10.1->flask->bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (1.1.1)
Requirement already satisfied: ply<4.0,>=3.4 in /opt/conda/lib/python3.6/site-packages (from thriftpy2>=0.4.0->py-zipkin->bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (3.11)
-Collecting importlib-metadata; python_version < "3.8"
/  Downloading importlib_metadata-2.0.0-py2.py3-none-any.whl (31 kB)
\Collecting zipp>=0.5
-  Downloading zipp-3.2.0-py3-none-any.whl (5.1 kB)
/Building wheels for collected packages: termcolor, wrapt, pyyaml
  Building wheel for termcolor (setup.py): started
\  Building wheel for termcolor (setup.py): finished with status 'done'
  Created wheel for termcolor: filename=termcolor-1.1.0-py3-none-any.whl size=4830 sha256=5bd4d4d616fa311a9e18aabf0dbc032cc242f224c0acf975abeb83a93ba4e723
  Stored in directory: /tmp/pip-ephem-wheel-cache-r27l5jel/wheels/93/2a/eb/e58dbcbc963549ee4f065ff80a59f274cc7210b6eab962acdc
  Building wheel for wrapt (setup.py): started
-  Building wheel for wrapt (setup.py): finished with status 'done'
  Created wheel for wrapt: filename=wrapt-1.12.1-cp36-cp36m-linux_x86_64.whl size=75212 sha256=a2f276f4c7fa5a6cbfd623e7b08d6461dc9324d69be925cf215cce9a23addad0
  Stored in directory: /tmp/pip-ephem-wheel-cache-r27l5jel/wheels/32/42/7f/23cae9ff6ef66798d00dc5d659088e57dbba01566f6c60db63
  Building wheel for pyyaml (setup.py): started
\  Building wheel for pyyaml (setup.py): finished with status 'done'
  Created wheel for pyyaml: filename=PyYAML-5.3.1-cp36-cp36m-linux_x86_64.whl size=44619 sha256=d23ad27a50c47224b6ef9d5a710002cca27b45b2794dfbd3d03106012c0b7873
  Stored in directory: /tmp/pip-ephem-wheel-cache-r27l5jel/wheels/e5/9d/ad/2ee53cf262cba1ffd8afe1487eef788ea3f260b7e6232a80fc
Successfully built termcolor wrapt pyyaml
|Installing collected packages: gast, h5py, keras-applications, absl-py, termcolor, wrapt, keras-preprocessing, zipp, importlib-metadata, markdown, tensorboard, google-pasta, astor, tensorflow-estimator, tensorflow, scipy, pyyaml, keras, pytz, pandas
|Successfully installed absl-py-0.10.0 astor-0.8.1 gast-0.4.0 google-pasta-0.2.0 h5py-2.10.0 importlib-metadata-2.0.0 keras-2.3.1 keras-applications-1.0.8 keras-preprocessing-1.1.2 markdown-3.2.2 pandas-1.0.3 pytz-2020.1 pyyaml-5.3.1 scipy-1.5.2 tensorboard-1.14.0 tensorflow-1.14.0 tensorflow-estimator-1.14.0 termcolor-1.1.0 wrapt-1.12.1 zipp-3.2.0
/ ---> c70dd9411d8d
Step 8/15 : COPY . /bento
| ---> 1e081b13cb25
Step 9/15 : RUN if [ -d /bento/bundled_pip_dependencies ]; then pip install -U bundled_pip_dependencies/* ;fi
\ ---> Running in 4e337f4ff3e3
-Processing ./bundled_pip_dependencies/BentoML-0.9.0rc0+7.g8af1c8b.tar.gz
  Installing build dependencies: started
|  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
|  Getting requirements to build wheel: finished with status 'done'
    Preparing wheel metadata: started
\    Preparing wheel metadata: finished with status 'done'
-Requirement already satisfied, skipping upgrade: dataclasses; python_version < "3.7" in /opt/conda/lib/python3.6/site-packages (from BentoML==0.9.0rc0+7.g8af1c8b) (0.7)
Requirement already satisfied, skipping upgrade: contextvars; python_version < "3.7" in /opt/conda/lib/python3.6/site-packages (from BentoML==0.9.0rc0+7.g8af1c8b) (2.4)
Requirement already satisfied, skipping upgrade: packaging in /opt/conda/lib/python3.6/site-packages (from BentoML==0.9.0rc0+7.g8af1c8b) (20.4)
Requirement already satisfied, skipping upgrade: cerberus in /opt/conda/lib/python3.6/site-packages (from BentoML==0.9.0rc0+7.g8af1c8b) (1.3.2)
Requirement already satisfied, skipping upgrade: requests in /opt/conda/lib/python3.6/site-packages (from BentoML==0.9.0rc0+7.g8af1c8b) (2.24.0)
/Requirement already satisfied, skipping upgrade: python-json-logger in /opt/conda/lib/python3.6/site-packages (from BentoML==0.9.0rc0+7.g8af1c8b) (0.1.11)
Requirement already satisfied, skipping upgrade: multidict in /opt/conda/lib/python3.6/site-packages (from BentoML==0.9.0rc0+7.g8af1c8b) (4.7.6)
Requirement already satisfied, skipping upgrade: python-dateutil<3.0.0,>=2.7.3 in /opt/conda/lib/python3.6/site-packages (from BentoML==0.9.0rc0+7.g8af1c8b) (2.8.1)
Requirement already satisfied, skipping upgrade: click>=7.0 in /opt/conda/lib/python3.6/site-packages (from BentoML==0.9.0rc0+7.g8af1c8b) (7.1.2)
Requirement already satisfied, skipping upgrade: sqlalchemy-utils<0.36.8 in /opt/conda/lib/python3.6/site-packages (from BentoML==0.9.0rc0+7.g8af1c8b) (0.36.7)
Requirement already satisfied, skipping upgrade: prometheus-client in /opt/conda/lib/python3.6/site-packages (from BentoML==0.9.0rc0+7.g8af1c8b) (0.8.0)
Requirement already satisfied, skipping upgrade: numpy in /opt/conda/lib/python3.6/site-packages (from BentoML==0.9.0rc0+7.g8af1c8b) (1.19.2)
Requirement already satisfied, skipping upgrade: configparser in /opt/conda/lib/python3.6/site-packages (from BentoML==0.9.0rc0+7.g8af1c8b) (5.0.0)
Requirement already satisfied, skipping upgrade: boto3 in /opt/conda/lib/python3.6/site-packages (from BentoML==0.9.0rc0+7.g8af1c8b) (1.15.2)
Requirement already satisfied, skipping upgrade: humanfriendly in /opt/conda/lib/python3.6/site-packages (from BentoML==0.9.0rc0+7.g8af1c8b) (8.2)
|Requirement already satisfied, skipping upgrade: ruamel.yaml>=0.15.0 in /opt/conda/lib/python3.6/site-packages (from BentoML==0.9.0rc0+7.g8af1c8b) (0.15.87)
Requirement already satisfied, skipping upgrade: certifi in /opt/conda/lib/python3.6/site-packages (from BentoML==0.9.0rc0+7.g8af1c8b) (2020.6.20)
Requirement already satisfied, skipping upgrade: docker in /opt/conda/lib/python3.6/site-packages (from BentoML==0.9.0rc0+7.g8af1c8b) (4.3.1)
Requirement already satisfied, skipping upgrade: aiohttp in /opt/conda/lib/python3.6/site-packages (from BentoML==0.9.0rc0+7.g8af1c8b) (3.6.2)
Requirement already satisfied, skipping upgrade: py-zipkin in /opt/conda/lib/python3.6/site-packages (from BentoML==0.9.0rc0+7.g8af1c8b) (0.20.0)
Requirement already satisfied, skipping upgrade: protobuf>=3.6.0 in /opt/conda/lib/python3.6/site-packages (from BentoML==0.9.0rc0+7.g8af1c8b) (3.13.0)
Requirement already satisfied, skipping upgrade: grpcio<=1.27.2 in /opt/conda/lib/python3.6/site-packages (from BentoML==0.9.0rc0+7.g8af1c8b) (1.27.2)
Requirement already satisfied, skipping upgrade: sqlalchemy>=1.3.0 in /opt/conda/lib/python3.6/site-packages (from BentoML==0.9.0rc0+7.g8af1c8b) (1.3.19)
Requirement already satisfied, skipping upgrade: gunicorn in /opt/conda/lib/python3.6/site-packages (from BentoML==0.9.0rc0+7.g8af1c8b) (20.0.4)
Requirement already satisfied, skipping upgrade: flask in /opt/conda/lib/python3.6/site-packages (from BentoML==0.9.0rc0+7.g8af1c8b) (1.1.2)
Requirement already satisfied, skipping upgrade: alembic in /opt/conda/lib/python3.6/site-packages (from BentoML==0.9.0rc0+7.g8af1c8b) (1.4.3)
Requirement already satisfied, skipping upgrade: tabulate in /opt/conda/lib/python3.6/site-packages (from BentoML==0.9.0rc0+7.g8af1c8b) (0.8.7)
Requirement already satisfied, skipping upgrade: psutil in /opt/conda/lib/python3.6/site-packages (from BentoML==0.9.0rc0+7.g8af1c8b) (5.7.2)
\Requirement already satisfied, skipping upgrade: immutables>=0.9 in /opt/conda/lib/python3.6/site-packages (from contextvars; python_version < "3.7"->BentoML==0.9.0rc0+7.g8af1c8b) (0.14)
Requirement already satisfied, skipping upgrade: six in /opt/conda/lib/python3.6/site-packages (from packaging->BentoML==0.9.0rc0+7.g8af1c8b) (1.15.0)
Requirement already satisfied, skipping upgrade: pyparsing>=2.0.2 in /opt/conda/lib/python3.6/site-packages (from packaging->BentoML==0.9.0rc0+7.g8af1c8b) (2.4.7)
Requirement already satisfied, skipping upgrade: setuptools in /opt/conda/lib/python3.6/site-packages (from cerberus->BentoML==0.9.0rc0+7.g8af1c8b) (49.6.0.post20200814)
Requirement already satisfied, skipping upgrade: idna<3,>=2.5 in /opt/conda/lib/python3.6/site-packages (from requests->BentoML==0.9.0rc0+7.g8af1c8b) (2.10)
Requirement already satisfied, skipping upgrade: chardet<4,>=3.0.2 in /opt/conda/lib/python3.6/site-packages (from requests->BentoML==0.9.0rc0+7.g8af1c8b) (3.0.4)
Requirement already satisfied, skipping upgrade: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /opt/conda/lib/python3.6/site-packages (from requests->BentoML==0.9.0rc0+7.g8af1c8b) (1.25.10)
Requirement already satisfied, skipping upgrade: botocore<1.19.0,>=1.18.2 in /opt/conda/lib/python3.6/site-packages (from boto3->BentoML==0.9.0rc0+7.g8af1c8b) (1.18.2)
Requirement already satisfied, skipping upgrade: jmespath<1.0.0,>=0.7.1 in /opt/conda/lib/python3.6/site-packages (from boto3->BentoML==0.9.0rc0+7.g8af1c8b) (0.10.0)
Requirement already satisfied, skipping upgrade: s3transfer<0.4.0,>=0.3.0 in /opt/conda/lib/python3.6/site-packages (from boto3->BentoML==0.9.0rc0+7.g8af1c8b) (0.3.3)
Requirement already satisfied, skipping upgrade: websocket-client>=0.32.0 in /opt/conda/lib/python3.6/site-packages (from docker->BentoML==0.9.0rc0+7.g8af1c8b) (0.57.0)
Requirement already satisfied, skipping upgrade: yarl<2.0,>=1.0 in /opt/conda/lib/python3.6/site-packages (from aiohttp->BentoML==0.9.0rc0+7.g8af1c8b) (1.5.1)
Requirement already satisfied, skipping upgrade: attrs>=17.3.0 in /opt/conda/lib/python3.6/site-packages (from aiohttp->BentoML==0.9.0rc0+7.g8af1c8b) (20.2.0)
Requirement already satisfied, skipping upgrade: typing-extensions>=3.6.5; python_version < "3.7" in /opt/conda/lib/python3.6/site-packages (from aiohttp->BentoML==0.9.0rc0+7.g8af1c8b) (3.7.4.3)
Requirement already satisfied, skipping upgrade: idna-ssl>=1.0; python_version < "3.7" in /opt/conda/lib/python3.6/site-packages (from aiohttp->BentoML==0.9.0rc0+7.g8af1c8b) (1.1.0)
Requirement already satisfied, skipping upgrade: async-timeout<4.0,>=3.0 in /opt/conda/lib/python3.6/site-packages (from aiohttp->BentoML==0.9.0rc0+7.g8af1c8b) (3.0.1)
Requirement already satisfied, skipping upgrade: thriftpy2>=0.4.0 in /opt/conda/lib/python3.6/site-packages (from py-zipkin->BentoML==0.9.0rc0+7.g8af1c8b) (0.4.11)
Requirement already satisfied, skipping upgrade: Jinja2>=2.10.1 in /opt/conda/lib/python3.6/site-packages (from flask->BentoML==0.9.0rc0+7.g8af1c8b) (2.11.2)
Requirement already satisfied, skipping upgrade: itsdangerous>=0.24 in /opt/conda/lib/python3.6/site-packages (from flask->BentoML==0.9.0rc0+7.g8af1c8b) (1.1.0)
-Requirement already satisfied, skipping upgrade: Werkzeug>=0.15 in /opt/conda/lib/python3.6/site-packages (from flask->BentoML==0.9.0rc0+7.g8af1c8b) (1.0.1)
Requirement already satisfied, skipping upgrade: Mako in /opt/conda/lib/python3.6/site-packages (from alembic->BentoML==0.9.0rc0+7.g8af1c8b) (1.1.3)
Requirement already satisfied, skipping upgrade: python-editor>=0.3 in /opt/conda/lib/python3.6/site-packages (from alembic->BentoML==0.9.0rc0+7.g8af1c8b) (1.0.4)
Requirement already satisfied, skipping upgrade: ply<4.0,>=3.4 in /opt/conda/lib/python3.6/site-packages (from thriftpy2>=0.4.0->py-zipkin->BentoML==0.9.0rc0+7.g8af1c8b) (3.11)
Requirement already satisfied, skipping upgrade: MarkupSafe>=0.23 in /opt/conda/lib/python3.6/site-packages (from Jinja2>=2.10.1->flask->BentoML==0.9.0rc0+7.g8af1c8b) (1.1.1)
Building wheels for collected packages: BentoML
  Building wheel for BentoML (PEP 517): started
\  Building wheel for BentoML (PEP 517): finished with status 'done'
  Created wheel for BentoML: filename=BentoML-0.9.0rc0+7.g8af1c8b-py3-none-any.whl size=620483 sha256=227fa6cf2cbe0ad404e2ca4e7d953ec634b4f697a731cc9835d5435e214fe98d
  Stored in directory: /root/.cache/pip/wheels/0f/c2/e2/64640eb406a012df3abf685e4e5fa441b87b3e609903d041b6
Successfully built BentoML
\Installing collected packages: BentoML
  Attempting uninstall: BentoML
    Found existing installation: BentoML 0.9.0rc0
-    Uninstalling BentoML-0.9.0rc0:
|      Successfully uninstalled BentoML-0.9.0rc0
/Successfully installed BentoML-0.9.0rc0+7.g8af1c8b
- ---> 4a97a40c390e
Step 10/15 : ENV PORT 5000
/ ---> Running in 6ce57e3db900
| ---> 4850998329db
Step 11/15 : EXPOSE $PORT
\ ---> Running in fdd4206ef8e3
- ---> 63182a42fd56
Step 12/15 : COPY docker-entrypoint.sh /usr/local/bin/
| ---> 6153d5b74653
Step 13/15 : RUN chmod +x /usr/local/bin/docker-entrypoint.sh
 ---> Running in b3824625907f
- ---> 97a5684fbee4
Step 14/15 : ENTRYPOINT [ "docker-entrypoint.sh" ]
/ ---> Running in c3028015d016
| ---> 9122a9474fe3
Step 15/15 : CMD ["bentoml", "serve-gunicorn", "/bento"]
\ ---> Running in f016847f562f
- ---> 2fe1155484ac
Successfully built 2fe1155484ac
Successfully tagged toxiccommentclassification:latest
Finished building toxiccommentclassification:latest from ToxicCommentClassification:latest
In [4]:
!docker run -p 5000:5000 toxiccommentclassification --workers 1 --enable-microbatch
[2020-09-23 05:19:42,011] INFO - Starting BentoML API server in production mode..
[2020-09-23 05:19:42,239] INFO - Running micro batch service on :5000
[2020-09-23 05:19:42 +0000] [12] [INFO] Starting gunicorn 20.0.4
[2020-09-23 05:19:42 +0000] [1] [INFO] Starting gunicorn 20.0.4
[2020-09-23 05:19:42 +0000] [12] [INFO] Listening at: http://0.0.0.0:5000 (12)
[2020-09-23 05:19:42 +0000] [12] [INFO] Using worker: aiohttp.worker.GunicornWebWorker
[2020-09-23 05:19:42 +0000] [1] [INFO] Listening at: http://0.0.0.0:55205 (1)
[2020-09-23 05:19:42 +0000] [1] [INFO] Using worker: sync
[2020-09-23 05:19:42 +0000] [14] [INFO] Booting worker with pid: 14
[2020-09-23 05:19:42 +0000] [13] [INFO] Booting worker with pid: 13
[2020-09-23 05:19:42,266] WARNING - Using BentoML not from official PyPI release. In order to find the same version of BentoML when deploying your BentoService, you must set the 'core/bentoml_deploy_version' config to a http/git location of your BentoML fork, e.g.: 'bentoml_deploy_version = git+https://github.com/{username}/[email protected]{branch}'
[2020-09-23 05:19:42,286] WARNING - Saved BentoService bundle version mismatch: loading BentoService bundle create with BentoML version 0.9.0.pre, but loading from BentoML version 0.9.0.pre+7.g8af1c8b
[2020-09-23 05:19:42,286] WARNING - Saved BentoService Python version mismatch: loading BentoService bundle created with Python version 3.6.10, but current environment version is 3.6.12.
[2020-09-23 05:19:42,291] INFO - Micro batch enabled for API `predict`
[2020-09-23 05:19:42,291] INFO - Your system nofile limit is 1048576, which means each instance of microbatch service is able to hold this number of connections at same time. You can increase the number of file descriptors for the server process, or launch more microbatch instances to accept more concurrent connection.
[2020-09-23 05:19:43,263] WARNING - Using BentoML not from official PyPI release. In order to find the same version of BentoML when deploying your BentoService, you must set the 'core/bentoml_deploy_version' config to a http/git location of your BentoML fork, e.g.: 'bentoml_deploy_version = git+https://github.com/{username}/[email protected]{branch}'
[2020-09-23 05:19:43,284] WARNING - Saved BentoService bundle version mismatch: loading BentoService bundle create with BentoML version 0.9.0.pre, but loading from BentoML version 0.9.0.pre+7.g8af1c8b
[2020-09-23 05:19:43,285] WARNING - Saved BentoService Python version mismatch: loading BentoService bundle created with Python version 3.6.10, but current environment version is 3.6.12.
2020-09-23 05:19:44.816335: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2020-09-23 05:19:44.837027: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2699905000 Hz
2020-09-23 05:19:44.837461: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x56193249be60 executing computations on platform Host. Devices:
2020-09-23 05:19:44.837520: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (0): <undefined>, <undefined>
Using TensorFlow backend.
/opt/conda/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:516: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint8 = np.dtype([("qint8", np.int8, 1)])
/opt/conda/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:517: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_quint8 = np.dtype([("quint8", np.uint8, 1)])
/opt/conda/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:518: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint16 = np.dtype([("qint16", np.int16, 1)])
/opt/conda/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:519: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_quint16 = np.dtype([("quint16", np.uint16, 1)])
/opt/conda/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:520: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint32 = np.dtype([("qint32", np.int32, 1)])
/opt/conda/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:525: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  np_resource = np.dtype([("resource", np.ubyte, 1)])
/opt/conda/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:541: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint8 = np.dtype([("qint8", np.int8, 1)])
/opt/conda/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:542: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_quint8 = np.dtype([("quint8", np.uint8, 1)])
/opt/conda/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:543: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint16 = np.dtype([("qint16", np.int16, 1)])
/opt/conda/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:544: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_quint16 = np.dtype([("quint16", np.uint16, 1)])
/opt/conda/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:545: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint32 = np.dtype([("qint32", np.int32, 1)])
/opt/conda/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:550: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  np_resource = np.dtype([("resource", np.ubyte, 1)])
WARNING:tensorflow:From /opt/conda/lib/python3.6/site-packages/bentoml/frameworks/keras.py:137: The name tf.keras.backend.set_session is deprecated. Please use tf.compat.v1.keras.backend.set_session instead.

2020-09-23 05:19:44.976531: W tensorflow/compiler/jit/mark_for_compilation_pass.cc:1412] (One-time warning): Not using XLA:CPU for cluster because envvar TF_XLA_FLAGS=--tf_xla_cpu_global_jit was not set.  If you want XLA:CPU, either set that envvar, or use experimental_jit_scope to enable XLA:CPU.  To confirm that XLA is active, pass --vmodule=xla_compilation_cache=1 (as a proper command-line flag, not via TF_XLA_FLAGS) or set the envvar XLA_FLAGS=--xla_hlo_profile.
WARNING:tensorflow:From /opt/conda/lib/python3.6/site-packages/tensorflow/python/ops/nn_impl.py:180: add_dispatch_support.<locals>.wrapper (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.where in 2.0, which has the same broadcast rule as np.where
WARNING:tensorflow:From /opt/conda/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py:422: The name tf.global_variables is deprecated. Please use tf.compat.v1.global_variables instead.

WARNING:tensorflow:From /opt/conda/lib/python3.6/site-packages/bentoml/frameworks/keras.py:123: The name tf.keras.backend.get_session is deprecated. Please use tf.compat.v1.keras.backend.get_session instead.

[2020-09-23 05:19:45,661] WARNING - pip package requirement pandas already exist
[2020-09-23 05:19:45,661] WARNING - pip package requirement tensorflow already exist
[2020-09-23 05:19:45,661] WARNING - pip package requirement keras already exist
[2020-09-23 05:19:59,931] INFO - {'service_name': 'ToxicCommentClassification', 'service_version': '20200923131400_B5BD19', 'api': 'predict', 'task': {'data': {}, 'task_id': 'aec6fe13-d1c7-47e0-b148-563b96e14662', 'batch': 1, 'http_headers': (('Host', '127.0.0.1:5000'), ('User-Agent', 'curl/7.72.0'), ('Content-Length', '34'), ('Accept', '*/*'), ('Content-Type', 'application/json'))}, 'result': {'data': '["toxic"]', 'http_status': 200, 'http_headers': (('Content-Type', 'application/json'),)}, 'request_id': 'aec6fe13-d1c7-47e0-b148-563b96e14662'}
^C
[2020-09-23 05:20:03 +0000] [1] [INFO] Handling signal: int
[2020-09-23 05:20:03 +0000] [14] [INFO] Worker exiting (pid: 14)

Load saved BentoService

bentoml.load is the API for loading a BentoML packaged model in python:

In [27]:
sample_test = test_df.iloc[40:42]
print(sample_test)
bento_service = bentoml.load(saved_path)

print(bento_service.predict(sample_test))
                  id                                       comment_text
40  0011cefc680993ba                      REDIRECT Talk:Mi Vida Eres TĂș
41  0011ef6aa33d42e6  " \n I'm not convinced that he was blind. Wher...
[2020-08-04 15:59:10,504] WARNING - Saved BentoService bundle version mismatch: loading BentoService bundle create with BentoML version 0.8.3, but loading from BentoML version 0.8.3+49.gdcc2e8b
[2020-08-04 15:59:10,505] WARNING - Module `toxic_comment_classifier` already loaded, using existing imported module.
WARNING:tensorflow:From /home/bentoml/BentoML/bentoml/artifact/keras_model_artifact.py:151: The name tf.keras.backend.set_session is deprecated. Please use tf.compat.v1.keras.backend.set_session instead.

['toxic', 'toxic']

Launch inference job from CLI

BentoML cli supports loading and running a packaged model from CLI. With the DataframeInput adapter, the CLI command supports reading input Dataframe data from CLI argument or local csv or json files:

In [5]:
!bentoml run ToxicCommentClassification:latest predict --input '[{"comment_text": "bad terrible"}]'
[2020-09-23 13:20:19,174] INFO - Getting latest version ToxicCommentClassification:20200923131400_B5BD19
[2020-09-23 13:20:19,564] WARNING - Using BentoML installed in `editable` model, the local BentoML repository including all code changes will be packaged together with saved bundle created, under the './bundled_pip_dependencies' directory of the saved bundle.
[2020-09-23 13:20:19,583] WARNING - Saved BentoService bundle version mismatch: loading BentoService bundle create with BentoML version 0.9.0.pre, but loading from BentoML version 0.9.0.pre+7.g8af1c8b
Using TensorFlow backend.
/opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:516: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint8 = np.dtype([("qint8", np.int8, 1)])
/opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:517: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_quint8 = np.dtype([("quint8", np.uint8, 1)])
/opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:518: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint16 = np.dtype([("qint16", np.int16, 1)])
/opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:519: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_quint16 = np.dtype([("quint16", np.uint16, 1)])
/opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:520: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint32 = np.dtype([("qint32", np.int32, 1)])
/opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:525: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  np_resource = np.dtype([("resource", np.ubyte, 1)])
/opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:541: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint8 = np.dtype([("qint8", np.int8, 1)])
/opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:542: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_quint8 = np.dtype([("quint8", np.uint8, 1)])
/opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:543: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint16 = np.dtype([("qint16", np.int16, 1)])
/opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:544: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_quint16 = np.dtype([("quint16", np.uint16, 1)])
/opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:545: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint32 = np.dtype([("qint32", np.int32, 1)])
/opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:550: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  np_resource = np.dtype([("resource", np.ubyte, 1)])
2020-09-23 13:20:20.862658: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcuda.so.1
2020-09-23 13:20:20.878224: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-09-23 13:20:20.878600: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 0 with properties: 
name: GeForce GTX 1060 major: 6 minor: 1 memoryClockRate(GHz): 1.6705
pciBusID: 0000:01:00.0
2020-09-23 13:20:20.878772: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.1
2020-09-23 13:20:20.880168: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10
2020-09-23 13:20:20.881398: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcufft.so.10
2020-09-23 13:20:20.881633: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcurand.so.10
2020-09-23 13:20:20.882788: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusolver.so.10
2020-09-23 13:20:20.883356: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusparse.so.10
2020-09-23 13:20:20.885821: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudnn.so.7
2020-09-23 13:20:20.885944: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-09-23 13:20:20.887855: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-09-23 13:20:20.888394: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1763] Adding visible gpu devices: 0
2020-09-23 13:20:20.888653: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.1 SSE4.2 AVX AVX2 FMA
2020-09-23 13:20:21.180837: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-09-23 13:20:21.181260: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x557a9136fce0 executing computations on platform CUDA. Devices:
2020-09-23 13:20:21.181283: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (0): GeForce GTX 1060, Compute Capability 6.1
2020-09-23 13:20:21.181482: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-09-23 13:20:21.181839: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 0 with properties: 
name: GeForce GTX 1060 major: 6 minor: 1 memoryClockRate(GHz): 1.6705
pciBusID: 0000:01:00.0
2020-09-23 13:20:21.181890: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.1
2020-09-23 13:20:21.181932: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10
2020-09-23 13:20:21.181946: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcufft.so.10
2020-09-23 13:20:21.181959: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcurand.so.10
2020-09-23 13:20:21.181999: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusolver.so.10
2020-09-23 13:20:21.182012: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusparse.so.10
2020-09-23 13:20:21.182038: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudnn.so.7
2020-09-23 13:20:21.182114: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-09-23 13:20:21.182481: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-09-23 13:20:21.182825: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1763] Adding visible gpu devices: 0
2020-09-23 13:20:21.182863: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.1
2020-09-23 13:20:21.183322: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1181] Device interconnect StreamExecutor with strength 1 edge matrix:
2020-09-23 13:20:21.183335: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1187]      0 
2020-09-23 13:20:21.183359: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 0:   N 
2020-09-23 13:20:21.183447: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-09-23 13:20:21.183792: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-09-23 13:20:21.184160: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 5683 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1060, pci bus id: 0000:01:00.0, compute capability: 6.1)
2020-09-23 13:20:21.204949: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2699905000 Hz
2020-09-23 13:20:21.205223: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x557a947a0590 executing computations on platform Host. Devices:
2020-09-23 13:20:21.205267: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (0): <undefined>, <undefined>
WARNING:tensorflow:From /home/bentoml/BentoML/bentoml/frameworks/keras.py:137: The name tf.keras.backend.set_session is deprecated. Please use tf.compat.v1.keras.backend.set_session instead.

WARNING:tensorflow:From /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages/tensorflow/python/ops/nn_impl.py:180: add_dispatch_support.<locals>.wrapper (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.where in 2.0, which has the same broadcast rule as np.where
WARNING:tensorflow:From /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py:422: The name tf.global_variables is deprecated. Please use tf.compat.v1.global_variables instead.

WARNING:tensorflow:From /home/bentoml/BentoML/bentoml/frameworks/keras.py:123: The name tf.keras.backend.get_session is deprecated. Please use tf.compat.v1.keras.backend.get_session instead.

[2020-09-23 13:20:22,320] WARNING - pip package requirement pandas already exist
[2020-09-23 13:20:22,321] WARNING - pip package requirement tensorflow already exist
[2020-09-23 13:20:22,321] WARNING - pip package requirement keras already exist
2020-09-23 13:20:22.776318: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10
2020-09-23 13:20:22.926447: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudnn.so.7
[2020-09-23 13:20:23,489] INFO - {'service_name': 'ToxicCommentClassification', 'service_version': '20200923131400_B5BD19', 'api': 'predict', 'task': {'data': {}, 'task_id': '68fffd50-0e72-4cfb-8a0e-39fd7dd4b3e7', 'batch': 1, 'cli_args': ('--input', '[{"comment_text": "bad terrible"}]')}, 'result': {'data': '["toxic"]', 'http_status': 200, 'http_headers': (('Content-Type', 'application/json'),)}, 'request_id': '68fffd50-0e72-4cfb-8a0e-39fd7dd4b3e7'}
["toxic"]

Deployment Options

If you are at a small team with limited engineering or DevOps resources, try out automated deployment with BentoML CLI, currently supporting AWS Lambda, AWS SageMaker, and Azure Functions:

If the cloud platform you are working with is not on the list above, try out these step-by-step guide on manually deploying BentoML packaged model to cloud platforms:

Lastly, if you have a DevOps or ML Engineering team who's operating a Kubernetes or OpenShift cluster, use the following guides as references for implementating your deployment strategy:

In [ ]: