BentoML Example: Keras Fashion MNIST

BentoML makes moving trained ML models to production easy:

  • Package models trained with any ML framework and reproduce them for model serving in production
  • Deploy anywhere for online API serving or offline batch serving
  • High-Performance API model server with adaptive micro-batching support
  • Central hub for managing models and deployment process via Web UI and APIs
  • Modular and flexible design making it adaptable to your infrastrcuture

BentoML is a framework for serving, managing, and deploying machine learning models. It is aiming to bridge the gap between Data Science and DevOps, and enable teams to deliver prediction services in a fast, repeatable, and scalable way.

Before reading this example project, be sure to check out the Getting started guide to learn about the basic concepts in BentoML.

This notebook demonstrates how to use BentoML to turn a Keras model into a docker image containing a REST API server serving this model, how to use your ML service built with BentoML as a CLI tool, and how to distribute it a pypi package.

Impression

In [1]:
%reload_ext autoreload
%autoreload 2
%matplotlib inline
In [2]:
!pip install -q bentoml tensorflow==1.14.0 numpy matplotlib keras==2.3.1 pillow
Requirement already satisfied: tensorflow==1.14.0 in /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages (1.14.0)
Requirement already satisfied: numpy in /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages (1.16.6)
Requirement already satisfied: matplotlib in /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages (3.1.3)
Requirement already satisfied: keras==2.3.1 in /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages (2.3.1)
Requirement already satisfied: pillow in /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages (7.2.0)
Requirement already satisfied: grpcio>=1.8.6 in /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages (from tensorflow==1.14.0) (1.27.2)
Requirement already satisfied: six>=1.10.0 in /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages (from tensorflow==1.14.0) (1.14.0)
Requirement already satisfied: termcolor>=1.1.0 in /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages (from tensorflow==1.14.0) (1.1.0)
Requirement already satisfied: tensorflow-estimator<1.15.0rc0,>=1.14.0rc0 in /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages (from tensorflow==1.14.0) (1.14.0)
Requirement already satisfied: absl-py>=0.7.0 in /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages (from tensorflow==1.14.0) (0.9.0)
Requirement already satisfied: wrapt>=1.11.1 in /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages (from tensorflow==1.14.0) (1.12.1)
Requirement already satisfied: wheel>=0.26 in /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages (from tensorflow==1.14.0) (0.34.2)
Requirement already satisfied: tensorboard<1.15.0,>=1.14.0 in /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages (from tensorflow==1.14.0) (1.14.0)
Requirement already satisfied: keras-applications>=1.0.6 in /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages (from tensorflow==1.14.0) (1.0.8)
Requirement already satisfied: keras-preprocessing>=1.0.5 in /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages (from tensorflow==1.14.0) (1.1.0)
Requirement already satisfied: google-pasta>=0.1.6 in /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages (from tensorflow==1.14.0) (0.2.0)
Requirement already satisfied: protobuf>=3.6.1 in /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages (from tensorflow==1.14.0) (3.11.4)
Requirement already satisfied: gast>=0.2.0 in /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages (from tensorflow==1.14.0) (0.3.3)
Requirement already satisfied: astor>=0.6.0 in /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages (from tensorflow==1.14.0) (0.8.0)
Requirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.1 in /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages (from matplotlib) (2.4.7)
Requirement already satisfied: kiwisolver>=1.0.1 in /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages (from matplotlib) (1.2.0)
Requirement already satisfied: cycler>=0.10 in /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages (from matplotlib) (0.10.0)
Requirement already satisfied: python-dateutil>=2.1 in /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages (from matplotlib) (2.8.0)
Requirement already satisfied: pyyaml in /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages (from keras==2.3.1) (5.3.1)
Requirement already satisfied: scipy>=0.14 in /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages (from keras==2.3.1) (1.4.1)
Requirement already satisfied: h5py in /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages (from keras==2.3.1) (2.10.0)
Requirement already satisfied: werkzeug>=0.11.15 in /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages (from tensorboard<1.15.0,>=1.14.0->tensorflow==1.14.0) (1.0.1)
Requirement already satisfied: setuptools>=41.0.0 in /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages (from tensorboard<1.15.0,>=1.14.0->tensorflow==1.14.0) (46.2.0.post20200511)
Requirement already satisfied: markdown>=2.6.8 in /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages (from tensorboard<1.15.0,>=1.14.0->tensorflow==1.14.0) (3.1.1)
In [2]:
from __future__ import absolute_import, division, print_function

import numpy as np
import matplotlib.pyplot as plt

import tensorflow as tf
print("Tensorflow Version: %s" % tf.__version__)

import keras
print("Keras Version: %s" % keras.__version__)

import bentoml
print("BentoML Version: %s" % bentoml.__version__)
Tensorflow Version: 1.14.0
Keras Version: 2.3.1
BentoML Version: 0.9.0.pre+7.g8af1c8b

Load Fashion MNIST dataset

In [3]:
fashion_mnist = keras.datasets.fashion_mnist

(train_images, train_labels), (test_images, test_labels) = fashion_mnist.load_data()

Each image is mapped to a single label. Since the class names are not included with the dataset, store them here to use later when plotting the images:

In [4]:
class_names = ['T-shirt/top', 'Trouser', 'Pullover', 'Dress', 'Coat',
               'Sandal', 'Shirt', 'Sneaker', 'Bag', 'Ankle boot']

Preprocess the data

The data must be preprocessed before training the network. If you inspect the first image in the training set, you will see that the pixel values fall in the range of 0 to 255:

In [5]:
plt.figure()
plt.imshow(train_images[0], cmap=plt.cm.binary)
plt.colorbar()
plt.grid(False)
plt.show()

We should scale these values to a range of 0 to 1 before feeding to the neural network model. For this, we divide the values by 255. It's important that the training set and the testing set are preprocessed in the same way:

In [6]:
train_images = train_images / 255.0
test_images = test_images / 255.0

Display the first 25 images from the training set and display the class name below each image. Verify that the data is in the correct format and we're ready to build and train the network.

In [7]:
plt.figure(figsize=(10,10))
for i in range(25):
    plt.subplot(5,5,i+1)
    plt.xticks([])
    plt.yticks([])
    plt.grid(False)
    plt.imshow(train_images[i], cmap=plt.cm.binary)
    plt.xlabel(class_names[train_labels[i]])
plt.show()
In [8]:
# Reshaping the images so that our model can accept it
train_images  = train_images.reshape((60000, 28, 28, 1))
test_images = test_images.reshape((10000, 28, 28, 1))

Build the model

In [9]:
model = keras.Sequential()
# Must define the input shape in the first layer of the neural network
model.add(keras.layers.Conv2D(filters=64, kernel_size=2, padding='same', activation='relu', input_shape=(28,28,1))) 
model.add(keras.layers.MaxPooling2D(pool_size=2))
model.add(keras.layers.Dropout(0.3))
model.add(keras.layers.Conv2D(filters=32, kernel_size=2, padding='same', activation='relu'))
model.add(keras.layers.MaxPooling2D(pool_size=2))
model.add(keras.layers.Dropout(0.3))
model.add(keras.layers.Flatten())
model.add(keras.layers.Dense(256, activation='relu'))
model.add(keras.layers.Dropout(0.5))
model.add(keras.layers.Dense(10, activation='softmax'))
# Take a look at the model summary
model.summary()
WARNING:tensorflow:From /opt/conda/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py:4070: The name tf.nn.max_pool is deprecated. Please use tf.nn.max_pool2d instead.

Model: "sequential_1"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
conv2d_1 (Conv2D)            (None, 28, 28, 64)        320       
_________________________________________________________________
max_pooling2d_1 (MaxPooling2 (None, 14, 14, 64)        0         
_________________________________________________________________
dropout_1 (Dropout)          (None, 14, 14, 64)        0         
_________________________________________________________________
conv2d_2 (Conv2D)            (None, 14, 14, 32)        8224      
_________________________________________________________________
max_pooling2d_2 (MaxPooling2 (None, 7, 7, 32)          0         
_________________________________________________________________
dropout_2 (Dropout)          (None, 7, 7, 32)          0         
_________________________________________________________________
flatten_1 (Flatten)          (None, 1568)              0         
_________________________________________________________________
dense_1 (Dense)              (None, 256)               401664    
_________________________________________________________________
dropout_3 (Dropout)          (None, 256)               0         
_________________________________________________________________
dense_2 (Dense)              (None, 10)                2570      
=================================================================
Total params: 412,778
Trainable params: 412,778
Non-trainable params: 0
_________________________________________________________________

Compile the model

In [10]:
model.compile(optimizer='adam',
              loss='sparse_categorical_crossentropy',
              metrics=['accuracy'])

Train the model

In [11]:
model.fit(train_images, train_labels, batch_size=64, epochs=5)
WARNING:tensorflow:From /opt/conda/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py:422: The name tf.global_variables is deprecated. Please use tf.compat.v1.global_variables instead.

Epoch 1/5
60000/60000 [==============================] - 6s 101us/step - loss: 0.5853 - accuracy: 0.7847
Epoch 2/5
60000/60000 [==============================] - 5s 86us/step - loss: 0.4095 - accuracy: 0.8508
Epoch 3/5
60000/60000 [==============================] - 5s 86us/step - loss: 0.3620 - accuracy: 0.8662
Epoch 4/5
60000/60000 [==============================] - 5s 86us/step - loss: 0.3354 - accuracy: 0.8784
Epoch 5/5
60000/60000 [==============================] - 5s 85us/step - loss: 0.3190 - accuracy: 0.8843
Out[11]:
<keras.callbacks.callbacks.History at 0x7f47f4624828>

Evaluate model accuracy

In [12]:
test_loss, test_acc = model.evaluate(test_images, test_labels)

print('Test accuracy:', test_acc)
10000/10000 [==============================] - 1s 52us/step
Test accuracy: 0.899399995803833
In [13]:
y_hat = model.predict_classes(test_images)

# Plot a random sample of 10 test images, their predicted labels and ground truth
figure = plt.figure(figsize=(20, 8))
for i, index in enumerate(np.random.choice(test_images.shape[0], size=15, replace=False)):
    ax = figure.add_subplot(3, 5, i + 1, xticks=[], yticks=[])
    # Display each image
    ax.imshow(np.squeeze(test_images[index]))
    predict_index = y_hat[index]
    true_index = test_labels[index]
    # Set the title for each image
    ax.set_title("{} ({})".format(class_names[predict_index], 
                                  class_names[true_index]),
                                  color=("green" if predict_index == true_index else "red"))

Predict on a single image

Let's first see the test image and and its label:

In [14]:
plt.imshow(test_images[0].squeeze(-1), cmap=plt.cm.binary)
plt.show()
print('Class:', class_names[test_labels[0]])
Class: Ankle boot
In [15]:
pred_idx = model.predict_classes(np.expand_dims(test_images[0], 0))[0]
print('Prediction:', class_names[pred_idx])
Prediction: Ankle boot

Define BentoService for model serving

In [18]:
%%writefile keras_fashion_mnist.py
from typing import List

import numpy as np
from PIL import Image
from bentoml import api, artifacts, env, BentoService
from bentoml.frameworks.keras import KerasModelArtifact
from bentoml.adapters import ImageInput

class_names = ['T-shirt/top', 'Trouser', 'Pullover', 'Dress', 'Coat',
               'Sandal', 'Shirt', 'Sneaker', 'Bag', 'Ankle boot']

@env(pip_packages=['keras==2.3.1', 'tensorflow==1.14.0', 'pillow', 'numpy'])
@artifacts([KerasModelArtifact('classifier')])
class KerasFashionMnistService(BentoService):
        
    @api(input=ImageInput(pilmode='L'), batch=True)
    def predict(self, imgs: List[np.ndarray]) -> List[str]:
        inputs = []
        for img in imgs:
            img = Image.fromarray(img).resize((28, 28))
            img = np.array(img.getdata()).reshape((28, 28, 1))
            inputs.append(img)
        inputs = np.stack(inputs)
        class_idxs = self.artifacts.classifier.predict_classes(inputs)
        return [class_names[class_idx] for class_idx in class_idxs]
Overwriting keras_fashion_mnist.py

Save BentoService to file archive

In [19]:
# 1) import the custom BentoService defined above
from keras_fashion_mnist import KerasFashionMnistService

# 2) `pack` it with required artifacts
mnist_svc = KerasFashionMnistService()
mnist_svc.pack('classifier', model)

# 3) save your BentoSerivce to file archive
saved_path = mnist_svc.save()
[2020-09-23 13:23:49,677] WARNING - Using BentoML installed in `editable` model, the local BentoML repository including all code changes will be packaged together with saved bundle created, under the './bundled_pip_dependencies' directory of the saved bundle.
[2020-09-23 13:23:49,704] WARNING - pip package requirement tensorflow already exist
WARNING:tensorflow:From /home/bentoml/BentoML/bentoml/frameworks/keras.py:123: The name tf.keras.backend.get_session is deprecated. Please use tf.compat.v1.keras.backend.get_session instead.

[2020-09-23 13:23:51,430] INFO - Detected non-PyPI-released BentoML installed, copying local BentoML modulefiles to target saved bundle path..
/opt/conda/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages/setuptools/dist.py:452: UserWarning: Normalizing '0.9.0.pre+7.g8af1c8b' to '0.9.0rc0+7.g8af1c8b'
  warnings.warn(tmpl.format(**locals()))
warning: no previously-included files matching '*~' found anywhere in distribution
warning: no previously-included files matching '*.pyo' found anywhere in distribution
warning: no previously-included files matching '.git' found anywhere in distribution
warning: no previously-included files matching '.ipynb_checkpoints' found anywhere in distribution
warning: no previously-included files matching '__pycache__' found anywhere in distribution
warning: no directories found matching 'bentoml/yatai/web/dist'
no previously-included directories found matching 'e2e_tests'
no previously-included directories found matching 'tests'
no previously-included directories found matching 'benchmark'
UPDATING BentoML-0.9.0rc0+7.g8af1c8b/bentoml/_version.py
set BentoML-0.9.0rc0+7.g8af1c8b/bentoml/_version.py to '0.9.0.pre+7.g8af1c8b'
[2020-09-23 13:23:52,274] INFO - BentoService bundle 'KerasFashionMnistService:20200923132350_974EEA' saved to: /home/bentoml/bentoml/repository/KerasFashionMnistService/20200923132350_974EEA

Load saved BentoService

bentoml.load is the API for loading a BentoML packaged model in python:

In [20]:
svc = bentoml.load(saved_path)
svc.predict([test_images[0].squeeze(-1)])
[2020-09-23 13:23:59,176] WARNING - Saved BentoService bundle version mismatch: loading BentoService bundle create with BentoML version 0.9.0.pre, but loading from BentoML version 0.9.0.pre+7.g8af1c8b
[2020-09-23 13:23:59,178] WARNING - Module `keras_fashion_mnist` already loaded, using existing imported module.
WARNING:tensorflow:From /home/bentoml/BentoML/bentoml/frameworks/keras.py:137: The name tf.keras.backend.set_session is deprecated. Please use tf.compat.v1.keras.backend.set_session instead.

[2020-09-23 13:24:00,085] WARNING - pip package requirement imageio already exist
[2020-09-23 13:24:00,086] WARNING - pip package requirement tensorflow already exist
[2020-09-23 13:24:00,087] WARNING - pip package requirement keras already exist
Out[20]:
['Ankle boot']

Launch inference job from CLI

BentoML cli supports loading and running a packaged model from CLI. With the DataframeInput adapter, the CLI command supports reading input Dataframe data from CLI argument or local csv or json files:

In [2]:
!bentoml run KerasFashionMnistService:latest predict --input-file sample_image.png
[2020-09-23 13:24:35,370] INFO - Getting latest version KerasFashionMnistService:20200923132350_974EEA
[2020-09-23 13:24:36,401] WARNING - Using BentoML installed in `editable` model, the local BentoML repository including all code changes will be packaged together with saved bundle created, under the './bundled_pip_dependencies' directory of the saved bundle.
[2020-09-23 13:24:36,419] WARNING - Saved BentoService bundle version mismatch: loading BentoService bundle create with BentoML version 0.9.0.pre, but loading from BentoML version 0.9.0.pre+7.g8af1c8b
/opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:516: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint8 = np.dtype([("qint8", np.int8, 1)])
/opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:517: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_quint8 = np.dtype([("quint8", np.uint8, 1)])
/opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:518: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint16 = np.dtype([("qint16", np.int16, 1)])
/opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:519: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_quint16 = np.dtype([("quint16", np.uint16, 1)])
/opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:520: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint32 = np.dtype([("qint32", np.int32, 1)])
/opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:525: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  np_resource = np.dtype([("resource", np.ubyte, 1)])
/opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:541: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint8 = np.dtype([("qint8", np.int8, 1)])
/opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:542: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_quint8 = np.dtype([("quint8", np.uint8, 1)])
/opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:543: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint16 = np.dtype([("qint16", np.int16, 1)])
/opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:544: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_quint16 = np.dtype([("quint16", np.uint16, 1)])
/opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:545: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint32 = np.dtype([("qint32", np.int32, 1)])
/opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:550: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  np_resource = np.dtype([("resource", np.ubyte, 1)])
Using TensorFlow backend.
2020-09-23 13:24:37.559712: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcuda.so.1
2020-09-23 13:24:37.574229: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-09-23 13:24:37.574603: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 0 with properties: 
name: GeForce GTX 1060 major: 6 minor: 1 memoryClockRate(GHz): 1.6705
pciBusID: 0000:01:00.0
2020-09-23 13:24:37.574778: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.1
2020-09-23 13:24:37.576014: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10
2020-09-23 13:24:37.577344: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcufft.so.10
2020-09-23 13:24:37.577571: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcurand.so.10
2020-09-23 13:24:37.579057: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusolver.so.10
2020-09-23 13:24:37.579956: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusparse.so.10
2020-09-23 13:24:37.583071: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudnn.so.7
2020-09-23 13:24:37.583207: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-09-23 13:24:37.583615: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-09-23 13:24:37.583945: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1763] Adding visible gpu devices: 0
2020-09-23 13:24:37.584206: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.1 SSE4.2 AVX AVX2 FMA
2020-09-23 13:24:37.728318: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-09-23 13:24:37.728789: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x5557595e9bd0 executing computations on platform CUDA. Devices:
2020-09-23 13:24:37.728809: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (0): GeForce GTX 1060, Compute Capability 6.1
2020-09-23 13:24:37.728961: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-09-23 13:24:37.729290: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 0 with properties: 
name: GeForce GTX 1060 major: 6 minor: 1 memoryClockRate(GHz): 1.6705
pciBusID: 0000:01:00.0
2020-09-23 13:24:37.729326: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.1
2020-09-23 13:24:37.729340: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10
2020-09-23 13:24:37.729352: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcufft.so.10
2020-09-23 13:24:37.729363: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcurand.so.10
2020-09-23 13:24:37.729376: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusolver.so.10
2020-09-23 13:24:37.729388: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusparse.so.10
2020-09-23 13:24:37.729400: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudnn.so.7
2020-09-23 13:24:37.729447: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-09-23 13:24:37.729796: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-09-23 13:24:37.730123: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1763] Adding visible gpu devices: 0
2020-09-23 13:24:37.730164: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.1
2020-09-23 13:24:37.730641: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1181] Device interconnect StreamExecutor with strength 1 edge matrix:
2020-09-23 13:24:37.730654: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1187]      0 
2020-09-23 13:24:37.730662: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 0:   N 
2020-09-23 13:24:37.730760: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-09-23 13:24:37.731141: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-09-23 13:24:37.731496: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 5683 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1060, pci bus id: 0000:01:00.0, compute capability: 6.1)
2020-09-23 13:24:37.756994: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2699905000 Hz
2020-09-23 13:24:37.757349: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x55575c5d4840 executing computations on platform Host. Devices:
2020-09-23 13:24:37.757417: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (0): <undefined>, <undefined>
WARNING:tensorflow:From /home/bentoml/BentoML/bentoml/frameworks/keras.py:137: The name tf.keras.backend.set_session is deprecated. Please use tf.compat.v1.keras.backend.set_session instead.

WARNING:tensorflow:From /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py:4070: The name tf.nn.max_pool is deprecated. Please use tf.nn.max_pool2d instead.

WARNING:tensorflow:From /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py:422: The name tf.global_variables is deprecated. Please use tf.compat.v1.global_variables instead.

WARNING:tensorflow:From /home/bentoml/BentoML/bentoml/frameworks/keras.py:123: The name tf.keras.backend.get_session is deprecated. Please use tf.compat.v1.keras.backend.get_session instead.

[2020-09-23 13:24:38,862] WARNING - pip package requirement tensorflow already exist
[2020-09-23 13:24:38,862] WARNING - pip package requirement keras already exist
2020-09-23 13:24:39.961990: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10
2020-09-23 13:24:40.109036: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudnn.so.7
[2020-09-23 13:24:40,664] INFO - {'service_name': 'KerasFashionMnistService', 'service_version': '20200923132350_974EEA', 'api': 'predict', 'task': {'data': {'uri': 'file:///home/bentoml/lab/gallery/legacy-keras/fashion-mnist/sample_image.png', 'name': 'sample_image.png'}, 'task_id': 'f350fdaa-9523-469b-8d73-8facb0945bb7', 'cli_args': ('--input-file', 'sample_image.png')}, 'result': {'data': '"Ankle boot"', 'http_status': 200, 'http_headers': (('Content-Type', 'application/json'),)}, 'request_id': 'f350fdaa-9523-469b-8d73-8facb0945bb7'}
"Ankle boot"

REST API Model Serving

To start a REST API model server with the BentoService saved above, use the bentoml serve command:

In [3]:
!bentoml serve KerasFashionMnistService:latest --enable-microbatch
[2020-09-23 13:24:48,477] INFO - Getting latest version KerasFashionMnistService:20200923132350_974EEA
[2020-09-23 13:24:48,477] INFO - Starting BentoML API server in development mode..
[2020-09-23 13:24:49,637] WARNING - Using BentoML installed in `editable` model, the local BentoML repository including all code changes will be packaged together with saved bundle created, under the './bundled_pip_dependencies' directory of the saved bundle.
[2020-09-23 13:24:49,654] WARNING - Saved BentoService bundle version mismatch: loading BentoService bundle create with BentoML version 0.9.0.pre, but loading from BentoML version 0.9.0.pre+7.g8af1c8b
/opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:516: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint8 = np.dtype([("qint8", np.int8, 1)])
/opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:517: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_quint8 = np.dtype([("quint8", np.uint8, 1)])
/opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:518: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint16 = np.dtype([("qint16", np.int16, 1)])
/opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:519: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_quint16 = np.dtype([("quint16", np.uint16, 1)])
/opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:520: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint32 = np.dtype([("qint32", np.int32, 1)])
/opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:525: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  np_resource = np.dtype([("resource", np.ubyte, 1)])
/opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:541: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint8 = np.dtype([("qint8", np.int8, 1)])
/opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:542: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_quint8 = np.dtype([("quint8", np.uint8, 1)])
/opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:543: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint16 = np.dtype([("qint16", np.int16, 1)])
/opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:544: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_quint16 = np.dtype([("quint16", np.uint16, 1)])
/opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:545: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint32 = np.dtype([("qint32", np.int32, 1)])
/opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:550: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  np_resource = np.dtype([("resource", np.ubyte, 1)])
Using TensorFlow backend.
2020-09-23 13:24:50.754920: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcuda.so.1
2020-09-23 13:24:50.769561: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-09-23 13:24:50.769934: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 0 with properties: 
name: GeForce GTX 1060 major: 6 minor: 1 memoryClockRate(GHz): 1.6705
pciBusID: 0000:01:00.0
2020-09-23 13:24:50.770104: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.1
2020-09-23 13:24:50.771331: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10
2020-09-23 13:24:50.772609: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcufft.so.10
2020-09-23 13:24:50.772876: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcurand.so.10
2020-09-23 13:24:50.774317: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusolver.so.10
2020-09-23 13:24:50.775157: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusparse.so.10
2020-09-23 13:24:50.778709: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudnn.so.7
2020-09-23 13:24:50.778855: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-09-23 13:24:50.779265: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-09-23 13:24:50.779583: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1763] Adding visible gpu devices: 0
2020-09-23 13:24:50.779837: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.1 SSE4.2 AVX AVX2 FMA
2020-09-23 13:24:50.950505: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-09-23 13:24:50.950963: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x5597103fd600 executing computations on platform CUDA. Devices:
2020-09-23 13:24:50.950987: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (0): GeForce GTX 1060, Compute Capability 6.1
2020-09-23 13:24:50.951170: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-09-23 13:24:50.951547: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 0 with properties: 
name: GeForce GTX 1060 major: 6 minor: 1 memoryClockRate(GHz): 1.6705
pciBusID: 0000:01:00.0
2020-09-23 13:24:50.951582: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.1
2020-09-23 13:24:50.951612: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10
2020-09-23 13:24:50.951638: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcufft.so.10
2020-09-23 13:24:50.951665: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcurand.so.10
2020-09-23 13:24:50.951679: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusolver.so.10
2020-09-23 13:24:50.951692: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusparse.so.10
2020-09-23 13:24:50.951731: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudnn.so.7
2020-09-23 13:24:50.951794: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-09-23 13:24:50.952186: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-09-23 13:24:50.952530: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1763] Adding visible gpu devices: 0
2020-09-23 13:24:50.952558: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.1
2020-09-23 13:24:50.953034: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1181] Device interconnect StreamExecutor with strength 1 edge matrix:
2020-09-23 13:24:50.953047: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1187]      0 
2020-09-23 13:24:50.953053: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 0:   N 
2020-09-23 13:24:50.953157: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-09-23 13:24:50.953528: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-09-23 13:24:50.953875: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 5683 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1060, pci bus id: 0000:01:00.0, compute capability: 6.1)
2020-09-23 13:24:50.972930: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2699905000 Hz
2020-09-23 13:24:50.973274: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x5597135723e0 executing computations on platform Host. Devices:
2020-09-23 13:24:50.973362: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (0): <undefined>, <undefined>
WARNING:tensorflow:From /home/bentoml/BentoML/bentoml/frameworks/keras.py:137: The name tf.keras.backend.set_session is deprecated. Please use tf.compat.v1.keras.backend.set_session instead.

WARNING:tensorflow:From /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py:4070: The name tf.nn.max_pool is deprecated. Please use tf.nn.max_pool2d instead.

WARNING:tensorflow:From /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py:422: The name tf.global_variables is deprecated. Please use tf.compat.v1.global_variables instead.

WARNING:tensorflow:From /home/bentoml/BentoML/bentoml/frameworks/keras.py:123: The name tf.keras.backend.get_session is deprecated. Please use tf.compat.v1.keras.backend.get_session instead.

[2020-09-23 13:24:52,198] WARNING - pip package requirement tensorflow already exist
[2020-09-23 13:24:52,198] WARNING - pip package requirement keras already exist
[2020-09-23 13:24:53,251] WARNING - Saved BentoService bundle version mismatch: loading BentoService bundle create with BentoML version 0.9.0.pre, but loading from BentoML version 0.9.0.pre+7.g8af1c8b
[2020-09-23 13:24:53,253] INFO - Micro batch enabled for API `predict`
[2020-09-23 13:24:53,253] INFO - Your system nofile limit is 10000, which means each instance of microbatch service is able to hold this number of connections at same time. You can increase the number of file descriptors for the server process, or launch more microbatch instances to accept more concurrent connection.
[2020-09-23 13:24:53,267] INFO - Running micro batch service on :5000
 * Serving Flask app "KerasFashionMnistService" (lazy loading)
 * Environment: production
   WARNING: This is a development server. Do not use it in a production deployment.
   Use a production WSGI server instead.
 * Debug mode: off
 * Running on http://127.0.0.1:48301/ (Press CTRL+C to quit)
======== Running on http://0.0.0.0:5000 ========
(Press CTRL+C to quit)
^C

If you are running this notebook from Google Colab, you can start the dev server with --run-with-ngrok option, to gain acccess to the API endpoint via a public endpoint managed by ngrok:

In [ ]:
!bentoml serve KerasFashionMnistService:latest --run-with-ngrok

Test sending request to API Server from terminal:

curl -X POST "http://127.0.0.1:5000/predict" -F image=@sample_image.png

or:

curl -X POST "http://127.0.0.1:5000/predict" -H "Content-Type: image/png" --data-binary @sample_image.png

Containerize model server with Docker

One common way of distributing this model API server for production deployment, is via Docker containers. And BentoML provides a convenient way to do that.

Note that docker is not available in Google Colab. You will need to download and run this notebook locally to try out this containerization with docker feature.

If you already have docker configured, simply run the follow command to product a docker container serving the IrisClassifier prediction service created above:

In [4]:
!bentoml containerize KerasFashionMnistService:latest -t kerasfashionmnistservice:latest
[2020-09-23 13:25:09,964] INFO - Getting latest version KerasFashionMnistService:20200923132350_974EEA
Found Bento: /home/bentoml/bentoml/repository/KerasFashionMnistService/20200923132350_974EEA
[2020-09-23 13:25:09,983] WARNING - Using BentoML installed in `editable` model, the local BentoML repository including all code changes will be packaged together with saved bundle created, under the './bundled_pip_dependencies' directory of the saved bundle.
[2020-09-23 13:25:09,999] WARNING - Saved BentoService bundle version mismatch: loading BentoService bundle create with BentoML version 0.9.0.pre, but loading from BentoML version 0.9.0.pre+7.g8af1c8b
Building Docker image kerasfashionmnistservice:latest from KerasFashionMnistService:latest 
-Step 1/15 : FROM bentoml/model-server:0.9.0.pre-py36
 ---> 4aac43d10e50
Step 2/15 : ARG EXTRA_PIP_INSTALL_ARGS=
 ---> Using cache
 ---> 790054f5ad85
Step 3/15 : ENV EXTRA_PIP_INSTALL_ARGS $EXTRA_PIP_INSTALL_ARGS
 ---> Using cache
 ---> 85b0a1b40542
Step 4/15 : COPY environment.yml requirements.txt setup.sh* bentoml-init.sh python_version* /bento/
| ---> 8729a1f5c821
Step 5/15 : WORKDIR /bento
 ---> Running in 60907eb65c99
\ ---> 6df2fc7b9f5b
Step 6/15 : RUN chmod +x /bento/bentoml-init.sh
 ---> Running in 5623e9822b02
| ---> 4c3fa98dc1df
Step 7/15 : RUN if [ -f /bento/bentoml-init.sh ]; then bash -c /bento/bentoml-init.sh; fi
 ---> Running in 050ae928ea3d
\+++ dirname /bento/bentoml-init.sh

++ cd /bento
++ pwd -P

+ SAVED_BUNDLE_PATH=/bento
+ cd /bento

+ '[' -f ./setup.sh ']'
+ '[' -f ./python_version ']'

++ cat ./python_version

+ PY_VERSION_SAVED=3.6.10
+ DESIRED_PY_VERSION=3.6

++ python -c 'import sys; print(f"{sys.version_info.major}.{sys.version_info.minor}")'

+ CURRENT_PY_VERSION=3.6
+ [[ 3.6 == \3\.\6 ]]
+ echo 'Python Version in docker base image 3.6 matches requirement python=3.6. Skipping.'

Python Version in docker base image 3.6 matches requirement python=3.6. Skipping.
+ command -v conda
+ echo 'Updating conda base environment with environment.yml'
+ conda env update -n base -f ./environment.yml

Updating conda base environment with environment.yml
|Collecting package metadata (repodata.json): ...working... 
|done
Solving environment: ...working... 
-done

Downloading and Extracting Packages
openssl-1.1.1h       | 2.1 MB    |            |   0% 
openssl-1.1.1h       | 2.1 MB    |            |   1% 
openssl-1.1.1h       | 2.1 MB    | 2          |   3% 
openssl-1.1.1h       | 2.1 MB    | 4          |   4% 
openssl-1.1.1h       | 2.1 MB    | 5          |   6% 
openssl-1.1.1h       | 2.1 MB    | 7          |   7% 
openssl-1.1.1h       | 2.1 MB    | #3         |  13% 
openssl-1.1.1h       | 2.1 MB    | #9         |  19% 
openssl-1.1.1h       | 2.1 MB    | ##7        |  27% 
openssl-1.1.1h       | 2.1 MB    | ##9        |  29% 
openssl-1.1.1h       | 2.1 MB    | ###3       |  34% 
openssl-1.1.1h       | 2.1 MB    | ###6       |  36% 
openssl-1.1.1h       | 2.1 MB    | ####       |  41% 
openssl-1.1.1h       | 2.1 MB    | ####1      |  42% 
openssl-1.1.1h       | 2.1 MB    | ####6      |  46% 
openssl-1.1.1h       | 2.1 MB    | ####8      |  49% 
openssl-1.1.1h       | 2.1 MB    | #####      |  51% 
openssl-1.1.1h       | 2.1 MB    | #####2     |  52% 
openssl-1.1.1h       | 2.1 MB    | #####5     |  55% 
openssl-1.1.1h       | 2.1 MB    | ######     |  60% 
openssl-1.1.1h       | 2.1 MB    | #######2   |  73% 
openssl-1.1.1h       | 2.1 MB    | ########   |  80% 
openssl-1.1.1h       | 2.1 MB    | ########1  |  82% 
openssl-1.1.1h       | 2.1 MB    | ########6  |  87% 
openssl-1.1.1h       | 2.1 MB    | ########9  |  89% 
openssl-1.1.1h       | 2.1 MB    | #########1 |  91% 
openssl-1.1.1h       | 2.1 MB    | #########5 |  96% 
openssl-1.1.1h       | 2.1 MB    | #########7 |  98% 
openssl-1.1.1h       | 2.1 MB    | ########## | 100% 
openssl-1.1.1h       | 2.1 MB    | ########## | 100% 

ca-certificates-2020 | 145 KB    |            |   0% 
ca-certificates-2020 | 145 KB    | #1         |  11% 
ca-certificates-2020 | 145 KB    | ##2        |  22% 
ca-certificates-2020 | 145 KB    | ###3       |  33% 
ca-certificates-2020 | 145 KB    | ######6    |  66% 
ca-certificates-2020 | 145 KB    | ########## | 100% 

python_abi-3.6       | 4 KB      |            |   0% 
python_abi-3.6       | 4 KB      | ########## | 100% 
python_abi-3.6       | 4 KB      | ########## | 100% 

pip-20.2.3           | 1.1 MB    |            |   0% 
pip-20.2.3           | 1.1 MB    | 1          |   1% 
pip-20.2.3           | 1.1 MB    | 5          |   6% 
pip-20.2.3           | 1.1 MB    | 8          |   9% 
pip-20.2.3           | 1.1 MB    | #8         |  19% 
pip-20.2.3           | 1.1 MB    | ##7        |  28% 
pip-20.2.3           | 1.1 MB    | ###6       |  37% 
pip-20.2.3           | 1.1 MB    | #####2     |  53% 
pip-20.2.3           | 1.1 MB    | #####8     |  58% 
pip-20.2.3           | 1.1 MB    | ######2    |  63% 
pip-20.2.3           | 1.1 MB    | ########   |  80% 
pip-20.2.3           | 1.1 MB    | ########4  |  85% 
pip-20.2.3           | 1.1 MB    | #########  |  91% 
pip-20.2.3           | 1.1 MB    | #########3 |  94% 
pip-20.2.3           | 1.1 MB    | ########## | 100% 
pip-20.2.3           | 1.1 MB    | ########## | 100% 

certifi-2020.6.20    | 151 KB    |            |   0% 
certifi-2020.6.20    | 151 KB    | #          |  11% 
certifi-2020.6.20    | 151 KB    | ##1        |  21% 
certifi-2020.6.20    | 151 KB    | ####2      |  42% 
certifi-2020.6.20    | 151 KB    | #####3     |  53% 
certifi-2020.6.20    | 151 KB    | ########4  |  85% 
certifi-2020.6.20    | 151 KB    | ########## | 100% 

Preparing transaction: ...working... 
/done
Verifying transaction: ...working... 
|done
Executing transaction: ...working... 
\done
/#
# To activate this environment, use
#
#     $ conda activate base
#
# To deactivate an active environment, use
#
#     $ conda deactivate
/+ pip install -r ./requirements.txt --no-cache-dir

-Requirement already satisfied: bentoml==0.9.0.pre in /opt/conda/lib/python3.6/site-packages (from -r ./requirements.txt (line 1)) (0.9.0rc0)
|Collecting keras==2.3.1
-  Downloading Keras-2.3.1-py2.py3-none-any.whl (377 kB)
/Collecting tensorflow==1.14.0
|  Downloading tensorflow-1.14.0-cp36-cp36m-manylinux1_x86_64.whl (109.2 MB)
-Collecting pillow==7.2.0
/  Downloading Pillow-7.2.0-cp36-cp36m-manylinux1_x86_64.whl (2.2 MB)
|Requirement already satisfied: numpy==1.19.2 in /opt/conda/lib/python3.6/site-packages (from -r ./requirements.txt (line 5)) (1.19.2)
\Collecting imageio==2.9.0
/  Downloading imageio-2.9.0-py3-none-any.whl (3.3 MB)
-Requirement already satisfied: tabulate in /opt/conda/lib/python3.6/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (0.8.7)
Requirement already satisfied: requests in /opt/conda/lib/python3.6/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (2.24.0)
Requirement already satisfied: cerberus in /opt/conda/lib/python3.6/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (1.3.2)
Requirement already satisfied: click>=7.0 in /opt/conda/lib/python3.6/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (7.1.2)
Requirement already satisfied: py-zipkin in /opt/conda/lib/python3.6/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (0.20.0)
Requirement already satisfied: prometheus-client in /opt/conda/lib/python3.6/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (0.8.0)
Requirement already satisfied: docker in /opt/conda/lib/python3.6/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (4.3.1)
Requirement already satisfied: certifi in /opt/conda/lib/python3.6/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (2020.6.20)
Requirement already satisfied: python-json-logger in /opt/conda/lib/python3.6/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (0.1.11)
Requirement already satisfied: alembic in /opt/conda/lib/python3.6/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (1.4.3)
Requirement already satisfied: boto3 in /opt/conda/lib/python3.6/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (1.15.2)
Requirement already satisfied: protobuf>=3.6.0 in /opt/conda/lib/python3.6/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (3.13.0)
Requirement already satisfied: python-dateutil<3.0.0,>=2.7.3 in /opt/conda/lib/python3.6/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (2.8.1)
/Requirement already satisfied: ruamel.yaml>=0.15.0 in /opt/conda/lib/python3.6/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (0.15.87)
Requirement already satisfied: sqlalchemy-utils<0.36.8 in /opt/conda/lib/python3.6/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (0.36.7)
Requirement already satisfied: humanfriendly in /opt/conda/lib/python3.6/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (8.2)
Requirement already satisfied: psutil in /opt/conda/lib/python3.6/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (5.7.2)
Requirement already satisfied: dataclasses; python_version < "3.7" in /opt/conda/lib/python3.6/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (0.7)
Requirement already satisfied: packaging in /opt/conda/lib/python3.6/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (20.4)
Requirement already satisfied: aiohttp in /opt/conda/lib/python3.6/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (3.6.2)
|Requirement already satisfied: contextvars; python_version < "3.7" in /opt/conda/lib/python3.6/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (2.4)
Requirement already satisfied: grpcio<=1.27.2 in /opt/conda/lib/python3.6/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (1.27.2)
Requirement already satisfied: multidict in /opt/conda/lib/python3.6/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (4.7.6)
Requirement already satisfied: sqlalchemy>=1.3.0 in /opt/conda/lib/python3.6/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (1.3.19)
Requirement already satisfied: gunicorn in /opt/conda/lib/python3.6/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (20.0.4)
Requirement already satisfied: configparser in /opt/conda/lib/python3.6/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (5.0.0)
Requirement already satisfied: flask in /opt/conda/lib/python3.6/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (1.1.2)
-Collecting keras-preprocessing>=1.0.5
/  Downloading Keras_Preprocessing-1.1.2-py2.py3-none-any.whl (42 kB)
\Requirement already satisfied: six>=1.9.0 in /opt/conda/lib/python3.6/site-packages (from keras==2.3.1->-r ./requirements.txt (line 2)) (1.15.0)
-Collecting scipy>=0.14
/  Downloading scipy-1.5.2-cp36-cp36m-manylinux1_x86_64.whl (25.9 MB)
-Collecting pyyaml
|  Downloading PyYAML-5.3.1.tar.gz (269 kB)
/Collecting keras-applications>=1.0.6
|  Downloading Keras_Applications-1.0.8-py3-none-any.whl (50 kB)
|Collecting h5py
\  Downloading h5py-2.10.0-cp36-cp36m-manylinux1_x86_64.whl (2.9 MB)
|Collecting tensorflow-estimator<1.15.0rc0,>=1.14.0rc0
\  Downloading tensorflow_estimator-1.14.0-py2.py3-none-any.whl (488 kB)
/Collecting astor>=0.6.0
\  Downloading astor-0.8.1-py2.py3-none-any.whl (27 kB)
Requirement already satisfied: wheel>=0.26 in /opt/conda/lib/python3.6/site-packages (from tensorflow==1.14.0->-r ./requirements.txt (line 3)) (0.35.1)
/Collecting absl-py>=0.7.0
|  Downloading absl_py-0.10.0-py3-none-any.whl (127 kB)
|Collecting tensorboard<1.15.0,>=1.14.0
-  Downloading tensorboard-1.14.0-py3-none-any.whl (3.1 MB)
-Collecting wrapt>=1.11.1
/  Downloading wrapt-1.12.1.tar.gz (27 kB)
|Collecting google-pasta>=0.1.6
\  Downloading google_pasta-0.2.0-py3-none-any.whl (57 kB)
/Collecting termcolor>=1.1.0
\  Downloading termcolor-1.1.0.tar.gz (3.9 kB)
\Collecting gast>=0.2.0
-  Downloading gast-0.4.0-py3-none-any.whl (9.8 kB)
Requirement already satisfied: chardet<4,>=3.0.2 in /opt/conda/lib/python3.6/site-packages (from requests->bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (3.0.4)
Requirement already satisfied: idna<3,>=2.5 in /opt/conda/lib/python3.6/site-packages (from requests->bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (2.10)
Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /opt/conda/lib/python3.6/site-packages (from requests->bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (1.25.10)
Requirement already satisfied: setuptools in /opt/conda/lib/python3.6/site-packages (from cerberus->bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (49.6.0.post20200814)
Requirement already satisfied: thriftpy2>=0.4.0 in /opt/conda/lib/python3.6/site-packages (from py-zipkin->bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (0.4.11)
Requirement already satisfied: websocket-client>=0.32.0 in /opt/conda/lib/python3.6/site-packages (from docker->bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (0.57.0)
Requirement already satisfied: python-editor>=0.3 in /opt/conda/lib/python3.6/site-packages (from alembic->bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (1.0.4)
Requirement already satisfied: Mako in /opt/conda/lib/python3.6/site-packages (from alembic->bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (1.1.3)
Requirement already satisfied: botocore<1.19.0,>=1.18.2 in /opt/conda/lib/python3.6/site-packages (from boto3->bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (1.18.2)
/Requirement already satisfied: jmespath<1.0.0,>=0.7.1 in /opt/conda/lib/python3.6/site-packages (from boto3->bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (0.10.0)
Requirement already satisfied: s3transfer<0.4.0,>=0.3.0 in /opt/conda/lib/python3.6/site-packages (from boto3->bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (0.3.3)
Requirement already satisfied: pyparsing>=2.0.2 in /opt/conda/lib/python3.6/site-packages (from packaging->bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (2.4.7)
Requirement already satisfied: attrs>=17.3.0 in /opt/conda/lib/python3.6/site-packages (from aiohttp->bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (20.2.0)
Requirement already satisfied: typing-extensions>=3.6.5; python_version < "3.7" in /opt/conda/lib/python3.6/site-packages (from aiohttp->bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (3.7.4.3)
Requirement already satisfied: yarl<2.0,>=1.0 in /opt/conda/lib/python3.6/site-packages (from aiohttp->bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (1.5.1)
Requirement already satisfied: idna-ssl>=1.0; python_version < "3.7" in /opt/conda/lib/python3.6/site-packages (from aiohttp->bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (1.1.0)
Requirement already satisfied: async-timeout<4.0,>=3.0 in /opt/conda/lib/python3.6/site-packages (from aiohttp->bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (3.0.1)
Requirement already satisfied: immutables>=0.9 in /opt/conda/lib/python3.6/site-packages (from contextvars; python_version < "3.7"->bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (0.14)
Requirement already satisfied: Werkzeug>=0.15 in /opt/conda/lib/python3.6/site-packages (from flask->bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (1.0.1)
Requirement already satisfied: itsdangerous>=0.24 in /opt/conda/lib/python3.6/site-packages (from flask->bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (1.1.0)
Requirement already satisfied: Jinja2>=2.10.1 in /opt/conda/lib/python3.6/site-packages (from flask->bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (2.11.2)
\Collecting markdown>=2.6.8
-  Downloading Markdown-3.2.2-py3-none-any.whl (88 kB)
|Requirement already satisfied: ply<4.0,>=3.4 in /opt/conda/lib/python3.6/site-packages (from thriftpy2>=0.4.0->py-zipkin->bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (3.11)
Requirement already satisfied: MarkupSafe>=0.9.2 in /opt/conda/lib/python3.6/site-packages (from Mako->alembic->bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (1.1.1)
/Collecting importlib-metadata; python_version < "3.8"
|  Downloading importlib_metadata-2.0.0-py2.py3-none-any.whl (31 kB)
-Collecting zipp>=0.5
|  Downloading zipp-3.2.0-py3-none-any.whl (5.1 kB)
Building wheels for collected packages: pyyaml, wrapt, termcolor
  Building wheel for pyyaml (setup.py): started
|  Building wheel for pyyaml (setup.py): finished with status 'done'
  Created wheel for pyyaml: filename=PyYAML-5.3.1-cp36-cp36m-linux_x86_64.whl size=44619 sha256=bc3a34861f160832e54226e9dfc5886ed6a1f193d13bc07bd8cc6c07e9252140
  Stored in directory: /tmp/pip-ephem-wheel-cache-n4qcgfhc/wheels/e5/9d/ad/2ee53cf262cba1ffd8afe1487eef788ea3f260b7e6232a80fc
  Building wheel for wrapt (setup.py): started
|  Building wheel for wrapt (setup.py): finished with status 'done'
  Created wheel for wrapt: filename=wrapt-1.12.1-cp36-cp36m-linux_x86_64.whl size=75206 sha256=3e1a885724afaa51a425c64bc576f9bcac3f850636e59c10ee14160a0d4d9340
  Stored in directory: /tmp/pip-ephem-wheel-cache-n4qcgfhc/wheels/32/42/7f/23cae9ff6ef66798d00dc5d659088e57dbba01566f6c60db63
  Building wheel for termcolor (setup.py): started
/  Building wheel for termcolor (setup.py): finished with status 'done'
  Created wheel for termcolor: filename=termcolor-1.1.0-py3-none-any.whl size=4830 sha256=66929de748b95c287b1919fc0cbd3c4337b4897dbc42ebcda4be7c6cbdc707fa
  Stored in directory: /tmp/pip-ephem-wheel-cache-n4qcgfhc/wheels/93/2a/eb/e58dbcbc963549ee4f065ff80a59f274cc7210b6eab962acdc
Successfully built pyyaml wrapt termcolor
\Installing collected packages: keras-preprocessing, scipy, pyyaml, h5py, keras-applications, keras, tensorflow-estimator, astor, absl-py, zipp, importlib-metadata, markdown, tensorboard, wrapt, google-pasta, termcolor, gast, tensorflow, pillow, imageio
/Successfully installed absl-py-0.10.0 astor-0.8.1 gast-0.4.0 google-pasta-0.2.0 h5py-2.10.0 imageio-2.9.0 importlib-metadata-2.0.0 keras-2.3.1 keras-applications-1.0.8 keras-preprocessing-1.1.2 markdown-3.2.2 pillow-7.2.0 pyyaml-5.3.1 scipy-1.5.2 tensorboard-1.14.0 tensorflow-1.14.0 tensorflow-estimator-1.14.0 termcolor-1.1.0 wrapt-1.12.1 zipp-3.2.0
| ---> f67cf901ef18
Step 8/15 : COPY . /bento
/ ---> 4a58550a0fad
Step 9/15 : RUN if [ -d /bento/bundled_pip_dependencies ]; then pip install -U bundled_pip_dependencies/* ;fi
| ---> Running in 98757f1e8214
-Processing ./bundled_pip_dependencies/BentoML-0.9.0rc0+7.g8af1c8b.tar.gz
/  Installing build dependencies: started
-  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
/  Getting requirements to build wheel: finished with status 'done'
    Preparing wheel metadata: started
|    Preparing wheel metadata: finished with status 'done'
\Requirement already satisfied, skipping upgrade: grpcio<=1.27.2 in /opt/conda/lib/python3.6/site-packages (from BentoML==0.9.0rc0+7.g8af1c8b) (1.27.2)
Requirement already satisfied, skipping upgrade: boto3 in /opt/conda/lib/python3.6/site-packages (from BentoML==0.9.0rc0+7.g8af1c8b) (1.15.2)
Requirement already satisfied, skipping upgrade: numpy in /opt/conda/lib/python3.6/site-packages (from BentoML==0.9.0rc0+7.g8af1c8b) (1.19.2)
-Requirement already satisfied, skipping upgrade: prometheus-client in /opt/conda/lib/python3.6/site-packages (from BentoML==0.9.0rc0+7.g8af1c8b) (0.8.0)
Requirement already satisfied, skipping upgrade: sqlalchemy-utils<0.36.8 in /opt/conda/lib/python3.6/site-packages (from BentoML==0.9.0rc0+7.g8af1c8b) (0.36.7)
Requirement already satisfied, skipping upgrade: gunicorn in /opt/conda/lib/python3.6/site-packages (from BentoML==0.9.0rc0+7.g8af1c8b) (20.0.4)
Requirement already satisfied, skipping upgrade: requests in /opt/conda/lib/python3.6/site-packages (from BentoML==0.9.0rc0+7.g8af1c8b) (2.24.0)
Requirement already satisfied, skipping upgrade: psutil in /opt/conda/lib/python3.6/site-packages (from BentoML==0.9.0rc0+7.g8af1c8b) (5.7.2)
/Requirement already satisfied, skipping upgrade: packaging in /opt/conda/lib/python3.6/site-packages (from BentoML==0.9.0rc0+7.g8af1c8b) (20.4)
Requirement already satisfied, skipping upgrade: contextvars; python_version < "3.7" in /opt/conda/lib/python3.6/site-packages (from BentoML==0.9.0rc0+7.g8af1c8b) (2.4)
Requirement already satisfied, skipping upgrade: docker in /opt/conda/lib/python3.6/site-packages (from BentoML==0.9.0rc0+7.g8af1c8b) (4.3.1)
Requirement already satisfied, skipping upgrade: aiohttp in /opt/conda/lib/python3.6/site-packages (from BentoML==0.9.0rc0+7.g8af1c8b) (3.6.2)
Requirement already satisfied, skipping upgrade: dataclasses; python_version < "3.7" in /opt/conda/lib/python3.6/site-packages (from BentoML==0.9.0rc0+7.g8af1c8b) (0.7)
Requirement already satisfied, skipping upgrade: humanfriendly in /opt/conda/lib/python3.6/site-packages (from BentoML==0.9.0rc0+7.g8af1c8b) (8.2)
Requirement already satisfied, skipping upgrade: tabulate in /opt/conda/lib/python3.6/site-packages (from BentoML==0.9.0rc0+7.g8af1c8b) (0.8.7)
Requirement already satisfied, skipping upgrade: ruamel.yaml>=0.15.0 in /opt/conda/lib/python3.6/site-packages (from BentoML==0.9.0rc0+7.g8af1c8b) (0.15.87)
Requirement already satisfied, skipping upgrade: py-zipkin in /opt/conda/lib/python3.6/site-packages (from BentoML==0.9.0rc0+7.g8af1c8b) (0.20.0)
Requirement already satisfied, skipping upgrade: python-dateutil<3.0.0,>=2.7.3 in /opt/conda/lib/python3.6/site-packages (from BentoML==0.9.0rc0+7.g8af1c8b) (2.8.1)
Requirement already satisfied, skipping upgrade: certifi in /opt/conda/lib/python3.6/site-packages (from BentoML==0.9.0rc0+7.g8af1c8b) (2020.6.20)
Requirement already satisfied, skipping upgrade: multidict in /opt/conda/lib/python3.6/site-packages (from BentoML==0.9.0rc0+7.g8af1c8b) (4.7.6)
Requirement already satisfied, skipping upgrade: protobuf>=3.6.0 in /opt/conda/lib/python3.6/site-packages (from BentoML==0.9.0rc0+7.g8af1c8b) (3.13.0)
Requirement already satisfied, skipping upgrade: flask in /opt/conda/lib/python3.6/site-packages (from BentoML==0.9.0rc0+7.g8af1c8b) (1.1.2)
Requirement already satisfied, skipping upgrade: alembic in /opt/conda/lib/python3.6/site-packages (from BentoML==0.9.0rc0+7.g8af1c8b) (1.4.3)
Requirement already satisfied, skipping upgrade: cerberus in /opt/conda/lib/python3.6/site-packages (from BentoML==0.9.0rc0+7.g8af1c8b) (1.3.2)
Requirement already satisfied, skipping upgrade: python-json-logger in /opt/conda/lib/python3.6/site-packages (from BentoML==0.9.0rc0+7.g8af1c8b) (0.1.11)
Requirement already satisfied, skipping upgrade: click>=7.0 in /opt/conda/lib/python3.6/site-packages (from BentoML==0.9.0rc0+7.g8af1c8b) (7.1.2)
Requirement already satisfied, skipping upgrade: sqlalchemy>=1.3.0 in /opt/conda/lib/python3.6/site-packages (from BentoML==0.9.0rc0+7.g8af1c8b) (1.3.19)
Requirement already satisfied, skipping upgrade: configparser in /opt/conda/lib/python3.6/site-packages (from BentoML==0.9.0rc0+7.g8af1c8b) (5.0.0)
|Requirement already satisfied, skipping upgrade: six>=1.5.2 in /opt/conda/lib/python3.6/site-packages (from grpcio<=1.27.2->BentoML==0.9.0rc0+7.g8af1c8b) (1.15.0)
Requirement already satisfied, skipping upgrade: s3transfer<0.4.0,>=0.3.0 in /opt/conda/lib/python3.6/site-packages (from boto3->BentoML==0.9.0rc0+7.g8af1c8b) (0.3.3)
Requirement already satisfied, skipping upgrade: jmespath<1.0.0,>=0.7.1 in /opt/conda/lib/python3.6/site-packages (from boto3->BentoML==0.9.0rc0+7.g8af1c8b) (0.10.0)
Requirement already satisfied, skipping upgrade: botocore<1.19.0,>=1.18.2 in /opt/conda/lib/python3.6/site-packages (from boto3->BentoML==0.9.0rc0+7.g8af1c8b) (1.18.2)
Requirement already satisfied, skipping upgrade: setuptools>=3.0 in /opt/conda/lib/python3.6/site-packages (from gunicorn->BentoML==0.9.0rc0+7.g8af1c8b) (49.6.0.post20200814)
Requirement already satisfied, skipping upgrade: idna<3,>=2.5 in /opt/conda/lib/python3.6/site-packages (from requests->BentoML==0.9.0rc0+7.g8af1c8b) (2.10)
Requirement already satisfied, skipping upgrade: chardet<4,>=3.0.2 in /opt/conda/lib/python3.6/site-packages (from requests->BentoML==0.9.0rc0+7.g8af1c8b) (3.0.4)
Requirement already satisfied, skipping upgrade: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /opt/conda/lib/python3.6/site-packages (from requests->BentoML==0.9.0rc0+7.g8af1c8b) (1.25.10)
Requirement already satisfied, skipping upgrade: pyparsing>=2.0.2 in /opt/conda/lib/python3.6/site-packages (from packaging->BentoML==0.9.0rc0+7.g8af1c8b) (2.4.7)
Requirement already satisfied, skipping upgrade: immutables>=0.9 in /opt/conda/lib/python3.6/site-packages (from contextvars; python_version < "3.7"->BentoML==0.9.0rc0+7.g8af1c8b) (0.14)
Requirement already satisfied, skipping upgrade: websocket-client>=0.32.0 in /opt/conda/lib/python3.6/site-packages (from docker->BentoML==0.9.0rc0+7.g8af1c8b) (0.57.0)
Requirement already satisfied, skipping upgrade: async-timeout<4.0,>=3.0 in /opt/conda/lib/python3.6/site-packages (from aiohttp->BentoML==0.9.0rc0+7.g8af1c8b) (3.0.1)
Requirement already satisfied, skipping upgrade: typing-extensions>=3.6.5; python_version < "3.7" in /opt/conda/lib/python3.6/site-packages (from aiohttp->BentoML==0.9.0rc0+7.g8af1c8b) (3.7.4.3)
Requirement already satisfied, skipping upgrade: attrs>=17.3.0 in /opt/conda/lib/python3.6/site-packages (from aiohttp->BentoML==0.9.0rc0+7.g8af1c8b) (20.2.0)
Requirement already satisfied, skipping upgrade: yarl<2.0,>=1.0 in /opt/conda/lib/python3.6/site-packages (from aiohttp->BentoML==0.9.0rc0+7.g8af1c8b) (1.5.1)
Requirement already satisfied, skipping upgrade: idna-ssl>=1.0; python_version < "3.7" in /opt/conda/lib/python3.6/site-packages (from aiohttp->BentoML==0.9.0rc0+7.g8af1c8b) (1.1.0)
Requirement already satisfied, skipping upgrade: thriftpy2>=0.4.0 in /opt/conda/lib/python3.6/site-packages (from py-zipkin->BentoML==0.9.0rc0+7.g8af1c8b) (0.4.11)
\Requirement already satisfied, skipping upgrade: Werkzeug>=0.15 in /opt/conda/lib/python3.6/site-packages (from flask->BentoML==0.9.0rc0+7.g8af1c8b) (1.0.1)
Requirement already satisfied, skipping upgrade: Jinja2>=2.10.1 in /opt/conda/lib/python3.6/site-packages (from flask->BentoML==0.9.0rc0+7.g8af1c8b) (2.11.2)
Requirement already satisfied, skipping upgrade: itsdangerous>=0.24 in /opt/conda/lib/python3.6/site-packages (from flask->BentoML==0.9.0rc0+7.g8af1c8b) (1.1.0)
Requirement already satisfied, skipping upgrade: python-editor>=0.3 in /opt/conda/lib/python3.6/site-packages (from alembic->BentoML==0.9.0rc0+7.g8af1c8b) (1.0.4)
Requirement already satisfied, skipping upgrade: Mako in /opt/conda/lib/python3.6/site-packages (from alembic->BentoML==0.9.0rc0+7.g8af1c8b) (1.1.3)
Requirement already satisfied, skipping upgrade: ply<4.0,>=3.4 in /opt/conda/lib/python3.6/site-packages (from thriftpy2>=0.4.0->py-zipkin->BentoML==0.9.0rc0+7.g8af1c8b) (3.11)
Requirement already satisfied, skipping upgrade: MarkupSafe>=0.23 in /opt/conda/lib/python3.6/site-packages (from Jinja2>=2.10.1->flask->BentoML==0.9.0rc0+7.g8af1c8b) (1.1.1)
Building wheels for collected packages: BentoML
  Building wheel for BentoML (PEP 517): started
|  Building wheel for BentoML (PEP 517): finished with status 'done'
  Created wheel for BentoML: filename=BentoML-0.9.0rc0+7.g8af1c8b-py3-none-any.whl size=620483 sha256=c0a193a2b7442065fb7abec76df7be020511af5c5a8ea34107d58bc21b9760e1
  Stored in directory: /root/.cache/pip/wheels/0f/c2/e2/64640eb406a012df3abf685e4e5fa441b87b3e609903d041b6
Successfully built BentoML
|Installing collected packages: BentoML
  Attempting uninstall: BentoML
    Found existing installation: BentoML 0.9.0rc0
\    Uninstalling BentoML-0.9.0rc0:
/      Successfully uninstalled BentoML-0.9.0rc0
-Successfully installed BentoML-0.9.0rc0+7.g8af1c8b
\ ---> 40090207374a
Step 10/15 : ENV PORT 5000
- ---> Running in bef4344724f5
| ---> b336b939b877
Step 11/15 : EXPOSE $PORT
 ---> Running in 94b0d4042e8c
- ---> 6dbdacacc9ee
Step 12/15 : COPY docker-entrypoint.sh /usr/local/bin/
| ---> 08f3c0134892
Step 13/15 : RUN chmod +x /usr/local/bin/docker-entrypoint.sh
\ ---> Running in 919280fa3744
| ---> 0c8922f41ad5
Step 14/15 : ENTRYPOINT [ "docker-entrypoint.sh" ]
\ ---> Running in cf1a37e5f312
/ ---> 084525a4fb10
Step 15/15 : CMD ["bentoml", "serve-gunicorn", "/bento"]
| ---> Running in ae9aab696766
- ---> df4a2b1bf7e6
Successfully built df4a2b1bf7e6
Successfully tagged kerasfashionmnistservice:latest
Finished building kerasfashionmnistservice:latest from KerasFashionMnistService:latest
In [5]:
!docker run -p 5000:5000 kerasfashionmnistservice
[2020-09-23 05:35:06,581] INFO - Starting BentoML API server in production mode..
[2020-09-23 05:35:06,806] INFO - get_gunicorn_num_of_workers: 3, calculated by cpu count
[2020-09-23 05:35:06 +0000] [1] [INFO] Starting gunicorn 20.0.4
[2020-09-23 05:35:06 +0000] [1] [INFO] Listening at: http://0.0.0.0:5000 (1)
[2020-09-23 05:35:06 +0000] [1] [INFO] Using worker: sync
[2020-09-23 05:35:06 +0000] [13] [INFO] Booting worker with pid: 13
[2020-09-23 05:35:06 +0000] [14] [INFO] Booting worker with pid: 14
[2020-09-23 05:35:06 +0000] [15] [INFO] Booting worker with pid: 15
[2020-09-23 05:35:07,826] WARNING - Using BentoML not from official PyPI release. In order to find the same version of BentoML when deploying your BentoService, you must set the 'core/bentoml_deploy_version' config to a http/git location of your BentoML fork, e.g.: 'bentoml_deploy_version = git+https://github.com/{username}/[email protected]{branch}'
[2020-09-23 05:35:07,847] WARNING - Saved BentoService bundle version mismatch: loading BentoService bundle create with BentoML version 0.9.0.pre, but loading from BentoML version 0.9.0.pre+7.g8af1c8b
[2020-09-23 05:35:07,847] WARNING - Saved BentoService Python version mismatch: loading BentoService bundle created with Python version 3.6.10, but current environment version is 3.6.12.
[2020-09-23 05:35:07,855] WARNING - Using BentoML not from official PyPI release. In order to find the same version of BentoML when deploying your BentoService, you must set the 'core/bentoml_deploy_version' config to a http/git location of your BentoML fork, e.g.: 'bentoml_deploy_version = git+https://github.com/{username}/[email protected]{branch}'
[2020-09-23 05:35:07,868] WARNING - Using BentoML not from official PyPI release. In order to find the same version of BentoML when deploying your BentoService, you must set the 'core/bentoml_deploy_version' config to a http/git location of your BentoML fork, e.g.: 'bentoml_deploy_version = git+https://github.com/{username}/[email protected]{branch}'
[2020-09-23 05:35:07,877] WARNING - Saved BentoService bundle version mismatch: loading BentoService bundle create with BentoML version 0.9.0.pre, but loading from BentoML version 0.9.0.pre+7.g8af1c8b
[2020-09-23 05:35:07,877] WARNING - Saved BentoService Python version mismatch: loading BentoService bundle created with Python version 3.6.10, but current environment version is 3.6.12.
[2020-09-23 05:35:07,900] WARNING - Saved BentoService bundle version mismatch: loading BentoService bundle create with BentoML version 0.9.0.pre, but loading from BentoML version 0.9.0.pre+7.g8af1c8b
[2020-09-23 05:35:07,900] WARNING - Saved BentoService Python version mismatch: loading BentoService bundle created with Python version 3.6.10, but current environment version is 3.6.12.
2020-09-23 05:35:09.265543: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2020-09-23 05:35:09.288840: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2699905000 Hz
2020-09-23 05:35:09.289083: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x5637744043a0 executing computations on platform Host. Devices:
2020-09-23 05:35:09.289103: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (0): <undefined>, <undefined>
/opt/conda/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:516: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint8 = np.dtype([("qint8", np.int8, 1)])
/opt/conda/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:517: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_quint8 = np.dtype([("quint8", np.uint8, 1)])
/opt/conda/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:518: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint16 = np.dtype([("qint16", np.int16, 1)])
/opt/conda/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:519: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_quint16 = np.dtype([("quint16", np.uint16, 1)])
/opt/conda/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:520: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint32 = np.dtype([("qint32", np.int32, 1)])
/opt/conda/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:525: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  np_resource = np.dtype([("resource", np.ubyte, 1)])
/opt/conda/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:541: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint8 = np.dtype([("qint8", np.int8, 1)])
/opt/conda/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:542: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_quint8 = np.dtype([("quint8", np.uint8, 1)])
/opt/conda/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:543: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint16 = np.dtype([("qint16", np.int16, 1)])
/opt/conda/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:544: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_quint16 = np.dtype([("quint16", np.uint16, 1)])
/opt/conda/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:545: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint32 = np.dtype([("qint32", np.int32, 1)])
/opt/conda/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:550: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  np_resource = np.dtype([("resource", np.ubyte, 1)])
Using TensorFlow backend.
WARNING:tensorflow:From /opt/conda/lib/python3.6/site-packages/bentoml/frameworks/keras.py:137: The name tf.keras.backend.set_session is deprecated. Please use tf.compat.v1.keras.backend.set_session instead.

WARNING:tensorflow:From /opt/conda/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py:4070: The name tf.nn.max_pool is deprecated. Please use tf.nn.max_pool2d instead.

2020-09-23 05:35:09.320438: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2020-09-23 05:35:09.328772: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2699905000 Hz
2020-09-23 05:35:09.328969: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x563774404be0 executing computations on platform Host. Devices:
2020-09-23 05:35:09.328987: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (0): <undefined>, <undefined>
/opt/conda/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:516: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint8 = np.dtype([("qint8", np.int8, 1)])
/opt/conda/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:517: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_quint8 = np.dtype([("quint8", np.uint8, 1)])
/opt/conda/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:518: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint16 = np.dtype([("qint16", np.int16, 1)])
/opt/conda/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:519: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_quint16 = np.dtype([("quint16", np.uint16, 1)])
/opt/conda/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:520: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint32 = np.dtype([("qint32", np.int32, 1)])
/opt/conda/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:525: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  np_resource = np.dtype([("resource", np.ubyte, 1)])
/opt/conda/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:541: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint8 = np.dtype([("qint8", np.int8, 1)])
/opt/conda/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:542: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_quint8 = np.dtype([("quint8", np.uint8, 1)])
/opt/conda/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:543: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint16 = np.dtype([("qint16", np.int16, 1)])
/opt/conda/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:544: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_quint16 = np.dtype([("quint16", np.uint16, 1)])
/opt/conda/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:545: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint32 = np.dtype([("qint32", np.int32, 1)])
/opt/conda/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:550: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  np_resource = np.dtype([("resource", np.ubyte, 1)])
Using TensorFlow backend.
WARNING:tensorflow:From /opt/conda/lib/python3.6/site-packages/bentoml/frameworks/keras.py:137: The name tf.keras.backend.set_session is deprecated. Please use tf.compat.v1.keras.backend.set_session instead.

WARNING:tensorflow:From /opt/conda/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py:4070: The name tf.nn.max_pool is deprecated. Please use tf.nn.max_pool2d instead.

2020-09-23 05:35:09.378158: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2020-09-23 05:35:09.388984: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2699905000 Hz
2020-09-23 05:35:09.389196: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x563774403ae0 executing computations on platform Host. Devices:
2020-09-23 05:35:09.389212: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (0): <undefined>, <undefined>
/opt/conda/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:516: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint8 = np.dtype([("qint8", np.int8, 1)])
/opt/conda/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:517: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_quint8 = np.dtype([("quint8", np.uint8, 1)])
/opt/conda/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:518: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint16 = np.dtype([("qint16", np.int16, 1)])
/opt/conda/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:519: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_quint16 = np.dtype([("quint16", np.uint16, 1)])
/opt/conda/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:520: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint32 = np.dtype([("qint32", np.int32, 1)])
/opt/conda/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:525: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  np_resource = np.dtype([("resource", np.ubyte, 1)])
/opt/conda/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:541: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint8 = np.dtype([("qint8", np.int8, 1)])
/opt/conda/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:542: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_quint8 = np.dtype([("quint8", np.uint8, 1)])
/opt/conda/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:543: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint16 = np.dtype([("qint16", np.int16, 1)])
/opt/conda/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:544: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_quint16 = np.dtype([("quint16", np.uint16, 1)])
/opt/conda/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:545: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint32 = np.dtype([("qint32", np.int32, 1)])
/opt/conda/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:550: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  np_resource = np.dtype([("resource", np.ubyte, 1)])
Using TensorFlow backend.
WARNING:tensorflow:From /opt/conda/lib/python3.6/site-packages/bentoml/frameworks/keras.py:137: The name tf.keras.backend.set_session is deprecated. Please use tf.compat.v1.keras.backend.set_session instead.

WARNING:tensorflow:From /opt/conda/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py:4070: The name tf.nn.max_pool is deprecated. Please use tf.nn.max_pool2d instead.

2020-09-23 05:35:09.477116: W tensorflow/compiler/jit/mark_for_compilation_pass.cc:1412] (One-time warning): Not using XLA:CPU for cluster because envvar TF_XLA_FLAGS=--tf_xla_cpu_global_jit was not set.  If you want XLA:CPU, either set that envvar, or use experimental_jit_scope to enable XLA:CPU.  To confirm that XLA is active, pass --vmodule=xla_compilation_cache=1 (as a proper command-line flag, not via TF_XLA_FLAGS) or set the envvar XLA_FLAGS=--xla_hlo_profile.
2020-09-23 05:35:09.535995: W tensorflow/compiler/jit/mark_for_compilation_pass.cc:1412] (One-time warning): Not using XLA:CPU for cluster because envvar TF_XLA_FLAGS=--tf_xla_cpu_global_jit was not set.  If you want XLA:CPU, either set that envvar, or use experimental_jit_scope to enable XLA:CPU.  To confirm that XLA is active, pass --vmodule=xla_compilation_cache=1 (as a proper command-line flag, not via TF_XLA_FLAGS) or set the envvar XLA_FLAGS=--xla_hlo_profile.
2020-09-23 05:35:09.582936: W tensorflow/compiler/jit/mark_for_compilation_pass.cc:1412] (One-time warning): Not using XLA:CPU for cluster because envvar TF_XLA_FLAGS=--tf_xla_cpu_global_jit was not set.  If you want XLA:CPU, either set that envvar, or use experimental_jit_scope to enable XLA:CPU.  To confirm that XLA is active, pass --vmodule=xla_compilation_cache=1 (as a proper command-line flag, not via TF_XLA_FLAGS) or set the envvar XLA_FLAGS=--xla_hlo_profile.
WARNING:tensorflow:From /opt/conda/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py:422: The name tf.global_variables is deprecated. Please use tf.compat.v1.global_variables instead.

WARNING:tensorflow:From /opt/conda/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py:422: The name tf.global_variables is deprecated. Please use tf.compat.v1.global_variables instead.

WARNING:tensorflow:From /opt/conda/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py:422: The name tf.global_variables is deprecated. Please use tf.compat.v1.global_variables instead.

WARNING:tensorflow:From /opt/conda/lib/python3.6/site-packages/bentoml/frameworks/keras.py:123: The name tf.keras.backend.get_session is deprecated. Please use tf.compat.v1.keras.backend.get_session instead.

[2020-09-23 05:35:10,507] WARNING - pip package requirement tensorflow already exist
[2020-09-23 05:35:10,508] WARNING - pip package requirement keras already exist
WARNING:tensorflow:From /opt/conda/lib/python3.6/site-packages/bentoml/frameworks/keras.py:123: The name tf.keras.backend.get_session is deprecated. Please use tf.compat.v1.keras.backend.get_session instead.

[2020-09-23 05:35:10,525] WARNING - pip package requirement tensorflow already exist
[2020-09-23 05:35:10,525] WARNING - pip package requirement keras already exist
WARNING:tensorflow:From /opt/conda/lib/python3.6/site-packages/bentoml/frameworks/keras.py:123: The name tf.keras.backend.get_session is deprecated. Please use tf.compat.v1.keras.backend.get_session instead.

[2020-09-23 05:35:10,615] WARNING - pip package requirement tensorflow already exist
[2020-09-23 05:35:10,616] WARNING - pip package requirement keras already exist
[2020-09-23 05:35:55,479] ERROR - Exception on /predict [POST]
Traceback (most recent call last):
  File "/opt/conda/lib/python3.6/site-packages/bentoml/server/api_server.py", line 288, in api_func
    response = api.handle_request(request)
  File "/opt/conda/lib/python3.6/site-packages/bentoml/service/inference_api.py", line 281, in handle_request
    results = self.infer((inf_task,))
  File "/opt/conda/lib/python3.6/site-packages/bentoml/service/inference_api.py", line 222, in infer
    user_args = self.input_adapter.extract_user_func_args(inf_tasks)
  File "/opt/conda/lib/python3.6/site-packages/bentoml/adapters/image_input.py", line 134, in extract_user_func_args
    img_array = imageio.imread(task.data, pilmode=self.pilmode)
  File "/opt/conda/lib/python3.6/site-packages/imageio/core/functions.py", line 265, in imread
    reader = read(uri, format, "i", **kwargs)
  File "/opt/conda/lib/python3.6/site-packages/imageio/core/functions.py", line 172, in get_reader
    request = Request(uri, "r" + mode, **kwargs)
  File "/opt/conda/lib/python3.6/site-packages/imageio/core/request.py", line 124, in __init__
    self._parse_uri(uri)
  File "/opt/conda/lib/python3.6/site-packages/imageio/core/request.py", line 222, in _parse_uri
    raise IOError("Cannot understand given URI: %s." % uri_r)
OSError: Cannot understand given URI: None.
[2020-09-23 05:36:09,200] INFO - {'service_name': 'KerasFashionMnistService', 'service_version': '20200923132350_974EEA', 'api': 'predict', 'task': {'data': {'name': 'sample_image.png'}, 'task_id': '895a16b1-c689-490f-b30e-c5601984093a', 'http_headers': (('Host', '127.0.0.1:5000'), ('User-Agent', 'curl/7.72.0'), ('Content-Length', '133877'), ('Accept', '*/*'), ('Content-Type', 'multipart/form-data; boundary=------------------------4f19d0a72380315e'))}, 'result': {'data': '"Ankle boot"', 'http_status': 200, 'http_headers': (('Content-Type', 'application/json'),)}, 'request_id': '895a16b1-c689-490f-b30e-c5601984093a'}
^C
[2020-09-23 05:36:20 +0000] [1] [INFO] Handling signal: int
[2020-09-23 05:36:20 +0000] [13] [INFO] Worker exiting (pid: 13)
[2020-09-23 05:36:20 +0000] [14] [INFO] Worker exiting (pid: 14)
[2020-09-23 05:36:20 +0000] [15] [INFO] Worker exiting (pid: 15)

Deployment Options

If you are at a small team with limited engineering or DevOps resources, try out automated deployment with BentoML CLI, currently supporting AWS Lambda, AWS SageMaker, and Azure Functions:

If the cloud platform you are working with is not on the list above, try out these step-by-step guide on manually deploying BentoML packaged model to cloud platforms:

Lastly, if you have a DevOps or ML Engineering team who's operating a Kubernetes or OpenShift cluster, use the following guides as references for implementating your deployment strategy:

In [ ]: