BentoML makes moving trained ML models to production easy:
BentoML is a framework for serving, managing, and deploying machine learning models. It is aiming to bridge the gap between Data Science and DevOps, and enable teams to deliver prediction services in a fast, repeatable, and scalable way.
Before reading this example project, be sure to check out the Getting started guide to learn about the basic concepts in BentoML.
This notebook demonstrates how to use BentoML to turn a Keras model into a docker image containing a REST API server serving this model, how to use your ML service built with BentoML as a CLI tool, and how to distribute it a pypi package.
%reload_ext autoreload
%autoreload 2
%matplotlib inline
!pip install -q bentoml tensorflow==1.14.0 numpy matplotlib keras==2.3.1 pillow
Requirement already satisfied: tensorflow==1.14.0 in /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages (1.14.0) Requirement already satisfied: numpy in /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages (1.16.6) Requirement already satisfied: matplotlib in /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages (3.1.3) Requirement already satisfied: keras==2.3.1 in /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages (2.3.1) Requirement already satisfied: pillow in /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages (7.2.0) Requirement already satisfied: grpcio>=1.8.6 in /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages (from tensorflow==1.14.0) (1.27.2) Requirement already satisfied: six>=1.10.0 in /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages (from tensorflow==1.14.0) (1.14.0) Requirement already satisfied: termcolor>=1.1.0 in /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages (from tensorflow==1.14.0) (1.1.0) Requirement already satisfied: tensorflow-estimator<1.15.0rc0,>=1.14.0rc0 in /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages (from tensorflow==1.14.0) (1.14.0) Requirement already satisfied: absl-py>=0.7.0 in /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages (from tensorflow==1.14.0) (0.9.0) Requirement already satisfied: wrapt>=1.11.1 in /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages (from tensorflow==1.14.0) (1.12.1) Requirement already satisfied: wheel>=0.26 in /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages (from tensorflow==1.14.0) (0.34.2) Requirement already satisfied: tensorboard<1.15.0,>=1.14.0 in /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages (from tensorflow==1.14.0) (1.14.0) Requirement already satisfied: keras-applications>=1.0.6 in /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages (from tensorflow==1.14.0) (1.0.8) Requirement already satisfied: keras-preprocessing>=1.0.5 in /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages (from tensorflow==1.14.0) (1.1.0) Requirement already satisfied: google-pasta>=0.1.6 in /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages (from tensorflow==1.14.0) (0.2.0) Requirement already satisfied: protobuf>=3.6.1 in /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages (from tensorflow==1.14.0) (3.11.4) Requirement already satisfied: gast>=0.2.0 in /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages (from tensorflow==1.14.0) (0.3.3) Requirement already satisfied: astor>=0.6.0 in /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages (from tensorflow==1.14.0) (0.8.0) Requirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.1 in /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages (from matplotlib) (2.4.7) Requirement already satisfied: kiwisolver>=1.0.1 in /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages (from matplotlib) (1.2.0) Requirement already satisfied: cycler>=0.10 in /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages (from matplotlib) (0.10.0) Requirement already satisfied: python-dateutil>=2.1 in /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages (from matplotlib) (2.8.0) Requirement already satisfied: pyyaml in /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages (from keras==2.3.1) (5.3.1) Requirement already satisfied: scipy>=0.14 in /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages (from keras==2.3.1) (1.4.1) Requirement already satisfied: h5py in /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages (from keras==2.3.1) (2.10.0) Requirement already satisfied: werkzeug>=0.11.15 in /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages (from tensorboard<1.15.0,>=1.14.0->tensorflow==1.14.0) (1.0.1) Requirement already satisfied: setuptools>=41.0.0 in /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages (from tensorboard<1.15.0,>=1.14.0->tensorflow==1.14.0) (46.2.0.post20200511) Requirement already satisfied: markdown>=2.6.8 in /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages (from tensorboard<1.15.0,>=1.14.0->tensorflow==1.14.0) (3.1.1)
from __future__ import absolute_import, division, print_function
import numpy as np
import matplotlib.pyplot as plt
import tensorflow as tf
print("Tensorflow Version: %s" % tf.__version__)
import keras
print("Keras Version: %s" % keras.__version__)
import bentoml
print("BentoML Version: %s" % bentoml.__version__)
Tensorflow Version: 1.14.0 Keras Version: 2.3.1 BentoML Version: 0.9.0.pre+7.g8af1c8b
fashion_mnist = keras.datasets.fashion_mnist
(train_images, train_labels), (test_images, test_labels) = fashion_mnist.load_data()
Each image is mapped to a single label. Since the class names are not included with the dataset, store them here to use later when plotting the images:
class_names = ['T-shirt/top', 'Trouser', 'Pullover', 'Dress', 'Coat',
'Sandal', 'Shirt', 'Sneaker', 'Bag', 'Ankle boot']
The data must be preprocessed before training the network. If you inspect the first image in the training set, you will see that the pixel values fall in the range of 0 to 255:
plt.figure()
plt.imshow(train_images[0], cmap=plt.cm.binary)
plt.colorbar()
plt.grid(False)
plt.show()
We should scale these values to a range of 0 to 1 before feeding to the neural network model. For this, we divide the values by 255. It's important that the training set and the testing set are preprocessed in the same way:
train_images = train_images / 255.0
test_images = test_images / 255.0
Display the first 25 images from the training set and display the class name below each image. Verify that the data is in the correct format and we're ready to build and train the network.
plt.figure(figsize=(10,10))
for i in range(25):
plt.subplot(5,5,i+1)
plt.xticks([])
plt.yticks([])
plt.grid(False)
plt.imshow(train_images[i], cmap=plt.cm.binary)
plt.xlabel(class_names[train_labels[i]])
plt.show()
# Reshaping the images so that our model can accept it
train_images = train_images.reshape((60000, 28, 28, 1))
test_images = test_images.reshape((10000, 28, 28, 1))
model = keras.Sequential()
# Must define the input shape in the first layer of the neural network
model.add(keras.layers.Conv2D(filters=64, kernel_size=2, padding='same', activation='relu', input_shape=(28,28,1)))
model.add(keras.layers.MaxPooling2D(pool_size=2))
model.add(keras.layers.Dropout(0.3))
model.add(keras.layers.Conv2D(filters=32, kernel_size=2, padding='same', activation='relu'))
model.add(keras.layers.MaxPooling2D(pool_size=2))
model.add(keras.layers.Dropout(0.3))
model.add(keras.layers.Flatten())
model.add(keras.layers.Dense(256, activation='relu'))
model.add(keras.layers.Dropout(0.5))
model.add(keras.layers.Dense(10, activation='softmax'))
# Take a look at the model summary
model.summary()
WARNING:tensorflow:From /opt/conda/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py:4070: The name tf.nn.max_pool is deprecated. Please use tf.nn.max_pool2d instead. Model: "sequential_1" _________________________________________________________________ Layer (type) Output Shape Param # ================================================================= conv2d_1 (Conv2D) (None, 28, 28, 64) 320 _________________________________________________________________ max_pooling2d_1 (MaxPooling2 (None, 14, 14, 64) 0 _________________________________________________________________ dropout_1 (Dropout) (None, 14, 14, 64) 0 _________________________________________________________________ conv2d_2 (Conv2D) (None, 14, 14, 32) 8224 _________________________________________________________________ max_pooling2d_2 (MaxPooling2 (None, 7, 7, 32) 0 _________________________________________________________________ dropout_2 (Dropout) (None, 7, 7, 32) 0 _________________________________________________________________ flatten_1 (Flatten) (None, 1568) 0 _________________________________________________________________ dense_1 (Dense) (None, 256) 401664 _________________________________________________________________ dropout_3 (Dropout) (None, 256) 0 _________________________________________________________________ dense_2 (Dense) (None, 10) 2570 ================================================================= Total params: 412,778 Trainable params: 412,778 Non-trainable params: 0 _________________________________________________________________
model.compile(optimizer='adam',
loss='sparse_categorical_crossentropy',
metrics=['accuracy'])
model.fit(train_images, train_labels, batch_size=64, epochs=5)
WARNING:tensorflow:From /opt/conda/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py:422: The name tf.global_variables is deprecated. Please use tf.compat.v1.global_variables instead. Epoch 1/5 60000/60000 [==============================] - 6s 101us/step - loss: 0.5853 - accuracy: 0.7847 Epoch 2/5 60000/60000 [==============================] - 5s 86us/step - loss: 0.4095 - accuracy: 0.8508 Epoch 3/5 60000/60000 [==============================] - 5s 86us/step - loss: 0.3620 - accuracy: 0.8662 Epoch 4/5 60000/60000 [==============================] - 5s 86us/step - loss: 0.3354 - accuracy: 0.8784 Epoch 5/5 60000/60000 [==============================] - 5s 85us/step - loss: 0.3190 - accuracy: 0.8843
<keras.callbacks.callbacks.History at 0x7f47f4624828>
test_loss, test_acc = model.evaluate(test_images, test_labels)
print('Test accuracy:', test_acc)
10000/10000 [==============================] - 1s 52us/step Test accuracy: 0.899399995803833
y_hat = model.predict_classes(test_images)
# Plot a random sample of 10 test images, their predicted labels and ground truth
figure = plt.figure(figsize=(20, 8))
for i, index in enumerate(np.random.choice(test_images.shape[0], size=15, replace=False)):
ax = figure.add_subplot(3, 5, i + 1, xticks=[], yticks=[])
# Display each image
ax.imshow(np.squeeze(test_images[index]))
predict_index = y_hat[index]
true_index = test_labels[index]
# Set the title for each image
ax.set_title("{} ({})".format(class_names[predict_index],
class_names[true_index]),
color=("green" if predict_index == true_index else "red"))
Let's first see the test image and and its label:
plt.imshow(test_images[0].squeeze(-1), cmap=plt.cm.binary)
plt.show()
print('Class:', class_names[test_labels[0]])
Class: Ankle boot
pred_idx = model.predict_classes(np.expand_dims(test_images[0], 0))[0]
print('Prediction:', class_names[pred_idx])
Prediction: Ankle boot
%%writefile keras_fashion_mnist.py
from typing import List
import numpy as np
from PIL import Image
from bentoml import api, artifacts, env, BentoService
from bentoml.frameworks.keras import KerasModelArtifact
from bentoml.adapters import ImageInput
class_names = ['T-shirt/top', 'Trouser', 'Pullover', 'Dress', 'Coat',
'Sandal', 'Shirt', 'Sneaker', 'Bag', 'Ankle boot']
@env(pip_packages=['keras==2.3.1', 'tensorflow==1.14.0', 'pillow', 'numpy'])
@artifacts([KerasModelArtifact('classifier')])
class KerasFashionMnistService(BentoService):
@api(input=ImageInput(pilmode='L'), batch=True)
def predict(self, imgs: List[np.ndarray]) -> List[str]:
inputs = []
for img in imgs:
img = Image.fromarray(img).resize((28, 28))
img = np.array(img.getdata()).reshape((28, 28, 1))
inputs.append(img)
inputs = np.stack(inputs)
class_idxs = self.artifacts.classifier.predict_classes(inputs)
return [class_names[class_idx] for class_idx in class_idxs]
Overwriting keras_fashion_mnist.py
# 1) import the custom BentoService defined above
from keras_fashion_mnist import KerasFashionMnistService
# 2) `pack` it with required artifacts
mnist_svc = KerasFashionMnistService()
mnist_svc.pack('classifier', model)
# 3) save your BentoSerivce to file archive
saved_path = mnist_svc.save()
[2020-09-23 13:23:49,677] WARNING - Using BentoML installed in `editable` model, the local BentoML repository including all code changes will be packaged together with saved bundle created, under the './bundled_pip_dependencies' directory of the saved bundle. [2020-09-23 13:23:49,704] WARNING - pip package requirement tensorflow already exist WARNING:tensorflow:From /home/bentoml/BentoML/bentoml/frameworks/keras.py:123: The name tf.keras.backend.get_session is deprecated. Please use tf.compat.v1.keras.backend.get_session instead. [2020-09-23 13:23:51,430] INFO - Detected non-PyPI-released BentoML installed, copying local BentoML modulefiles to target saved bundle path..
/opt/conda/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages/setuptools/dist.py:452: UserWarning: Normalizing '0.9.0.pre+7.g8af1c8b' to '0.9.0rc0+7.g8af1c8b' warnings.warn(tmpl.format(**locals())) warning: no previously-included files matching '*~' found anywhere in distribution warning: no previously-included files matching '*.pyo' found anywhere in distribution warning: no previously-included files matching '.git' found anywhere in distribution warning: no previously-included files matching '.ipynb_checkpoints' found anywhere in distribution warning: no previously-included files matching '__pycache__' found anywhere in distribution warning: no directories found matching 'bentoml/yatai/web/dist' no previously-included directories found matching 'e2e_tests' no previously-included directories found matching 'tests' no previously-included directories found matching 'benchmark'
UPDATING BentoML-0.9.0rc0+7.g8af1c8b/bentoml/_version.py set BentoML-0.9.0rc0+7.g8af1c8b/bentoml/_version.py to '0.9.0.pre+7.g8af1c8b' [2020-09-23 13:23:52,274] INFO - BentoService bundle 'KerasFashionMnistService:20200923132350_974EEA' saved to: /home/bentoml/bentoml/repository/KerasFashionMnistService/20200923132350_974EEA
bentoml.load is the API for loading a BentoML packaged model in python:
svc = bentoml.load(saved_path)
svc.predict([test_images[0].squeeze(-1)])
[2020-09-23 13:23:59,176] WARNING - Saved BentoService bundle version mismatch: loading BentoService bundle create with BentoML version 0.9.0.pre, but loading from BentoML version 0.9.0.pre+7.g8af1c8b [2020-09-23 13:23:59,178] WARNING - Module `keras_fashion_mnist` already loaded, using existing imported module. WARNING:tensorflow:From /home/bentoml/BentoML/bentoml/frameworks/keras.py:137: The name tf.keras.backend.set_session is deprecated. Please use tf.compat.v1.keras.backend.set_session instead. [2020-09-23 13:24:00,085] WARNING - pip package requirement imageio already exist [2020-09-23 13:24:00,086] WARNING - pip package requirement tensorflow already exist [2020-09-23 13:24:00,087] WARNING - pip package requirement keras already exist
['Ankle boot']
BentoML cli supports loading and running a packaged model from CLI. With the DataframeInput adapter, the CLI command supports reading input Dataframe data from CLI argument or local csv or json files:
!bentoml run KerasFashionMnistService:latest predict --input-file sample_image.png
[2020-09-23 13:24:35,370] INFO - Getting latest version KerasFashionMnistService:20200923132350_974EEA [2020-09-23 13:24:36,401] WARNING - Using BentoML installed in `editable` model, the local BentoML repository including all code changes will be packaged together with saved bundle created, under the './bundled_pip_dependencies' directory of the saved bundle. [2020-09-23 13:24:36,419] WARNING - Saved BentoService bundle version mismatch: loading BentoService bundle create with BentoML version 0.9.0.pre, but loading from BentoML version 0.9.0.pre+7.g8af1c8b /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:516: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. _np_qint8 = np.dtype([("qint8", np.int8, 1)]) /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:517: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. _np_quint8 = np.dtype([("quint8", np.uint8, 1)]) /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:518: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. _np_qint16 = np.dtype([("qint16", np.int16, 1)]) /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:519: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. _np_quint16 = np.dtype([("quint16", np.uint16, 1)]) /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:520: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. _np_qint32 = np.dtype([("qint32", np.int32, 1)]) /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:525: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. np_resource = np.dtype([("resource", np.ubyte, 1)]) /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:541: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. _np_qint8 = np.dtype([("qint8", np.int8, 1)]) /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:542: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. _np_quint8 = np.dtype([("quint8", np.uint8, 1)]) /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:543: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. _np_qint16 = np.dtype([("qint16", np.int16, 1)]) /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:544: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. _np_quint16 = np.dtype([("quint16", np.uint16, 1)]) /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:545: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. _np_qint32 = np.dtype([("qint32", np.int32, 1)]) /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:550: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. np_resource = np.dtype([("resource", np.ubyte, 1)]) Using TensorFlow backend. 2020-09-23 13:24:37.559712: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcuda.so.1 2020-09-23 13:24:37.574229: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2020-09-23 13:24:37.574603: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 0 with properties: name: GeForce GTX 1060 major: 6 minor: 1 memoryClockRate(GHz): 1.6705 pciBusID: 0000:01:00.0 2020-09-23 13:24:37.574778: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.1 2020-09-23 13:24:37.576014: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10 2020-09-23 13:24:37.577344: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcufft.so.10 2020-09-23 13:24:37.577571: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcurand.so.10 2020-09-23 13:24:37.579057: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusolver.so.10 2020-09-23 13:24:37.579956: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusparse.so.10 2020-09-23 13:24:37.583071: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudnn.so.7 2020-09-23 13:24:37.583207: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2020-09-23 13:24:37.583615: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2020-09-23 13:24:37.583945: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1763] Adding visible gpu devices: 0 2020-09-23 13:24:37.584206: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.1 SSE4.2 AVX AVX2 FMA 2020-09-23 13:24:37.728318: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2020-09-23 13:24:37.728789: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x5557595e9bd0 executing computations on platform CUDA. Devices: 2020-09-23 13:24:37.728809: I tensorflow/compiler/xla/service/service.cc:175] StreamExecutor device (0): GeForce GTX 1060, Compute Capability 6.1 2020-09-23 13:24:37.728961: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2020-09-23 13:24:37.729290: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 0 with properties: name: GeForce GTX 1060 major: 6 minor: 1 memoryClockRate(GHz): 1.6705 pciBusID: 0000:01:00.0 2020-09-23 13:24:37.729326: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.1 2020-09-23 13:24:37.729340: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10 2020-09-23 13:24:37.729352: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcufft.so.10 2020-09-23 13:24:37.729363: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcurand.so.10 2020-09-23 13:24:37.729376: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusolver.so.10 2020-09-23 13:24:37.729388: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusparse.so.10 2020-09-23 13:24:37.729400: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudnn.so.7 2020-09-23 13:24:37.729447: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2020-09-23 13:24:37.729796: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2020-09-23 13:24:37.730123: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1763] Adding visible gpu devices: 0 2020-09-23 13:24:37.730164: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.1 2020-09-23 13:24:37.730641: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1181] Device interconnect StreamExecutor with strength 1 edge matrix: 2020-09-23 13:24:37.730654: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1187] 0 2020-09-23 13:24:37.730662: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 0: N 2020-09-23 13:24:37.730760: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2020-09-23 13:24:37.731141: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2020-09-23 13:24:37.731496: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 5683 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1060, pci bus id: 0000:01:00.0, compute capability: 6.1) 2020-09-23 13:24:37.756994: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2699905000 Hz 2020-09-23 13:24:37.757349: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x55575c5d4840 executing computations on platform Host. Devices: 2020-09-23 13:24:37.757417: I tensorflow/compiler/xla/service/service.cc:175] StreamExecutor device (0): <undefined>, <undefined> WARNING:tensorflow:From /home/bentoml/BentoML/bentoml/frameworks/keras.py:137: The name tf.keras.backend.set_session is deprecated. Please use tf.compat.v1.keras.backend.set_session instead. WARNING:tensorflow:From /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py:4070: The name tf.nn.max_pool is deprecated. Please use tf.nn.max_pool2d instead. WARNING:tensorflow:From /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py:422: The name tf.global_variables is deprecated. Please use tf.compat.v1.global_variables instead. WARNING:tensorflow:From /home/bentoml/BentoML/bentoml/frameworks/keras.py:123: The name tf.keras.backend.get_session is deprecated. Please use tf.compat.v1.keras.backend.get_session instead. [2020-09-23 13:24:38,862] WARNING - pip package requirement tensorflow already exist [2020-09-23 13:24:38,862] WARNING - pip package requirement keras already exist 2020-09-23 13:24:39.961990: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10 2020-09-23 13:24:40.109036: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudnn.so.7 [2020-09-23 13:24:40,664] INFO - {'service_name': 'KerasFashionMnistService', 'service_version': '20200923132350_974EEA', 'api': 'predict', 'task': {'data': {'uri': 'file:///home/bentoml/lab/gallery/legacy-keras/fashion-mnist/sample_image.png', 'name': 'sample_image.png'}, 'task_id': 'f350fdaa-9523-469b-8d73-8facb0945bb7', 'cli_args': ('--input-file', 'sample_image.png')}, 'result': {'data': '"Ankle boot"', 'http_status': 200, 'http_headers': (('Content-Type', 'application/json'),)}, 'request_id': 'f350fdaa-9523-469b-8d73-8facb0945bb7'} "Ankle boot"
To start a REST API model server with the BentoService saved above, use the bentoml serve command:
!bentoml serve KerasFashionMnistService:latest --enable-microbatch
[2020-09-23 13:24:48,477] INFO - Getting latest version KerasFashionMnistService:20200923132350_974EEA
[2020-09-23 13:24:48,477] INFO - Starting BentoML API server in development mode..
[2020-09-23 13:24:49,637] WARNING - Using BentoML installed in `editable` model, the local BentoML repository including all code changes will be packaged together with saved bundle created, under the './bundled_pip_dependencies' directory of the saved bundle.
[2020-09-23 13:24:49,654] WARNING - Saved BentoService bundle version mismatch: loading BentoService bundle create with BentoML version 0.9.0.pre, but loading from BentoML version 0.9.0.pre+7.g8af1c8b
/opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:516: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_qint8 = np.dtype([("qint8", np.int8, 1)])
/opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:517: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_quint8 = np.dtype([("quint8", np.uint8, 1)])
/opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:518: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_qint16 = np.dtype([("qint16", np.int16, 1)])
/opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:519: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_quint16 = np.dtype([("quint16", np.uint16, 1)])
/opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:520: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_qint32 = np.dtype([("qint32", np.int32, 1)])
/opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:525: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
np_resource = np.dtype([("resource", np.ubyte, 1)])
/opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:541: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_qint8 = np.dtype([("qint8", np.int8, 1)])
/opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:542: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_quint8 = np.dtype([("quint8", np.uint8, 1)])
/opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:543: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_qint16 = np.dtype([("qint16", np.int16, 1)])
/opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:544: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_quint16 = np.dtype([("quint16", np.uint16, 1)])
/opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:545: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_qint32 = np.dtype([("qint32", np.int32, 1)])
/opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:550: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
np_resource = np.dtype([("resource", np.ubyte, 1)])
Using TensorFlow backend.
2020-09-23 13:24:50.754920: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcuda.so.1
2020-09-23 13:24:50.769561: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-09-23 13:24:50.769934: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 0 with properties:
name: GeForce GTX 1060 major: 6 minor: 1 memoryClockRate(GHz): 1.6705
pciBusID: 0000:01:00.0
2020-09-23 13:24:50.770104: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.1
2020-09-23 13:24:50.771331: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10
2020-09-23 13:24:50.772609: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcufft.so.10
2020-09-23 13:24:50.772876: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcurand.so.10
2020-09-23 13:24:50.774317: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusolver.so.10
2020-09-23 13:24:50.775157: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusparse.so.10
2020-09-23 13:24:50.778709: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudnn.so.7
2020-09-23 13:24:50.778855: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-09-23 13:24:50.779265: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-09-23 13:24:50.779583: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1763] Adding visible gpu devices: 0
2020-09-23 13:24:50.779837: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.1 SSE4.2 AVX AVX2 FMA
2020-09-23 13:24:50.950505: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-09-23 13:24:50.950963: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x5597103fd600 executing computations on platform CUDA. Devices:
2020-09-23 13:24:50.950987: I tensorflow/compiler/xla/service/service.cc:175] StreamExecutor device (0): GeForce GTX 1060, Compute Capability 6.1
2020-09-23 13:24:50.951170: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-09-23 13:24:50.951547: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 0 with properties:
name: GeForce GTX 1060 major: 6 minor: 1 memoryClockRate(GHz): 1.6705
pciBusID: 0000:01:00.0
2020-09-23 13:24:50.951582: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.1
2020-09-23 13:24:50.951612: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10
2020-09-23 13:24:50.951638: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcufft.so.10
2020-09-23 13:24:50.951665: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcurand.so.10
2020-09-23 13:24:50.951679: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusolver.so.10
2020-09-23 13:24:50.951692: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusparse.so.10
2020-09-23 13:24:50.951731: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudnn.so.7
2020-09-23 13:24:50.951794: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-09-23 13:24:50.952186: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-09-23 13:24:50.952530: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1763] Adding visible gpu devices: 0
2020-09-23 13:24:50.952558: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.1
2020-09-23 13:24:50.953034: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1181] Device interconnect StreamExecutor with strength 1 edge matrix:
2020-09-23 13:24:50.953047: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1187] 0
2020-09-23 13:24:50.953053: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 0: N
2020-09-23 13:24:50.953157: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-09-23 13:24:50.953528: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-09-23 13:24:50.953875: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 5683 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1060, pci bus id: 0000:01:00.0, compute capability: 6.1)
2020-09-23 13:24:50.972930: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2699905000 Hz
2020-09-23 13:24:50.973274: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x5597135723e0 executing computations on platform Host. Devices:
2020-09-23 13:24:50.973362: I tensorflow/compiler/xla/service/service.cc:175] StreamExecutor device (0): <undefined>, <undefined>
WARNING:tensorflow:From /home/bentoml/BentoML/bentoml/frameworks/keras.py:137: The name tf.keras.backend.set_session is deprecated. Please use tf.compat.v1.keras.backend.set_session instead.
WARNING:tensorflow:From /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py:4070: The name tf.nn.max_pool is deprecated. Please use tf.nn.max_pool2d instead.
WARNING:tensorflow:From /opt/anaconda3/envs/bentoml-py3_6-tf1_4/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py:422: The name tf.global_variables is deprecated. Please use tf.compat.v1.global_variables instead.
WARNING:tensorflow:From /home/bentoml/BentoML/bentoml/frameworks/keras.py:123: The name tf.keras.backend.get_session is deprecated. Please use tf.compat.v1.keras.backend.get_session instead.
[2020-09-23 13:24:52,198] WARNING - pip package requirement tensorflow already exist
[2020-09-23 13:24:52,198] WARNING - pip package requirement keras already exist
[2020-09-23 13:24:53,251] WARNING - Saved BentoService bundle version mismatch: loading BentoService bundle create with BentoML version 0.9.0.pre, but loading from BentoML version 0.9.0.pre+7.g8af1c8b
[2020-09-23 13:24:53,253] INFO - Micro batch enabled for API `predict`
[2020-09-23 13:24:53,253] INFO - Your system nofile limit is 10000, which means each instance of microbatch service is able to hold this number of connections at same time. You can increase the number of file descriptors for the server process, or launch more microbatch instances to accept more concurrent connection.
[2020-09-23 13:24:53,267] INFO - Running micro batch service on :5000
* Serving Flask app "KerasFashionMnistService" (lazy loading)
* Environment: production
WARNING: This is a development server. Do not use it in a production deployment.
Use a production WSGI server instead.
* Debug mode: off
* Running on http://127.0.0.1:48301/ (Press CTRL+C to quit)
======== Running on http://0.0.0.0:5000 ========
(Press CTRL+C to quit)
^C
If you are running this notebook from Google Colab, you can start the dev server with --run-with-ngrok
option, to gain acccess to the API endpoint via a public endpoint managed by ngrok:
!bentoml serve KerasFashionMnistService:latest --run-with-ngrok
Test sending request to API Server from terminal:
curl -X POST "http://127.0.0.1:5000/predict" -F image=@sample_image.png
or:
curl -X POST "http://127.0.0.1:5000/predict" -H "Content-Type: image/png" --data-binary @sample_image.png
One common way of distributing this model API server for production deployment, is via Docker containers. And BentoML provides a convenient way to do that.
Note that docker is not available in Google Colab. You will need to download and run this notebook locally to try out this containerization with docker feature.
If you already have docker configured, simply run the follow command to product a docker container serving the IrisClassifier prediction service created above:
!bentoml containerize KerasFashionMnistService:latest -t kerasfashionmnistservice:latest
[2020-09-23 13:25:09,964] INFO - Getting latest version KerasFashionMnistService:20200923132350_974EEA Found Bento: /home/bentoml/bentoml/repository/KerasFashionMnistService/20200923132350_974EEA [2020-09-23 13:25:09,983] WARNING - Using BentoML installed in `editable` model, the local BentoML repository including all code changes will be packaged together with saved bundle created, under the './bundled_pip_dependencies' directory of the saved bundle. [2020-09-23 13:25:09,999] WARNING - Saved BentoService bundle version mismatch: loading BentoService bundle create with BentoML version 0.9.0.pre, but loading from BentoML version 0.9.0.pre+7.g8af1c8b Building Docker image kerasfashionmnistservice:latest from KerasFashionMnistService:latest -Step 1/15 : FROM bentoml/model-server:0.9.0.pre-py36 ---> 4aac43d10e50 Step 2/15 : ARG EXTRA_PIP_INSTALL_ARGS= ---> Using cache ---> 790054f5ad85 Step 3/15 : ENV EXTRA_PIP_INSTALL_ARGS $EXTRA_PIP_INSTALL_ARGS ---> Using cache ---> 85b0a1b40542 Step 4/15 : COPY environment.yml requirements.txt setup.sh* bentoml-init.sh python_version* /bento/ | ---> 8729a1f5c821 Step 5/15 : WORKDIR /bento ---> Running in 60907eb65c99 \ ---> 6df2fc7b9f5b Step 6/15 : RUN chmod +x /bento/bentoml-init.sh ---> Running in 5623e9822b02 | ---> 4c3fa98dc1df Step 7/15 : RUN if [ -f /bento/bentoml-init.sh ]; then bash -c /bento/bentoml-init.sh; fi ---> Running in 050ae928ea3d \+++ dirname /bento/bentoml-init.sh ++ cd /bento ++ pwd -P + SAVED_BUNDLE_PATH=/bento + cd /bento + '[' -f ./setup.sh ']' + '[' -f ./python_version ']' ++ cat ./python_version + PY_VERSION_SAVED=3.6.10 + DESIRED_PY_VERSION=3.6 ++ python -c 'import sys; print(f"{sys.version_info.major}.{sys.version_info.minor}")' + CURRENT_PY_VERSION=3.6 + [[ 3.6 == \3\.\6 ]] + echo 'Python Version in docker base image 3.6 matches requirement python=3.6. Skipping.' Python Version in docker base image 3.6 matches requirement python=3.6. Skipping. + command -v conda + echo 'Updating conda base environment with environment.yml' + conda env update -n base -f ./environment.yml Updating conda base environment with environment.yml |Collecting package metadata (repodata.json): ...working... |done Solving environment: ...working... -done Downloading and Extracting Packages openssl-1.1.1h | 2.1 MB | | 0% openssl-1.1.1h | 2.1 MB | | 1% openssl-1.1.1h | 2.1 MB | 2 | 3% openssl-1.1.1h | 2.1 MB | 4 | 4% openssl-1.1.1h | 2.1 MB | 5 | 6% openssl-1.1.1h | 2.1 MB | 7 | 7% openssl-1.1.1h | 2.1 MB | #3 | 13% openssl-1.1.1h | 2.1 MB | #9 | 19% openssl-1.1.1h | 2.1 MB | ##7 | 27% openssl-1.1.1h | 2.1 MB | ##9 | 29% openssl-1.1.1h | 2.1 MB | ###3 | 34% openssl-1.1.1h | 2.1 MB | ###6 | 36% openssl-1.1.1h | 2.1 MB | #### | 41% openssl-1.1.1h | 2.1 MB | ####1 | 42% openssl-1.1.1h | 2.1 MB | ####6 | 46% openssl-1.1.1h | 2.1 MB | ####8 | 49% openssl-1.1.1h | 2.1 MB | ##### | 51% openssl-1.1.1h | 2.1 MB | #####2 | 52% openssl-1.1.1h | 2.1 MB | #####5 | 55% openssl-1.1.1h | 2.1 MB | ###### | 60% openssl-1.1.1h | 2.1 MB | #######2 | 73% openssl-1.1.1h | 2.1 MB | ######## | 80% openssl-1.1.1h | 2.1 MB | ########1 | 82% openssl-1.1.1h | 2.1 MB | ########6 | 87% openssl-1.1.1h | 2.1 MB | ########9 | 89% openssl-1.1.1h | 2.1 MB | #########1 | 91% openssl-1.1.1h | 2.1 MB | #########5 | 96% openssl-1.1.1h | 2.1 MB | #########7 | 98% openssl-1.1.1h | 2.1 MB | ########## | 100% openssl-1.1.1h | 2.1 MB | ########## | 100% ca-certificates-2020 | 145 KB | | 0% ca-certificates-2020 | 145 KB | #1 | 11% ca-certificates-2020 | 145 KB | ##2 | 22% ca-certificates-2020 | 145 KB | ###3 | 33% ca-certificates-2020 | 145 KB | ######6 | 66% ca-certificates-2020 | 145 KB | ########## | 100% python_abi-3.6 | 4 KB | | 0% python_abi-3.6 | 4 KB | ########## | 100% python_abi-3.6 | 4 KB | ########## | 100% pip-20.2.3 | 1.1 MB | | 0% pip-20.2.3 | 1.1 MB | 1 | 1% pip-20.2.3 | 1.1 MB | 5 | 6% pip-20.2.3 | 1.1 MB | 8 | 9% pip-20.2.3 | 1.1 MB | #8 | 19% pip-20.2.3 | 1.1 MB | ##7 | 28% pip-20.2.3 | 1.1 MB | ###6 | 37% pip-20.2.3 | 1.1 MB | #####2 | 53% pip-20.2.3 | 1.1 MB | #####8 | 58% pip-20.2.3 | 1.1 MB | ######2 | 63% pip-20.2.3 | 1.1 MB | ######## | 80% pip-20.2.3 | 1.1 MB | ########4 | 85% pip-20.2.3 | 1.1 MB | ######### | 91% pip-20.2.3 | 1.1 MB | #########3 | 94% pip-20.2.3 | 1.1 MB | ########## | 100% pip-20.2.3 | 1.1 MB | ########## | 100% certifi-2020.6.20 | 151 KB | | 0% certifi-2020.6.20 | 151 KB | # | 11% certifi-2020.6.20 | 151 KB | ##1 | 21% certifi-2020.6.20 | 151 KB | ####2 | 42% certifi-2020.6.20 | 151 KB | #####3 | 53% certifi-2020.6.20 | 151 KB | ########4 | 85% certifi-2020.6.20 | 151 KB | ########## | 100% Preparing transaction: ...working... /done Verifying transaction: ...working... |done Executing transaction: ...working... \done /# # To activate this environment, use # # $ conda activate base # # To deactivate an active environment, use # # $ conda deactivate /+ pip install -r ./requirements.txt --no-cache-dir -Requirement already satisfied: bentoml==0.9.0.pre in /opt/conda/lib/python3.6/site-packages (from -r ./requirements.txt (line 1)) (0.9.0rc0) |Collecting keras==2.3.1 - Downloading Keras-2.3.1-py2.py3-none-any.whl (377 kB) /Collecting tensorflow==1.14.0 | Downloading tensorflow-1.14.0-cp36-cp36m-manylinux1_x86_64.whl (109.2 MB) -Collecting pillow==7.2.0 / Downloading Pillow-7.2.0-cp36-cp36m-manylinux1_x86_64.whl (2.2 MB) |Requirement already satisfied: numpy==1.19.2 in /opt/conda/lib/python3.6/site-packages (from -r ./requirements.txt (line 5)) (1.19.2) \Collecting imageio==2.9.0 / Downloading imageio-2.9.0-py3-none-any.whl (3.3 MB) -Requirement already satisfied: tabulate in /opt/conda/lib/python3.6/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (0.8.7) Requirement already satisfied: requests in /opt/conda/lib/python3.6/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (2.24.0) Requirement already satisfied: cerberus in /opt/conda/lib/python3.6/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (1.3.2) Requirement already satisfied: click>=7.0 in /opt/conda/lib/python3.6/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (7.1.2) Requirement already satisfied: py-zipkin in /opt/conda/lib/python3.6/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (0.20.0) Requirement already satisfied: prometheus-client in /opt/conda/lib/python3.6/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (0.8.0) Requirement already satisfied: docker in /opt/conda/lib/python3.6/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (4.3.1) Requirement already satisfied: certifi in /opt/conda/lib/python3.6/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (2020.6.20) Requirement already satisfied: python-json-logger in /opt/conda/lib/python3.6/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (0.1.11) Requirement already satisfied: alembic in /opt/conda/lib/python3.6/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (1.4.3) Requirement already satisfied: boto3 in /opt/conda/lib/python3.6/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (1.15.2) Requirement already satisfied: protobuf>=3.6.0 in /opt/conda/lib/python3.6/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (3.13.0) Requirement already satisfied: python-dateutil<3.0.0,>=2.7.3 in /opt/conda/lib/python3.6/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (2.8.1) /Requirement already satisfied: ruamel.yaml>=0.15.0 in /opt/conda/lib/python3.6/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (0.15.87) Requirement already satisfied: sqlalchemy-utils<0.36.8 in /opt/conda/lib/python3.6/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (0.36.7) Requirement already satisfied: humanfriendly in /opt/conda/lib/python3.6/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (8.2) Requirement already satisfied: psutil in /opt/conda/lib/python3.6/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (5.7.2) Requirement already satisfied: dataclasses; python_version < "3.7" in /opt/conda/lib/python3.6/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (0.7) Requirement already satisfied: packaging in /opt/conda/lib/python3.6/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (20.4) Requirement already satisfied: aiohttp in /opt/conda/lib/python3.6/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (3.6.2) |Requirement already satisfied: contextvars; python_version < "3.7" in /opt/conda/lib/python3.6/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (2.4) Requirement already satisfied: grpcio<=1.27.2 in /opt/conda/lib/python3.6/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (1.27.2) Requirement already satisfied: multidict in /opt/conda/lib/python3.6/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (4.7.6) Requirement already satisfied: sqlalchemy>=1.3.0 in /opt/conda/lib/python3.6/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (1.3.19) Requirement already satisfied: gunicorn in /opt/conda/lib/python3.6/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (20.0.4) Requirement already satisfied: configparser in /opt/conda/lib/python3.6/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (5.0.0) Requirement already satisfied: flask in /opt/conda/lib/python3.6/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (1.1.2) -Collecting keras-preprocessing>=1.0.5 / Downloading Keras_Preprocessing-1.1.2-py2.py3-none-any.whl (42 kB) \Requirement already satisfied: six>=1.9.0 in /opt/conda/lib/python3.6/site-packages (from keras==2.3.1->-r ./requirements.txt (line 2)) (1.15.0) -Collecting scipy>=0.14 / Downloading scipy-1.5.2-cp36-cp36m-manylinux1_x86_64.whl (25.9 MB) -Collecting pyyaml | Downloading PyYAML-5.3.1.tar.gz (269 kB) /Collecting keras-applications>=1.0.6 | Downloading Keras_Applications-1.0.8-py3-none-any.whl (50 kB) |Collecting h5py \ Downloading h5py-2.10.0-cp36-cp36m-manylinux1_x86_64.whl (2.9 MB) |Collecting tensorflow-estimator<1.15.0rc0,>=1.14.0rc0 \ Downloading tensorflow_estimator-1.14.0-py2.py3-none-any.whl (488 kB) /Collecting astor>=0.6.0 \ Downloading astor-0.8.1-py2.py3-none-any.whl (27 kB) Requirement already satisfied: wheel>=0.26 in /opt/conda/lib/python3.6/site-packages (from tensorflow==1.14.0->-r ./requirements.txt (line 3)) (0.35.1) /Collecting absl-py>=0.7.0 | Downloading absl_py-0.10.0-py3-none-any.whl (127 kB) |Collecting tensorboard<1.15.0,>=1.14.0 - Downloading tensorboard-1.14.0-py3-none-any.whl (3.1 MB) -Collecting wrapt>=1.11.1 / Downloading wrapt-1.12.1.tar.gz (27 kB) |Collecting google-pasta>=0.1.6 \ Downloading google_pasta-0.2.0-py3-none-any.whl (57 kB) /Collecting termcolor>=1.1.0 \ Downloading termcolor-1.1.0.tar.gz (3.9 kB) \Collecting gast>=0.2.0 - Downloading gast-0.4.0-py3-none-any.whl (9.8 kB) Requirement already satisfied: chardet<4,>=3.0.2 in /opt/conda/lib/python3.6/site-packages (from requests->bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (3.0.4) Requirement already satisfied: idna<3,>=2.5 in /opt/conda/lib/python3.6/site-packages (from requests->bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (2.10) Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /opt/conda/lib/python3.6/site-packages (from requests->bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (1.25.10) Requirement already satisfied: setuptools in /opt/conda/lib/python3.6/site-packages (from cerberus->bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (49.6.0.post20200814) Requirement already satisfied: thriftpy2>=0.4.0 in /opt/conda/lib/python3.6/site-packages (from py-zipkin->bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (0.4.11) Requirement already satisfied: websocket-client>=0.32.0 in /opt/conda/lib/python3.6/site-packages (from docker->bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (0.57.0) Requirement already satisfied: python-editor>=0.3 in /opt/conda/lib/python3.6/site-packages (from alembic->bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (1.0.4) Requirement already satisfied: Mako in /opt/conda/lib/python3.6/site-packages (from alembic->bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (1.1.3) Requirement already satisfied: botocore<1.19.0,>=1.18.2 in /opt/conda/lib/python3.6/site-packages (from boto3->bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (1.18.2) /Requirement already satisfied: jmespath<1.0.0,>=0.7.1 in /opt/conda/lib/python3.6/site-packages (from boto3->bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (0.10.0) Requirement already satisfied: s3transfer<0.4.0,>=0.3.0 in /opt/conda/lib/python3.6/site-packages (from boto3->bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (0.3.3) Requirement already satisfied: pyparsing>=2.0.2 in /opt/conda/lib/python3.6/site-packages (from packaging->bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (2.4.7) Requirement already satisfied: attrs>=17.3.0 in /opt/conda/lib/python3.6/site-packages (from aiohttp->bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (20.2.0) Requirement already satisfied: typing-extensions>=3.6.5; python_version < "3.7" in /opt/conda/lib/python3.6/site-packages (from aiohttp->bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (3.7.4.3) Requirement already satisfied: yarl<2.0,>=1.0 in /opt/conda/lib/python3.6/site-packages (from aiohttp->bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (1.5.1) Requirement already satisfied: idna-ssl>=1.0; python_version < "3.7" in /opt/conda/lib/python3.6/site-packages (from aiohttp->bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (1.1.0) Requirement already satisfied: async-timeout<4.0,>=3.0 in /opt/conda/lib/python3.6/site-packages (from aiohttp->bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (3.0.1) Requirement already satisfied: immutables>=0.9 in /opt/conda/lib/python3.6/site-packages (from contextvars; python_version < "3.7"->bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (0.14) Requirement already satisfied: Werkzeug>=0.15 in /opt/conda/lib/python3.6/site-packages (from flask->bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (1.0.1) Requirement already satisfied: itsdangerous>=0.24 in /opt/conda/lib/python3.6/site-packages (from flask->bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (1.1.0) Requirement already satisfied: Jinja2>=2.10.1 in /opt/conda/lib/python3.6/site-packages (from flask->bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (2.11.2) \Collecting markdown>=2.6.8 - Downloading Markdown-3.2.2-py3-none-any.whl (88 kB) |Requirement already satisfied: ply<4.0,>=3.4 in /opt/conda/lib/python3.6/site-packages (from thriftpy2>=0.4.0->py-zipkin->bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (3.11) Requirement already satisfied: MarkupSafe>=0.9.2 in /opt/conda/lib/python3.6/site-packages (from Mako->alembic->bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (1.1.1) /Collecting importlib-metadata; python_version < "3.8" | Downloading importlib_metadata-2.0.0-py2.py3-none-any.whl (31 kB) -Collecting zipp>=0.5 | Downloading zipp-3.2.0-py3-none-any.whl (5.1 kB) Building wheels for collected packages: pyyaml, wrapt, termcolor Building wheel for pyyaml (setup.py): started | Building wheel for pyyaml (setup.py): finished with status 'done' Created wheel for pyyaml: filename=PyYAML-5.3.1-cp36-cp36m-linux_x86_64.whl size=44619 sha256=bc3a34861f160832e54226e9dfc5886ed6a1f193d13bc07bd8cc6c07e9252140 Stored in directory: /tmp/pip-ephem-wheel-cache-n4qcgfhc/wheels/e5/9d/ad/2ee53cf262cba1ffd8afe1487eef788ea3f260b7e6232a80fc Building wheel for wrapt (setup.py): started | Building wheel for wrapt (setup.py): finished with status 'done' Created wheel for wrapt: filename=wrapt-1.12.1-cp36-cp36m-linux_x86_64.whl size=75206 sha256=3e1a885724afaa51a425c64bc576f9bcac3f850636e59c10ee14160a0d4d9340 Stored in directory: /tmp/pip-ephem-wheel-cache-n4qcgfhc/wheels/32/42/7f/23cae9ff6ef66798d00dc5d659088e57dbba01566f6c60db63 Building wheel for termcolor (setup.py): started / Building wheel for termcolor (setup.py): finished with status 'done' Created wheel for termcolor: filename=termcolor-1.1.0-py3-none-any.whl size=4830 sha256=66929de748b95c287b1919fc0cbd3c4337b4897dbc42ebcda4be7c6cbdc707fa Stored in directory: /tmp/pip-ephem-wheel-cache-n4qcgfhc/wheels/93/2a/eb/e58dbcbc963549ee4f065ff80a59f274cc7210b6eab962acdc Successfully built pyyaml wrapt termcolor \Installing collected packages: keras-preprocessing, scipy, pyyaml, h5py, keras-applications, keras, tensorflow-estimator, astor, absl-py, zipp, importlib-metadata, markdown, tensorboard, wrapt, google-pasta, termcolor, gast, tensorflow, pillow, imageio /Successfully installed absl-py-0.10.0 astor-0.8.1 gast-0.4.0 google-pasta-0.2.0 h5py-2.10.0 imageio-2.9.0 importlib-metadata-2.0.0 keras-2.3.1 keras-applications-1.0.8 keras-preprocessing-1.1.2 markdown-3.2.2 pillow-7.2.0 pyyaml-5.3.1 scipy-1.5.2 tensorboard-1.14.0 tensorflow-1.14.0 tensorflow-estimator-1.14.0 termcolor-1.1.0 wrapt-1.12.1 zipp-3.2.0 | ---> f67cf901ef18 Step 8/15 : COPY . /bento / ---> 4a58550a0fad Step 9/15 : RUN if [ -d /bento/bundled_pip_dependencies ]; then pip install -U bundled_pip_dependencies/* ;fi | ---> Running in 98757f1e8214 -Processing ./bundled_pip_dependencies/BentoML-0.9.0rc0+7.g8af1c8b.tar.gz / Installing build dependencies: started - Installing build dependencies: finished with status 'done' Getting requirements to build wheel: started / Getting requirements to build wheel: finished with status 'done' Preparing wheel metadata: started | Preparing wheel metadata: finished with status 'done' \Requirement already satisfied, skipping upgrade: grpcio<=1.27.2 in /opt/conda/lib/python3.6/site-packages (from BentoML==0.9.0rc0+7.g8af1c8b) (1.27.2) Requirement already satisfied, skipping upgrade: boto3 in /opt/conda/lib/python3.6/site-packages (from BentoML==0.9.0rc0+7.g8af1c8b) (1.15.2) Requirement already satisfied, skipping upgrade: numpy in /opt/conda/lib/python3.6/site-packages (from BentoML==0.9.0rc0+7.g8af1c8b) (1.19.2) -Requirement already satisfied, skipping upgrade: prometheus-client in /opt/conda/lib/python3.6/site-packages (from BentoML==0.9.0rc0+7.g8af1c8b) (0.8.0) Requirement already satisfied, skipping upgrade: sqlalchemy-utils<0.36.8 in /opt/conda/lib/python3.6/site-packages (from BentoML==0.9.0rc0+7.g8af1c8b) (0.36.7) Requirement already satisfied, skipping upgrade: gunicorn in /opt/conda/lib/python3.6/site-packages (from BentoML==0.9.0rc0+7.g8af1c8b) (20.0.4) Requirement already satisfied, skipping upgrade: requests in /opt/conda/lib/python3.6/site-packages (from BentoML==0.9.0rc0+7.g8af1c8b) (2.24.0) Requirement already satisfied, skipping upgrade: psutil in /opt/conda/lib/python3.6/site-packages (from BentoML==0.9.0rc0+7.g8af1c8b) (5.7.2) /Requirement already satisfied, skipping upgrade: packaging in /opt/conda/lib/python3.6/site-packages (from BentoML==0.9.0rc0+7.g8af1c8b) (20.4) Requirement already satisfied, skipping upgrade: contextvars; python_version < "3.7" in /opt/conda/lib/python3.6/site-packages (from BentoML==0.9.0rc0+7.g8af1c8b) (2.4) Requirement already satisfied, skipping upgrade: docker in /opt/conda/lib/python3.6/site-packages (from BentoML==0.9.0rc0+7.g8af1c8b) (4.3.1) Requirement already satisfied, skipping upgrade: aiohttp in /opt/conda/lib/python3.6/site-packages (from BentoML==0.9.0rc0+7.g8af1c8b) (3.6.2) Requirement already satisfied, skipping upgrade: dataclasses; python_version < "3.7" in /opt/conda/lib/python3.6/site-packages (from BentoML==0.9.0rc0+7.g8af1c8b) (0.7) Requirement already satisfied, skipping upgrade: humanfriendly in /opt/conda/lib/python3.6/site-packages (from BentoML==0.9.0rc0+7.g8af1c8b) (8.2) Requirement already satisfied, skipping upgrade: tabulate in /opt/conda/lib/python3.6/site-packages (from BentoML==0.9.0rc0+7.g8af1c8b) (0.8.7) Requirement already satisfied, skipping upgrade: ruamel.yaml>=0.15.0 in /opt/conda/lib/python3.6/site-packages (from BentoML==0.9.0rc0+7.g8af1c8b) (0.15.87) Requirement already satisfied, skipping upgrade: py-zipkin in /opt/conda/lib/python3.6/site-packages (from BentoML==0.9.0rc0+7.g8af1c8b) (0.20.0) Requirement already satisfied, skipping upgrade: python-dateutil<3.0.0,>=2.7.3 in /opt/conda/lib/python3.6/site-packages (from BentoML==0.9.0rc0+7.g8af1c8b) (2.8.1) Requirement already satisfied, skipping upgrade: certifi in /opt/conda/lib/python3.6/site-packages (from BentoML==0.9.0rc0+7.g8af1c8b) (2020.6.20) Requirement already satisfied, skipping upgrade: multidict in /opt/conda/lib/python3.6/site-packages (from BentoML==0.9.0rc0+7.g8af1c8b) (4.7.6) Requirement already satisfied, skipping upgrade: protobuf>=3.6.0 in /opt/conda/lib/python3.6/site-packages (from BentoML==0.9.0rc0+7.g8af1c8b) (3.13.0) Requirement already satisfied, skipping upgrade: flask in /opt/conda/lib/python3.6/site-packages (from BentoML==0.9.0rc0+7.g8af1c8b) (1.1.2) Requirement already satisfied, skipping upgrade: alembic in /opt/conda/lib/python3.6/site-packages (from BentoML==0.9.0rc0+7.g8af1c8b) (1.4.3) Requirement already satisfied, skipping upgrade: cerberus in /opt/conda/lib/python3.6/site-packages (from BentoML==0.9.0rc0+7.g8af1c8b) (1.3.2) Requirement already satisfied, skipping upgrade: python-json-logger in /opt/conda/lib/python3.6/site-packages (from BentoML==0.9.0rc0+7.g8af1c8b) (0.1.11) Requirement already satisfied, skipping upgrade: click>=7.0 in /opt/conda/lib/python3.6/site-packages (from BentoML==0.9.0rc0+7.g8af1c8b) (7.1.2) Requirement already satisfied, skipping upgrade: sqlalchemy>=1.3.0 in /opt/conda/lib/python3.6/site-packages (from BentoML==0.9.0rc0+7.g8af1c8b) (1.3.19) Requirement already satisfied, skipping upgrade: configparser in /opt/conda/lib/python3.6/site-packages (from BentoML==0.9.0rc0+7.g8af1c8b) (5.0.0) |Requirement already satisfied, skipping upgrade: six>=1.5.2 in /opt/conda/lib/python3.6/site-packages (from grpcio<=1.27.2->BentoML==0.9.0rc0+7.g8af1c8b) (1.15.0) Requirement already satisfied, skipping upgrade: s3transfer<0.4.0,>=0.3.0 in /opt/conda/lib/python3.6/site-packages (from boto3->BentoML==0.9.0rc0+7.g8af1c8b) (0.3.3) Requirement already satisfied, skipping upgrade: jmespath<1.0.0,>=0.7.1 in /opt/conda/lib/python3.6/site-packages (from boto3->BentoML==0.9.0rc0+7.g8af1c8b) (0.10.0) Requirement already satisfied, skipping upgrade: botocore<1.19.0,>=1.18.2 in /opt/conda/lib/python3.6/site-packages (from boto3->BentoML==0.9.0rc0+7.g8af1c8b) (1.18.2) Requirement already satisfied, skipping upgrade: setuptools>=3.0 in /opt/conda/lib/python3.6/site-packages (from gunicorn->BentoML==0.9.0rc0+7.g8af1c8b) (49.6.0.post20200814) Requirement already satisfied, skipping upgrade: idna<3,>=2.5 in /opt/conda/lib/python3.6/site-packages (from requests->BentoML==0.9.0rc0+7.g8af1c8b) (2.10) Requirement already satisfied, skipping upgrade: chardet<4,>=3.0.2 in /opt/conda/lib/python3.6/site-packages (from requests->BentoML==0.9.0rc0+7.g8af1c8b) (3.0.4) Requirement already satisfied, skipping upgrade: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /opt/conda/lib/python3.6/site-packages (from requests->BentoML==0.9.0rc0+7.g8af1c8b) (1.25.10) Requirement already satisfied, skipping upgrade: pyparsing>=2.0.2 in /opt/conda/lib/python3.6/site-packages (from packaging->BentoML==0.9.0rc0+7.g8af1c8b) (2.4.7) Requirement already satisfied, skipping upgrade: immutables>=0.9 in /opt/conda/lib/python3.6/site-packages (from contextvars; python_version < "3.7"->BentoML==0.9.0rc0+7.g8af1c8b) (0.14) Requirement already satisfied, skipping upgrade: websocket-client>=0.32.0 in /opt/conda/lib/python3.6/site-packages (from docker->BentoML==0.9.0rc0+7.g8af1c8b) (0.57.0) Requirement already satisfied, skipping upgrade: async-timeout<4.0,>=3.0 in /opt/conda/lib/python3.6/site-packages (from aiohttp->BentoML==0.9.0rc0+7.g8af1c8b) (3.0.1) Requirement already satisfied, skipping upgrade: typing-extensions>=3.6.5; python_version < "3.7" in /opt/conda/lib/python3.6/site-packages (from aiohttp->BentoML==0.9.0rc0+7.g8af1c8b) (3.7.4.3) Requirement already satisfied, skipping upgrade: attrs>=17.3.0 in /opt/conda/lib/python3.6/site-packages (from aiohttp->BentoML==0.9.0rc0+7.g8af1c8b) (20.2.0) Requirement already satisfied, skipping upgrade: yarl<2.0,>=1.0 in /opt/conda/lib/python3.6/site-packages (from aiohttp->BentoML==0.9.0rc0+7.g8af1c8b) (1.5.1) Requirement already satisfied, skipping upgrade: idna-ssl>=1.0; python_version < "3.7" in /opt/conda/lib/python3.6/site-packages (from aiohttp->BentoML==0.9.0rc0+7.g8af1c8b) (1.1.0) Requirement already satisfied, skipping upgrade: thriftpy2>=0.4.0 in /opt/conda/lib/python3.6/site-packages (from py-zipkin->BentoML==0.9.0rc0+7.g8af1c8b) (0.4.11) \Requirement already satisfied, skipping upgrade: Werkzeug>=0.15 in /opt/conda/lib/python3.6/site-packages (from flask->BentoML==0.9.0rc0+7.g8af1c8b) (1.0.1) Requirement already satisfied, skipping upgrade: Jinja2>=2.10.1 in /opt/conda/lib/python3.6/site-packages (from flask->BentoML==0.9.0rc0+7.g8af1c8b) (2.11.2) Requirement already satisfied, skipping upgrade: itsdangerous>=0.24 in /opt/conda/lib/python3.6/site-packages (from flask->BentoML==0.9.0rc0+7.g8af1c8b) (1.1.0) Requirement already satisfied, skipping upgrade: python-editor>=0.3 in /opt/conda/lib/python3.6/site-packages (from alembic->BentoML==0.9.0rc0+7.g8af1c8b) (1.0.4) Requirement already satisfied, skipping upgrade: Mako in /opt/conda/lib/python3.6/site-packages (from alembic->BentoML==0.9.0rc0+7.g8af1c8b) (1.1.3) Requirement already satisfied, skipping upgrade: ply<4.0,>=3.4 in /opt/conda/lib/python3.6/site-packages (from thriftpy2>=0.4.0->py-zipkin->BentoML==0.9.0rc0+7.g8af1c8b) (3.11) Requirement already satisfied, skipping upgrade: MarkupSafe>=0.23 in /opt/conda/lib/python3.6/site-packages (from Jinja2>=2.10.1->flask->BentoML==0.9.0rc0+7.g8af1c8b) (1.1.1) Building wheels for collected packages: BentoML Building wheel for BentoML (PEP 517): started | Building wheel for BentoML (PEP 517): finished with status 'done' Created wheel for BentoML: filename=BentoML-0.9.0rc0+7.g8af1c8b-py3-none-any.whl size=620483 sha256=c0a193a2b7442065fb7abec76df7be020511af5c5a8ea34107d58bc21b9760e1 Stored in directory: /root/.cache/pip/wheels/0f/c2/e2/64640eb406a012df3abf685e4e5fa441b87b3e609903d041b6 Successfully built BentoML |Installing collected packages: BentoML Attempting uninstall: BentoML Found existing installation: BentoML 0.9.0rc0 \ Uninstalling BentoML-0.9.0rc0: / Successfully uninstalled BentoML-0.9.0rc0 -Successfully installed BentoML-0.9.0rc0+7.g8af1c8b \ ---> 40090207374a Step 10/15 : ENV PORT 5000 - ---> Running in bef4344724f5 | ---> b336b939b877 Step 11/15 : EXPOSE $PORT ---> Running in 94b0d4042e8c - ---> 6dbdacacc9ee Step 12/15 : COPY docker-entrypoint.sh /usr/local/bin/ | ---> 08f3c0134892 Step 13/15 : RUN chmod +x /usr/local/bin/docker-entrypoint.sh \ ---> Running in 919280fa3744 | ---> 0c8922f41ad5 Step 14/15 : ENTRYPOINT [ "docker-entrypoint.sh" ] \ ---> Running in cf1a37e5f312 / ---> 084525a4fb10 Step 15/15 : CMD ["bentoml", "serve-gunicorn", "/bento"] | ---> Running in ae9aab696766 - ---> df4a2b1bf7e6 Successfully built df4a2b1bf7e6 Successfully tagged kerasfashionmnistservice:latest Finished building kerasfashionmnistservice:latest from KerasFashionMnistService:latest
!docker run -p 5000:5000 kerasfashionmnistservice
[2020-09-23 05:35:06,581] INFO - Starting BentoML API server in production mode.. [2020-09-23 05:35:06,806] INFO - get_gunicorn_num_of_workers: 3, calculated by cpu count [2020-09-23 05:35:06 +0000] [1] [INFO] Starting gunicorn 20.0.4 [2020-09-23 05:35:06 +0000] [1] [INFO] Listening at: http://0.0.0.0:5000 (1) [2020-09-23 05:35:06 +0000] [1] [INFO] Using worker: sync [2020-09-23 05:35:06 +0000] [13] [INFO] Booting worker with pid: 13 [2020-09-23 05:35:06 +0000] [14] [INFO] Booting worker with pid: 14 [2020-09-23 05:35:06 +0000] [15] [INFO] Booting worker with pid: 15 [2020-09-23 05:35:07,826] WARNING - Using BentoML not from official PyPI release. In order to find the same version of BentoML when deploying your BentoService, you must set the 'core/bentoml_deploy_version' config to a http/git location of your BentoML fork, e.g.: 'bentoml_deploy_version = git+https://github.com/{username}/bentoml.git@{branch}' [2020-09-23 05:35:07,847] WARNING - Saved BentoService bundle version mismatch: loading BentoService bundle create with BentoML version 0.9.0.pre, but loading from BentoML version 0.9.0.pre+7.g8af1c8b [2020-09-23 05:35:07,847] WARNING - Saved BentoService Python version mismatch: loading BentoService bundle created with Python version 3.6.10, but current environment version is 3.6.12. [2020-09-23 05:35:07,855] WARNING - Using BentoML not from official PyPI release. In order to find the same version of BentoML when deploying your BentoService, you must set the 'core/bentoml_deploy_version' config to a http/git location of your BentoML fork, e.g.: 'bentoml_deploy_version = git+https://github.com/{username}/bentoml.git@{branch}' [2020-09-23 05:35:07,868] WARNING - Using BentoML not from official PyPI release. In order to find the same version of BentoML when deploying your BentoService, you must set the 'core/bentoml_deploy_version' config to a http/git location of your BentoML fork, e.g.: 'bentoml_deploy_version = git+https://github.com/{username}/bentoml.git@{branch}' [2020-09-23 05:35:07,877] WARNING - Saved BentoService bundle version mismatch: loading BentoService bundle create with BentoML version 0.9.0.pre, but loading from BentoML version 0.9.0.pre+7.g8af1c8b [2020-09-23 05:35:07,877] WARNING - Saved BentoService Python version mismatch: loading BentoService bundle created with Python version 3.6.10, but current environment version is 3.6.12. [2020-09-23 05:35:07,900] WARNING - Saved BentoService bundle version mismatch: loading BentoService bundle create with BentoML version 0.9.0.pre, but loading from BentoML version 0.9.0.pre+7.g8af1c8b [2020-09-23 05:35:07,900] WARNING - Saved BentoService Python version mismatch: loading BentoService bundle created with Python version 3.6.10, but current environment version is 3.6.12. 2020-09-23 05:35:09.265543: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA 2020-09-23 05:35:09.288840: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2699905000 Hz 2020-09-23 05:35:09.289083: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x5637744043a0 executing computations on platform Host. Devices: 2020-09-23 05:35:09.289103: I tensorflow/compiler/xla/service/service.cc:175] StreamExecutor device (0): <undefined>, <undefined> /opt/conda/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:516: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. _np_qint8 = np.dtype([("qint8", np.int8, 1)]) /opt/conda/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:517: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. _np_quint8 = np.dtype([("quint8", np.uint8, 1)]) /opt/conda/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:518: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. _np_qint16 = np.dtype([("qint16", np.int16, 1)]) /opt/conda/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:519: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. _np_quint16 = np.dtype([("quint16", np.uint16, 1)]) /opt/conda/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:520: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. _np_qint32 = np.dtype([("qint32", np.int32, 1)]) /opt/conda/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:525: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. np_resource = np.dtype([("resource", np.ubyte, 1)]) /opt/conda/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:541: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. _np_qint8 = np.dtype([("qint8", np.int8, 1)]) /opt/conda/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:542: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. _np_quint8 = np.dtype([("quint8", np.uint8, 1)]) /opt/conda/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:543: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. _np_qint16 = np.dtype([("qint16", np.int16, 1)]) /opt/conda/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:544: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. _np_quint16 = np.dtype([("quint16", np.uint16, 1)]) /opt/conda/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:545: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. _np_qint32 = np.dtype([("qint32", np.int32, 1)]) /opt/conda/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:550: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. np_resource = np.dtype([("resource", np.ubyte, 1)]) Using TensorFlow backend. WARNING:tensorflow:From /opt/conda/lib/python3.6/site-packages/bentoml/frameworks/keras.py:137: The name tf.keras.backend.set_session is deprecated. Please use tf.compat.v1.keras.backend.set_session instead. WARNING:tensorflow:From /opt/conda/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py:4070: The name tf.nn.max_pool is deprecated. Please use tf.nn.max_pool2d instead. 2020-09-23 05:35:09.320438: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA 2020-09-23 05:35:09.328772: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2699905000 Hz 2020-09-23 05:35:09.328969: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x563774404be0 executing computations on platform Host. Devices: 2020-09-23 05:35:09.328987: I tensorflow/compiler/xla/service/service.cc:175] StreamExecutor device (0): <undefined>, <undefined> /opt/conda/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:516: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. _np_qint8 = np.dtype([("qint8", np.int8, 1)]) /opt/conda/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:517: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. _np_quint8 = np.dtype([("quint8", np.uint8, 1)]) /opt/conda/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:518: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. _np_qint16 = np.dtype([("qint16", np.int16, 1)]) /opt/conda/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:519: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. _np_quint16 = np.dtype([("quint16", np.uint16, 1)]) /opt/conda/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:520: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. _np_qint32 = np.dtype([("qint32", np.int32, 1)]) /opt/conda/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:525: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. np_resource = np.dtype([("resource", np.ubyte, 1)]) /opt/conda/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:541: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. _np_qint8 = np.dtype([("qint8", np.int8, 1)]) /opt/conda/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:542: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. _np_quint8 = np.dtype([("quint8", np.uint8, 1)]) /opt/conda/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:543: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. _np_qint16 = np.dtype([("qint16", np.int16, 1)]) /opt/conda/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:544: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. _np_quint16 = np.dtype([("quint16", np.uint16, 1)]) /opt/conda/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:545: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. _np_qint32 = np.dtype([("qint32", np.int32, 1)]) /opt/conda/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:550: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. np_resource = np.dtype([("resource", np.ubyte, 1)]) Using TensorFlow backend. WARNING:tensorflow:From /opt/conda/lib/python3.6/site-packages/bentoml/frameworks/keras.py:137: The name tf.keras.backend.set_session is deprecated. Please use tf.compat.v1.keras.backend.set_session instead. WARNING:tensorflow:From /opt/conda/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py:4070: The name tf.nn.max_pool is deprecated. Please use tf.nn.max_pool2d instead. 2020-09-23 05:35:09.378158: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA 2020-09-23 05:35:09.388984: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2699905000 Hz 2020-09-23 05:35:09.389196: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x563774403ae0 executing computations on platform Host. Devices: 2020-09-23 05:35:09.389212: I tensorflow/compiler/xla/service/service.cc:175] StreamExecutor device (0): <undefined>, <undefined> /opt/conda/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:516: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. _np_qint8 = np.dtype([("qint8", np.int8, 1)]) /opt/conda/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:517: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. _np_quint8 = np.dtype([("quint8", np.uint8, 1)]) /opt/conda/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:518: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. _np_qint16 = np.dtype([("qint16", np.int16, 1)]) /opt/conda/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:519: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. _np_quint16 = np.dtype([("quint16", np.uint16, 1)]) /opt/conda/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:520: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. _np_qint32 = np.dtype([("qint32", np.int32, 1)]) /opt/conda/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:525: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. np_resource = np.dtype([("resource", np.ubyte, 1)]) /opt/conda/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:541: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. _np_qint8 = np.dtype([("qint8", np.int8, 1)]) /opt/conda/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:542: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. _np_quint8 = np.dtype([("quint8", np.uint8, 1)]) /opt/conda/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:543: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. _np_qint16 = np.dtype([("qint16", np.int16, 1)]) /opt/conda/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:544: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. _np_quint16 = np.dtype([("quint16", np.uint16, 1)]) /opt/conda/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:545: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. _np_qint32 = np.dtype([("qint32", np.int32, 1)]) /opt/conda/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:550: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'. np_resource = np.dtype([("resource", np.ubyte, 1)]) Using TensorFlow backend. WARNING:tensorflow:From /opt/conda/lib/python3.6/site-packages/bentoml/frameworks/keras.py:137: The name tf.keras.backend.set_session is deprecated. Please use tf.compat.v1.keras.backend.set_session instead. WARNING:tensorflow:From /opt/conda/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py:4070: The name tf.nn.max_pool is deprecated. Please use tf.nn.max_pool2d instead. 2020-09-23 05:35:09.477116: W tensorflow/compiler/jit/mark_for_compilation_pass.cc:1412] (One-time warning): Not using XLA:CPU for cluster because envvar TF_XLA_FLAGS=--tf_xla_cpu_global_jit was not set. If you want XLA:CPU, either set that envvar, or use experimental_jit_scope to enable XLA:CPU. To confirm that XLA is active, pass --vmodule=xla_compilation_cache=1 (as a proper command-line flag, not via TF_XLA_FLAGS) or set the envvar XLA_FLAGS=--xla_hlo_profile. 2020-09-23 05:35:09.535995: W tensorflow/compiler/jit/mark_for_compilation_pass.cc:1412] (One-time warning): Not using XLA:CPU for cluster because envvar TF_XLA_FLAGS=--tf_xla_cpu_global_jit was not set. If you want XLA:CPU, either set that envvar, or use experimental_jit_scope to enable XLA:CPU. To confirm that XLA is active, pass --vmodule=xla_compilation_cache=1 (as a proper command-line flag, not via TF_XLA_FLAGS) or set the envvar XLA_FLAGS=--xla_hlo_profile. 2020-09-23 05:35:09.582936: W tensorflow/compiler/jit/mark_for_compilation_pass.cc:1412] (One-time warning): Not using XLA:CPU for cluster because envvar TF_XLA_FLAGS=--tf_xla_cpu_global_jit was not set. If you want XLA:CPU, either set that envvar, or use experimental_jit_scope to enable XLA:CPU. To confirm that XLA is active, pass --vmodule=xla_compilation_cache=1 (as a proper command-line flag, not via TF_XLA_FLAGS) or set the envvar XLA_FLAGS=--xla_hlo_profile. WARNING:tensorflow:From /opt/conda/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py:422: The name tf.global_variables is deprecated. Please use tf.compat.v1.global_variables instead. WARNING:tensorflow:From /opt/conda/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py:422: The name tf.global_variables is deprecated. Please use tf.compat.v1.global_variables instead. WARNING:tensorflow:From /opt/conda/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py:422: The name tf.global_variables is deprecated. Please use tf.compat.v1.global_variables instead. WARNING:tensorflow:From /opt/conda/lib/python3.6/site-packages/bentoml/frameworks/keras.py:123: The name tf.keras.backend.get_session is deprecated. Please use tf.compat.v1.keras.backend.get_session instead. [2020-09-23 05:35:10,507] WARNING - pip package requirement tensorflow already exist [2020-09-23 05:35:10,508] WARNING - pip package requirement keras already exist WARNING:tensorflow:From /opt/conda/lib/python3.6/site-packages/bentoml/frameworks/keras.py:123: The name tf.keras.backend.get_session is deprecated. Please use tf.compat.v1.keras.backend.get_session instead. [2020-09-23 05:35:10,525] WARNING - pip package requirement tensorflow already exist [2020-09-23 05:35:10,525] WARNING - pip package requirement keras already exist WARNING:tensorflow:From /opt/conda/lib/python3.6/site-packages/bentoml/frameworks/keras.py:123: The name tf.keras.backend.get_session is deprecated. Please use tf.compat.v1.keras.backend.get_session instead. [2020-09-23 05:35:10,615] WARNING - pip package requirement tensorflow already exist [2020-09-23 05:35:10,616] WARNING - pip package requirement keras already exist [2020-09-23 05:35:55,479] ERROR - Exception on /predict [POST] Traceback (most recent call last): File "/opt/conda/lib/python3.6/site-packages/bentoml/server/api_server.py", line 288, in api_func response = api.handle_request(request) File "/opt/conda/lib/python3.6/site-packages/bentoml/service/inference_api.py", line 281, in handle_request results = self.infer((inf_task,)) File "/opt/conda/lib/python3.6/site-packages/bentoml/service/inference_api.py", line 222, in infer user_args = self.input_adapter.extract_user_func_args(inf_tasks) File "/opt/conda/lib/python3.6/site-packages/bentoml/adapters/image_input.py", line 134, in extract_user_func_args img_array = imageio.imread(task.data, pilmode=self.pilmode) File "/opt/conda/lib/python3.6/site-packages/imageio/core/functions.py", line 265, in imread reader = read(uri, format, "i", **kwargs) File "/opt/conda/lib/python3.6/site-packages/imageio/core/functions.py", line 172, in get_reader request = Request(uri, "r" + mode, **kwargs) File "/opt/conda/lib/python3.6/site-packages/imageio/core/request.py", line 124, in __init__ self._parse_uri(uri) File "/opt/conda/lib/python3.6/site-packages/imageio/core/request.py", line 222, in _parse_uri raise IOError("Cannot understand given URI: %s." % uri_r) OSError: Cannot understand given URI: None. [2020-09-23 05:36:09,200] INFO - {'service_name': 'KerasFashionMnistService', 'service_version': '20200923132350_974EEA', 'api': 'predict', 'task': {'data': {'name': 'sample_image.png'}, 'task_id': '895a16b1-c689-490f-b30e-c5601984093a', 'http_headers': (('Host', '127.0.0.1:5000'), ('User-Agent', 'curl/7.72.0'), ('Content-Length', '133877'), ('Accept', '*/*'), ('Content-Type', 'multipart/form-data; boundary=------------------------4f19d0a72380315e'))}, 'result': {'data': '"Ankle boot"', 'http_status': 200, 'http_headers': (('Content-Type', 'application/json'),)}, 'request_id': '895a16b1-c689-490f-b30e-c5601984093a'} ^C [2020-09-23 05:36:20 +0000] [1] [INFO] Handling signal: int [2020-09-23 05:36:20 +0000] [13] [INFO] Worker exiting (pid: 13) [2020-09-23 05:36:20 +0000] [14] [INFO] Worker exiting (pid: 14) [2020-09-23 05:36:20 +0000] [15] [INFO] Worker exiting (pid: 15)
If you are at a small team with limited engineering or DevOps resources, try out automated deployment with BentoML CLI, currently supporting AWS Lambda, AWS SageMaker, and Azure Functions:
If the cloud platform you are working with is not on the list above, try out these step-by-step guide on manually deploying BentoML packaged model to cloud platforms:
Lastly, if you have a DevOps or ML Engineering team who's operating a Kubernetes or OpenShift cluster, use the following guides as references for implementating your deployment strategy: