BentoML Demo: Iris Classifier with custom web UI

BentoML makes moving trained ML models to production easy:

  • Package models trained with any ML framework and reproduce them for model serving in production
  • Deploy anywhere for online API serving or offline batch serving
  • High-Performance API model server with adaptive micro-batching support
  • Central hub for managing models and deployment process via Web UI and APIs
  • Modular and flexible design making it adaptable to your infrastrcuture

BentoML is a framework for serving, managing, and deploying machine learning models. It is aiming to bridge the gap between Data Science and DevOps, and enable teams to deliver prediction services in a fast, repeatable, and scalable way.

Before reading this example project, be sure to check out the Getting started guide to learn about the basic concepts in BentoML.

This notebook demonstrates how to use BentoML to serve a Iris Classification model containing a REST API server with Custom Web UI.

Impression

In [1]:
%reload_ext autoreload
%autoreload 2
%matplotlib inline

import warnings
warnings.filterwarnings("ignore")
In [2]:
!pip install -q bentoml "scikit-learn>=0.23.2"

Create BentoService for model serving

In [1]:
%%writefile iris_classifier.py
from bentoml import env, artifacts, api, BentoService
from bentoml.adapters import DataframeInput
from bentoml.frameworks.sklearn import SklearnModelArtifact

@env(infer_pip_packages=True)
@artifacts([SklearnModelArtifact('model')])
class IrisClassifier(BentoService):

    @api(input=DataframeInput(), batch=True)
    def predict(self, df):
        # Optional pre-processing, post-processing code goes here
        return self.artifacts.model.predict(df)
Writing iris_classifier.py
In [2]:
%%writefile main.py
from sklearn import svm
from sklearn import datasets

from iris_classifier import IrisClassifier

if __name__ == "__main__":
    # Load training data
    iris = datasets.load_iris()
    X, y = iris.data, iris.target

    # Model Training
    clf = svm.SVC(gamma='scale')
    clf.fit(X, y)

    # Create a iris classifier service instance
    iris_classifier_service = IrisClassifier()

    # Pack the newly trained model artifact
    iris_classifier_service.pack('model', clf)

    # Save the prediction service to disk for model serving
    saved_path = iris_classifier_service.save()
Writing main.py
In [3]:
!python main.py
[2020-09-09 11:50:39,692] WARNING - Using BentoML installed in `editable` model, the local BentoML repository including all code changes will be packaged together with saved bundle created, under the './bundled_pip_dependencies' directory of the saved bundle.
[2020-09-09 11:50:41,005] INFO - Detected non-PyPI-released BentoML installed, copying local BentoML modulefiles to target saved bundle path..
warning: no previously-included files matching '*~' found anywhere in distribution
warning: no previously-included files matching '*.pyo' found anywhere in distribution
warning: no previously-included files matching '.git' found anywhere in distribution
warning: no previously-included files matching '.ipynb_checkpoints' found anywhere in distribution
warning: no previously-included files matching '__pycache__' found anywhere in distribution
no previously-included directories found matching 'e2e_tests'
no previously-included directories found matching 'tests'
no previously-included directories found matching 'benchmark'
UPDATING BentoML-0.8.6+34.g6123b8c6/bentoml/_version.py
set BentoML-0.8.6+34.g6123b8c6/bentoml/_version.py to '0.8.6+34.g6123b8c6'
[2020-09-09 11:50:45,682] INFO - BentoService bundle 'IrisClassifier:20200909115040_9FC7F4' saved to: /Users/bozhaoyu/bentoml/repository/IrisClassifier/20200909115040_9FC7F4

REST API Model Serving

To start a REST API model server with the BentoService saved above, use the bentoml serve command:

In [6]:
!bentoml serve IrisClassifier:latest
[2020-07-28 16:38:55,465] INFO - Getting latest version IrisClassifier:20200728163753_DE85E9
[2020-07-28 16:38:56,409] WARNING - Using BentoML installed in `editable` model, the local BentoML repository including all code changes will be packaged together with saved bundle created, under the './bundled_pip_dependencies' directory of the saved bundle.
[2020-07-28 16:38:56,422] WARNING - Saved BentoService bundle version mismatch: loading BentoService bundle create with BentoML version 0.8.1, but loading from BentoML version 0.8.1+27.g8e155f5.dirty
 * Serving Flask app "IrisClassifier" (lazy loading)
 * Environment: production
   WARNING: This is a development server. Do not use it in a production deployment.
   Use a production WSGI server instead.
 * Debug mode: off
 * Running on http://127.0.0.1:5000/ (Press CTRL+C to quit)
^C

If you are running this notebook from Google Colab, you can start the dev server with --run-with-ngrok option, to gain acccess to the API endpoint via a public endpoint managed by ngrok:

In [ ]:
!bentoml serveIrisiClassifier:latest --run-with-ngrok

At this point you can test out the Rest API server either by opening http://localhost:5000 in a new tab which will serve the swagger docs:

alt text

or by making a curl request in a another terminal window:

curl -i \
--header "Content-Type: application/json" \
--request POST \
--data '[[1,2,1,2]]' \
localhost:5000/predict

Adding Custom Web Static Content

In [7]:
!curl https://raw.githubusercontent.com/bentoml/gallery/master/scikit-learn/iris-classifier/static.tar.xz -o static.tar.xz
!tar --xz -xf static.tar.xz
!rm static.tar.xz
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  197k  100  197k    0     0   200k      0 --:--:-- --:--:-- --:--:--  200k

Here we have a very simple web ui as our static content that bento will serve. Now we will edit our bento service to point to this static directory.

Add @web_static_content('./static') to iris_classifier.py

Note: The path can be both relative or absolute.

In [9]:
%%writefile iris_classifier.py
from bentoml import env, artifacts, api, BentoService, web_static_content
from bentoml.adapters import DataframeInput
from bentoml.artifact import SklearnModelArtifact

@env(auto_pip_dependencies=True)
@artifacts([SklearnModelArtifact('model')])
@web_static_content('./static')
class IrisClassifier(BentoService):

    @api(input=DataframeInput(), batch=True)
    def test(self, df):
        # Optional pre-processing, post-processing code goes here
        return self.artifacts.model.predict(df)
Overwriting iris_classifier.py
In [10]:
!python main.py
[2020-07-30 09:09:28,868] WARNING - Using BentoML installed in `editable` model, the local BentoML repository including all code changes will be packaged together with saved bundle created, under the './bundled_pip_dependencies' directory of the saved bundle.
[2020-07-30 09:09:39,344] INFO - Detect BentoML installed in development model, copying local BentoML module file to target saved bundle path
running sdist
running egg_info
writing BentoML.egg-info/PKG-INFO
writing dependency_links to BentoML.egg-info/dependency_links.txt
writing entry points to BentoML.egg-info/entry_points.txt
writing requirements to BentoML.egg-info/requires.txt
writing top-level names to BentoML.egg-info/top_level.txt
reading manifest file 'BentoML.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
warning: no previously-included files matching '*~' found anywhere in distribution
warning: no previously-included files matching '*.pyo' found anywhere in distribution
warning: no previously-included files matching '.git' found anywhere in distribution
warning: no previously-included files matching '.ipynb_checkpoints' found anywhere in distribution
warning: no previously-included files matching '__pycache__' found anywhere in distribution
warning: no directories found matching 'bentoml/server/static'
warning: no directories found matching 'bentoml/yatai/web/dist'
no previously-included directories found matching 'e2e_tests'
no previously-included directories found matching 'tests'
no previously-included directories found matching 'benchmark'
writing manifest file 'BentoML.egg-info/SOURCES.txt'
running check
creating BentoML-0.8.3+42.gb8d36b6
creating BentoML-0.8.3+42.gb8d36b6/BentoML.egg-info
creating BentoML-0.8.3+42.gb8d36b6/bentoml
creating BentoML-0.8.3+42.gb8d36b6/bentoml/adapters
creating BentoML-0.8.3+42.gb8d36b6/bentoml/artifact
creating BentoML-0.8.3+42.gb8d36b6/bentoml/cli
creating BentoML-0.8.3+42.gb8d36b6/bentoml/clipper
creating BentoML-0.8.3+42.gb8d36b6/bentoml/configuration
creating BentoML-0.8.3+42.gb8d36b6/bentoml/configuration/__pycache__
creating BentoML-0.8.3+42.gb8d36b6/bentoml/handlers
creating BentoML-0.8.3+42.gb8d36b6/bentoml/marshal
creating BentoML-0.8.3+42.gb8d36b6/bentoml/saved_bundle
creating BentoML-0.8.3+42.gb8d36b6/bentoml/server
creating BentoML-0.8.3+42.gb8d36b6/bentoml/utils
creating BentoML-0.8.3+42.gb8d36b6/bentoml/yatai
creating BentoML-0.8.3+42.gb8d36b6/bentoml/yatai/client
creating BentoML-0.8.3+42.gb8d36b6/bentoml/yatai/deployment
creating BentoML-0.8.3+42.gb8d36b6/bentoml/yatai/deployment/aws_lambda
creating BentoML-0.8.3+42.gb8d36b6/bentoml/yatai/deployment/azure_functions
creating BentoML-0.8.3+42.gb8d36b6/bentoml/yatai/deployment/sagemaker
creating BentoML-0.8.3+42.gb8d36b6/bentoml/yatai/migrations
creating BentoML-0.8.3+42.gb8d36b6/bentoml/yatai/migrations/__pycache__
creating BentoML-0.8.3+42.gb8d36b6/bentoml/yatai/migrations/versions
creating BentoML-0.8.3+42.gb8d36b6/bentoml/yatai/migrations/versions/__pycache__
creating BentoML-0.8.3+42.gb8d36b6/bentoml/yatai/proto
creating BentoML-0.8.3+42.gb8d36b6/bentoml/yatai/repository
creating BentoML-0.8.3+42.gb8d36b6/bentoml/yatai/validator
copying files to BentoML-0.8.3+42.gb8d36b6...
copying LICENSE -> BentoML-0.8.3+42.gb8d36b6
copying MANIFEST.in -> BentoML-0.8.3+42.gb8d36b6
copying README.md -> BentoML-0.8.3+42.gb8d36b6
copying pyproject.toml -> BentoML-0.8.3+42.gb8d36b6
copying setup.cfg -> BentoML-0.8.3+42.gb8d36b6
copying setup.py -> BentoML-0.8.3+42.gb8d36b6
copying versioneer.py -> BentoML-0.8.3+42.gb8d36b6
copying BentoML.egg-info/PKG-INFO -> BentoML-0.8.3+42.gb8d36b6/BentoML.egg-info
copying BentoML.egg-info/SOURCES.txt -> BentoML-0.8.3+42.gb8d36b6/BentoML.egg-info
copying BentoML.egg-info/dependency_links.txt -> BentoML-0.8.3+42.gb8d36b6/BentoML.egg-info
copying BentoML.egg-info/entry_points.txt -> BentoML-0.8.3+42.gb8d36b6/BentoML.egg-info
copying BentoML.egg-info/requires.txt -> BentoML-0.8.3+42.gb8d36b6/BentoML.egg-info
copying BentoML.egg-info/top_level.txt -> BentoML-0.8.3+42.gb8d36b6/BentoML.egg-info
copying bentoml/__init__.py -> BentoML-0.8.3+42.gb8d36b6/bentoml
copying bentoml/_version.py -> BentoML-0.8.3+42.gb8d36b6/bentoml
copying bentoml/exceptions.py -> BentoML-0.8.3+42.gb8d36b6/bentoml
copying bentoml/service.py -> BentoML-0.8.3+42.gb8d36b6/bentoml
copying bentoml/service_env.py -> BentoML-0.8.3+42.gb8d36b6/bentoml
copying bentoml/adapters/__init__.py -> BentoML-0.8.3+42.gb8d36b6/bentoml/adapters
copying bentoml/adapters/base_input.py -> BentoML-0.8.3+42.gb8d36b6/bentoml/adapters
copying bentoml/adapters/base_output.py -> BentoML-0.8.3+42.gb8d36b6/bentoml/adapters
copying bentoml/adapters/clipper_input.py -> BentoML-0.8.3+42.gb8d36b6/bentoml/adapters
copying bentoml/adapters/dataframe_input.py -> BentoML-0.8.3+42.gb8d36b6/bentoml/adapters
copying bentoml/adapters/dataframe_output.py -> BentoML-0.8.3+42.gb8d36b6/bentoml/adapters
copying bentoml/adapters/default_output.py -> BentoML-0.8.3+42.gb8d36b6/bentoml/adapters
copying bentoml/adapters/fastai_image_input.py -> BentoML-0.8.3+42.gb8d36b6/bentoml/adapters
copying bentoml/adapters/file_input.py -> BentoML-0.8.3+42.gb8d36b6/bentoml/adapters
copying bentoml/adapters/image_input.py -> BentoML-0.8.3+42.gb8d36b6/bentoml/adapters
copying bentoml/adapters/json_input.py -> BentoML-0.8.3+42.gb8d36b6/bentoml/adapters
copying bentoml/adapters/json_output.py -> BentoML-0.8.3+42.gb8d36b6/bentoml/adapters
copying bentoml/adapters/legacy_image_input.py -> BentoML-0.8.3+42.gb8d36b6/bentoml/adapters
copying bentoml/adapters/legacy_json_input.py -> BentoML-0.8.3+42.gb8d36b6/bentoml/adapters
copying bentoml/adapters/multi_image_input.py -> BentoML-0.8.3+42.gb8d36b6/bentoml/adapters
copying bentoml/adapters/pytorch_tensor_input.py -> BentoML-0.8.3+42.gb8d36b6/bentoml/adapters
copying bentoml/adapters/tensorflow_tensor_input.py -> BentoML-0.8.3+42.gb8d36b6/bentoml/adapters
copying bentoml/adapters/tensorflow_tensor_output.py -> BentoML-0.8.3+42.gb8d36b6/bentoml/adapters
copying bentoml/adapters/utils.py -> BentoML-0.8.3+42.gb8d36b6/bentoml/adapters
copying bentoml/artifact/__init__.py -> BentoML-0.8.3+42.gb8d36b6/bentoml/artifact
copying bentoml/artifact/artifact.py -> BentoML-0.8.3+42.gb8d36b6/bentoml/artifact
copying bentoml/artifact/fastai2_model_artifact.py -> BentoML-0.8.3+42.gb8d36b6/bentoml/artifact
copying bentoml/artifact/fastai_model_artifact.py -> BentoML-0.8.3+42.gb8d36b6/bentoml/artifact
copying bentoml/artifact/fasttext_model_artifact.py -> BentoML-0.8.3+42.gb8d36b6/bentoml/artifact
copying bentoml/artifact/h2o_model_artifact.py -> BentoML-0.8.3+42.gb8d36b6/bentoml/artifact
copying bentoml/artifact/json_artifact.py -> BentoML-0.8.3+42.gb8d36b6/bentoml/artifact
copying bentoml/artifact/keras_model_artifact.py -> BentoML-0.8.3+42.gb8d36b6/bentoml/artifact
copying bentoml/artifact/lightgbm_model_artifact.py -> BentoML-0.8.3+42.gb8d36b6/bentoml/artifact
copying bentoml/artifact/onnx_model_artifact.py -> BentoML-0.8.3+42.gb8d36b6/bentoml/artifact
copying bentoml/artifact/pickle_artifact.py -> BentoML-0.8.3+42.gb8d36b6/bentoml/artifact
copying bentoml/artifact/pytorch_model_artifact.py -> BentoML-0.8.3+42.gb8d36b6/bentoml/artifact
copying bentoml/artifact/sklearn_model_artifact.py -> BentoML-0.8.3+42.gb8d36b6/bentoml/artifact
copying bentoml/artifact/spacy_model_artifact.py -> BentoML-0.8.3+42.gb8d36b6/bentoml/artifact
copying bentoml/artifact/text_file_artifact.py -> BentoML-0.8.3+42.gb8d36b6/bentoml/artifact
copying bentoml/artifact/tf_savedmodel_artifact.py -> BentoML-0.8.3+42.gb8d36b6/bentoml/artifact
copying bentoml/artifact/xgboost_model_artifact.py -> BentoML-0.8.3+42.gb8d36b6/bentoml/artifact
copying bentoml/cli/__init__.py -> BentoML-0.8.3+42.gb8d36b6/bentoml/cli
copying bentoml/cli/aws_lambda.py -> BentoML-0.8.3+42.gb8d36b6/bentoml/cli
copying bentoml/cli/aws_sagemaker.py -> BentoML-0.8.3+42.gb8d36b6/bentoml/cli
copying bentoml/cli/azure_functions.py -> BentoML-0.8.3+42.gb8d36b6/bentoml/cli
copying bentoml/cli/bento_management.py -> BentoML-0.8.3+42.gb8d36b6/bentoml/cli
copying bentoml/cli/bento_service.py -> BentoML-0.8.3+42.gb8d36b6/bentoml/cli
copying bentoml/cli/click_utils.py -> BentoML-0.8.3+42.gb8d36b6/bentoml/cli
copying bentoml/cli/config.py -> BentoML-0.8.3+42.gb8d36b6/bentoml/cli
copying bentoml/cli/deployment.py -> BentoML-0.8.3+42.gb8d36b6/bentoml/cli
copying bentoml/cli/utils.py -> BentoML-0.8.3+42.gb8d36b6/bentoml/cli
copying bentoml/cli/yatai_service.py -> BentoML-0.8.3+42.gb8d36b6/bentoml/cli
copying bentoml/clipper/__init__.py -> BentoML-0.8.3+42.gb8d36b6/bentoml/clipper
copying bentoml/configuration/__init__.py -> BentoML-0.8.3+42.gb8d36b6/bentoml/configuration
copying bentoml/configuration/configparser.py -> BentoML-0.8.3+42.gb8d36b6/bentoml/configuration
copying bentoml/configuration/default_bentoml.cfg -> BentoML-0.8.3+42.gb8d36b6/bentoml/configuration
copying bentoml/configuration/__pycache__/__init__.cpython-36.pyc -> BentoML-0.8.3+42.gb8d36b6/bentoml/configuration/__pycache__
copying bentoml/configuration/__pycache__/__init__.cpython-37.pyc -> BentoML-0.8.3+42.gb8d36b6/bentoml/configuration/__pycache__
copying bentoml/configuration/__pycache__/__init__.cpython-38.pyc -> BentoML-0.8.3+42.gb8d36b6/bentoml/configuration/__pycache__
copying bentoml/configuration/__pycache__/configparser.cpython-36.pyc -> BentoML-0.8.3+42.gb8d36b6/bentoml/configuration/__pycache__
copying bentoml/configuration/__pycache__/configparser.cpython-37.pyc -> BentoML-0.8.3+42.gb8d36b6/bentoml/configuration/__pycache__
copying bentoml/configuration/__pycache__/configparser.cpython-38.pyc -> BentoML-0.8.3+42.gb8d36b6/bentoml/configuration/__pycache__
copying bentoml/handlers/__init__.py -> BentoML-0.8.3+42.gb8d36b6/bentoml/handlers
copying bentoml/marshal/__init__.py -> BentoML-0.8.3+42.gb8d36b6/bentoml/marshal
copying bentoml/marshal/dispatcher.py -> BentoML-0.8.3+42.gb8d36b6/bentoml/marshal
copying bentoml/marshal/marshal.py -> BentoML-0.8.3+42.gb8d36b6/bentoml/marshal
copying bentoml/marshal/utils.py -> BentoML-0.8.3+42.gb8d36b6/bentoml/marshal
copying bentoml/saved_bundle/__init__.py -> BentoML-0.8.3+42.gb8d36b6/bentoml/saved_bundle
copying bentoml/saved_bundle/bentoml-init.sh -> BentoML-0.8.3+42.gb8d36b6/bentoml/saved_bundle
copying bentoml/saved_bundle/bundler.py -> BentoML-0.8.3+42.gb8d36b6/bentoml/saved_bundle
copying bentoml/saved_bundle/config.py -> BentoML-0.8.3+42.gb8d36b6/bentoml/saved_bundle
copying bentoml/saved_bundle/docker-entrypoint.sh -> BentoML-0.8.3+42.gb8d36b6/bentoml/saved_bundle
copying bentoml/saved_bundle/loader.py -> BentoML-0.8.3+42.gb8d36b6/bentoml/saved_bundle
copying bentoml/saved_bundle/pip_pkg.py -> BentoML-0.8.3+42.gb8d36b6/bentoml/saved_bundle
copying bentoml/saved_bundle/py_module_utils.py -> BentoML-0.8.3+42.gb8d36b6/bentoml/saved_bundle
copying bentoml/saved_bundle/templates.py -> BentoML-0.8.3+42.gb8d36b6/bentoml/saved_bundle
copying bentoml/server/__init__.py -> BentoML-0.8.3+42.gb8d36b6/bentoml/server
copying bentoml/server/api_server.py -> BentoML-0.8.3+42.gb8d36b6/bentoml/server
copying bentoml/server/gunicorn_config.py -> BentoML-0.8.3+42.gb8d36b6/bentoml/server
copying bentoml/server/gunicorn_server.py -> BentoML-0.8.3+42.gb8d36b6/bentoml/server
copying bentoml/server/instruments.py -> BentoML-0.8.3+42.gb8d36b6/bentoml/server
copying bentoml/server/marshal_server.py -> BentoML-0.8.3+42.gb8d36b6/bentoml/server
copying bentoml/server/open_api.py -> BentoML-0.8.3+42.gb8d36b6/bentoml/server
copying bentoml/server/trace.py -> BentoML-0.8.3+42.gb8d36b6/bentoml/server
copying bentoml/server/utils.py -> BentoML-0.8.3+42.gb8d36b6/bentoml/server
copying bentoml/utils/__init__.py -> BentoML-0.8.3+42.gb8d36b6/bentoml/utils
copying bentoml/utils/alg.py -> BentoML-0.8.3+42.gb8d36b6/bentoml/utils
copying bentoml/utils/benchmark.py -> BentoML-0.8.3+42.gb8d36b6/bentoml/utils
copying bentoml/utils/cloudpickle.py -> BentoML-0.8.3+42.gb8d36b6/bentoml/utils
copying bentoml/utils/dataframe_util.py -> BentoML-0.8.3+42.gb8d36b6/bentoml/utils
copying bentoml/utils/flask_ngrok.py -> BentoML-0.8.3+42.gb8d36b6/bentoml/utils
copying bentoml/utils/hybridmethod.py -> BentoML-0.8.3+42.gb8d36b6/bentoml/utils
copying bentoml/utils/lazy_loader.py -> BentoML-0.8.3+42.gb8d36b6/bentoml/utils
copying bentoml/utils/log.py -> BentoML-0.8.3+42.gb8d36b6/bentoml/utils
copying bentoml/utils/s3.py -> BentoML-0.8.3+42.gb8d36b6/bentoml/utils
copying bentoml/utils/tempdir.py -> BentoML-0.8.3+42.gb8d36b6/bentoml/utils
copying bentoml/utils/usage_stats.py -> BentoML-0.8.3+42.gb8d36b6/bentoml/utils
copying bentoml/yatai/__init__.py -> BentoML-0.8.3+42.gb8d36b6/bentoml/yatai
copying bentoml/yatai/alembic.ini -> BentoML-0.8.3+42.gb8d36b6/bentoml/yatai
copying bentoml/yatai/db.py -> BentoML-0.8.3+42.gb8d36b6/bentoml/yatai
copying bentoml/yatai/deployment_utils.py -> BentoML-0.8.3+42.gb8d36b6/bentoml/yatai
copying bentoml/yatai/status.py -> BentoML-0.8.3+42.gb8d36b6/bentoml/yatai
copying bentoml/yatai/utils.py -> BentoML-0.8.3+42.gb8d36b6/bentoml/yatai
copying bentoml/yatai/yatai_service.py -> BentoML-0.8.3+42.gb8d36b6/bentoml/yatai
copying bentoml/yatai/yatai_service_impl.py -> BentoML-0.8.3+42.gb8d36b6/bentoml/yatai
copying bentoml/yatai/client/__init__.py -> BentoML-0.8.3+42.gb8d36b6/bentoml/yatai/client
copying bentoml/yatai/client/bento_repository_api.py -> BentoML-0.8.3+42.gb8d36b6/bentoml/yatai/client
copying bentoml/yatai/client/deployment_api.py -> BentoML-0.8.3+42.gb8d36b6/bentoml/yatai/client
copying bentoml/yatai/deployment/__init__.py -> BentoML-0.8.3+42.gb8d36b6/bentoml/yatai/deployment
copying bentoml/yatai/deployment/operator.py -> BentoML-0.8.3+42.gb8d36b6/bentoml/yatai/deployment
copying bentoml/yatai/deployment/store.py -> BentoML-0.8.3+42.gb8d36b6/bentoml/yatai/deployment
copying bentoml/yatai/deployment/utils.py -> BentoML-0.8.3+42.gb8d36b6/bentoml/yatai/deployment
copying bentoml/yatai/deployment/aws_lambda/__init__.py -> BentoML-0.8.3+42.gb8d36b6/bentoml/yatai/deployment/aws_lambda
copying bentoml/yatai/deployment/aws_lambda/download_extra_resources.py -> BentoML-0.8.3+42.gb8d36b6/bentoml/yatai/deployment/aws_lambda
copying bentoml/yatai/deployment/aws_lambda/lambda_app.py -> BentoML-0.8.3+42.gb8d36b6/bentoml/yatai/deployment/aws_lambda
copying bentoml/yatai/deployment/aws_lambda/operator.py -> BentoML-0.8.3+42.gb8d36b6/bentoml/yatai/deployment/aws_lambda
copying bentoml/yatai/deployment/aws_lambda/utils.py -> BentoML-0.8.3+42.gb8d36b6/bentoml/yatai/deployment/aws_lambda
copying bentoml/yatai/deployment/azure_functions/Dockerfile -> BentoML-0.8.3+42.gb8d36b6/bentoml/yatai/deployment/azure_functions
copying bentoml/yatai/deployment/azure_functions/__init__.py -> BentoML-0.8.3+42.gb8d36b6/bentoml/yatai/deployment/azure_functions
copying bentoml/yatai/deployment/azure_functions/app_init.py -> BentoML-0.8.3+42.gb8d36b6/bentoml/yatai/deployment/azure_functions
copying bentoml/yatai/deployment/azure_functions/constants.py -> BentoML-0.8.3+42.gb8d36b6/bentoml/yatai/deployment/azure_functions
copying bentoml/yatai/deployment/azure_functions/host.json -> BentoML-0.8.3+42.gb8d36b6/bentoml/yatai/deployment/azure_functions
copying bentoml/yatai/deployment/azure_functions/local.settings.json -> BentoML-0.8.3+42.gb8d36b6/bentoml/yatai/deployment/azure_functions
copying bentoml/yatai/deployment/azure_functions/operator.py -> BentoML-0.8.3+42.gb8d36b6/bentoml/yatai/deployment/azure_functions
copying bentoml/yatai/deployment/azure_functions/templates.py -> BentoML-0.8.3+42.gb8d36b6/bentoml/yatai/deployment/azure_functions
copying bentoml/yatai/deployment/sagemaker/__init__.py -> BentoML-0.8.3+42.gb8d36b6/bentoml/yatai/deployment/sagemaker
copying bentoml/yatai/deployment/sagemaker/model_server.py -> BentoML-0.8.3+42.gb8d36b6/bentoml/yatai/deployment/sagemaker
copying bentoml/yatai/deployment/sagemaker/nginx.conf -> BentoML-0.8.3+42.gb8d36b6/bentoml/yatai/deployment/sagemaker
copying bentoml/yatai/deployment/sagemaker/operator.py -> BentoML-0.8.3+42.gb8d36b6/bentoml/yatai/deployment/sagemaker
copying bentoml/yatai/deployment/sagemaker/serve -> BentoML-0.8.3+42.gb8d36b6/bentoml/yatai/deployment/sagemaker
copying bentoml/yatai/deployment/sagemaker/wsgi.py -> BentoML-0.8.3+42.gb8d36b6/bentoml/yatai/deployment/sagemaker
copying bentoml/yatai/migrations/README -> BentoML-0.8.3+42.gb8d36b6/bentoml/yatai/migrations
copying bentoml/yatai/migrations/__init__.py -> BentoML-0.8.3+42.gb8d36b6/bentoml/yatai/migrations
copying bentoml/yatai/migrations/env.py -> BentoML-0.8.3+42.gb8d36b6/bentoml/yatai/migrations
copying bentoml/yatai/migrations/script.py.mako -> BentoML-0.8.3+42.gb8d36b6/bentoml/yatai/migrations
copying bentoml/yatai/migrations/__pycache__/env.cpython-36.pyc -> BentoML-0.8.3+42.gb8d36b6/bentoml/yatai/migrations/__pycache__
copying bentoml/yatai/migrations/versions/__init__.py -> BentoML-0.8.3+42.gb8d36b6/bentoml/yatai/migrations/versions
copying bentoml/yatai/migrations/versions/a6b00ae45279_add_last_updated_at_for_deployments.py -> BentoML-0.8.3+42.gb8d36b6/bentoml/yatai/migrations/versions
copying bentoml/yatai/migrations/versions/__pycache__/a6b00ae45279_add_last_updated_at_for_deployments.cpython-36.pyc -> BentoML-0.8.3+42.gb8d36b6/bentoml/yatai/migrations/versions/__pycache__
copying bentoml/yatai/proto/__init__.py -> BentoML-0.8.3+42.gb8d36b6/bentoml/yatai/proto
copying bentoml/yatai/proto/deployment_pb2.py -> BentoML-0.8.3+42.gb8d36b6/bentoml/yatai/proto
copying bentoml/yatai/proto/repository_pb2.py -> BentoML-0.8.3+42.gb8d36b6/bentoml/yatai/proto
copying bentoml/yatai/proto/status_pb2.py -> BentoML-0.8.3+42.gb8d36b6/bentoml/yatai/proto
copying bentoml/yatai/proto/yatai_service_pb2.py -> BentoML-0.8.3+42.gb8d36b6/bentoml/yatai/proto
copying bentoml/yatai/proto/yatai_service_pb2_grpc.py -> BentoML-0.8.3+42.gb8d36b6/bentoml/yatai/proto
copying bentoml/yatai/repository/__init__.py -> BentoML-0.8.3+42.gb8d36b6/bentoml/yatai/repository
copying bentoml/yatai/repository/base_repository.py -> BentoML-0.8.3+42.gb8d36b6/bentoml/yatai/repository
copying bentoml/yatai/repository/local_repository.py -> BentoML-0.8.3+42.gb8d36b6/bentoml/yatai/repository
copying bentoml/yatai/repository/metadata_store.py -> BentoML-0.8.3+42.gb8d36b6/bentoml/yatai/repository
copying bentoml/yatai/repository/repository.py -> BentoML-0.8.3+42.gb8d36b6/bentoml/yatai/repository
copying bentoml/yatai/repository/s3_repository.py -> BentoML-0.8.3+42.gb8d36b6/bentoml/yatai/repository
copying bentoml/yatai/validator/__init__.py -> BentoML-0.8.3+42.gb8d36b6/bentoml/yatai/validator
copying bentoml/yatai/validator/deployment_pb_validator.py -> BentoML-0.8.3+42.gb8d36b6/bentoml/yatai/validator
Writing BentoML-0.8.3+42.gb8d36b6/setup.cfg
UPDATING BentoML-0.8.3+42.gb8d36b6/bentoml/_version.py
set BentoML-0.8.3+42.gb8d36b6/bentoml/_version.py to '0.8.3+42.gb8d36b6'
Creating tar archive
removing 'BentoML-0.8.3+42.gb8d36b6' (and everything under it)
[2020-07-30 09:09:40,124] INFO - BentoService bundle 'IrisClassifier:20200730090929_C02FB7' saved to: /home/bentoml/bentoml/repository/IrisClassifier/20200730090929_C02FB7
In [ ]:
!bentoml serve IrisClassifier:latest
[2020-07-30 09:09:43,808] INFO - Getting latest version IrisClassifier:20200730090929_C02FB7
[2020-07-30 09:09:43,808] INFO - Starting BentoML API server in development mode..
[2020-07-30 09:09:44,916] WARNING - Using BentoML installed in `editable` model, the local BentoML repository including all code changes will be packaged together with saved bundle created, under the './bundled_pip_dependencies' directory of the saved bundle.
[2020-07-30 09:09:44,949] WARNING - Saved BentoService bundle version mismatch: loading BentoService bundle create with BentoML version 0.8.3, but loading from BentoML version 0.8.3+42.gb8d36b6
 * Serving Flask app "IrisClassifier" (lazy loading)
 * Environment: production
   WARNING: This is a development server. Do not use it in a production deployment.
   Use a production WSGI server instead.
 * Debug mode: off
 * Running on http://127.0.0.1:5000/ (Press CTRL+C to quit)
127.0.0.1 - - [30/Jul/2020 09:09:52] "POST /test HTTP/1.1" 200 -
127.0.0.1 - - [30/Jul/2020 09:09:55] "POST /test HTTP/1.1" 200 -
127.0.0.1 - - [30/Jul/2020 09:09:57] "POST /test HTTP/1.1" 200 -
127.0.0.1 - - [30/Jul/2020 09:10:00] "POST /test HTTP/1.1" 200 -

Now if you visit http://localhost:5000/, you should be served with a beautiful UI:

Custom UI

It's still possible to access the swagger docs at /docs

Containerize model server with Docker

One common way of distributing this model API server for production deployment, is via Docker containers. And BentoML provides a convenient way to do that.

Note that docker is not available in Google Colab. You will need to download and run this notebook locally to try out this containerization with docker feature.

If you already have docker configured, simply run the follow command to product a docker container serving the IrisClassifier prediction service created above:

In [4]:
!bentoml containerize IrisClassifier:latest
[2020-09-09 11:52:05,072] INFO - Getting latest version IrisClassifier:20200909115040_9FC7F4
Found Bento: /Users/bozhaoyu/bentoml/repository/IrisClassifier/20200909115040_9FC7F4
[2020-09-09 11:52:05,109] WARNING - Using BentoML installed in `editable` model, the local BentoML repository including all code changes will be packaged together with saved bundle created, under the './bundled_pip_dependencies' directory of the saved bundle.
[2020-09-09 11:52:05,126] WARNING - Saved BentoService bundle version mismatch: loading BentoService bundle create with BentoML version 0.8.6, but loading from BentoML version 0.8.6+34.g6123b8c6
Tag not specified, using tag parsed from BentoService: 'irisclassifier:20200909115040_9FC7F4'
Building Docker image irisclassifier:20200909115040_9FC7F4 from IrisClassifier:latest 
-we in here
processed docker file
(None, None)
root in create archive /Users/bozhaoyu/bentoml/repository/IrisClassifier/20200909115040_9FC7F4 ['Dockerfile', 'IrisClassifier', 'IrisClassifier/__init__.py', 'IrisClassifier/artifacts', 'IrisClassifier/artifacts/__init__.py', 'IrisClassifier/artifacts/model.pkl', 'IrisClassifier/bentoml.yml', 'IrisClassifier/iris_classifier.py', 'MANIFEST.in', 'README.md', 'bentoml-init.sh', 'bentoml.yml', 'bundled_pip_dependencies', 'bundled_pip_dependencies/BentoML-0.8.6+34.g6123b8c6.tar.gz', 'docker-entrypoint.sh', 'environment.yml', 'requirements.txt', 'setup.py']
about to build
about to upgrade params
check each param and update
if use config proxy
if buildargs
if shmsize
if labels
if cache from
if target
if network_mode
if squash
if extra hosts is not None
if platform is not None
if isolcation is not None
if context is not None
setting auth {'Content-Type': 'application/tar'}
|docker build <tempfile._TemporaryFileWrapper object at 0x7fba46590be0> {'t': 'irisclassifier:20200909115040_9FC7F4', 'remote': None, 'q': False, 'nocache': False, 'rm': False, 'forcerm': False, 'pull': False, 'dockerfile': (None, None)}
-docker response <Response [200]>
context closes
print responses
Step 1/15 : FROM bentoml/model-server:0.8.6
 ---> 71644b758bed
Step 2/15 : COPY . /bento
\ ---> ac6b4936bb75
Step 3/15 : WORKDIR /bento
 ---> Running in ab9341b14333
- ---> dc2c710d4a0f
Step 4/15 : ARG PIP_INDEX_URL=https://pypi.python.org/simple/
/ ---> Running in f21c5c697580
| ---> 6e05f460fbc5
Step 5/15 : ARG PIP_TRUSTED_HOST=pypi.python.org
 ---> Running in 900a926e8c53
\ ---> 3cc575c53975
Step 6/15 : ENV PIP_INDEX_URL $PIP_INDEX_URL
- ---> Running in 3367fdca94cc
/ ---> e20c93c7f376
Step 7/15 : ENV PIP_TRUSTED_HOST $PIP_TRUSTED_HOST
 ---> Running in d14c545e8549
| ---> b5fcb21c7843
Step 8/15 : RUN chmod +x /bento/bentoml-init.sh
\ ---> Running in 107df63f664e
- ---> 9073da44fe2d
Step 9/15 : RUN if [ -f /bento/bentoml-init.sh ]; then bash -c /bento/bentoml-init.sh; fi
 ---> Running in e9185848bc40
\+++ dirname /bento/bentoml-init.sh

++ cd /bento
++ pwd -P

+ SAVED_BUNDLE_PATH=/bento
+ cd /bento
+ '[' -f ./setup.sh ']'
+ command -v conda
+ echo 'Updating conda base environment with environment.yml'
+ conda env update -n base -f ./environment.yml

Updating conda base environment with environment.yml
/Collecting package metadata (repodata.json): ...working... 
\done
Solving environment: ...working... 
|done
\
Downloading and Extracting Packages
openssl-1.1.1g       | 2.1 MB    |            |   0% 
openssl-1.1.1g       | 2.1 MB    |            |   1% 
openssl-1.1.1g       | 2.1 MB    | #9         |  19% 
openssl-1.1.1g       | 2.1 MB    | ####7      |  47% 
openssl-1.1.1g       | 2.1 MB    | ######8    |  68% 
openssl-1.1.1g       | 2.1 MB    | ########## | 100% 

python_abi-3.7       | 4 KB      |            |   0% 
python_abi-3.7       | 4 KB      | ########## | 100% 

certifi-2020.6.20    | 151 KB    |            |   0% 
certifi-2020.6.20    | 151 KB    | #          |  11% 
certifi-2020.6.20    | 151 KB    | ########## | 100% 

python-3.7.3         | 32.1 MB   |            |   0% 
python-3.7.3         | 32.1 MB   |            |   0% 
python-3.7.3         | 32.1 MB   | 2          |   3% 
python-3.7.3         | 32.1 MB   | 5          |   6% 
python-3.7.3         | 32.1 MB   | 9          |   9% 
python-3.7.3         | 32.1 MB   | #1         |  12% 
python-3.7.3         | 32.1 MB   | #5         |  15% 
python-3.7.3         | 32.1 MB   | ##         |  21% 
python-3.7.3         | 32.1 MB   | ##5        |  25% 
python-3.7.3         | 32.1 MB   | ##9        |  30% 
python-3.7.3         | 32.1 MB   | ###5       |  35% 
python-3.7.3         | 32.1 MB   | ###9       |  40% 
python-3.7.3         | 32.1 MB   | ####4      |  45% 
python-3.7.3         | 32.1 MB   | ####9      |  50% 
python-3.7.3         | 32.1 MB   | #####4     |  55% 
python-3.7.3         | 32.1 MB   | ######     |  60% 
python-3.7.3         | 32.1 MB   | ######4    |  65% 
python-3.7.3         | 32.1 MB   | ######9    |  70% 
python-3.7.3         | 32.1 MB   | #######5   |  75% 
python-3.7.3         | 32.1 MB   | ########   |  80% 
python-3.7.3         | 32.1 MB   | ########6  |  86% 
python-3.7.3         | 32.1 MB   | #########1 |  91% 
python-3.7.3         | 32.1 MB   | #########7 |  97% 
python-3.7.3         | 32.1 MB   | ########## | 100% 

ca-certificates-2020 | 145 KB    |            |   0% 
ca-certificates-2020 | 145 KB    | ########## | 100% 

pip-20.2.3           | 1.1 MB    |            |   0% 
pip-20.2.3           | 1.1 MB    | ###9       |  39% 
pip-20.2.3           | 1.1 MB    | ########## | 100% 

Preparing transaction: ...working... 
\done
Verifying transaction: ...working... 
|done
Executing transaction: ...working... 
\done
\

==> WARNING: A newer version of conda exists. <==
  current version: 4.8.2
  latest version: 4.8.4

Please update conda by running

    $ conda update -n base -c defaults conda



#
# To activate this environment, use
#
#     $ conda activate base
#
# To deactivate an active environment, use
#
#     $ conda deactivate
\+ pip install -r ./requirements.txt --no-cache-dir

\Looking in indexes: https://pypi.python.org/simple/
/Collecting scikit-learn
|  Downloading scikit_learn-0.23.2-cp37-cp37m-manylinux1_x86_64.whl (6.8 MB)
-Collecting pandas
/  Downloading pandas-1.1.2-cp37-cp37m-manylinux1_x86_64.whl (10.5 MB)
\Requirement already satisfied: bentoml==0.8.6 in /opt/conda/lib/python3.7/site-packages (from -r ./requirements.txt (line 3)) (0.8.6)
|Collecting threadpoolctl>=2.0.0
  Downloading threadpoolctl-2.1.0-py3-none-any.whl (12 kB)
-Collecting joblib>=0.11
  Downloading joblib-0.16.0-py3-none-any.whl (300 kB)
-Collecting scipy>=0.19.1
/  Downloading scipy-1.5.2-cp37-cp37m-manylinux1_x86_64.whl (25.9 MB)
-Requirement already satisfied: numpy>=1.13.3 in /opt/conda/lib/python3.7/site-packages (from scikit-learn->-r ./requirements.txt (line 1)) (1.19.1)
Requirement already satisfied: python-dateutil>=2.7.3 in /opt/conda/lib/python3.7/site-packages (from pandas->-r ./requirements.txt (line 2)) (2.8.0)
\Collecting pytz>=2017.2
-  Downloading pytz-2020.1-py2.py3-none-any.whl (510 kB)
Requirement already satisfied: python-json-logger in /opt/conda/lib/python3.7/site-packages (from bentoml==0.8.6->-r ./requirements.txt (line 3)) (0.1.11)
Requirement already satisfied: psutil in /opt/conda/lib/python3.7/site-packages (from bentoml==0.8.6->-r ./requirements.txt (line 3)) (5.7.2)
Requirement already satisfied: protobuf>=3.6.0 in /opt/conda/lib/python3.7/site-packages (from bentoml==0.8.6->-r ./requirements.txt (line 3)) (3.13.0)
Requirement already satisfied: certifi in /opt/conda/lib/python3.7/site-packages (from bentoml==0.8.6->-r ./requirements.txt (line 3)) (2020.6.20)
Requirement already satisfied: aiohttp in /opt/conda/lib/python3.7/site-packages (from bentoml==0.8.6->-r ./requirements.txt (line 3)) (3.6.2)
/Requirement already satisfied: prometheus-client in /opt/conda/lib/python3.7/site-packages (from bentoml==0.8.6->-r ./requirements.txt (line 3)) (0.8.0)
Requirement already satisfied: requests in /opt/conda/lib/python3.7/site-packages (from bentoml==0.8.6->-r ./requirements.txt (line 3)) (2.22.0)
Requirement already satisfied: configparser in /opt/conda/lib/python3.7/site-packages (from bentoml==0.8.6->-r ./requirements.txt (line 3)) (5.0.0)
Requirement already satisfied: sqlalchemy-utils in /opt/conda/lib/python3.7/site-packages (from bentoml==0.8.6->-r ./requirements.txt (line 3)) (0.36.8)
|Requirement already satisfied: alembic in /opt/conda/lib/python3.7/site-packages (from bentoml==0.8.6->-r ./requirements.txt (line 3)) (1.4.2)
Requirement already satisfied: flask in /opt/conda/lib/python3.7/site-packages (from bentoml==0.8.6->-r ./requirements.txt (line 3)) (1.1.2)
Requirement already satisfied: click>=7.0 in /opt/conda/lib/python3.7/site-packages (from bentoml==0.8.6->-r ./requirements.txt (line 3)) (7.1.2)
Requirement already satisfied: packaging in /opt/conda/lib/python3.7/site-packages (from bentoml==0.8.6->-r ./requirements.txt (line 3)) (20.4)
Requirement already satisfied: py-zipkin in /opt/conda/lib/python3.7/site-packages (from bentoml==0.8.6->-r ./requirements.txt (line 3)) (0.20.0)
Requirement already satisfied: ruamel.yaml>=0.15.0 in /opt/conda/lib/python3.7/site-packages (from bentoml==0.8.6->-r ./requirements.txt (line 3)) (0.15.87)
Requirement already satisfied: gunicorn in /opt/conda/lib/python3.7/site-packages (from bentoml==0.8.6->-r ./requirements.txt (line 3)) (20.0.4)
Requirement already satisfied: cerberus in /opt/conda/lib/python3.7/site-packages (from bentoml==0.8.6->-r ./requirements.txt (line 3)) (1.3.2)
Requirement already satisfied: tabulate in /opt/conda/lib/python3.7/site-packages (from bentoml==0.8.6->-r ./requirements.txt (line 3)) (0.8.7)
Requirement already satisfied: boto3 in /opt/conda/lib/python3.7/site-packages (from bentoml==0.8.6->-r ./requirements.txt (line 3)) (1.14.48)
Requirement already satisfied: grpcio<=1.27.2 in /opt/conda/lib/python3.7/site-packages (from bentoml==0.8.6->-r ./requirements.txt (line 3)) (1.27.2)
Requirement already satisfied: sqlalchemy>=1.3.0 in /opt/conda/lib/python3.7/site-packages (from bentoml==0.8.6->-r ./requirements.txt (line 3)) (1.3.19)
Requirement already satisfied: docker in /opt/conda/lib/python3.7/site-packages (from bentoml==0.8.6->-r ./requirements.txt (line 3)) (4.3.1)
\Requirement already satisfied: humanfriendly in /opt/conda/lib/python3.7/site-packages (from bentoml==0.8.6->-r ./requirements.txt (line 3)) (8.2)
Requirement already satisfied: multidict in /opt/conda/lib/python3.7/site-packages (from bentoml==0.8.6->-r ./requirements.txt (line 3)) (4.7.6)
Requirement already satisfied: six>=1.5 in /opt/conda/lib/python3.7/site-packages (from python-dateutil>=2.7.3->pandas->-r ./requirements.txt (line 2)) (1.14.0)
Requirement already satisfied: setuptools in /opt/conda/lib/python3.7/site-packages (from protobuf>=3.6.0->bentoml==0.8.6->-r ./requirements.txt (line 3)) (45.2.0.post20200210)
Requirement already satisfied: attrs>=17.3.0 in /opt/conda/lib/python3.7/site-packages (from aiohttp->bentoml==0.8.6->-r ./requirements.txt (line 3)) (20.1.0)
Requirement already satisfied: chardet<4.0,>=2.0 in /opt/conda/lib/python3.7/site-packages (from aiohttp->bentoml==0.8.6->-r ./requirements.txt (line 3)) (3.0.4)
Requirement already satisfied: async-timeout<4.0,>=3.0 in /opt/conda/lib/python3.7/site-packages (from aiohttp->bentoml==0.8.6->-r ./requirements.txt (line 3)) (3.0.1)
Requirement already satisfied: yarl<2.0,>=1.0 in /opt/conda/lib/python3.7/site-packages (from aiohttp->bentoml==0.8.6->-r ./requirements.txt (line 3)) (1.5.1)
Requirement already satisfied: idna<2.9,>=2.5 in /opt/conda/lib/python3.7/site-packages (from requests->bentoml==0.8.6->-r ./requirements.txt (line 3)) (2.8)
Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /opt/conda/lib/python3.7/site-packages (from requests->bentoml==0.8.6->-r ./requirements.txt (line 3)) (1.25.8)
Requirement already satisfied: Mako in /opt/conda/lib/python3.7/site-packages (from alembic->bentoml==0.8.6->-r ./requirements.txt (line 3)) (1.1.3)
Requirement already satisfied: python-editor>=0.3 in /opt/conda/lib/python3.7/site-packages (from alembic->bentoml==0.8.6->-r ./requirements.txt (line 3)) (1.0.4)
Requirement already satisfied: itsdangerous>=0.24 in /opt/conda/lib/python3.7/site-packages (from flask->bentoml==0.8.6->-r ./requirements.txt (line 3)) (1.1.0)
Requirement already satisfied: Jinja2>=2.10.1 in /opt/conda/lib/python3.7/site-packages (from flask->bentoml==0.8.6->-r ./requirements.txt (line 3)) (2.11.2)
Requirement already satisfied: Werkzeug>=0.15 in /opt/conda/lib/python3.7/site-packages (from flask->bentoml==0.8.6->-r ./requirements.txt (line 3)) (1.0.1)
Requirement already satisfied: pyparsing>=2.0.2 in /opt/conda/lib/python3.7/site-packages (from packaging->bentoml==0.8.6->-r ./requirements.txt (line 3)) (2.4.7)
-Requirement already satisfied: thriftpy2>=0.4.0 in /opt/conda/lib/python3.7/site-packages (from py-zipkin->bentoml==0.8.6->-r ./requirements.txt (line 3)) (0.4.11)
Requirement already satisfied: botocore<1.18.0,>=1.17.48 in /opt/conda/lib/python3.7/site-packages (from boto3->bentoml==0.8.6->-r ./requirements.txt (line 3)) (1.17.48)
Requirement already satisfied: jmespath<1.0.0,>=0.7.1 in /opt/conda/lib/python3.7/site-packages (from boto3->bentoml==0.8.6->-r ./requirements.txt (line 3)) (0.10.0)
Requirement already satisfied: s3transfer<0.4.0,>=0.3.0 in /opt/conda/lib/python3.7/site-packages (from boto3->bentoml==0.8.6->-r ./requirements.txt (line 3)) (0.3.3)
Requirement already satisfied: websocket-client>=0.32.0 in /opt/conda/lib/python3.7/site-packages (from docker->bentoml==0.8.6->-r ./requirements.txt (line 3)) (0.57.0)
Requirement already satisfied: typing-extensions>=3.7.4; python_version < "3.8" in /opt/conda/lib/python3.7/site-packages (from yarl<2.0,>=1.0->aiohttp->bentoml==0.8.6->-r ./requirements.txt (line 3)) (3.7.4.3)
Requirement already satisfied: MarkupSafe>=0.9.2 in /opt/conda/lib/python3.7/site-packages (from Mako->alembic->bentoml==0.8.6->-r ./requirements.txt (line 3)) (1.1.1)
Requirement already satisfied: ply<4.0,>=3.4 in /opt/conda/lib/python3.7/site-packages (from thriftpy2>=0.4.0->py-zipkin->bentoml==0.8.6->-r ./requirements.txt (line 3)) (3.11)
Requirement already satisfied: docutils<0.16,>=0.10 in /opt/conda/lib/python3.7/site-packages (from botocore<1.18.0,>=1.17.48->boto3->bentoml==0.8.6->-r ./requirements.txt (line 3)) (0.15.2)
/Installing collected packages: threadpoolctl, joblib, scipy, scikit-learn, pytz, pandas
-Successfully installed joblib-0.16.0 pandas-1.1.2 pytz-2020.1 scikit-learn-0.23.2 scipy-1.5.2 threadpoolctl-2.1.0
-+ for filename in ./bundled_pip_dependencies/*.tar.gz
+ '[' -e ./bundled_pip_dependencies/BentoML-0.8.6+34.g6123b8c6.tar.gz ']'
+ pip install -U ./bundled_pip_dependencies/BentoML-0.8.6+34.g6123b8c6.tar.gz --no-cache-dir

\Looking in indexes: https://pypi.python.org/simple/
Processing ./bundled_pip_dependencies/BentoML-0.8.6+34.g6123b8c6.tar.gz
-  Installing build dependencies: started
-  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
/  Getting requirements to build wheel: finished with status 'done'
    Preparing wheel metadata: started
|    Preparing wheel metadata: finished with status 'done'
-Requirement already satisfied, skipping upgrade: packaging in /opt/conda/lib/python3.7/site-packages (from BentoML==0.8.6+34.g6123b8c6) (20.4)
Requirement already satisfied, skipping upgrade: tabulate in /opt/conda/lib/python3.7/site-packages (from BentoML==0.8.6+34.g6123b8c6) (0.8.7)
Requirement already satisfied, skipping upgrade: grpcio<=1.27.2 in /opt/conda/lib/python3.7/site-packages (from BentoML==0.8.6+34.g6123b8c6) (1.27.2)
Requirement already satisfied, skipping upgrade: protobuf>=3.6.0 in /opt/conda/lib/python3.7/site-packages (from BentoML==0.8.6+34.g6123b8c6) (3.13.0)
Requirement already satisfied, skipping upgrade: configparser in /opt/conda/lib/python3.7/site-packages (from BentoML==0.8.6+34.g6123b8c6) (5.0.0)
Requirement already satisfied, skipping upgrade: boto3 in /opt/conda/lib/python3.7/site-packages (from BentoML==0.8.6+34.g6123b8c6) (1.14.48)
Requirement already satisfied, skipping upgrade: aiohttp in /opt/conda/lib/python3.7/site-packages (from BentoML==0.8.6+34.g6123b8c6) (3.6.2)
/Requirement already satisfied, skipping upgrade: numpy in /opt/conda/lib/python3.7/site-packages (from BentoML==0.8.6+34.g6123b8c6) (1.19.1)
Requirement already satisfied, skipping upgrade: prometheus-client in /opt/conda/lib/python3.7/site-packages (from BentoML==0.8.6+34.g6123b8c6) (0.8.0)
Requirement already satisfied, skipping upgrade: psutil in /opt/conda/lib/python3.7/site-packages (from BentoML==0.8.6+34.g6123b8c6) (5.7.2)
Requirement already satisfied, skipping upgrade: alembic in /opt/conda/lib/python3.7/site-packages (from BentoML==0.8.6+34.g6123b8c6) (1.4.2)
|Collecting sqlalchemy-utils<0.36.8
-  Downloading SQLAlchemy-Utils-0.36.7.tar.gz (131 kB)
|Requirement already satisfied, skipping upgrade: ruamel.yaml>=0.15.0 in /opt/conda/lib/python3.7/site-packages (from BentoML==0.8.6+34.g6123b8c6) (0.15.87)
Requirement already satisfied, skipping upgrade: multidict in /opt/conda/lib/python3.7/site-packages (from BentoML==0.8.6+34.g6123b8c6) (4.7.6)
Requirement already satisfied, skipping upgrade: cerberus in /opt/conda/lib/python3.7/site-packages (from BentoML==0.8.6+34.g6123b8c6) (1.3.2)
Requirement already satisfied, skipping upgrade: flask in /opt/conda/lib/python3.7/site-packages (from BentoML==0.8.6+34.g6123b8c6) (1.1.2)
Requirement already satisfied, skipping upgrade: python-json-logger in /opt/conda/lib/python3.7/site-packages (from BentoML==0.8.6+34.g6123b8c6) (0.1.11)
Requirement already satisfied, skipping upgrade: python-dateutil<3.0.0,>=2.7.3 in /opt/conda/lib/python3.7/site-packages (from BentoML==0.8.6+34.g6123b8c6) (2.8.0)
Requirement already satisfied, skipping upgrade: py-zipkin in /opt/conda/lib/python3.7/site-packages (from BentoML==0.8.6+34.g6123b8c6) (0.20.0)
Requirement already satisfied, skipping upgrade: sqlalchemy>=1.3.0 in /opt/conda/lib/python3.7/site-packages (from BentoML==0.8.6+34.g6123b8c6) (1.3.19)
Requirement already satisfied, skipping upgrade: requests in /opt/conda/lib/python3.7/site-packages (from BentoML==0.8.6+34.g6123b8c6) (2.22.0)
\Requirement already satisfied, skipping upgrade: humanfriendly in /opt/conda/lib/python3.7/site-packages (from BentoML==0.8.6+34.g6123b8c6) (8.2)
Requirement already satisfied, skipping upgrade: docker in /opt/conda/lib/python3.7/site-packages (from BentoML==0.8.6+34.g6123b8c6) (4.3.1)
Requirement already satisfied, skipping upgrade: click>=7.0 in /opt/conda/lib/python3.7/site-packages (from BentoML==0.8.6+34.g6123b8c6) (7.1.2)
Requirement already satisfied, skipping upgrade: gunicorn in /opt/conda/lib/python3.7/site-packages (from BentoML==0.8.6+34.g6123b8c6) (20.0.4)
Requirement already satisfied, skipping upgrade: certifi in /opt/conda/lib/python3.7/site-packages (from BentoML==0.8.6+34.g6123b8c6) (2020.6.20)
Requirement already satisfied, skipping upgrade: six in /opt/conda/lib/python3.7/site-packages (from packaging->BentoML==0.8.6+34.g6123b8c6) (1.14.0)
Requirement already satisfied, skipping upgrade: pyparsing>=2.0.2 in /opt/conda/lib/python3.7/site-packages (from packaging->BentoML==0.8.6+34.g6123b8c6) (2.4.7)
Requirement already satisfied, skipping upgrade: setuptools in /opt/conda/lib/python3.7/site-packages (from protobuf>=3.6.0->BentoML==0.8.6+34.g6123b8c6) (45.2.0.post20200210)
Requirement already satisfied, skipping upgrade: jmespath<1.0.0,>=0.7.1 in /opt/conda/lib/python3.7/site-packages (from boto3->BentoML==0.8.6+34.g6123b8c6) (0.10.0)
Requirement already satisfied, skipping upgrade: s3transfer<0.4.0,>=0.3.0 in /opt/conda/lib/python3.7/site-packages (from boto3->BentoML==0.8.6+34.g6123b8c6) (0.3.3)
Requirement already satisfied, skipping upgrade: botocore<1.18.0,>=1.17.48 in /opt/conda/lib/python3.7/site-packages (from boto3->BentoML==0.8.6+34.g6123b8c6) (1.17.48)
Requirement already satisfied, skipping upgrade: chardet<4.0,>=2.0 in /opt/conda/lib/python3.7/site-packages (from aiohttp->BentoML==0.8.6+34.g6123b8c6) (3.0.4)
Requirement already satisfied, skipping upgrade: yarl<2.0,>=1.0 in /opt/conda/lib/python3.7/site-packages (from aiohttp->BentoML==0.8.6+34.g6123b8c6) (1.5.1)
Requirement already satisfied, skipping upgrade: async-timeout<4.0,>=3.0 in /opt/conda/lib/python3.7/site-packages (from aiohttp->BentoML==0.8.6+34.g6123b8c6) (3.0.1)
Requirement already satisfied, skipping upgrade: attrs>=17.3.0 in /opt/conda/lib/python3.7/site-packages (from aiohttp->BentoML==0.8.6+34.g6123b8c6) (20.1.0)
-Requirement already satisfied, skipping upgrade: Mako in /opt/conda/lib/python3.7/site-packages (from alembic->BentoML==0.8.6+34.g6123b8c6) (1.1.3)
Requirement already satisfied, skipping upgrade: python-editor>=0.3 in /opt/conda/lib/python3.7/site-packages (from alembic->BentoML==0.8.6+34.g6123b8c6) (1.0.4)
Requirement already satisfied, skipping upgrade: Jinja2>=2.10.1 in /opt/conda/lib/python3.7/site-packages (from flask->BentoML==0.8.6+34.g6123b8c6) (2.11.2)
Requirement already satisfied, skipping upgrade: itsdangerous>=0.24 in /opt/conda/lib/python3.7/site-packages (from flask->BentoML==0.8.6+34.g6123b8c6) (1.1.0)
Requirement already satisfied, skipping upgrade: Werkzeug>=0.15 in /opt/conda/lib/python3.7/site-packages (from flask->BentoML==0.8.6+34.g6123b8c6) (1.0.1)
Requirement already satisfied, skipping upgrade: thriftpy2>=0.4.0 in /opt/conda/lib/python3.7/site-packages (from py-zipkin->BentoML==0.8.6+34.g6123b8c6) (0.4.11)
Requirement already satisfied, skipping upgrade: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /opt/conda/lib/python3.7/site-packages (from requests->BentoML==0.8.6+34.g6123b8c6) (1.25.8)
Requirement already satisfied, skipping upgrade: idna<2.9,>=2.5 in /opt/conda/lib/python3.7/site-packages (from requests->BentoML==0.8.6+34.g6123b8c6) (2.8)
Requirement already satisfied, skipping upgrade: websocket-client>=0.32.0 in /opt/conda/lib/python3.7/site-packages (from docker->BentoML==0.8.6+34.g6123b8c6) (0.57.0)
Requirement already satisfied, skipping upgrade: docutils<0.16,>=0.10 in /opt/conda/lib/python3.7/site-packages (from botocore<1.18.0,>=1.17.48->boto3->BentoML==0.8.6+34.g6123b8c6) (0.15.2)
Requirement already satisfied, skipping upgrade: typing-extensions>=3.7.4; python_version < "3.8" in /opt/conda/lib/python3.7/site-packages (from yarl<2.0,>=1.0->aiohttp->BentoML==0.8.6+34.g6123b8c6) (3.7.4.3)
Requirement already satisfied, skipping upgrade: MarkupSafe>=0.9.2 in /opt/conda/lib/python3.7/site-packages (from Mako->alembic->BentoML==0.8.6+34.g6123b8c6) (1.1.1)
Requirement already satisfied, skipping upgrade: ply<4.0,>=3.4 in /opt/conda/lib/python3.7/site-packages (from thriftpy2>=0.4.0->py-zipkin->BentoML==0.8.6+34.g6123b8c6) (3.11)
Building wheels for collected packages: BentoML, sqlalchemy-utils
  Building wheel for BentoML (PEP 517): started
\  Building wheel for BentoML (PEP 517): finished with status 'done'
-  Created wheel for BentoML: filename=BentoML-0.8.6+34.g6123b8c6-py3-none-any.whl size=4712455 sha256=e6944330cc4d24e21e20865bf3b08f780a9c2fde3fbc129cbda87c7f36aef89b
  Stored in directory: /tmp/pip-ephem-wheel-cache-n87zjgsc/wheels/be/7b/58/8207840666d87408400c426e983365fbdcb71015a400a124ea
  Building wheel for sqlalchemy-utils (setup.py): started
-  Building wheel for sqlalchemy-utils (setup.py): finished with status 'done'
  Created wheel for sqlalchemy-utils: filename=SQLAlchemy_Utils-0.36.7-py2.py3-none-any.whl size=93226 sha256=2f7ba4b0a4e6875f0b834144d9850d2a7e7e952d19eb97b5225e8fe8849415cb
  Stored in directory: /tmp/pip-ephem-wheel-cache-n87zjgsc/wheels/85/a9/d2/3377194742a9ad5eb57ee36f934438c15e1aaa0843dd41c899
Successfully built BentoML sqlalchemy-utils
|Installing collected packages: sqlalchemy-utils, BentoML
  Attempting uninstall: sqlalchemy-utils
    Found existing installation: SQLAlchemy-Utils 0.36.8
    Uninstalling SQLAlchemy-Utils-0.36.8:
\      Successfully uninstalled SQLAlchemy-Utils-0.36.8
-  Attempting uninstall: BentoML
    Found existing installation: BentoML 0.8.6
/    Uninstalling BentoML-0.8.6:
\      Successfully uninstalled BentoML-0.8.6
/Successfully installed BentoML-0.8.6+34.g6123b8c6 sqlalchemy-utils-0.36.7
| ---> 0048faff09db
Step 10/15 : ENV PORT 5000
\ ---> Running in 1d42a7a4b733
- ---> 26ba558b2fd9
Step 11/15 : EXPOSE $PORT
| ---> Running in fafe7833175e
\ ---> e0a7df8be789
Step 12/15 : COPY docker-entrypoint.sh /usr/local/bin/
/ ---> 4d4bf73560df
Step 13/15 : RUN chmod +x /usr/local/bin/docker-entrypoint.sh
| ---> Running in e29127c9dd49
- ---> f27486f42c1e
Step 14/15 : ENTRYPOINT [ "docker-entrypoint.sh" ]
 ---> Running in 03d8e4ff5f2a
\ ---> 5b46c9c1baeb
Step 15/15 : CMD ["bentoml", "serve-gunicorn", "/bento"]
 ---> Running in ad2385425be6
/ ---> a541a3c92322
Successfully built a541a3c92322
Successfully tagged irisclassifier:20200909115040_9FC7F4
Finished building irisclassifier:20200909115040_9FC7F4 from IrisClassifier:latest
In [ ]:
!docker run --rm -p 5000:5000 irisclassifier:20200909115040_9FC7F4 

Launch inference job from CLI

BentoML cli supports loading and running a packaged model from CLI. With the DataframeInput adapter, the CLI command supports reading input Dataframe data from CLI argument or local csv or json files:

In [ ]:
!bentoml run IrisClassifier:latest predict --input '[[5, 4, 3, 2]]'

Load saved BentoService

bentoml.load is the API for loading a BentoML packaged model in python:

In [ ]:
from bentoml import load

service = load(saved_path)

print(service.predict([[5,4,3,2]]))

Deployment Options

If you are at a small team with limited engineering or DevOps resources, try out automated deployment with BentoML CLI, currently supporting AWS Lambda, AWS SageMaker, and Azure Functions:

If the cloud platform you are working with is not on the list above, try out these step-by-step guide on manually deploying BentoML packaged model to cloud platforms:

Lastly, if you have a DevOps or ML Engineering team who's operating a Kubernetes or OpenShift cluster, use the following guides as references for implementating your deployment strategy:

In [ ]: