BentoML Example: SKLearn to ONNX model

BentoML makes moving trained ML models to production easy:

  • Package models trained with any ML framework and reproduce them for model serving in production
  • Deploy anywhere for online API serving or offline batch serving
  • High-Performance API model server with adaptive micro-batching support
  • Central hub for managing models and deployment process via Web UI and APIs
  • Modular and flexible design making it adaptable to your infrastrcuture

BentoML is a framework for serving, managing, and deploying machine learning models. It is aiming to bridge the gap between Data Science and DevOps, and enable teams to deliver prediction services in a fast, repeatable, and scalable way.

Before reading this example project, be sure to check out the Getting started guide to learn about the basic concepts in BentoML.

This example notebooks demonstrates how to build SK model and packed as ONNX model for BentoML

Impression

In [1]:
!pip install -q bentoml "skl2onnx==1.7.0" "onnx==1.7.0" "onnxmltools==1.7.0"
Collecting skl2onnx
  Using cached skl2onnx-1.7.0-py2.py3-none-any.whl (191 kB)
Requirement already satisfied: onnx in /opt/anaconda3/lib/python3.7/site-packages (1.7.0)
Collecting onnxmltools
  Using cached onnxmltools-1.7.0-py2.py3-none-any.whl (252 kB)
Requirement already satisfied: protobuf in /opt/anaconda3/lib/python3.7/site-packages (from skl2onnx) (3.11.3)
Collecting onnxconverter-common>=1.5.1
  Using cached onnxconverter_common-1.7.0-py2.py3-none-any.whl (64 kB)
Requirement already satisfied: scipy>=1.0 in /opt/anaconda3/lib/python3.7/site-packages (from skl2onnx) (1.3.1)
Requirement already satisfied: six in /opt/anaconda3/lib/python3.7/site-packages (from skl2onnx) (1.15.0)
Requirement already satisfied: scikit-learn>=0.19 in /opt/anaconda3/lib/python3.7/site-packages (from skl2onnx) (0.21.3)
Requirement already satisfied: numpy>=1.15 in /opt/anaconda3/lib/python3.7/site-packages (from skl2onnx) (1.18.5)
Requirement already satisfied: typing-extensions>=3.6.2.1 in /opt/anaconda3/lib/python3.7/site-packages (from onnx) (3.7.4.2)
Collecting keras2onnx
  Using cached keras2onnx-1.7.0-py3-none-any.whl (96 kB)
Requirement already satisfied: setuptools in /opt/anaconda3/lib/python3.7/site-packages (from protobuf->skl2onnx) (49.1.0.post20200710)
Requirement already satisfied: joblib>=0.11 in /opt/anaconda3/lib/python3.7/site-packages (from scikit-learn>=0.19->skl2onnx) (0.13.2)
Requirement already satisfied: requests in /opt/anaconda3/lib/python3.7/site-packages (from keras2onnx->onnxmltools) (2.24.0)
Collecting fire
  Using cached fire-0.3.1.tar.gz (81 kB)
Requirement already satisfied: chardet<4,>=3.0.2 in /opt/anaconda3/lib/python3.7/site-packages (from requests->keras2onnx->onnxmltools) (3.0.4)
Requirement already satisfied: idna<3,>=2.5 in /opt/anaconda3/lib/python3.7/site-packages (from requests->keras2onnx->onnxmltools) (2.10)
Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /opt/anaconda3/lib/python3.7/site-packages (from requests->keras2onnx->onnxmltools) (1.24.3)
Requirement already satisfied: certifi>=2017.4.17 in /opt/anaconda3/lib/python3.7/site-packages (from requests->keras2onnx->onnxmltools) (2020.6.20)
Collecting termcolor
  Downloading termcolor-1.1.0.tar.gz (3.9 kB)
Building wheels for collected packages: fire, termcolor
  Building wheel for fire (setup.py) ... done
  Created wheel for fire: filename=fire-0.3.1-py2.py3-none-any.whl size=111005 sha256=771d43d8451dfcbda6c1066b6e619b3f1ddec020d8ecc3fb0d8c0e70bbf8e526
  Stored in directory: /home/bentoml/.cache/pip/wheels/95/38/e1/8b62337a8ecf5728bdc1017e828f253f7a9cf25db999861bec
  Building wheel for termcolor (setup.py) ... done
  Created wheel for termcolor: filename=termcolor-1.1.0-py3-none-any.whl size=4830 sha256=4551679660493ab6f79c2ab0add35ed6f744f524e43039ab340d2abc7d796707
  Stored in directory: /home/bentoml/.cache/pip/wheels/3f/e3/ec/8a8336ff196023622fbcb36de0c5a5c218cbb24111d1d4c7f2
Successfully built fire termcolor
Installing collected packages: onnxconverter-common, skl2onnx, termcolor, fire, keras2onnx, onnxmltools
Successfully installed fire-0.3.1 keras2onnx-1.7.0 onnxconverter-common-1.7.0 onnxmltools-1.7.0 skl2onnx-1.7.0 termcolor-1.1.0
In [1]:
from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split
iris = load_iris()
X, y = iris.data, iris.target
X_train, X_test, y_train, y_test = train_test_split(X, y)

from sklearn.linear_model import LogisticRegression
clr = LogisticRegression()
clr.fit(X_train, y_train)
/usr/local/anaconda3/envs/dev-py3/lib/python3.7/site-packages/sklearn/linear_model/_logistic.py:762: ConvergenceWarning: lbfgs failed to converge (status=1):
STOP: TOTAL NO. of ITERATIONS REACHED LIMIT.

Increase the number of iterations (max_iter) or scale the data as shown in:
    https://scikit-learn.org/stable/modules/preprocessing.html
Please also refer to the documentation for alternative solver options:
    https://scikit-learn.org/stable/modules/linear_model.html#logistic-regression
  extra_warning_msg=_LOGISTIC_SOLVER_CONVERGENCE_MSG)
Out[1]:
LogisticRegression()
In [2]:
clr.predict([[5.1, 3.5, 1.4, 0.2]])
Out[2]:
array([0])
In [3]:
%%writefile iris_classifier_onnx.py

import numpy
import bentoml
from bentoml.adapters import DataframeInput
from bentoml.frameworks.onnx import OnnxModelArtifact

@bentoml.artifacts([OnnxModelArtifact('model')])
@bentoml.env(infer_pip_packages=True)
class IrisClassifierOnnx(bentoml.BentoService):
    
    @bentoml.api(input=DataframeInput(), batch=True)
    def predict(self, df):
        input_data = df.to_numpy().astype(numpy.float32)
        input_name = self.artifacts.model.get_inputs()[0].name
        output_name = self.artifacts.model.get_outputs()[0].name
        outputs = numpy.zeros(input_data.shape[0])
        for i in range(input_data.shape[0]):
            outputs[i] = self.artifacts.model.run([output_name], {input_name: input_data[i: i + 1]})[0]
        return outputs
Writing iris_classifier_onnx.py
In [4]:
from iris_classifier_onnx import IrisClassifierOnnx
from skl2onnx.common.data_types import FloatTensorType
import onnxmltools



initial_type = [('float_input', FloatTensorType([None, 4]))]
onnx_model = onnxmltools.convert_sklearn(clr, initial_types=initial_type)

svc = IrisClassifierOnnx()
svc.pack('model', onnx_model)

saved_path = svc.save()
[2020-09-22 14:32:58,569] WARNING - Using BentoML installed in `editable` model, the local BentoML repository including all code changes will be packaged together with saved bundle created, under the './bundled_pip_dependencies' directory of the saved bundle.
[2020-09-22 14:32:58,902] INFO - Using default docker base image: `None` specified inBentoML config file or env var. User must make sure that the docker base image either has Python 3.7 or conda installed.
[2020-09-22 14:33:01,032] INFO - Detected non-PyPI-released BentoML installed, copying local BentoML modulefiles to target saved bundle path..
/usr/local/anaconda3/envs/dev-py3/lib/python3.7/site-packages/setuptools/dist.py:476: UserWarning: Normalizing '0.9.0.pre+3.gcebf2015' to '0.9.0rc0+3.gcebf2015'
  normalized_version,
warning: no previously-included files matching '*~' found anywhere in distribution
warning: no previously-included files matching '*.pyo' found anywhere in distribution
warning: no previously-included files matching '.git' found anywhere in distribution
warning: no previously-included files matching '.ipynb_checkpoints' found anywhere in distribution
warning: no previously-included files matching '__pycache__' found anywhere in distribution
no previously-included directories found matching 'e2e_tests'
no previously-included directories found matching 'tests'
no previously-included directories found matching 'benchmark'
UPDATING BentoML-0.9.0rc0+3.gcebf2015/bentoml/_version.py
set BentoML-0.9.0rc0+3.gcebf2015/bentoml/_version.py to '0.9.0.pre+3.gcebf2015'
[2020-09-22 14:33:06,941] INFO - BentoService bundle 'IrisClassifierOnnx:20200922143300_5EB8CF' saved to: /Users/bozhaoyu/bentoml/repository/IrisClassifierOnnx/20200922143300_5EB8CF

REST API Model Serving

To start a REST API model server with the BentoService saved above, use the bentoml serve command:

In [2]:
!bentoml serve IrisClassifierOnnx:latest --enable-microbatch
[2020-08-04 09:58:22,230] INFO - Getting latest version IrisClassifierOnnx:20200804094903_8746D7
[2020-08-04 09:58:22,230] INFO - Starting BentoML API server in development mode..
[2020-08-04 09:58:23,377] WARNING - Using BentoML installed in `editable` model, the local BentoML repository including all code changes will be packaged together with saved bundle created, under the './bundled_pip_dependencies' directory of the saved bundle.
[2020-08-04 09:58:23,405] WARNING - Saved BentoService bundle version mismatch: loading BentoService bundle create with BentoML version 0.8.1, but loading from BentoML version 0.8.3+47.g5daa71b
[2020-08-04 09:58:23,406] WARNING - Saved BentoService Python version mismatch: loading BentoService bundle created with Python version 3.7.4, but current environment version is 3.6.10.
[2020-08-04 09:58:24,464] WARNING - Saved BentoService bundle version mismatch: loading BentoService bundle create with BentoML version 0.8.1, but loading from BentoML version 0.8.3+47.g5daa71b
[2020-08-04 09:58:24,464] WARNING - Saved BentoService Python version mismatch: loading BentoService bundle created with Python version 3.7.4, but current environment version is 3.6.10.
[2020-08-04 09:58:24,466] INFO - Micro batch enabled for API `predict`
[2020-08-04 09:58:24,466] INFO - Your system nofile limit is 10000, which means each instance of microbatch service is able to hold this number of connections at same time. You can increase the number of file descriptors for the server process, or launch more microbatch instances to accept more concurrent connection.
[2020-08-04 09:58:24,475] INFO - Running micro batch service on :5000
 * Serving Flask app "IrisClassifierOnnx" (lazy loading)
 * Environment: production
   WARNING: This is a development server. Do not use it in a production deployment.
   Use a production WSGI server instead.
 * Debug mode: off
 * Running on http://127.0.0.1:40091/ (Press CTRL+C to quit)
======== Running on http://0.0.0.0:5000 ========
(Press CTRL+C to quit)
[2020-08-04 09:58:27,265] INFO - Initializing onnxruntime InferenceSession from onnx file:'/home/bentoml/bentoml/repository/IrisClassifierOnnx/20200804094903_8746D7/IrisClassifierOnnx/artifacts/model.onnx'
127.0.0.1 - - [04/Aug/2020 09:58:28] "POST /predict HTTP/1.1" 200 -
127.0.0.1 - - [04/Aug/2020 09:58:29] "POST /predict HTTP/1.1" 200 -
127.0.0.1 - - [04/Aug/2020 09:58:29] "POST /predict HTTP/1.1" 200 -
127.0.0.1 - - [04/Aug/2020 09:58:30] "POST /predict HTTP/1.1" 200 -
127.0.0.1 - - [04/Aug/2020 09:58:41] "POST /predict HTTP/1.1" 200 -
^C

If you are running this notebook from Google Colab, you can start the dev server with --run-with-ngrok option, to gain acccess to the API endpoint via a public endpoint managed by ngrok:

In [ ]:
!bentoml serve IrisClassifierOnnx:latest --run-with-ngrok

Send prediction requeset to the REST API server

curl -X POST \
  http://localhost:5000/predict \
  -H 'Content-Type: application/json' \
  -d '[[5.1, 3.5, 1.4, 0.2]]'

Containerize model server with Docker

One common way of distributing this model API server for production deployment, is via Docker containers. And BentoML provides a convenient way to do that.

Note that docker is not available in Google Colab. You will need to download and run this notebook locally to try out this containerization with docker feature.

If you already have docker configured, simply run the follow command to product a docker container serving the IrisClassifier prediction service created above:

In [6]:
!bentoml containerize IrisClassifierOnnx:latest
[2020-09-22 14:38:45,682] INFO - Getting latest version IrisClassifierOnnx:20200922143300_5EB8CF
Found Bento: /Users/bozhaoyu/bentoml/repository/IrisClassifierOnnx/20200922143300_5EB8CF
[2020-09-22 14:38:45,725] WARNING - Using BentoML installed in `editable` model, the local BentoML repository including all code changes will be packaged together with saved bundle created, under the './bundled_pip_dependencies' directory of the saved bundle.
[2020-09-22 14:38:45,742] WARNING - Saved BentoService bundle version mismatch: loading BentoService bundle create with BentoML version 0.9.0.pre, but loading from BentoML version 0.9.0.pre+3.gcebf2015
Tag not specified, using tag parsed from BentoService: 'irisclassifieronnx:20200922143300_5EB8CF'
Building Docker image irisclassifieronnx:20200922143300_5EB8CF from IrisClassifierOnnx:latest 
-we in here
processed docker file
(None, None)
root in create archive /Users/bozhaoyu/bentoml/repository/IrisClassifierOnnx/20200922143300_5EB8CF ['Dockerfile', 'IrisClassifierOnnx', 'IrisClassifierOnnx/__init__.py', 'IrisClassifierOnnx/artifacts', 'IrisClassifierOnnx/artifacts/__init__.py', 'IrisClassifierOnnx/artifacts/model.onnx', 'IrisClassifierOnnx/bentoml.yml', 'IrisClassifierOnnx/iris_classifier_onnx.py', 'MANIFEST.in', 'README.md', 'bentoml-init.sh', 'bentoml.yml', 'bundled_pip_dependencies', 'bundled_pip_dependencies/BentoML-0.9.0rc0+3.gcebf2015.tar.gz', 'docker-entrypoint.sh', 'environment.yml', 'python_version', 'requirements.txt', 'setup.py']
about to build
about to upgrade params
check each param and update
if use config proxy
if buildargs
if shmsize
if labels
if cache from
if target
if network_mode
if squash
if extra hosts is not None
if platform is not None
if isolcation is not None
if context is not None
setting auth {'Content-Type': 'application/tar'}
|docker build <tempfile._TemporaryFileWrapper object at 0x7ff8e85b3e48> {'t': 'irisclassifieronnx:20200922143300_5EB8CF', 'remote': None, 'q': False, 'nocache': False, 'rm': False, 'forcerm': False, 'pull': False, 'dockerfile': (None, None)}
\docker response <Response [200]>
context closes
print responses
Step 1/15 : FROM bentoml/model-server:0.9.0.pre
 ---> a25066aa8b0e
Step 2/15 : ARG EXTRA_PIP_INSTALL_ARGS=
 ---> Using cache
 ---> fc6e47d06522
Step 3/15 : ENV EXTRA_PIP_INSTALL_ARGS $EXTRA_PIP_INSTALL_ARGS
 ---> Using cache
 ---> db8172e98571
Step 4/15 : COPY environment.yml requirements.txt setup.sh* bentoml-init.sh python_version* /bento/
\ ---> 8c3896cf0c52
Step 5/15 : WORKDIR /bento
- ---> Running in dfa90bd500df
/ ---> 728e3aac9702
Step 6/15 : RUN chmod +x /bento/bentoml-init.sh
 ---> Running in 41b3849a6c04
\ ---> b0dd49ef9cf4
Step 7/15 : RUN if [ -f /bento/bentoml-init.sh ]; then bash -c /bento/bentoml-init.sh; fi
 ---> Running in 3ebbaf28821e
|+++ dirname /bento/bentoml-init.sh
++ cd /bento
++ pwd -P

+ SAVED_BUNDLE_PATH=/bento
+ cd /bento
+ '[' -f ./setup.sh ']'

+ '[' -f ./python_version ']'

++ cat ./python_version

+ PY_VERSION_SAVED=3.7.3
+ DESIRED_PY_VERSION=3.7

++ python -c 'import sys; print(f"{sys.version_info.major}.{sys.version_info.minor}")'

Python Version in docker base image 3.7 matches requirement python=3.7. Skipping.
+ CURRENT_PY_VERSION=3.7
+ [[ 3.7 == \3\.\7 ]]
+ echo 'Python Version in docker base image 3.7 matches requirement python=3.7. Skipping.'
+ command -v conda

Updating conda base environment with environment.yml
+ echo 'Updating conda base environment with environment.yml'
+ conda env update -n base -f ./environment.yml

|Collecting package metadata (repodata.json): ...working... 
-done
Solving environment: ...working... 
-done
/
Downloading and Extracting Packages
cffi-1.14.3          | 223 KB    |            |   0% 
cffi-1.14.3          | 223 KB    | 7          |   7% 
cffi-1.14.3          | 223 KB    | ########## | 100% 

ca-certificates-2020 | 145 KB    |            |   0% 
ca-certificates-2020 | 145 KB    | ########## | 100% 

openssl-1.1.1h       | 2.1 MB    |            |   0% 
openssl-1.1.1h       | 2.1 MB    | ##3        |  24% 
openssl-1.1.1h       | 2.1 MB    | #########3 |  94% 
openssl-1.1.1h       | 2.1 MB    | ########## | 100% 

certifi-2020.6.20    | 151 KB    |            |   0% 
certifi-2020.6.20    | 151 KB    | ########## | 100% 

python_abi-3.7       | 4 KB      |            |   0% 
python_abi-3.7       | 4 KB      | ########## | 100% 

libffi-3.2.1         | 47 KB     |            |   0% 
libffi-3.2.1         | 47 KB     | ########## | 100% 

pip-20.2.3           | 1.1 MB    |            |   0% 
pip-20.2.3           | 1.1 MB    | #########7 |  98% 
pip-20.2.3           | 1.1 MB    | ########## | 100% 
Preparing transaction: ...working... 
-done
Verifying transaction: ...working... 
\done
Executing transaction: ...working... 
\done
/#
# To activate this environment, use
#
#     $ conda activate base
#
# To deactivate an active environment, use
#
#     $ conda deactivate
-+ pip install -r ./requirements.txt --no-cache-dir

/Requirement already satisfied: bentoml==0.9.0.pre in /opt/conda/lib/python3.7/site-packages (from -r ./requirements.txt (line 1)) (0.9.0rc0)
\Collecting pandas==0.24.2
/  Downloading pandas-0.24.2-cp37-cp37m-manylinux1_x86_64.whl (10.1 MB)
/Collecting onnxruntime==1.3.0
|  Downloading onnxruntime-1.3.0-cp37-cp37m-manylinux1_x86_64.whl (3.9 MB)
/Collecting numpy==1.18.4
  Downloading numpy-1.18.4-cp37-cp37m-manylinux1_x86_64.whl (20.2 MB)
-Requirement already satisfied: sqlalchemy-utils<0.36.8 in /opt/conda/lib/python3.7/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (0.36.7)
/Requirement already satisfied: multidict in /opt/conda/lib/python3.7/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (4.7.6)
Requirement already satisfied: configparser in /opt/conda/lib/python3.7/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (5.0.0)
|Requirement already satisfied: psutil in /opt/conda/lib/python3.7/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (5.7.2)
Requirement already satisfied: humanfriendly in /opt/conda/lib/python3.7/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (8.2)
Requirement already satisfied: boto3 in /opt/conda/lib/python3.7/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (1.15.2)
Requirement already satisfied: certifi in /opt/conda/lib/python3.7/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (2020.6.20)
Requirement already satisfied: protobuf>=3.6.0 in /opt/conda/lib/python3.7/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (3.13.0)
Requirement already satisfied: aiohttp in /opt/conda/lib/python3.7/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (3.6.2)
Requirement already satisfied: requests in /opt/conda/lib/python3.7/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (2.24.0)
Requirement already satisfied: prometheus-client in /opt/conda/lib/python3.7/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (0.8.0)
Requirement already satisfied: flask in /opt/conda/lib/python3.7/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (1.1.2)
Requirement already satisfied: cerberus in /opt/conda/lib/python3.7/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (1.3.2)
Requirement already satisfied: sqlalchemy>=1.3.0 in /opt/conda/lib/python3.7/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (1.3.19)
\Requirement already satisfied: packaging in /opt/conda/lib/python3.7/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (20.4)
Requirement already satisfied: click>=7.0 in /opt/conda/lib/python3.7/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (7.1.2)
Requirement already satisfied: py-zipkin in /opt/conda/lib/python3.7/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (0.20.0)
Requirement already satisfied: ruamel.yaml>=0.15.0 in /opt/conda/lib/python3.7/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (0.15.87)
Requirement already satisfied: python-dateutil<3.0.0,>=2.7.3 in /opt/conda/lib/python3.7/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (2.8.1)
Requirement already satisfied: gunicorn in /opt/conda/lib/python3.7/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (20.0.4)
Requirement already satisfied: python-json-logger in /opt/conda/lib/python3.7/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (0.1.11)
Requirement already satisfied: docker in /opt/conda/lib/python3.7/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (4.3.1)
Requirement already satisfied: grpcio<=1.27.2 in /opt/conda/lib/python3.7/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (1.27.2)
Requirement already satisfied: tabulate in /opt/conda/lib/python3.7/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (0.8.7)
Requirement already satisfied: alembic in /opt/conda/lib/python3.7/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (1.4.3)
/Collecting pytz>=2011k
  Downloading pytz-2020.1-py2.py3-none-any.whl (510 kB)
\Collecting onnx>=1.2.3
  Downloading onnx-1.7.0-cp37-cp37m-manylinux1_x86_64.whl (7.4 MB)
-Requirement already satisfied: six in /opt/conda/lib/python3.7/site-packages (from sqlalchemy-utils<0.36.8->bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (1.15.0)
Requirement already satisfied: botocore<1.19.0,>=1.18.2 in /opt/conda/lib/python3.7/site-packages (from boto3->bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (1.18.2)
Requirement already satisfied: jmespath<1.0.0,>=0.7.1 in /opt/conda/lib/python3.7/site-packages (from boto3->bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (0.10.0)
Requirement already satisfied: s3transfer<0.4.0,>=0.3.0 in /opt/conda/lib/python3.7/site-packages (from boto3->bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (0.3.3)
Requirement already satisfied: setuptools in /opt/conda/lib/python3.7/site-packages (from protobuf>=3.6.0->bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (49.6.0.post20200814)
Requirement already satisfied: yarl<2.0,>=1.0 in /opt/conda/lib/python3.7/site-packages (from aiohttp->bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (1.5.1)
/Requirement already satisfied: async-timeout<4.0,>=3.0 in /opt/conda/lib/python3.7/site-packages (from aiohttp->bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (3.0.1)
Requirement already satisfied: attrs>=17.3.0 in /opt/conda/lib/python3.7/site-packages (from aiohttp->bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (20.2.0)
Requirement already satisfied: chardet<4.0,>=2.0 in /opt/conda/lib/python3.7/site-packages (from aiohttp->bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (3.0.4)
Requirement already satisfied: idna<3,>=2.5 in /opt/conda/lib/python3.7/site-packages (from requests->bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (2.10)
Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /opt/conda/lib/python3.7/site-packages (from requests->bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (1.25.10)
Requirement already satisfied: Werkzeug>=0.15 in /opt/conda/lib/python3.7/site-packages (from flask->bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (1.0.1)
Requirement already satisfied: Jinja2>=2.10.1 in /opt/conda/lib/python3.7/site-packages (from flask->bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (2.11.2)
Requirement already satisfied: itsdangerous>=0.24 in /opt/conda/lib/python3.7/site-packages (from flask->bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (1.1.0)
Requirement already satisfied: pyparsing>=2.0.2 in /opt/conda/lib/python3.7/site-packages (from packaging->bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (2.4.7)
Requirement already satisfied: thriftpy2>=0.4.0 in /opt/conda/lib/python3.7/site-packages (from py-zipkin->bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (0.4.11)
Requirement already satisfied: websocket-client>=0.32.0 in /opt/conda/lib/python3.7/site-packages (from docker->bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (0.57.0)
Requirement already satisfied: Mako in /opt/conda/lib/python3.7/site-packages (from alembic->bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (1.1.3)
Requirement already satisfied: python-editor>=0.3 in /opt/conda/lib/python3.7/site-packages (from alembic->bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (1.0.4)
Requirement already satisfied: typing-extensions>=3.6.2.1 in /opt/conda/lib/python3.7/site-packages (from onnx>=1.2.3->onnxruntime==1.3.0->-r ./requirements.txt (line 3)) (3.7.4.3)
Requirement already satisfied: MarkupSafe>=0.23 in /opt/conda/lib/python3.7/site-packages (from Jinja2>=2.10.1->flask->bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (1.1.1)
Requirement already satisfied: ply<4.0,>=3.4 in /opt/conda/lib/python3.7/site-packages (from thriftpy2>=0.4.0->py-zipkin->bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (3.11)
\Installing collected packages: numpy, pytz, pandas, onnx, onnxruntime
  Attempting uninstall: numpy
    Found existing installation: numpy 1.19.2
/    Uninstalling numpy-1.19.2:
\      Successfully uninstalled numpy-1.19.2
-Successfully installed numpy-1.18.4 onnx-1.7.0 onnxruntime-1.3.0 pandas-0.24.2 pytz-2020.1
\ ---> 866c51f54760
Step 8/15 : COPY . /bento
/ ---> 6e11b086aa71
Step 9/15 : RUN if [ -d /bento/bundled_pip_dependencies ]; then pip install -U bundled_pip_dependencies/* ;fi
 ---> Running in 675f3fde37a6
-Processing ./bundled_pip_dependencies/BentoML-0.9.0rc0+3.gcebf2015.tar.gz
-  Installing build dependencies: started
-  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
/  Getting requirements to build wheel: finished with status 'done'
    Preparing wheel metadata: started
\    Preparing wheel metadata: finished with status 'done'
/Requirement already satisfied, skipping upgrade: protobuf>=3.6.0 in /opt/conda/lib/python3.7/site-packages (from BentoML==0.9.0rc0+3.gcebf2015) (3.13.0)
Requirement already satisfied, skipping upgrade: py-zipkin in /opt/conda/lib/python3.7/site-packages (from BentoML==0.9.0rc0+3.gcebf2015) (0.20.0)
Requirement already satisfied, skipping upgrade: requests in /opt/conda/lib/python3.7/site-packages (from BentoML==0.9.0rc0+3.gcebf2015) (2.24.0)
Requirement already satisfied, skipping upgrade: gunicorn in /opt/conda/lib/python3.7/site-packages (from BentoML==0.9.0rc0+3.gcebf2015) (20.0.4)
Requirement already satisfied, skipping upgrade: grpcio<=1.27.2 in /opt/conda/lib/python3.7/site-packages (from BentoML==0.9.0rc0+3.gcebf2015) (1.27.2)
Requirement already satisfied, skipping upgrade: python-json-logger in /opt/conda/lib/python3.7/site-packages (from BentoML==0.9.0rc0+3.gcebf2015) (0.1.11)
Requirement already satisfied, skipping upgrade: packaging in /opt/conda/lib/python3.7/site-packages (from BentoML==0.9.0rc0+3.gcebf2015) (20.4)
Requirement already satisfied, skipping upgrade: click>=7.0 in /opt/conda/lib/python3.7/site-packages (from BentoML==0.9.0rc0+3.gcebf2015) (7.1.2)
Requirement already satisfied, skipping upgrade: alembic in /opt/conda/lib/python3.7/site-packages (from BentoML==0.9.0rc0+3.gcebf2015) (1.4.3)
|Requirement already satisfied, skipping upgrade: psutil in /opt/conda/lib/python3.7/site-packages (from BentoML==0.9.0rc0+3.gcebf2015) (5.7.2)
Requirement already satisfied, skipping upgrade: sqlalchemy>=1.3.0 in /opt/conda/lib/python3.7/site-packages (from BentoML==0.9.0rc0+3.gcebf2015) (1.3.19)
Requirement already satisfied, skipping upgrade: multidict in /opt/conda/lib/python3.7/site-packages (from BentoML==0.9.0rc0+3.gcebf2015) (4.7.6)
Requirement already satisfied, skipping upgrade: aiohttp in /opt/conda/lib/python3.7/site-packages (from BentoML==0.9.0rc0+3.gcebf2015) (3.6.2)
Requirement already satisfied, skipping upgrade: humanfriendly in /opt/conda/lib/python3.7/site-packages (from BentoML==0.9.0rc0+3.gcebf2015) (8.2)
Requirement already satisfied, skipping upgrade: flask in /opt/conda/lib/python3.7/site-packages (from BentoML==0.9.0rc0+3.gcebf2015) (1.1.2)
Requirement already satisfied, skipping upgrade: prometheus-client in /opt/conda/lib/python3.7/site-packages (from BentoML==0.9.0rc0+3.gcebf2015) (0.8.0)
Requirement already satisfied, skipping upgrade: sqlalchemy-utils<0.36.8 in /opt/conda/lib/python3.7/site-packages (from BentoML==0.9.0rc0+3.gcebf2015) (0.36.7)
\Requirement already satisfied, skipping upgrade: cerberus in /opt/conda/lib/python3.7/site-packages (from BentoML==0.9.0rc0+3.gcebf2015) (1.3.2)
Requirement already satisfied, skipping upgrade: boto3 in /opt/conda/lib/python3.7/site-packages (from BentoML==0.9.0rc0+3.gcebf2015) (1.15.2)
Requirement already satisfied, skipping upgrade: configparser in /opt/conda/lib/python3.7/site-packages (from BentoML==0.9.0rc0+3.gcebf2015) (5.0.0)
Requirement already satisfied, skipping upgrade: python-dateutil<3.0.0,>=2.7.3 in /opt/conda/lib/python3.7/site-packages (from BentoML==0.9.0rc0+3.gcebf2015) (2.8.1)
Requirement already satisfied, skipping upgrade: docker in /opt/conda/lib/python3.7/site-packages (from BentoML==0.9.0rc0+3.gcebf2015) (4.3.1)
Requirement already satisfied, skipping upgrade: ruamel.yaml>=0.15.0 in /opt/conda/lib/python3.7/site-packages (from BentoML==0.9.0rc0+3.gcebf2015) (0.15.87)
Requirement already satisfied, skipping upgrade: numpy in /opt/conda/lib/python3.7/site-packages (from BentoML==0.9.0rc0+3.gcebf2015) (1.18.4)
Requirement already satisfied, skipping upgrade: tabulate in /opt/conda/lib/python3.7/site-packages (from BentoML==0.9.0rc0+3.gcebf2015) (0.8.7)
Requirement already satisfied, skipping upgrade: certifi in /opt/conda/lib/python3.7/site-packages (from BentoML==0.9.0rc0+3.gcebf2015) (2020.6.20)
Requirement already satisfied, skipping upgrade: setuptools in /opt/conda/lib/python3.7/site-packages (from protobuf>=3.6.0->BentoML==0.9.0rc0+3.gcebf2015) (49.6.0.post20200814)
-Requirement already satisfied, skipping upgrade: six>=1.9 in /opt/conda/lib/python3.7/site-packages (from protobuf>=3.6.0->BentoML==0.9.0rc0+3.gcebf2015) (1.15.0)
Requirement already satisfied, skipping upgrade: thriftpy2>=0.4.0 in /opt/conda/lib/python3.7/site-packages (from py-zipkin->BentoML==0.9.0rc0+3.gcebf2015) (0.4.11)
Requirement already satisfied, skipping upgrade: chardet<4,>=3.0.2 in /opt/conda/lib/python3.7/site-packages (from requests->BentoML==0.9.0rc0+3.gcebf2015) (3.0.4)
Requirement already satisfied, skipping upgrade: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /opt/conda/lib/python3.7/site-packages (from requests->BentoML==0.9.0rc0+3.gcebf2015) (1.25.10)
Requirement already satisfied, skipping upgrade: idna<3,>=2.5 in /opt/conda/lib/python3.7/site-packages (from requests->BentoML==0.9.0rc0+3.gcebf2015) (2.10)
Requirement already satisfied, skipping upgrade: pyparsing>=2.0.2 in /opt/conda/lib/python3.7/site-packages (from packaging->BentoML==0.9.0rc0+3.gcebf2015) (2.4.7)
Requirement already satisfied, skipping upgrade: Mako in /opt/conda/lib/python3.7/site-packages (from alembic->BentoML==0.9.0rc0+3.gcebf2015) (1.1.3)
Requirement already satisfied, skipping upgrade: python-editor>=0.3 in /opt/conda/lib/python3.7/site-packages (from alembic->BentoML==0.9.0rc0+3.gcebf2015) (1.0.4)
Requirement already satisfied, skipping upgrade: yarl<2.0,>=1.0 in /opt/conda/lib/python3.7/site-packages (from aiohttp->BentoML==0.9.0rc0+3.gcebf2015) (1.5.1)
Requirement already satisfied, skipping upgrade: async-timeout<4.0,>=3.0 in /opt/conda/lib/python3.7/site-packages (from aiohttp->BentoML==0.9.0rc0+3.gcebf2015) (3.0.1)
Requirement already satisfied, skipping upgrade: attrs>=17.3.0 in /opt/conda/lib/python3.7/site-packages (from aiohttp->BentoML==0.9.0rc0+3.gcebf2015) (20.2.0)
Requirement already satisfied, skipping upgrade: Werkzeug>=0.15 in /opt/conda/lib/python3.7/site-packages (from flask->BentoML==0.9.0rc0+3.gcebf2015) (1.0.1)
Requirement already satisfied, skipping upgrade: itsdangerous>=0.24 in /opt/conda/lib/python3.7/site-packages (from flask->BentoML==0.9.0rc0+3.gcebf2015) (1.1.0)
Requirement already satisfied, skipping upgrade: Jinja2>=2.10.1 in /opt/conda/lib/python3.7/site-packages (from flask->BentoML==0.9.0rc0+3.gcebf2015) (2.11.2)
/Requirement already satisfied, skipping upgrade: botocore<1.19.0,>=1.18.2 in /opt/conda/lib/python3.7/site-packages (from boto3->BentoML==0.9.0rc0+3.gcebf2015) (1.18.2)
Requirement already satisfied, skipping upgrade: jmespath<1.0.0,>=0.7.1 in /opt/conda/lib/python3.7/site-packages (from boto3->BentoML==0.9.0rc0+3.gcebf2015) (0.10.0)
Requirement already satisfied, skipping upgrade: s3transfer<0.4.0,>=0.3.0 in /opt/conda/lib/python3.7/site-packages (from boto3->BentoML==0.9.0rc0+3.gcebf2015) (0.3.3)
Requirement already satisfied, skipping upgrade: websocket-client>=0.32.0 in /opt/conda/lib/python3.7/site-packages (from docker->BentoML==0.9.0rc0+3.gcebf2015) (0.57.0)
Requirement already satisfied, skipping upgrade: ply<4.0,>=3.4 in /opt/conda/lib/python3.7/site-packages (from thriftpy2>=0.4.0->py-zipkin->BentoML==0.9.0rc0+3.gcebf2015) (3.11)
Requirement already satisfied, skipping upgrade: MarkupSafe>=0.9.2 in /opt/conda/lib/python3.7/site-packages (from Mako->alembic->BentoML==0.9.0rc0+3.gcebf2015) (1.1.1)
Requirement already satisfied, skipping upgrade: typing-extensions>=3.7.4; python_version < "3.8" in /opt/conda/lib/python3.7/site-packages (from yarl<2.0,>=1.0->aiohttp->BentoML==0.9.0rc0+3.gcebf2015) (3.7.4.3)
Building wheels for collected packages: BentoML
  Building wheel for BentoML (PEP 517): started
\  Building wheel for BentoML (PEP 517): finished with status 'done'
  Created wheel for BentoML: filename=BentoML-0.9.0rc0+3.gcebf2015-py3-none-any.whl size=3064091 sha256=b065620707c341cb307a75ee66e4c24eafeb59170d1bc3c344f040dcad703f83
  Stored in directory: /root/.cache/pip/wheels/a0/45/41/62152db705af4ff47e7a3d6abf6247986eef4aa1b94a58d3b9
Successfully built BentoML
\Installing collected packages: BentoML
  Attempting uninstall: BentoML
    Found existing installation: BentoML 0.9.0rc0
-    Uninstalling BentoML-0.9.0rc0:
\      Successfully uninstalled BentoML-0.9.0rc0
-Successfully installed BentoML-0.9.0rc0+3.gcebf2015
/ ---> 9d4f82cb1019
Step 10/15 : ENV PORT 5000
 ---> Running in 676f60e8ba7c
| ---> 572602250521
Step 11/15 : EXPOSE $PORT
 ---> Running in e16a57365266
- ---> 9b17a261cfb5
Step 12/15 : COPY docker-entrypoint.sh /usr/local/bin/
/ ---> e86df949cff8
Step 13/15 : RUN chmod +x /usr/local/bin/docker-entrypoint.sh
| ---> Running in 1fa5f562e6d0
- ---> 9633179ac813
Step 14/15 : ENTRYPOINT [ "docker-entrypoint.sh" ]
 ---> Running in 33c5e3e62cab
/ ---> c858376192ca
Step 15/15 : CMD ["bentoml", "serve-gunicorn", "/bento"]
| ---> Running in dd023c74d8f9
 ---> f2eab90cb68d
Successfully built f2eab90cb68d
\Successfully tagged irisclassifieronnx:20200922143300_5EB8CF
Finished building irisclassifieronnx:20200922143300_5EB8CF from IrisClassifierOnnx:latest
In [7]:
!docker run --rm -p 5000:5000 irisclassifieronnx:20200922143300_5EB8CF
[2020-09-22 21:40:47,259] INFO - Starting BentoML API server in production mode..
[2020-09-22 21:40:47,615] INFO - get_gunicorn_num_of_workers: 3, calculated by cpu count
[2020-09-22 21:40:47 +0000] [1] [INFO] Starting gunicorn 20.0.4
[2020-09-22 21:40:47 +0000] [1] [INFO] Listening at: http://0.0.0.0:5000 (1)
[2020-09-22 21:40:47 +0000] [1] [INFO] Using worker: sync
[2020-09-22 21:40:47 +0000] [11] [INFO] Booting worker with pid: 11
[2020-09-22 21:40:47 +0000] [12] [INFO] Booting worker with pid: 12
[2020-09-22 21:40:47 +0000] [13] [INFO] Booting worker with pid: 13
[2020-09-22 21:40:47,877] WARNING - Using BentoML not from official PyPI release. In order to find the same version of BentoML when deploying your BentoService, you must set the 'core/bentoml_deploy_version' config to a http/git location of your BentoML fork, e.g.: 'bentoml_deploy_version = git+https://github.com/{username}/[email protected]{branch}'
[2020-09-22 21:40:47,888] WARNING - Using BentoML not from official PyPI release. In order to find the same version of BentoML when deploying your BentoService, you must set the 'core/bentoml_deploy_version' config to a http/git location of your BentoML fork, e.g.: 'bentoml_deploy_version = git+https://github.com/{username}/[email protected]{branch}'
[2020-09-22 21:40:47,909] WARNING - Saved BentoService bundle version mismatch: loading BentoService bundle create with BentoML version 0.9.0.pre, but loading from BentoML version 0.9.0.pre+3.gcebf2015
[2020-09-22 21:40:47,910] WARNING - Saved BentoService Python version mismatch: loading BentoService bundle created with Python version 3.7.3, but current environment version is 3.7.6.
[2020-09-22 21:40:47,915] WARNING - Saved BentoService bundle version mismatch: loading BentoService bundle create with BentoML version 0.9.0.pre, but loading from BentoML version 0.9.0.pre+3.gcebf2015
[2020-09-22 21:40:47,916] WARNING - Saved BentoService Python version mismatch: loading BentoService bundle created with Python version 3.7.3, but current environment version is 3.7.6.
[2020-09-22 21:40:47,983] WARNING - Using BentoML not from official PyPI release. In order to find the same version of BentoML when deploying your BentoService, you must set the 'core/bentoml_deploy_version' config to a http/git location of your BentoML fork, e.g.: 'bentoml_deploy_version = git+https://github.com/{username}/[email protected]{branch}'
[2020-09-22 21:40:48,010] WARNING - Saved BentoService bundle version mismatch: loading BentoService bundle create with BentoML version 0.9.0.pre, but loading from BentoML version 0.9.0.pre+3.gcebf2015
[2020-09-22 21:40:48,010] WARNING - Saved BentoService Python version mismatch: loading BentoService bundle created with Python version 3.7.3, but current environment version is 3.7.6.
^C
[2020-09-22 21:40:51 +0000] [1] [INFO] Handling signal: int
[2020-09-22 21:40:51 +0000] [11] [INFO] Worker exiting (pid: 11)
[2020-09-22 21:40:51 +0000] [13] [INFO] Worker exiting (pid: 13)
[2020-09-22 21:40:51 +0000] [12] [INFO] Worker exiting (pid: 12)

Load saved BentoService

bentoml.load is the API for loading a BentoML packaged model in python:

In [ ]:
from bentoml import load


svc = load(saved_path)

print(svc.predict([[5.1, 3.5, 1.4, 0.2]]))

Launch inference job from CLI

BentoML cli supports loading and running a packaged model from CLI. With the DataframeInput adapter, the CLI command supports reading input Dataframe data from CLI argument or local csv or json files:

In [9]:
!bentoml run IrisClassifierOnnx:latest predict --input '[[5.1, 3.5, 1.4, 0.2], [5.1, 3.5, 1.4, 0.2]]'
[2020-09-22 14:41:28,131] INFO - Getting latest version IrisClassifierOnnx:20200922143300_5EB8CF
[2020-09-22 14:41:28,177] WARNING - Using BentoML installed in `editable` model, the local BentoML repository including all code changes will be packaged together with saved bundle created, under the './bundled_pip_dependencies' directory of the saved bundle.
[2020-09-22 14:41:28,192] WARNING - Saved BentoService bundle version mismatch: loading BentoService bundle create with BentoML version 0.9.0.pre, but loading from BentoML version 0.9.0.pre+3.gcebf2015
[2020-09-22 14:41:28,584] INFO - Using default docker base image: `None` specified inBentoML config file or env var. User must make sure that the docker base image either has Python 3.7 or conda installed.
[2020-09-22 14:41:29,074] INFO - Initializing onnxruntime InferenceSession from onnx file:'/Users/bozhaoyu/bentoml/repository/IrisClassifierOnnx/20200922143300_5EB8CF/IrisClassifierOnnx/artifacts/model.onnx'
[2020-09-22 14:41:33,065] INFO - {'service_name': 'IrisClassifierOnnx', 'service_version': '20200922143300_5EB8CF', 'api': 'predict', 'task': {'data': {}, 'task_id': '1c20a573-fe40-44b2-b28b-7dd907927996', 'batch': 2, 'cli_args': ('--input', '[[5.1, 3.5, 1.4, 0.2], [5.1, 3.5, 1.4, 0.2]]')}, 'result': {'data': '[0.0, 0.0]', 'http_status': 200, 'http_headers': (('Content-Type', 'application/json'),)}, 'request_id': '1c20a573-fe40-44b2-b28b-7dd907927996'}
[0.0, 0.0]

Deployment Options

If you are at a small team with limited engineering or DevOps resources, try out automated deployment with BentoML CLI, currently supporting AWS Lambda, AWS SageMaker, and Azure Functions:

If the cloud platform you are working with is not on the list above, try out these step-by-step guide on manually deploying BentoML packaged model to cloud platforms:

Lastly, if you have a DevOps or ML Engineering team who's operating a Kubernetes or OpenShift cluster, use the following guides as references for implementating your deployment strategy:

In [ ]: