BentoML Example: Time-Series Statistical Model

BentoML makes moving trained ML models to production easy:

  • Package models trained with any ML framework and reproduce them for model serving in production
  • Deploy anywhere for online API serving or offline batch serving
  • High-Performance API model server with adaptive micro-batching support
  • Central hub for managing models and deployment process via Web UI and APIs
  • Modular and flexible design making it adaptable to your infrastrcuture

BentoML is a framework for serving, managing, and deploying machine learning models. It is aiming to bridge the gap between Data Science and DevOps, and enable teams to deliver prediction services in a fast, repeatable, and scalable way.

Before reading this example project, be sure to check out the Getting started guide to learn about the basic concepts in BentoML.

Make sure to use GPU runtime when running this notebook in Google Colab, you can set it in top menu: Runtime > Change Runtime Type > Hardware accelerator.

Impression

In [1]:
%reload_ext autoreload
%autoreload 2
%matplotlib inline
In [2]:
!pip install -q bentoml statsmodels==0.10.1
WARNING: You are using pip version 20.2.2; however, version 20.2.3 is available.
You should consider upgrading via the '/usr/local/anaconda3/envs/dev-py3/bin/python -m pip install --upgrade pip' command.
In [3]:
%%writefile holt.py

# holt.py
from bentoml import env, artifacts, api, BentoService
from bentoml.adapters import DataframeInput
from bentoml.service.artifacts.common import PickleArtifact
import numpy as np

@env(pip_dependencies=["statsmodels==0.10.1","joblib","numpy"])
@artifacts([PickleArtifact('model')])
class holt(BentoService):
    @api(input=DataframeInput(), batch=True)
    def predict(self, df):

        # Printing the dataframe to cross-cjheck
        print(df.head())

        # Parsing the dataframe values
        weeks=int(df.iat[0,0])
        print(weeks)
        return((self.artifacts.model).forecast(weeks))
  
Overwriting holt.py

The bentoml.api decorator defines a service API, which is the entry point for accessing the prediction service. The DataframeInput here denotes that this service API will convert HTTP JSON request into pandas.DataFrame object before passing it to the user-defined API function code for inference.

The pip_dependencies specify the libraries that you would need in your code. Any library specified here will automatically get added to the requirements.txt folder .

Here we're using the PickleArtifact. However, BentoML also provide model artifact for other frameworks such as PytorchModelArtifact, KerasModelArtifact, FastaiModelArtifact, and XgboostModelArtifact etc.

The following code trains a scikit-learn model and bundles the trained model with an Holt instance. The Holt instance is then saved to disk in the BentoML SavedBundle format, which is a versioned file archive that is ready for production models serving deployment,we've considered a shampoo sales data which is available publicly.

In [5]:
import pandas as pd
from statsmodels.tsa.holtwinters import ExponentialSmoothing
import numpy as np
import joblib
from holt import holt

df=pd.read_csv('https://raw.githubusercontent.com/jbrownlee/Datasets/master/shampoo.csv')

#Taking a test-train split of 80 %
train=df[0:int(len(df)*0.8)] 
test=df[int(len(df)*0.8):]

#Pre-processing the  Month  field
train.Timestamp = pd.to_datetime(train.Month,format='%m-%d') 
train.index = train.Timestamp 
test.Timestamp = pd.to_datetime(test.Month,format='%m-%d') 
test.index = test.Timestamp 

#fitting the model based on  optimal parameters
model = ExponentialSmoothing(np.asarray(train['Sales']) ,seasonal_periods=7 ,trend='add', seasonal='add',).fit()

#creating an instance of the holt class
holt_obj = holt()


# Pack the newly trained model artifact
holt_obj.pack('model', model)
saved_path = holt_obj.save()
[2020-09-22 21:28:55,608] WARNING - Using BentoML installed in `editable` model, the local BentoML repository including all code changes will be packaged together with saved bundle created, under the './bundled_pip_dependencies' directory of the saved bundle.
[2020-09-22 21:28:56,042] INFO - Using default docker base image: `None` specified inBentoML config file or env var. User must make sure that the docker base image either has Python 3.7 or conda installed.
/usr/local/anaconda3/envs/dev-py3/lib/python3.7/site-packages/ipykernel_launcher.py:14: UserWarning: Pandas doesn't allow columns to be created via a new attribute name - see https://pandas.pydata.org/pandas-docs/stable/indexing.html#attribute-access
  
/usr/local/anaconda3/envs/dev-py3/lib/python3.7/site-packages/ipykernel_launcher.py:16: UserWarning: Pandas doesn't allow columns to be created via a new attribute name - see https://pandas.pydata.org/pandas-docs/stable/indexing.html#attribute-access
  app.launch_new_instance()
/usr/local/anaconda3/envs/dev-py3/lib/python3.7/site-packages/statsmodels/tsa/holtwinters.py:712: ConvergenceWarning: Optimization failed to converge. Check mle_retvals.
  ConvergenceWarning)
[2020-09-22 21:28:57,412] INFO - Detected non-PyPI-released BentoML installed, copying local BentoML modulefiles to target saved bundle path..
/usr/local/anaconda3/envs/dev-py3/lib/python3.7/site-packages/setuptools/dist.py:476: UserWarning: Normalizing '0.9.0.pre+6.g4beee0a8.dirty' to '0.9.0rc0+6.g4beee0a8.dirty'
  normalized_version,
warning: no previously-included files matching '*~' found anywhere in distribution
warning: no previously-included files matching '*.pyo' found anywhere in distribution
warning: no previously-included files matching '.git' found anywhere in distribution
warning: no previously-included files matching '.ipynb_checkpoints' found anywhere in distribution
warning: no previously-included files matching '__pycache__' found anywhere in distribution
no previously-included directories found matching 'e2e_tests'
no previously-included directories found matching 'tests'
no previously-included directories found matching 'benchmark'
UPDATING BentoML-0.9.0rc0+6.g4beee0a8.dirty/bentoml/_version.py
set BentoML-0.9.0rc0+6.g4beee0a8.dirty/bentoml/_version.py to '0.9.0.pre+6.g4beee0a8.dirty'
[2020-09-22 21:29:01,475] INFO - BentoService bundle 'holt:20200922212857_02D15F' saved to: /Users/bozhaoyu/bentoml/repository/holt/20200922212857_02D15F

REST API Model Serving

To start a REST API model server with the BentoService saved above, use the bentoml serve command:

In [6]:
!bentoml serve holt:latest
[2020-09-22 21:29:25,793] INFO - Getting latest version holt:20200922212857_02D15F
[2020-09-22 21:29:25,793] INFO - Starting BentoML API server in development mode..
[2020-09-22 21:29:26,096] WARNING - Using BentoML installed in `editable` model, the local BentoML repository including all code changes will be packaged together with saved bundle created, under the './bundled_pip_dependencies' directory of the saved bundle.
[2020-09-22 21:29:26,114] WARNING - Saved BentoService bundle version mismatch: loading BentoService bundle create with BentoML version 0.9.0.pre, but loading from BentoML version 0.9.0.pre+6.g4beee0a8.dirty
[2020-09-22 21:29:26,424] INFO - Using default docker base image: `None` specified inBentoML config file or env var. User must make sure that the docker base image either has Python 3.7 or conda installed.
 * Serving Flask app "holt" (lazy loading)
 * Environment: production
   WARNING: This is a development server. Do not use it in a production deployment.
   Use a production WSGI server instead.
 * Debug mode: off
 * Running on http://127.0.0.1:5000/ (Press CTRL+C to quit)
^C

If you are running this notebook from Google Colab, you can start the dev server with --run-with-ngrok option, to gain acccess to the API endpoint via a public endpoint managed by ngrok:

In [ ]:
!bentoml serve holt:latest --run-with-ngrok

Open http://127.0.0.1:5000 to see more information about the REST APIs server in your browser.

Send prediction requeset to the REST API server

Run the following curl command to send the image to REST API server and get a prediction result:

curl -i \
  --header "Content-Type: application/json" \
  --request POST \
  --data '[[2]]' \
  http://localhost:5000/predict

Containerize model server with Docker

One common way of distributing this model API server for production deployment, is via Docker containers. And BentoML provides a convenient way to do that.

Note that docker is not available in Google Colab. You will need to download and run this notebook locally to try out this containerization with docker feature.

If you already have docker configured, simply run the follow command to product a docker container serving the IrisClassifier prediction service created above:

In [7]:
!bentoml containerize holt:latest 
[2020-09-22 21:30:48,575] INFO - Getting latest version holt:20200922212857_02D15F
Found Bento: /Users/bozhaoyu/bentoml/repository/holt/20200922212857_02D15F
[2020-09-22 21:30:48,613] WARNING - Using BentoML installed in `editable` model, the local BentoML repository including all code changes will be packaged together with saved bundle created, under the './bundled_pip_dependencies' directory of the saved bundle.
[2020-09-22 21:30:48,627] WARNING - Saved BentoService bundle version mismatch: loading BentoService bundle create with BentoML version 0.9.0.pre, but loading from BentoML version 0.9.0.pre+6.g4beee0a8.dirty
Tag not specified, using tag parsed from BentoService: 'holt:20200922212857_02D15F'
Building Docker image holt:20200922212857_02D15F from holt:latest 
-we in here
processed docker file
(None, None)
root in create archive /Users/bozhaoyu/bentoml/repository/holt/20200922212857_02D15F ['Dockerfile', 'MANIFEST.in', 'README.md', 'bentoml-init.sh', 'bentoml.yml', 'bundled_pip_dependencies', 'bundled_pip_dependencies/BentoML-0.9.0rc0+6.g4beee0a8.dirty.tar.gz', 'docker-entrypoint.sh', 'environment.yml', 'holt', 'holt/__init__.py', 'holt/__pycache__', 'holt/__pycache__/holt.cpython-37.pyc', 'holt/artifacts', 'holt/artifacts/__init__.py', 'holt/artifacts/model.pkl', 'holt/bentoml.yml', 'holt/holt.py', 'python_version', 'requirements.txt', 'setup.py']
about to build
about to upgrade params
check each param and update
if use config proxy
if buildargs
if shmsize
if labels
if cache from
if target
if network_mode
if squash
if extra hosts is not None
if platform is not None
if isolcation is not None
if context is not None
setting auth {'Content-Type': 'application/tar'}
\docker build <tempfile._TemporaryFileWrapper object at 0x7faf2be326a0> {'t': 'holt:20200922212857_02D15F', 'remote': None, 'q': False, 'nocache': False, 'rm': False, 'forcerm': False, 'pull': False, 'dockerfile': (None, None)}
/docker response <Response [200]>
context closes
print responses
Step 1/15 : FROM bentoml/model-server:0.9.0.pre
 ---> a25066aa8b0e
Step 2/15 : ARG EXTRA_PIP_INSTALL_ARGS=
 ---> Using cache
 ---> fe99068ff045
Step 3/15 : ENV EXTRA_PIP_INSTALL_ARGS $EXTRA_PIP_INSTALL_ARGS
 ---> Using cache
 ---> d2e44bb9c5c9
Step 4/15 : COPY environment.yml requirements.txt setup.sh* bentoml-init.sh python_version* /bento/
/ ---> d4f6fb31f660
Step 5/15 : WORKDIR /bento
| ---> Running in d84427ac5714
- ---> 030c7183da8b
Step 6/15 : RUN chmod +x /bento/bentoml-init.sh
 ---> Running in 55a7adc8ff2f
| ---> 3b48beda6cb9
Step 7/15 : RUN if [ -f /bento/bentoml-init.sh ]; then bash -c /bento/bentoml-init.sh; fi
 ---> Running in 871d69335602
|+++ dirname /bento/bentoml-init.sh

++ cd /bento
++ pwd -P

+ SAVED_BUNDLE_PATH=/bento

+ cd /bento
+ '[' -f ./setup.sh ']'

+ '[' -f ./python_version ']'

++ cat ./python_version

+ PY_VERSION_SAVED=3.7.3
+ DESIRED_PY_VERSION=3.7

++ python -c 'import sys; print(f"{sys.version_info.major}.{sys.version_info.minor}")'

\+ CURRENT_PY_VERSION=3.7
+ [[ 3.7 == \3\.\7 ]]
+ echo 'Python Version in docker base image 3.7 matches requirement python=3.7. Skipping.'
+ command -v conda
+ echo 'Updating conda base environment with environment.yml'
+ conda env update -n base -f ./environment.yml

Python Version in docker base image 3.7 matches requirement python=3.7. Skipping.
Updating conda base environment with environment.yml
/Collecting package metadata (repodata.json): ...working... 
|done
Solving environment: ...working... 
|done
\
Downloading and Extracting Packages
python_abi-3.7       | 4 KB      |            |   0% 
python_abi-3.7       | 4 KB      | ########## | 100% 
python_abi-3.7       | 4 KB      | ########## | 100% 

certifi-2020.6.20    | 151 KB    |            |   0% 
certifi-2020.6.20    | 151 KB    | ########## | 100% 

pip-20.2.3           | 1.1 MB    |            |   0% 
pip-20.2.3           | 1.1 MB    | ########## | 100% 
pip-20.2.3           | 1.1 MB    | ########## | 100% 

libffi-3.2.1         | 47 KB     |            |   0% 
libffi-3.2.1         | 47 KB     | ########## | 100% 

cffi-1.14.3          | 223 KB    |            |   0% 
cffi-1.14.3          | 223 KB    | ########## | 100% 
cffi-1.14.3          | 223 KB    | ########## | 100% 

openssl-1.1.1h       | 2.1 MB    |            |   0% 
openssl-1.1.1h       | 2.1 MB    | #####      |  50% 
openssl-1.1.1h       | 2.1 MB    | ########## | 100% 
openssl-1.1.1h       | 2.1 MB    | ########## | 100% 

ca-certificates-2020 | 145 KB    |            |   0% 
ca-certificates-2020 | 145 KB    | ########## | 100% 
Preparing transaction: ...working... 
done
Verifying transaction: ...working... 
/done
Executing transaction: ...working... 
/done
\#
# To activate this environment, use
#
#     $ conda activate base
#
# To deactivate an active environment, use
#
#     $ conda deactivate
/+ pip install -r ./requirements.txt --no-cache-dir

/Requirement already satisfied: bentoml==0.9.0.pre in /opt/conda/lib/python3.7/site-packages (from -r ./requirements.txt (line 1)) (0.9.0rc0)
/Collecting statsmodels==0.10.1
|  Downloading statsmodels-0.10.1-cp37-cp37m-manylinux1_x86_64.whl (8.1 MB)
-Collecting joblib==0.14.1
  Downloading joblib-0.14.1-py2.py3-none-any.whl (294 kB)
/Collecting numpy==1.18.4
|  Downloading numpy-1.18.4-cp37-cp37m-manylinux1_x86_64.whl (20.2 MB)
\Collecting pandas==0.24.2
-  Downloading pandas-0.24.2-cp37-cp37m-manylinux1_x86_64.whl (10.1 MB)
\Requirement already satisfied: humanfriendly in /opt/conda/lib/python3.7/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (8.2)
Requirement already satisfied: packaging in /opt/conda/lib/python3.7/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (20.4)
Requirement already satisfied: cerberus in /opt/conda/lib/python3.7/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (1.3.2)
Requirement already satisfied: prometheus-client in /opt/conda/lib/python3.7/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (0.8.0)
Requirement already satisfied: docker in /opt/conda/lib/python3.7/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (4.3.1)
Requirement already satisfied: configparser in /opt/conda/lib/python3.7/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (5.0.0)
Requirement already satisfied: sqlalchemy>=1.3.0 in /opt/conda/lib/python3.7/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (1.3.19)
-Requirement already satisfied: certifi in /opt/conda/lib/python3.7/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (2020.6.20)
Requirement already satisfied: python-json-logger in /opt/conda/lib/python3.7/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (0.1.11)
Requirement already satisfied: requests in /opt/conda/lib/python3.7/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (2.24.0)
Requirement already satisfied: tabulate in /opt/conda/lib/python3.7/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (0.8.7)
Requirement already satisfied: flask in /opt/conda/lib/python3.7/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (1.1.2)
Requirement already satisfied: aiohttp in /opt/conda/lib/python3.7/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (3.6.2)
Requirement already satisfied: py-zipkin in /opt/conda/lib/python3.7/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (0.20.0)
Requirement already satisfied: gunicorn in /opt/conda/lib/python3.7/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (20.0.4)
Requirement already satisfied: psutil in /opt/conda/lib/python3.7/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (5.7.2)
Requirement already satisfied: ruamel.yaml>=0.15.0 in /opt/conda/lib/python3.7/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (0.15.87)
/Requirement already satisfied: grpcio<=1.27.2 in /opt/conda/lib/python3.7/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (1.27.2)
Requirement already satisfied: boto3 in /opt/conda/lib/python3.7/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (1.15.2)
Requirement already satisfied: alembic in /opt/conda/lib/python3.7/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (1.4.3)
Requirement already satisfied: multidict in /opt/conda/lib/python3.7/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (4.7.6)
Requirement already satisfied: protobuf>=3.6.0 in /opt/conda/lib/python3.7/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (3.13.0)
Requirement already satisfied: sqlalchemy-utils<0.36.8 in /opt/conda/lib/python3.7/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (0.36.7)
|Requirement already satisfied: python-dateutil<3.0.0,>=2.7.3 in /opt/conda/lib/python3.7/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (2.8.1)
Requirement already satisfied: click>=7.0 in /opt/conda/lib/python3.7/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (7.1.2)
Collecting patsy>=0.4.0
  Downloading patsy-0.5.1-py2.py3-none-any.whl (231 kB)
/Collecting scipy>=0.18
|  Downloading scipy-1.5.2-cp37-cp37m-manylinux1_x86_64.whl (25.9 MB)
/Collecting pytz>=2011k
  Downloading pytz-2020.1-py2.py3-none-any.whl (510 kB)
Requirement already satisfied: pyparsing>=2.0.2 in /opt/conda/lib/python3.7/site-packages (from packaging->bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (2.4.7)
|Requirement already satisfied: six in /opt/conda/lib/python3.7/site-packages (from packaging->bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (1.15.0)
Requirement already satisfied: setuptools in /opt/conda/lib/python3.7/site-packages (from cerberus->bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (49.6.0.post20200814)
Requirement already satisfied: websocket-client>=0.32.0 in /opt/conda/lib/python3.7/site-packages (from docker->bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (0.57.0)
Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /opt/conda/lib/python3.7/site-packages (from requests->bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (1.25.10)
Requirement already satisfied: idna<3,>=2.5 in /opt/conda/lib/python3.7/site-packages (from requests->bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (2.10)
Requirement already satisfied: chardet<4,>=3.0.2 in /opt/conda/lib/python3.7/site-packages (from requests->bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (3.0.4)
Requirement already satisfied: itsdangerous>=0.24 in /opt/conda/lib/python3.7/site-packages (from flask->bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (1.1.0)
Requirement already satisfied: Werkzeug>=0.15 in /opt/conda/lib/python3.7/site-packages (from flask->bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (1.0.1)
Requirement already satisfied: Jinja2>=2.10.1 in /opt/conda/lib/python3.7/site-packages (from flask->bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (2.11.2)
Requirement already satisfied: async-timeout<4.0,>=3.0 in /opt/conda/lib/python3.7/site-packages (from aiohttp->bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (3.0.1)
Requirement already satisfied: yarl<2.0,>=1.0 in /opt/conda/lib/python3.7/site-packages (from aiohttp->bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (1.5.1)
Requirement already satisfied: attrs>=17.3.0 in /opt/conda/lib/python3.7/site-packages (from aiohttp->bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (20.2.0)
\Requirement already satisfied: thriftpy2>=0.4.0 in /opt/conda/lib/python3.7/site-packages (from py-zipkin->bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (0.4.11)
Requirement already satisfied: botocore<1.19.0,>=1.18.2 in /opt/conda/lib/python3.7/site-packages (from boto3->bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (1.18.2)
Requirement already satisfied: jmespath<1.0.0,>=0.7.1 in /opt/conda/lib/python3.7/site-packages (from boto3->bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (0.10.0)
Requirement already satisfied: s3transfer<0.4.0,>=0.3.0 in /opt/conda/lib/python3.7/site-packages (from boto3->bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (0.3.3)
Requirement already satisfied: python-editor>=0.3 in /opt/conda/lib/python3.7/site-packages (from alembic->bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (1.0.4)
Requirement already satisfied: Mako in /opt/conda/lib/python3.7/site-packages (from alembic->bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (1.1.3)
Requirement already satisfied: MarkupSafe>=0.23 in /opt/conda/lib/python3.7/site-packages (from Jinja2>=2.10.1->flask->bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (1.1.1)
Requirement already satisfied: typing-extensions>=3.7.4; python_version < "3.8" in /opt/conda/lib/python3.7/site-packages (from yarl<2.0,>=1.0->aiohttp->bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (3.7.4.3)
Requirement already satisfied: ply<4.0,>=3.4 in /opt/conda/lib/python3.7/site-packages (from thriftpy2>=0.4.0->py-zipkin->bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (3.11)
-Installing collected packages: pytz, numpy, pandas, patsy, scipy, statsmodels, joblib
|  Attempting uninstall: numpy
    Found existing installation: numpy 1.19.2
-    Uninstalling numpy-1.19.2:
\      Successfully uninstalled numpy-1.19.2
\Successfully installed joblib-0.14.1 numpy-1.18.4 pandas-0.24.2 patsy-0.5.1 pytz-2020.1 scipy-1.5.2 statsmodels-0.10.1
- ---> 47503115aacd
Step 8/15 : COPY . /bento
| ---> 683ecd7746c3
Step 9/15 : RUN if [ -d /bento/bundled_pip_dependencies ]; then pip install -U bundled_pip_dependencies/* ;fi
\ ---> Running in 75c0de695bbf
/Processing ./bundled_pip_dependencies/BentoML-0.9.0rc0+6.g4beee0a8.dirty.tar.gz
/  Installing build dependencies: started
/  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
|  Getting requirements to build wheel: finished with status 'done'
    Preparing wheel metadata: started
-    Preparing wheel metadata: finished with status 'done'
|Requirement already satisfied, skipping upgrade: numpy in /opt/conda/lib/python3.7/site-packages (from BentoML==0.9.0rc0+6.g4beee0a8.dirty) (1.18.4)
Requirement already satisfied, skipping upgrade: sqlalchemy-utils<0.36.8 in /opt/conda/lib/python3.7/site-packages (from BentoML==0.9.0rc0+6.g4beee0a8.dirty) (0.36.7)
Requirement already satisfied, skipping upgrade: psutil in /opt/conda/lib/python3.7/site-packages (from BentoML==0.9.0rc0+6.g4beee0a8.dirty) (5.7.2)
\Requirement already satisfied, skipping upgrade: packaging in /opt/conda/lib/python3.7/site-packages (from BentoML==0.9.0rc0+6.g4beee0a8.dirty) (20.4)
Requirement already satisfied, skipping upgrade: docker in /opt/conda/lib/python3.7/site-packages (from BentoML==0.9.0rc0+6.g4beee0a8.dirty) (4.3.1)
Requirement already satisfied, skipping upgrade: sqlalchemy>=1.3.0 in /opt/conda/lib/python3.7/site-packages (from BentoML==0.9.0rc0+6.g4beee0a8.dirty) (1.3.19)
Requirement already satisfied, skipping upgrade: ruamel.yaml>=0.15.0 in /opt/conda/lib/python3.7/site-packages (from BentoML==0.9.0rc0+6.g4beee0a8.dirty) (0.15.87)
Requirement already satisfied, skipping upgrade: grpcio<=1.27.2 in /opt/conda/lib/python3.7/site-packages (from BentoML==0.9.0rc0+6.g4beee0a8.dirty) (1.27.2)
Requirement already satisfied, skipping upgrade: protobuf>=3.6.0 in /opt/conda/lib/python3.7/site-packages (from BentoML==0.9.0rc0+6.g4beee0a8.dirty) (3.13.0)
Requirement already satisfied, skipping upgrade: python-json-logger in /opt/conda/lib/python3.7/site-packages (from BentoML==0.9.0rc0+6.g4beee0a8.dirty) (0.1.11)
Requirement already satisfied, skipping upgrade: humanfriendly in /opt/conda/lib/python3.7/site-packages (from BentoML==0.9.0rc0+6.g4beee0a8.dirty) (8.2)
Requirement already satisfied, skipping upgrade: prometheus-client in /opt/conda/lib/python3.7/site-packages (from BentoML==0.9.0rc0+6.g4beee0a8.dirty) (0.8.0)
Requirement already satisfied, skipping upgrade: requests in /opt/conda/lib/python3.7/site-packages (from BentoML==0.9.0rc0+6.g4beee0a8.dirty) (2.24.0)
Requirement already satisfied, skipping upgrade: cerberus in /opt/conda/lib/python3.7/site-packages (from BentoML==0.9.0rc0+6.g4beee0a8.dirty) (1.3.2)
Requirement already satisfied, skipping upgrade: click>=7.0 in /opt/conda/lib/python3.7/site-packages (from BentoML==0.9.0rc0+6.g4beee0a8.dirty) (7.1.2)
Requirement already satisfied, skipping upgrade: aiohttp in /opt/conda/lib/python3.7/site-packages (from BentoML==0.9.0rc0+6.g4beee0a8.dirty) (3.6.2)
Requirement already satisfied, skipping upgrade: tabulate in /opt/conda/lib/python3.7/site-packages (from BentoML==0.9.0rc0+6.g4beee0a8.dirty) (0.8.7)
Requirement already satisfied, skipping upgrade: python-dateutil<3.0.0,>=2.7.3 in /opt/conda/lib/python3.7/site-packages (from BentoML==0.9.0rc0+6.g4beee0a8.dirty) (2.8.1)
Requirement already satisfied, skipping upgrade: py-zipkin in /opt/conda/lib/python3.7/site-packages (from BentoML==0.9.0rc0+6.g4beee0a8.dirty) (0.20.0)
-Requirement already satisfied, skipping upgrade: certifi in /opt/conda/lib/python3.7/site-packages (from BentoML==0.9.0rc0+6.g4beee0a8.dirty) (2020.6.20)
Requirement already satisfied, skipping upgrade: alembic in /opt/conda/lib/python3.7/site-packages (from BentoML==0.9.0rc0+6.g4beee0a8.dirty) (1.4.3)
Requirement already satisfied, skipping upgrade: configparser in /opt/conda/lib/python3.7/site-packages (from BentoML==0.9.0rc0+6.g4beee0a8.dirty) (5.0.0)
Requirement already satisfied, skipping upgrade: boto3 in /opt/conda/lib/python3.7/site-packages (from BentoML==0.9.0rc0+6.g4beee0a8.dirty) (1.15.2)
Requirement already satisfied, skipping upgrade: gunicorn in /opt/conda/lib/python3.7/site-packages (from BentoML==0.9.0rc0+6.g4beee0a8.dirty) (20.0.4)
Requirement already satisfied, skipping upgrade: flask in /opt/conda/lib/python3.7/site-packages (from BentoML==0.9.0rc0+6.g4beee0a8.dirty) (1.1.2)
Requirement already satisfied, skipping upgrade: multidict in /opt/conda/lib/python3.7/site-packages (from BentoML==0.9.0rc0+6.g4beee0a8.dirty) (4.7.6)
Requirement already satisfied, skipping upgrade: six in /opt/conda/lib/python3.7/site-packages (from sqlalchemy-utils<0.36.8->BentoML==0.9.0rc0+6.g4beee0a8.dirty) (1.15.0)
Requirement already satisfied, skipping upgrade: pyparsing>=2.0.2 in /opt/conda/lib/python3.7/site-packages (from packaging->BentoML==0.9.0rc0+6.g4beee0a8.dirty) (2.4.7)
Requirement already satisfied, skipping upgrade: websocket-client>=0.32.0 in /opt/conda/lib/python3.7/site-packages (from docker->BentoML==0.9.0rc0+6.g4beee0a8.dirty) (0.57.0)
Requirement already satisfied, skipping upgrade: setuptools in /opt/conda/lib/python3.7/site-packages (from protobuf>=3.6.0->BentoML==0.9.0rc0+6.g4beee0a8.dirty) (49.6.0.post20200814)
Requirement already satisfied, skipping upgrade: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /opt/conda/lib/python3.7/site-packages (from requests->BentoML==0.9.0rc0+6.g4beee0a8.dirty) (1.25.10)
Requirement already satisfied, skipping upgrade: chardet<4,>=3.0.2 in /opt/conda/lib/python3.7/site-packages (from requests->BentoML==0.9.0rc0+6.g4beee0a8.dirty) (3.0.4)
Requirement already satisfied, skipping upgrade: idna<3,>=2.5 in /opt/conda/lib/python3.7/site-packages (from requests->BentoML==0.9.0rc0+6.g4beee0a8.dirty) (2.10)
Requirement already satisfied, skipping upgrade: yarl<2.0,>=1.0 in /opt/conda/lib/python3.7/site-packages (from aiohttp->BentoML==0.9.0rc0+6.g4beee0a8.dirty) (1.5.1)
/Requirement already satisfied, skipping upgrade: async-timeout<4.0,>=3.0 in /opt/conda/lib/python3.7/site-packages (from aiohttp->BentoML==0.9.0rc0+6.g4beee0a8.dirty) (3.0.1)
Requirement already satisfied, skipping upgrade: attrs>=17.3.0 in /opt/conda/lib/python3.7/site-packages (from aiohttp->BentoML==0.9.0rc0+6.g4beee0a8.dirty) (20.2.0)
Requirement already satisfied, skipping upgrade: thriftpy2>=0.4.0 in /opt/conda/lib/python3.7/site-packages (from py-zipkin->BentoML==0.9.0rc0+6.g4beee0a8.dirty) (0.4.11)
Requirement already satisfied, skipping upgrade: python-editor>=0.3 in /opt/conda/lib/python3.7/site-packages (from alembic->BentoML==0.9.0rc0+6.g4beee0a8.dirty) (1.0.4)
Requirement already satisfied, skipping upgrade: Mako in /opt/conda/lib/python3.7/site-packages (from alembic->BentoML==0.9.0rc0+6.g4beee0a8.dirty) (1.1.3)
Requirement already satisfied, skipping upgrade: botocore<1.19.0,>=1.18.2 in /opt/conda/lib/python3.7/site-packages (from boto3->BentoML==0.9.0rc0+6.g4beee0a8.dirty) (1.18.2)
Requirement already satisfied, skipping upgrade: jmespath<1.0.0,>=0.7.1 in /opt/conda/lib/python3.7/site-packages (from boto3->BentoML==0.9.0rc0+6.g4beee0a8.dirty) (0.10.0)
Requirement already satisfied, skipping upgrade: s3transfer<0.4.0,>=0.3.0 in /opt/conda/lib/python3.7/site-packages (from boto3->BentoML==0.9.0rc0+6.g4beee0a8.dirty) (0.3.3)
Requirement already satisfied, skipping upgrade: itsdangerous>=0.24 in /opt/conda/lib/python3.7/site-packages (from flask->BentoML==0.9.0rc0+6.g4beee0a8.dirty) (1.1.0)
Requirement already satisfied, skipping upgrade: Werkzeug>=0.15 in /opt/conda/lib/python3.7/site-packages (from flask->BentoML==0.9.0rc0+6.g4beee0a8.dirty) (1.0.1)
Requirement already satisfied, skipping upgrade: Jinja2>=2.10.1 in /opt/conda/lib/python3.7/site-packages (from flask->BentoML==0.9.0rc0+6.g4beee0a8.dirty) (2.11.2)
Requirement already satisfied, skipping upgrade: typing-extensions>=3.7.4; python_version < "3.8" in /opt/conda/lib/python3.7/site-packages (from yarl<2.0,>=1.0->aiohttp->BentoML==0.9.0rc0+6.g4beee0a8.dirty) (3.7.4.3)
Requirement already satisfied, skipping upgrade: ply<4.0,>=3.4 in /opt/conda/lib/python3.7/site-packages (from thriftpy2>=0.4.0->py-zipkin->BentoML==0.9.0rc0+6.g4beee0a8.dirty) (3.11)
Requirement already satisfied, skipping upgrade: MarkupSafe>=0.9.2 in /opt/conda/lib/python3.7/site-packages (from Mako->alembic->BentoML==0.9.0rc0+6.g4beee0a8.dirty) (1.1.1)
Building wheels for collected packages: BentoML
  Building wheel for BentoML (PEP 517): started
|  Building wheel for BentoML (PEP 517): finished with status 'done'
  Created wheel for BentoML: filename=BentoML-0.9.0rc0+6.g4beee0a8.dirty-py3-none-any.whl size=3058751 sha256=70111534774d574d512f67e82acc43c08e3255b8f20a6268093b77ec2d730339
  Stored in directory: /root/.cache/pip/wheels/04/cb/be/16eecf14d2539252672c27d69a3b88c96604cf94c4c32a2ba7
Successfully built BentoML
|Installing collected packages: BentoML
  Attempting uninstall: BentoML
    Found existing installation: BentoML 0.9.0rc0
-    Uninstalling BentoML-0.9.0rc0:
\      Successfully uninstalled BentoML-0.9.0rc0
/Successfully installed BentoML-0.9.0rc0+6.g4beee0a8.dirty
/ ---> 860141a5b29e
Step 10/15 : ENV PORT 5000
| ---> Running in f4ea0beb335d
- ---> 9e8069c86aae
Step 11/15 : EXPOSE $PORT
/ ---> Running in 80ec14a4aa26
| ---> 7e690b4b6aaa
Step 12/15 : COPY docker-entrypoint.sh /usr/local/bin/
- ---> 7159051061cf
Step 13/15 : RUN chmod +x /usr/local/bin/docker-entrypoint.sh
/ ---> Running in e32ad386b9d8
- ---> adf678a1d0e1
Step 14/15 : ENTRYPOINT [ "docker-entrypoint.sh" ]
/ ---> Running in c5881f6d7757
| ---> ec36c7c0d7b7
Step 15/15 : CMD ["bentoml", "serve-gunicorn", "/bento"]
\ ---> Running in a851a4d1fa6f
- ---> df8295435529
Successfully built df8295435529
Successfully tagged holt:20200922212857_02D15F
Finished building holt:20200922212857_02D15F from holt:latest
In [8]:
!docker run -p 5000:5000 holt:20200922212857_02D15F
[2020-09-23 04:32:30,915] INFO - Starting BentoML API server in production mode..
[2020-09-23 04:32:31,267] INFO - get_gunicorn_num_of_workers: 3, calculated by cpu count
[2020-09-23 04:32:31,277] INFO - Running micro batch service on :5000
[2020-09-23 04:32:31 +0000] [11] [INFO] Starting gunicorn 20.0.4
[2020-09-23 04:32:31 +0000] [1] [INFO] Starting gunicorn 20.0.4
[2020-09-23 04:32:31 +0000] [11] [INFO] Listening at: http://0.0.0.0:5000 (11)
[2020-09-23 04:32:31 +0000] [1] [INFO] Listening at: http://0.0.0.0:59489 (1)
[2020-09-23 04:32:31 +0000] [1] [INFO] Using worker: sync
[2020-09-23 04:32:31 +0000] [11] [INFO] Using worker: aiohttp.worker.GunicornWebWorker
[2020-09-23 04:32:31 +0000] [13] [INFO] Booting worker with pid: 13
[2020-09-23 04:32:31 +0000] [12] [INFO] Booting worker with pid: 12
[2020-09-23 04:32:31,325] WARNING - Using BentoML installed in `editable` model, the local BentoML repository including all code changes will be packaged together with saved bundle created, under the './bundled_pip_dependencies' directory of the saved bundle.
[2020-09-23 04:32:31 +0000] [14] [INFO] Booting worker with pid: 14
[2020-09-23 04:32:31,345] WARNING - Saved BentoService bundle version mismatch: loading BentoService bundle create with BentoML version 0.9.0.pre, but loading from BentoML version 0.9.0.pre+6.g4beee0a8.dirty
[2020-09-23 04:32:31,345] WARNING - Saved BentoService Python version mismatch: loading BentoService bundle created with Python version 3.7.3, but current environment version is 3.7.6.
[2020-09-23 04:32:31,355] INFO - Micro batch enabled for API `predict`
[2020-09-23 04:32:31,355] INFO - Your system nofile limit is 1048576, which means each instance of microbatch service is able to hold this number of connections at same time. You can increase the number of file descriptors for the server process, or launch more microbatch instances to accept more concurrent connection.
[2020-09-23 04:32:31 +0000] [15] [INFO] Booting worker with pid: 15
[2020-09-23 04:32:31,561] WARNING - Using BentoML installed in `editable` model, the local BentoML repository including all code changes will be packaged together with saved bundle created, under the './bundled_pip_dependencies' directory of the saved bundle.
[2020-09-23 04:32:31,566] WARNING - Using BentoML installed in `editable` model, the local BentoML repository including all code changes will be packaged together with saved bundle created, under the './bundled_pip_dependencies' directory of the saved bundle.
[2020-09-23 04:32:31,568] WARNING - Using BentoML installed in `editable` model, the local BentoML repository including all code changes will be packaged together with saved bundle created, under the './bundled_pip_dependencies' directory of the saved bundle.
[2020-09-23 04:32:31,585] WARNING - Saved BentoService bundle version mismatch: loading BentoService bundle create with BentoML version 0.9.0.pre, but loading from BentoML version 0.9.0.pre+6.g4beee0a8.dirty
[2020-09-23 04:32:31,585] WARNING - Saved BentoService Python version mismatch: loading BentoService bundle created with Python version 3.7.3, but current environment version is 3.7.6.
[2020-09-23 04:32:31,592] WARNING - Saved BentoService bundle version mismatch: loading BentoService bundle create with BentoML version 0.9.0.pre, but loading from BentoML version 0.9.0.pre+6.g4beee0a8.dirty
[2020-09-23 04:32:31,592] WARNING - Saved BentoService Python version mismatch: loading BentoService bundle created with Python version 3.7.3, but current environment version is 3.7.6.
[2020-09-23 04:32:31,594] WARNING - Saved BentoService bundle version mismatch: loading BentoService bundle create with BentoML version 0.9.0.pre, but loading from BentoML version 0.9.0.pre+6.g4beee0a8.dirty
[2020-09-23 04:32:31,595] WARNING - Saved BentoService Python version mismatch: loading BentoService bundle created with Python version 3.7.3, but current environment version is 3.7.6.
^C
[2020-09-23 04:32:39 +0000] [1] [INFO] Handling signal: int
[2020-09-23 04:32:39 +0000] [15] [INFO] Worker exiting (pid: 15)
[2020-09-23 04:32:39 +0000] [13] [INFO] Worker exiting (pid: 13)
[2020-09-23 04:32:39 +0000] [14] [INFO] Worker exiting (pid: 14)

Load saved BentoService

bentoml.load is the API for loading a BentoML packaged model in python:

In [13]:
from bentoml import load
import pandas as pd

loaded_svc = load(saved_path)

print(loaded_svc.predict(pd.DataFrame(data=[[2]])))
[2020-09-22 21:34:39,426] WARNING - Saved BentoService bundle version mismatch: loading BentoService bundle create with BentoML version 0.9.0.pre, but loading from BentoML version 0.9.0.pre+6.g4beee0a8.dirty
[2020-09-22 21:34:39,428] WARNING - Module `holt` already loaded, using existing imported module.
[2020-09-22 21:34:39,432] WARNING - pip package requirement pandas already exist
   0
0  2
2
[487.86681173 415.82743026]

Launch inference job from CLI

BentoML cli supports loading and running a packaged model from CLI. With the DataframeInput adapter, the CLI command supports reading input Dataframe data from CLI argument or local csv or json files:

In [10]:
!bentoml run holt:latest predict --input '[[2]]'
[2020-09-22 21:33:21,460] INFO - Getting latest version holt:20200922212857_02D15F
[2020-09-22 21:33:21,499] WARNING - Using BentoML installed in `editable` model, the local BentoML repository including all code changes will be packaged together with saved bundle created, under the './bundled_pip_dependencies' directory of the saved bundle.
[2020-09-22 21:33:21,515] WARNING - Saved BentoService bundle version mismatch: loading BentoService bundle create with BentoML version 0.9.0.pre, but loading from BentoML version 0.9.0.pre+6.g4beee0a8.dirty
[2020-09-22 21:33:21,831] INFO - Using default docker base image: `None` specified inBentoML config file or env var. User must make sure that the docker base image either has Python 3.7 or conda installed.
   0
0  2
2
[2020-09-22 21:33:26,555] INFO - {'service_name': 'holt', 'service_version': '20200922212857_02D15F', 'api': 'predict', 'task': {'data': {}, 'task_id': 'f6ab7f63-0dcc-42c1-8bf3-769572ad51dc', 'batch': 1, 'cli_args': ('--input', '[[2]]')}, 'result': {'data': '[487.8668117296382]', 'http_status': 200, 'http_headers': (('Content-Type', 'application/json'),)}, 'request_id': 'f6ab7f63-0dcc-42c1-8bf3-769572ad51dc'}
[487.8668117296382]

Deployment Options

If you are at a small team with limited engineering or DevOps resources, try out automated deployment with BentoML CLI, currently supporting AWS Lambda, AWS SageMaker, and Azure Functions:

If the cloud platform you are working with is not on the list above, try out these step-by-step guide on manually deploying BentoML packaged model to cloud platforms:

Lastly, if you have a DevOps or ML Engineering team who's operating a Kubernetes or OpenShift cluster, use the following guides as references for implementating your deployment strategy:

In [ ]: