BentoML makes moving trained ML models to production easy:
BentoML is a framework for serving, managing, and deploying machine learning models. It is aiming to bridge the gap between Data Science and DevOps, and enable teams to deliver prediction services in a fast, repeatable, and scalable way.
Before reading this example project, be sure to check out the Getting started guide to learn about the basic concepts in BentoML.
This is a BentoML Demo Project demonstrating how to package and serve LightBGM model for production using BentoML.
BentoML is an open source platform for machine learning model serving and deployment.
In this example, we will use scikit-learn API for both xgboost
and lightgbm
. In general, we can use any python model.
%reload_ext autoreload
%autoreload 2
%matplotlib inline
import warnings
warnings.filterwarnings("ignore")
import bentoml
import lightgbm as lgb
import numpy as np
import pandas as pd
import xgboost as xgb
from sklearn.model_selection import train_test_split
download dataset from https://www.kaggle.com/c/titanic/data
%%sh
mkdir data
curl https://raw.githubusercontent.com/agconti/kaggle-titanic/master/data/train.csv -o ./data/train.csv
curl https://raw.githubusercontent.com/agconti/kaggle-titanic/master/data/test.csv -o ./data/test.csv
% Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 60302 100 60302 0 0 129k 0 --:--:-- --:--:-- --:--:-- 128k % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 28210 100 28210 0 0 86006 0 --:--:-- --:--:-- --:--:-- 86006
train_df = pd.read_csv("./data/train.csv")
test_df = pd.read_csv("./data/test.csv")
train_df.head()
PassengerId | Survived | Pclass | Name | Sex | Age | SibSp | Parch | Ticket | Fare | Cabin | Embarked | |
---|---|---|---|---|---|---|---|---|---|---|---|---|
0 | 1 | 0 | 3 | Braund, Mr. Owen Harris | male | 22.0 | 1 | 0 | A/5 21171 | 7.2500 | NaN | S |
1 | 2 | 1 | 1 | Cumings, Mrs. John Bradley (Florence Briggs Th... | female | 38.0 | 1 | 0 | PC 17599 | 71.2833 | C85 | C |
2 | 3 | 1 | 3 | Heikkinen, Miss. Laina | female | 26.0 | 0 | 0 | STON/O2. 3101282 | 7.9250 | NaN | S |
3 | 4 | 1 | 1 | Futrelle, Mrs. Jacques Heath (Lily May Peel) | female | 35.0 | 1 | 0 | 113803 | 53.1000 | C123 | S |
4 | 5 | 0 | 3 | Allen, Mr. William Henry | male | 35.0 | 0 | 0 | 373450 | 8.0500 | NaN | S |
y = train_df.pop("Survived")
cols = ["Pclass", "Age", "Fare", "SibSp", "Parch"]
X_train, X_test, y_train, y_test = train_test_split(
train_df[cols], y, test_size=0.2, random_state=42
)
lgb_model = lgb.LGBMClassifier()
lgb_model.fit(X_train, y_train)
LGBMClassifier()
xgb_model = xgb.sklearn.XGBRFClassifier()
xgb_model.fit(X_train, y_train)
models = {"xgb": xgb_model, "lgb": lgb_model}
We are going to use JsonInput
and return the data as JSON
object. JSON
objects are passed as a list.
%%writefile multiple_models_titanic_bento_service.py
import json
import bentoml
import lightgbm as lgb
import pandas as pd
import xgboost as xgb
from bentoml.adapters import JsonInput
from bentoml.frameworks.sklearn import SklearnModelArtifact
@bentoml.artifacts([SklearnModelArtifact("xgb"), SklearnModelArtifact("lgb")])
@bentoml.env(
conda_channels=["conda-forge"],
conda_dependencies=["lightgbm==2.3.*", "pandas==1.0.*", "xgboost==1.2.*"],
)
class TitanicSurvivalPredictionService(bentoml.BentoService):
@bentoml.api(input=JsonInput(), batch=True)
def predict(self, datain):
# datain is a list of a json object.
df = pd.read_json(json.dumps(datain[0]), orient="table")
data = df[["Pclass", "Age", "Fare", "SibSp", "Parch"]]
result = pd.DataFrame()
result["xgb_proba"] = self.artifacts.xgb.predict_proba(data)[:, 1]
result["lgb_proba"] = self.artifacts.lgb.predict_proba(data)[:, 1]
# make sure to return as a list of json
return [result.to_json(orient="table")]
Overwriting multiple_models_titanic_bento_service.py
# 1) import the custom BentoService defined above
from multiple_models_titanic_bento_service import TitanicSurvivalPredictionService
# 2) `pack` it with required artifacts
bento_service = TitanicSurvivalPredictionService()
bento_service.pack("xgb", xgb_model)
bento_service.pack("lgb", lgb_model)
# 3) save your BentoSerivce
saved_path = bento_service.save()
[2020-08-25 13:47:58,242] INFO - BentoService bundle 'TitanicSurvivalPredictionService:20200825134757_086746' saved to: /Users/thein/bentoml/repository/TitanicSurvivalPredictionService/20200825134757_086746
One common way of distributing this model API server for production deployment, is via Docker containers. And BentoML provides a convenient way to do that.
Note that docker is not available in Google Colab. You will need to download and run this notebook locally to try out this containerization with docker feature.
If you already have docker configured, simply run the follow command to product a docker container serving the IrisClassifier prediction service created above:
!bentoml containerize TitanicSurvivalPredictionService:latest
Sending build context to Docker daemon 551.4kB Step 1/15 : FROM bentoml/model-server:0.8.5 ---> 6639eed59dc6 Step 2/15 : COPY . /bento ---> 8916c6323930 Step 3/15 : WORKDIR /bento ---> Running in e8a222116d35 Removing intermediate container e8a222116d35 ---> ba725f7ade92 Step 4/15 : ARG PIP_INDEX_URL=https://pypi.python.org/simple/ ---> Running in e194a9e5087c Removing intermediate container e194a9e5087c ---> bfe333f7f373 Step 5/15 : ARG PIP_TRUSTED_HOST=pypi.python.org ---> Running in dbf5f680e79d Removing intermediate container dbf5f680e79d ---> 0c9fc4ae7489 Step 6/15 : ENV PIP_INDEX_URL $PIP_INDEX_URL ---> Running in cb44a811259f Removing intermediate container cb44a811259f ---> 7a6e44b43f45 Step 7/15 : ENV PIP_TRUSTED_HOST $PIP_TRUSTED_HOST ---> Running in bca323581f0b Removing intermediate container bca323581f0b ---> 8f1f8b11a2d2 Step 8/15 : RUN chmod +x /bento/bentoml-init.sh ---> Running in 3ddc42cb25c2 Removing intermediate container 3ddc42cb25c2 ---> 9b890a10990e Step 9/15 : RUN if [ -f /bento/bentoml-init.sh ]; then bash -c /bento/bentoml-init.sh; fi ---> Running in 5c32936f764e +++ dirname /bento/bentoml-init.sh ++ cd /bento ++ pwd -P + SAVED_BUNDLE_PATH=/bento + cd /bento + '[' -f ./setup.sh ']' + command -v conda + conda env update -n base -f ./environment.yml Collecting package metadata (repodata.json): ...working... done Solving environment: ...working... done Downloading and Extracting Packages mkl-service-2.3.0 | 218 KB | ########## | 100% xgboost-1.2.0 | 11 KB | ########## | 100% intel-openmp-2020.1 | 780 KB | ########## | 100% pip-20.2.2 | 1.8 MB | ########## | 100% mkl-2020.1 | 129.0 MB | ########## | 100% python-dateutil-2.8. | 215 KB | ########## | 100% pytz-2020.1 | 184 KB | ########## | 100% scipy-1.5.2 | 14.3 MB | ########## | 100% certifi-2020.6.20 | 156 KB | ########## | 100% python_abi-3.7 | 4 KB | ########## | 100% libxgboost-1.2.0 | 3.1 MB | ########## | 100% numpy-1.19.1 | 21 KB | ########## | 100% py-xgboost-1.2.0 | 1.7 MB | ########## | 100% openssl-1.1.1g | 2.5 MB | ########## | 100% mkl_random-1.1.1 | 322 KB | ########## | 100% pandas-1.0.5 | 7.8 MB | ########## | 100% joblib-0.16.0 | 210 KB | ########## | 100% scikit-learn-0.23.1 | 5.0 MB | ########## | 100% _py-xgboost-mutex-2. | 9 KB | ########## | 100% numpy-base-1.19.1 | 4.1 MB | ########## | 100% blas-1.0 | 6 KB | ########## | 100% ca-certificates-2020 | 125 KB | ########## | 100% threadpoolctl-2.1.0 | 17 KB | ########## | 100% lightgbm-2.3.0 | 936 KB | ########## | 100% libgfortran-ng-7.3.0 | 1006 KB | ########## | 100% mkl_fft-1.1.0 | 143 KB | ########## | 100% Preparing transaction: ...working... done Verifying transaction: ...working... done Executing transaction: ...working... done # # To activate this environment, use # # $ conda activate base # # To deactivate an active environment, use # # $ conda deactivate ==> WARNING: A newer version of conda exists. <== current version: 4.8.2 latest version: 4.8.4 Please update conda by running $ conda update -n base -c defaults conda + pip install -r ./requirements.txt --no-cache-dir Looking in indexes: https://pypi.python.org/simple/ Requirement already satisfied: scikit-learn in /opt/conda/lib/python3.7/site-packages (from -r ./requirements.txt (line 1)) (0.23.1) Requirement already satisfied: bentoml==0.8.5 in /opt/conda/lib/python3.7/site-packages (from -r ./requirements.txt (line 2)) (0.8.5) Requirement already satisfied: numpy>=1.13.3 in /opt/conda/lib/python3.7/site-packages (from scikit-learn->-r ./requirements.txt (line 1)) (1.19.1) Requirement already satisfied: threadpoolctl>=2.0.0 in /opt/conda/lib/python3.7/site-packages (from scikit-learn->-r ./requirements.txt (line 1)) (2.1.0) Requirement already satisfied: scipy>=0.19.1 in /opt/conda/lib/python3.7/site-packages (from scikit-learn->-r ./requirements.txt (line 1)) (1.5.2) Requirement already satisfied: joblib>=0.11 in /opt/conda/lib/python3.7/site-packages (from scikit-learn->-r ./requirements.txt (line 1)) (0.16.0) Requirement already satisfied: tabulate in /opt/conda/lib/python3.7/site-packages (from bentoml==0.8.5->-r ./requirements.txt (line 2)) (0.8.7) Requirement already satisfied: configparser in /opt/conda/lib/python3.7/site-packages (from bentoml==0.8.5->-r ./requirements.txt (line 2)) (5.0.0) Requirement already satisfied: sqlalchemy>=1.3.0 in /opt/conda/lib/python3.7/site-packages (from bentoml==0.8.5->-r ./requirements.txt (line 2)) (1.3.18) Requirement already satisfied: certifi in /opt/conda/lib/python3.7/site-packages (from bentoml==0.8.5->-r ./requirements.txt (line 2)) (2020.6.20) Requirement already satisfied: cerberus in /opt/conda/lib/python3.7/site-packages (from bentoml==0.8.5->-r ./requirements.txt (line 2)) (1.3.2) Requirement already satisfied: aiohttp in /opt/conda/lib/python3.7/site-packages (from bentoml==0.8.5->-r ./requirements.txt (line 2)) (3.6.2) Requirement already satisfied: packaging in /opt/conda/lib/python3.7/site-packages (from bentoml==0.8.5->-r ./requirements.txt (line 2)) (20.4) Requirement already satisfied: protobuf>=3.6.0 in /opt/conda/lib/python3.7/site-packages (from bentoml==0.8.5->-r ./requirements.txt (line 2)) (3.12.4) Requirement already satisfied: docker in /opt/conda/lib/python3.7/site-packages (from bentoml==0.8.5->-r ./requirements.txt (line 2)) (4.3.0) Requirement already satisfied: grpcio<=1.27.2 in /opt/conda/lib/python3.7/site-packages (from bentoml==0.8.5->-r ./requirements.txt (line 2)) (1.27.2) Collecting python-dateutil<2.8.1,>=2.1 Downloading python_dateutil-2.8.0-py2.py3-none-any.whl (226 kB) Requirement already satisfied: requests in /opt/conda/lib/python3.7/site-packages (from bentoml==0.8.5->-r ./requirements.txt (line 2)) (2.22.0) Requirement already satisfied: prometheus-client in /opt/conda/lib/python3.7/site-packages (from bentoml==0.8.5->-r ./requirements.txt (line 2)) (0.8.0) Requirement already satisfied: gunicorn in /opt/conda/lib/python3.7/site-packages (from bentoml==0.8.5->-r ./requirements.txt (line 2)) (20.0.4) Requirement already satisfied: humanfriendly in /opt/conda/lib/python3.7/site-packages (from bentoml==0.8.5->-r ./requirements.txt (line 2)) (8.2) Requirement already satisfied: flask in /opt/conda/lib/python3.7/site-packages (from bentoml==0.8.5->-r ./requirements.txt (line 2)) (1.1.2) Requirement already satisfied: click>=7.0 in /opt/conda/lib/python3.7/site-packages (from bentoml==0.8.5->-r ./requirements.txt (line 2)) (7.1.2) Requirement already satisfied: psutil in /opt/conda/lib/python3.7/site-packages (from bentoml==0.8.5->-r ./requirements.txt (line 2)) (5.7.2) Requirement already satisfied: boto3 in /opt/conda/lib/python3.7/site-packages (from bentoml==0.8.5->-r ./requirements.txt (line 2)) (1.14.39) Requirement already satisfied: multidict in /opt/conda/lib/python3.7/site-packages (from bentoml==0.8.5->-r ./requirements.txt (line 2)) (4.7.6) Requirement already satisfied: python-json-logger in /opt/conda/lib/python3.7/site-packages (from bentoml==0.8.5->-r ./requirements.txt (line 2)) (0.1.11) Requirement already satisfied: alembic in /opt/conda/lib/python3.7/site-packages (from bentoml==0.8.5->-r ./requirements.txt (line 2)) (1.4.2) Requirement already satisfied: py-zipkin in /opt/conda/lib/python3.7/site-packages (from bentoml==0.8.5->-r ./requirements.txt (line 2)) (0.20.0) Requirement already satisfied: sqlalchemy-utils in /opt/conda/lib/python3.7/site-packages (from bentoml==0.8.5->-r ./requirements.txt (line 2)) (0.36.8) Requirement already satisfied: ruamel.yaml>=0.15.0 in /opt/conda/lib/python3.7/site-packages (from bentoml==0.8.5->-r ./requirements.txt (line 2)) (0.15.87) Requirement already satisfied: setuptools in /opt/conda/lib/python3.7/site-packages (from cerberus->bentoml==0.8.5->-r ./requirements.txt (line 2)) (45.2.0.post20200210) Requirement already satisfied: yarl<2.0,>=1.0 in /opt/conda/lib/python3.7/site-packages (from aiohttp->bentoml==0.8.5->-r ./requirements.txt (line 2)) (1.5.1) Requirement already satisfied: async-timeout<4.0,>=3.0 in /opt/conda/lib/python3.7/site-packages (from aiohttp->bentoml==0.8.5->-r ./requirements.txt (line 2)) (3.0.1) Requirement already satisfied: chardet<4.0,>=2.0 in /opt/conda/lib/python3.7/site-packages (from aiohttp->bentoml==0.8.5->-r ./requirements.txt (line 2)) (3.0.4) Requirement already satisfied: attrs>=17.3.0 in /opt/conda/lib/python3.7/site-packages (from aiohttp->bentoml==0.8.5->-r ./requirements.txt (line 2)) (19.3.0) Requirement already satisfied: six in /opt/conda/lib/python3.7/site-packages (from packaging->bentoml==0.8.5->-r ./requirements.txt (line 2)) (1.14.0) Requirement already satisfied: pyparsing>=2.0.2 in /opt/conda/lib/python3.7/site-packages (from packaging->bentoml==0.8.5->-r ./requirements.txt (line 2)) (2.4.7) Requirement already satisfied: websocket-client>=0.32.0 in /opt/conda/lib/python3.7/site-packages (from docker->bentoml==0.8.5->-r ./requirements.txt (line 2)) (0.57.0) Requirement already satisfied: idna<2.9,>=2.5 in /opt/conda/lib/python3.7/site-packages (from requests->bentoml==0.8.5->-r ./requirements.txt (line 2)) (2.8) Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /opt/conda/lib/python3.7/site-packages (from requests->bentoml==0.8.5->-r ./requirements.txt (line 2)) (1.25.8) Requirement already satisfied: Werkzeug>=0.15 in /opt/conda/lib/python3.7/site-packages (from flask->bentoml==0.8.5->-r ./requirements.txt (line 2)) (1.0.1) Requirement already satisfied: Jinja2>=2.10.1 in /opt/conda/lib/python3.7/site-packages (from flask->bentoml==0.8.5->-r ./requirements.txt (line 2)) (2.11.2) Requirement already satisfied: itsdangerous>=0.24 in /opt/conda/lib/python3.7/site-packages (from flask->bentoml==0.8.5->-r ./requirements.txt (line 2)) (1.1.0) Requirement already satisfied: jmespath<1.0.0,>=0.7.1 in /opt/conda/lib/python3.7/site-packages (from boto3->bentoml==0.8.5->-r ./requirements.txt (line 2)) (0.10.0) Requirement already satisfied: s3transfer<0.4.0,>=0.3.0 in /opt/conda/lib/python3.7/site-packages (from boto3->bentoml==0.8.5->-r ./requirements.txt (line 2)) (0.3.3) Requirement already satisfied: botocore<1.18.0,>=1.17.39 in /opt/conda/lib/python3.7/site-packages (from boto3->bentoml==0.8.5->-r ./requirements.txt (line 2)) (1.17.39) Requirement already satisfied: Mako in /opt/conda/lib/python3.7/site-packages (from alembic->bentoml==0.8.5->-r ./requirements.txt (line 2)) (1.1.3) Requirement already satisfied: python-editor>=0.3 in /opt/conda/lib/python3.7/site-packages (from alembic->bentoml==0.8.5->-r ./requirements.txt (line 2)) (1.0.4) Requirement already satisfied: thriftpy2>=0.4.0 in /opt/conda/lib/python3.7/site-packages (from py-zipkin->bentoml==0.8.5->-r ./requirements.txt (line 2)) (0.4.11) Requirement already satisfied: typing-extensions>=3.7.4; python_version < "3.8" in /opt/conda/lib/python3.7/site-packages (from yarl<2.0,>=1.0->aiohttp->bentoml==0.8.5->-r ./requirements.txt (line 2)) (3.7.4.2) Requirement already satisfied: MarkupSafe>=0.23 in /opt/conda/lib/python3.7/site-packages (from Jinja2>=2.10.1->flask->bentoml==0.8.5->-r ./requirements.txt (line 2)) (1.1.1) Requirement already satisfied: docutils<0.16,>=0.10 in /opt/conda/lib/python3.7/site-packages (from botocore<1.18.0,>=1.17.39->boto3->bentoml==0.8.5->-r ./requirements.txt (line 2)) (0.15.2) Requirement already satisfied: ply<4.0,>=3.4 in /opt/conda/lib/python3.7/site-packages (from thriftpy2>=0.4.0->py-zipkin->bentoml==0.8.5->-r ./requirements.txt (line 2)) (3.11) Installing collected packages: python-dateutil Attempting uninstall: python-dateutil Found existing installation: python-dateutil 2.8.1 Uninstalling python-dateutil-2.8.1: Successfully uninstalled python-dateutil-2.8.1 Successfully installed python-dateutil-2.8.0 + for filename in ./bundled_pip_dependencies/*.tar.gz + '[' -e './bundled_pip_dependencies/*.tar.gz' ']' + continue Removing intermediate container 5c32936f764e ---> 089fdb91acb3 Step 10/15 : ENV PORT 5000 ---> Running in f2c62158d05f Removing intermediate container f2c62158d05f ---> 0d33795106f8 Step 11/15 : EXPOSE $PORT ---> Running in b4fb56d2e1c6 Removing intermediate container b4fb56d2e1c6 ---> df14a798ba76 Step 12/15 : COPY docker-entrypoint.sh /usr/local/bin/ ---> 9e24b68d273d Step 13/15 : RUN chmod +x /usr/local/bin/docker-entrypoint.sh ---> Running in 0ad73aa6f6f0 Removing intermediate container 0ad73aa6f6f0 ---> c608332dd924 Step 14/15 : ENTRYPOINT [ "docker-entrypoint.sh" ] ---> Running in efee5b711dbf Removing intermediate container efee5b711dbf ---> d055e40fe3f9 Step 15/15 : CMD ["bentoml", "serve-gunicorn", "/bento"] ---> Running in 3e0c629cfb1f Removing intermediate container 3e0c629cfb1f ---> fa9a266c2e10 Successfully built fa9a266c2e10 Successfully tagged multi_models_titanic:latest
# port forward to 7000
!docker run -d -p 7000:5000 TitanicSurvivalPredictionService --enable-microbatch
bentoml.load is the API for loading a BentoML packaged model in python:
import json
import bentoml
bento_model = bentoml.load(saved_path)
print(bento_model.predict(X_tests.to_json(orient='table')))
[2020-08-25 13:47:58,780] WARNING - Module `multiple_models_titanic_bento_service` already loaded, using existing imported module.
If you are at a small team with limited engineering or DevOps resources, try out automated deployment with BentoML CLI, currently supporting AWS Lambda, AWS SageMaker, and Azure Functions:
If the cloud platform you are working with is not on the list above, try out these step-by-step guide on manually deploying BentoML packaged model to cloud platforms:
Lastly, if you have a DevOps or ML Engineering team who's operating a Kubernetes or OpenShift cluster, use the following guides as references for implementating your deployment strategy: