BentoML Example : Multiple Models with JsonInput

Titanic Survival Prediction with Xgboost and Lightgbm

BentoML makes moving trained ML models to production easy:

  • Package models trained with any ML framework and reproduce them for model serving in production
  • Deploy anywhere for online API serving or offline batch serving
  • High-Performance API model server with adaptive micro-batching support
  • Central hub for managing models and deployment process via Web UI and APIs
  • Modular and flexible design making it adaptable to your infrastrcuture

BentoML is a framework for serving, managing, and deploying machine learning models. It is aiming to bridge the gap between Data Science and DevOps, and enable teams to deliver prediction services in a fast, repeatable, and scalable way.

Before reading this example project, be sure to check out the Getting started guide to learn about the basic concepts in BentoML.

This is a BentoML Demo Project demonstrating how to package and serve LightBGM model for production using BentoML.

BentoML is an open source platform for machine learning model serving and deployment.

In this example, we will use scikit-learn API for both xgboost and lightgbm. In general, we can use any python model.

Impression

In [1]:
%reload_ext autoreload
%autoreload 2
%matplotlib inline

import warnings

warnings.filterwarnings("ignore")
In [2]:
import bentoml
import lightgbm as lgb
import numpy as np
import pandas as pd
import xgboost as xgb
from sklearn.model_selection import train_test_split

Prepare Dataset

download dataset from https://www.kaggle.com/c/titanic/data

In [3]:
%%sh
mkdir data
curl https://raw.githubusercontent.com/agconti/kaggle-titanic/master/data/train.csv -o ./data/train.csv
curl https://raw.githubusercontent.com/agconti/kaggle-titanic/master/data/test.csv -o ./data/test.csv
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 60302  100 60302    0     0   129k      0 --:--:-- --:--:-- --:--:--  128k
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 28210  100 28210    0     0  86006      0 --:--:-- --:--:-- --:--:-- 86006
In [4]:
train_df = pd.read_csv("./data/train.csv")
test_df = pd.read_csv("./data/test.csv")
train_df.head()
Out[4]:
PassengerId Survived Pclass Name Sex Age SibSp Parch Ticket Fare Cabin Embarked
0 1 0 3 Braund, Mr. Owen Harris male 22.0 1 0 A/5 21171 7.2500 NaN S
1 2 1 1 Cumings, Mrs. John Bradley (Florence Briggs Th... female 38.0 1 0 PC 17599 71.2833 C85 C
2 3 1 3 Heikkinen, Miss. Laina female 26.0 0 0 STON/O2. 3101282 7.9250 NaN S
3 4 1 1 Futrelle, Mrs. Jacques Heath (Lily May Peel) female 35.0 1 0 113803 53.1000 C123 S
4 5 0 3 Allen, Mr. William Henry male 35.0 0 0 373450 8.0500 NaN S
In [5]:
y = train_df.pop("Survived")
cols = ["Pclass", "Age", "Fare", "SibSp", "Parch"]
X_train, X_test, y_train, y_test = train_test_split(
    train_df[cols], y, test_size=0.2, random_state=42
)

Model Training

In [6]:
lgb_model = lgb.LGBMClassifier()
lgb_model.fit(X_train, y_train)
Out[6]:
LGBMClassifier()
In [ ]:
xgb_model = xgb.sklearn.XGBRFClassifier()
xgb_model.fit(X_train, y_train)
In [8]:
models = {"xgb": xgb_model, "lgb": lgb_model}

Create BentoService for model serving

We are going to use JsonInput and return the data as JSON object. JSON objects are passed as a list.

In [9]:
%%writefile multiple_models_titanic_bento_service.py

import json

import bentoml
import lightgbm as lgb
import pandas as pd
import xgboost as xgb
from bentoml.adapters import JsonInput
from bentoml.frameworks.sklearn import SklearnModelArtifact


@bentoml.artifacts([SklearnModelArtifact("xgb"), SklearnModelArtifact("lgb")])
@bentoml.env(
    conda_channels=["conda-forge"],
    conda_dependencies=["lightgbm==2.3.*", "pandas==1.0.*", "xgboost==1.2.*"],
)
class TitanicSurvivalPredictionService(bentoml.BentoService):
    @bentoml.api(input=JsonInput(), batch=True)
    def predict(self, datain):
        # datain is a list of a json object.
        df = pd.read_json(json.dumps(datain[0]), orient="table")

        data = df[["Pclass", "Age", "Fare", "SibSp", "Parch"]]
        result = pd.DataFrame()
        result["xgb_proba"] = self.artifacts.xgb.predict_proba(data)[:, 1]
        result["lgb_proba"] = self.artifacts.lgb.predict_proba(data)[:, 1]
        # make sure to return as a list of json
        return [result.to_json(orient="table")]
Overwriting multiple_models_titanic_bento_service.py

Save BentoML service archive

In [10]:
# 1) import the custom BentoService defined above
from multiple_models_titanic_bento_service import TitanicSurvivalPredictionService

# 2) `pack` it with required artifacts
bento_service = TitanicSurvivalPredictionService()
bento_service.pack("xgb", xgb_model)
bento_service.pack("lgb", lgb_model)

# 3) save your BentoSerivce
saved_path = bento_service.save()
[2020-08-25 13:47:58,242] INFO - BentoService bundle 'TitanicSurvivalPredictionService:20200825134757_086746' saved to: /Users/thein/bentoml/repository/TitanicSurvivalPredictionService/20200825134757_086746

Containerize model server with Docker

One common way of distributing this model API server for production deployment, is via Docker containers. And BentoML provides a convenient way to do that.

Note that docker is not available in Google Colab. You will need to download and run this notebook locally to try out this containerization with docker feature.

If you already have docker configured, simply run the follow command to product a docker container serving the IrisClassifier prediction service created above:

In [12]:
!bentoml containerize TitanicSurvivalPredictionService:latest
Sending build context to Docker daemon  551.4kB
Step 1/15 : FROM bentoml/model-server:0.8.5
 ---> 6639eed59dc6
Step 2/15 : COPY . /bento
 ---> 8916c6323930
Step 3/15 : WORKDIR /bento
 ---> Running in e8a222116d35
Removing intermediate container e8a222116d35
 ---> ba725f7ade92
Step 4/15 : ARG PIP_INDEX_URL=https://pypi.python.org/simple/
 ---> Running in e194a9e5087c
Removing intermediate container e194a9e5087c
 ---> bfe333f7f373
Step 5/15 : ARG PIP_TRUSTED_HOST=pypi.python.org
 ---> Running in dbf5f680e79d
Removing intermediate container dbf5f680e79d
 ---> 0c9fc4ae7489
Step 6/15 : ENV PIP_INDEX_URL $PIP_INDEX_URL
 ---> Running in cb44a811259f
Removing intermediate container cb44a811259f
 ---> 7a6e44b43f45
Step 7/15 : ENV PIP_TRUSTED_HOST $PIP_TRUSTED_HOST
 ---> Running in bca323581f0b
Removing intermediate container bca323581f0b
 ---> 8f1f8b11a2d2
Step 8/15 : RUN chmod +x /bento/bentoml-init.sh
 ---> Running in 3ddc42cb25c2
Removing intermediate container 3ddc42cb25c2
 ---> 9b890a10990e
Step 9/15 : RUN if [ -f /bento/bentoml-init.sh ]; then bash -c /bento/bentoml-init.sh; fi
 ---> Running in 5c32936f764e
+++ dirname /bento/bentoml-init.sh
++ cd /bento
++ pwd -P
+ SAVED_BUNDLE_PATH=/bento
+ cd /bento
+ '[' -f ./setup.sh ']'
+ command -v conda
+ conda env update -n base -f ./environment.yml
Collecting package metadata (repodata.json): ...working... done
Solving environment: ...working... done

Downloading and Extracting Packages
mkl-service-2.3.0    | 218 KB    | ########## | 100% 
xgboost-1.2.0        | 11 KB     | ########## | 100% 
intel-openmp-2020.1  | 780 KB    | ########## | 100% 
pip-20.2.2           | 1.8 MB    | ########## | 100% 
mkl-2020.1           | 129.0 MB  | ########## | 100% 
python-dateutil-2.8. | 215 KB    | ########## | 100% 
pytz-2020.1          | 184 KB    | ########## | 100% 
scipy-1.5.2          | 14.3 MB   | ########## | 100% 
certifi-2020.6.20    | 156 KB    | ########## | 100% 
python_abi-3.7       | 4 KB      | ########## | 100% 
libxgboost-1.2.0     | 3.1 MB    | ########## | 100% 
numpy-1.19.1         | 21 KB     | ########## | 100% 
py-xgboost-1.2.0     | 1.7 MB    | ########## | 100% 
openssl-1.1.1g       | 2.5 MB    | ########## | 100% 
mkl_random-1.1.1     | 322 KB    | ########## | 100% 
pandas-1.0.5         | 7.8 MB    | ########## | 100% 
joblib-0.16.0        | 210 KB    | ########## | 100% 
scikit-learn-0.23.1  | 5.0 MB    | ########## | 100% 
_py-xgboost-mutex-2. | 9 KB      | ########## | 100% 
numpy-base-1.19.1    | 4.1 MB    | ########## | 100% 
blas-1.0             | 6 KB      | ########## | 100% 
ca-certificates-2020 | 125 KB    | ########## | 100% 
threadpoolctl-2.1.0  | 17 KB     | ########## | 100% 
lightgbm-2.3.0       | 936 KB    | ########## | 100% 
libgfortran-ng-7.3.0 | 1006 KB   | ########## | 100% 
mkl_fft-1.1.0        | 143 KB    | ########## | 100% 
Preparing transaction: ...working... done
Verifying transaction: ...working... done
Executing transaction: ...working... done
#
# To activate this environment, use
#
#     $ conda activate base
#
# To deactivate an active environment, use
#
#     $ conda deactivate



==> WARNING: A newer version of conda exists. <==
  current version: 4.8.2
  latest version: 4.8.4

Please update conda by running

    $ conda update -n base -c defaults conda


+ pip install -r ./requirements.txt --no-cache-dir
Looking in indexes: https://pypi.python.org/simple/
Requirement already satisfied: scikit-learn in /opt/conda/lib/python3.7/site-packages (from -r ./requirements.txt (line 1)) (0.23.1)
Requirement already satisfied: bentoml==0.8.5 in /opt/conda/lib/python3.7/site-packages (from -r ./requirements.txt (line 2)) (0.8.5)
Requirement already satisfied: numpy>=1.13.3 in /opt/conda/lib/python3.7/site-packages (from scikit-learn->-r ./requirements.txt (line 1)) (1.19.1)
Requirement already satisfied: threadpoolctl>=2.0.0 in /opt/conda/lib/python3.7/site-packages (from scikit-learn->-r ./requirements.txt (line 1)) (2.1.0)
Requirement already satisfied: scipy>=0.19.1 in /opt/conda/lib/python3.7/site-packages (from scikit-learn->-r ./requirements.txt (line 1)) (1.5.2)
Requirement already satisfied: joblib>=0.11 in /opt/conda/lib/python3.7/site-packages (from scikit-learn->-r ./requirements.txt (line 1)) (0.16.0)
Requirement already satisfied: tabulate in /opt/conda/lib/python3.7/site-packages (from bentoml==0.8.5->-r ./requirements.txt (line 2)) (0.8.7)
Requirement already satisfied: configparser in /opt/conda/lib/python3.7/site-packages (from bentoml==0.8.5->-r ./requirements.txt (line 2)) (5.0.0)
Requirement already satisfied: sqlalchemy>=1.3.0 in /opt/conda/lib/python3.7/site-packages (from bentoml==0.8.5->-r ./requirements.txt (line 2)) (1.3.18)
Requirement already satisfied: certifi in /opt/conda/lib/python3.7/site-packages (from bentoml==0.8.5->-r ./requirements.txt (line 2)) (2020.6.20)
Requirement already satisfied: cerberus in /opt/conda/lib/python3.7/site-packages (from bentoml==0.8.5->-r ./requirements.txt (line 2)) (1.3.2)
Requirement already satisfied: aiohttp in /opt/conda/lib/python3.7/site-packages (from bentoml==0.8.5->-r ./requirements.txt (line 2)) (3.6.2)
Requirement already satisfied: packaging in /opt/conda/lib/python3.7/site-packages (from bentoml==0.8.5->-r ./requirements.txt (line 2)) (20.4)
Requirement already satisfied: protobuf>=3.6.0 in /opt/conda/lib/python3.7/site-packages (from bentoml==0.8.5->-r ./requirements.txt (line 2)) (3.12.4)
Requirement already satisfied: docker in /opt/conda/lib/python3.7/site-packages (from bentoml==0.8.5->-r ./requirements.txt (line 2)) (4.3.0)
Requirement already satisfied: grpcio<=1.27.2 in /opt/conda/lib/python3.7/site-packages (from bentoml==0.8.5->-r ./requirements.txt (line 2)) (1.27.2)
Collecting python-dateutil<2.8.1,>=2.1
  Downloading python_dateutil-2.8.0-py2.py3-none-any.whl (226 kB)
Requirement already satisfied: requests in /opt/conda/lib/python3.7/site-packages (from bentoml==0.8.5->-r ./requirements.txt (line 2)) (2.22.0)
Requirement already satisfied: prometheus-client in /opt/conda/lib/python3.7/site-packages (from bentoml==0.8.5->-r ./requirements.txt (line 2)) (0.8.0)
Requirement already satisfied: gunicorn in /opt/conda/lib/python3.7/site-packages (from bentoml==0.8.5->-r ./requirements.txt (line 2)) (20.0.4)
Requirement already satisfied: humanfriendly in /opt/conda/lib/python3.7/site-packages (from bentoml==0.8.5->-r ./requirements.txt (line 2)) (8.2)
Requirement already satisfied: flask in /opt/conda/lib/python3.7/site-packages (from bentoml==0.8.5->-r ./requirements.txt (line 2)) (1.1.2)
Requirement already satisfied: click>=7.0 in /opt/conda/lib/python3.7/site-packages (from bentoml==0.8.5->-r ./requirements.txt (line 2)) (7.1.2)
Requirement already satisfied: psutil in /opt/conda/lib/python3.7/site-packages (from bentoml==0.8.5->-r ./requirements.txt (line 2)) (5.7.2)
Requirement already satisfied: boto3 in /opt/conda/lib/python3.7/site-packages (from bentoml==0.8.5->-r ./requirements.txt (line 2)) (1.14.39)
Requirement already satisfied: multidict in /opt/conda/lib/python3.7/site-packages (from bentoml==0.8.5->-r ./requirements.txt (line 2)) (4.7.6)
Requirement already satisfied: python-json-logger in /opt/conda/lib/python3.7/site-packages (from bentoml==0.8.5->-r ./requirements.txt (line 2)) (0.1.11)
Requirement already satisfied: alembic in /opt/conda/lib/python3.7/site-packages (from bentoml==0.8.5->-r ./requirements.txt (line 2)) (1.4.2)
Requirement already satisfied: py-zipkin in /opt/conda/lib/python3.7/site-packages (from bentoml==0.8.5->-r ./requirements.txt (line 2)) (0.20.0)
Requirement already satisfied: sqlalchemy-utils in /opt/conda/lib/python3.7/site-packages (from bentoml==0.8.5->-r ./requirements.txt (line 2)) (0.36.8)
Requirement already satisfied: ruamel.yaml>=0.15.0 in /opt/conda/lib/python3.7/site-packages (from bentoml==0.8.5->-r ./requirements.txt (line 2)) (0.15.87)
Requirement already satisfied: setuptools in /opt/conda/lib/python3.7/site-packages (from cerberus->bentoml==0.8.5->-r ./requirements.txt (line 2)) (45.2.0.post20200210)
Requirement already satisfied: yarl<2.0,>=1.0 in /opt/conda/lib/python3.7/site-packages (from aiohttp->bentoml==0.8.5->-r ./requirements.txt (line 2)) (1.5.1)
Requirement already satisfied: async-timeout<4.0,>=3.0 in /opt/conda/lib/python3.7/site-packages (from aiohttp->bentoml==0.8.5->-r ./requirements.txt (line 2)) (3.0.1)
Requirement already satisfied: chardet<4.0,>=2.0 in /opt/conda/lib/python3.7/site-packages (from aiohttp->bentoml==0.8.5->-r ./requirements.txt (line 2)) (3.0.4)
Requirement already satisfied: attrs>=17.3.0 in /opt/conda/lib/python3.7/site-packages (from aiohttp->bentoml==0.8.5->-r ./requirements.txt (line 2)) (19.3.0)
Requirement already satisfied: six in /opt/conda/lib/python3.7/site-packages (from packaging->bentoml==0.8.5->-r ./requirements.txt (line 2)) (1.14.0)
Requirement already satisfied: pyparsing>=2.0.2 in /opt/conda/lib/python3.7/site-packages (from packaging->bentoml==0.8.5->-r ./requirements.txt (line 2)) (2.4.7)
Requirement already satisfied: websocket-client>=0.32.0 in /opt/conda/lib/python3.7/site-packages (from docker->bentoml==0.8.5->-r ./requirements.txt (line 2)) (0.57.0)
Requirement already satisfied: idna<2.9,>=2.5 in /opt/conda/lib/python3.7/site-packages (from requests->bentoml==0.8.5->-r ./requirements.txt (line 2)) (2.8)
Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /opt/conda/lib/python3.7/site-packages (from requests->bentoml==0.8.5->-r ./requirements.txt (line 2)) (1.25.8)
Requirement already satisfied: Werkzeug>=0.15 in /opt/conda/lib/python3.7/site-packages (from flask->bentoml==0.8.5->-r ./requirements.txt (line 2)) (1.0.1)
Requirement already satisfied: Jinja2>=2.10.1 in /opt/conda/lib/python3.7/site-packages (from flask->bentoml==0.8.5->-r ./requirements.txt (line 2)) (2.11.2)
Requirement already satisfied: itsdangerous>=0.24 in /opt/conda/lib/python3.7/site-packages (from flask->bentoml==0.8.5->-r ./requirements.txt (line 2)) (1.1.0)
Requirement already satisfied: jmespath<1.0.0,>=0.7.1 in /opt/conda/lib/python3.7/site-packages (from boto3->bentoml==0.8.5->-r ./requirements.txt (line 2)) (0.10.0)
Requirement already satisfied: s3transfer<0.4.0,>=0.3.0 in /opt/conda/lib/python3.7/site-packages (from boto3->bentoml==0.8.5->-r ./requirements.txt (line 2)) (0.3.3)
Requirement already satisfied: botocore<1.18.0,>=1.17.39 in /opt/conda/lib/python3.7/site-packages (from boto3->bentoml==0.8.5->-r ./requirements.txt (line 2)) (1.17.39)
Requirement already satisfied: Mako in /opt/conda/lib/python3.7/site-packages (from alembic->bentoml==0.8.5->-r ./requirements.txt (line 2)) (1.1.3)
Requirement already satisfied: python-editor>=0.3 in /opt/conda/lib/python3.7/site-packages (from alembic->bentoml==0.8.5->-r ./requirements.txt (line 2)) (1.0.4)
Requirement already satisfied: thriftpy2>=0.4.0 in /opt/conda/lib/python3.7/site-packages (from py-zipkin->bentoml==0.8.5->-r ./requirements.txt (line 2)) (0.4.11)
Requirement already satisfied: typing-extensions>=3.7.4; python_version < "3.8" in /opt/conda/lib/python3.7/site-packages (from yarl<2.0,>=1.0->aiohttp->bentoml==0.8.5->-r ./requirements.txt (line 2)) (3.7.4.2)
Requirement already satisfied: MarkupSafe>=0.23 in /opt/conda/lib/python3.7/site-packages (from Jinja2>=2.10.1->flask->bentoml==0.8.5->-r ./requirements.txt (line 2)) (1.1.1)
Requirement already satisfied: docutils<0.16,>=0.10 in /opt/conda/lib/python3.7/site-packages (from botocore<1.18.0,>=1.17.39->boto3->bentoml==0.8.5->-r ./requirements.txt (line 2)) (0.15.2)
Requirement already satisfied: ply<4.0,>=3.4 in /opt/conda/lib/python3.7/site-packages (from thriftpy2>=0.4.0->py-zipkin->bentoml==0.8.5->-r ./requirements.txt (line 2)) (3.11)
Installing collected packages: python-dateutil
  Attempting uninstall: python-dateutil
    Found existing installation: python-dateutil 2.8.1
    Uninstalling python-dateutil-2.8.1:
      Successfully uninstalled python-dateutil-2.8.1
Successfully installed python-dateutil-2.8.0
+ for filename in ./bundled_pip_dependencies/*.tar.gz
+ '[' -e './bundled_pip_dependencies/*.tar.gz' ']'
+ continue
Removing intermediate container 5c32936f764e
 ---> 089fdb91acb3
Step 10/15 : ENV PORT 5000
 ---> Running in f2c62158d05f
Removing intermediate container f2c62158d05f
 ---> 0d33795106f8
Step 11/15 : EXPOSE $PORT
 ---> Running in b4fb56d2e1c6
Removing intermediate container b4fb56d2e1c6
 ---> df14a798ba76
Step 12/15 : COPY docker-entrypoint.sh /usr/local/bin/
 ---> 9e24b68d273d
Step 13/15 : RUN chmod +x /usr/local/bin/docker-entrypoint.sh
 ---> Running in 0ad73aa6f6f0
Removing intermediate container 0ad73aa6f6f0
 ---> c608332dd924
Step 14/15 : ENTRYPOINT [ "docker-entrypoint.sh" ]
 ---> Running in efee5b711dbf
Removing intermediate container efee5b711dbf
 ---> d055e40fe3f9
Step 15/15 : CMD ["bentoml", "serve-gunicorn", "/bento"]
 ---> Running in 3e0c629cfb1f
Removing intermediate container 3e0c629cfb1f
 ---> fa9a266c2e10
Successfully built fa9a266c2e10
Successfully tagged multi_models_titanic:latest
In [ ]:
# port forward to 7000 
!docker run -d -p 7000:5000 TitanicSurvivalPredictionService --enable-microbatch

Load saved BentoService

bentoml.load is the API for loading a BentoML packaged model in python:

In [11]:
import json

import bentoml

bento_model = bentoml.load(saved_path)

print(bento_model.predict(X_tests.to_json(orient='table')))
[2020-08-25 13:47:58,780] WARNING - Module `multiple_models_titanic_bento_service` already loaded, using existing imported module.

Deployment Options

If you are at a small team with limited engineering or DevOps resources, try out automated deployment with BentoML CLI, currently supporting AWS Lambda, AWS SageMaker, and Azure Functions:

If the cloud platform you are working with is not on the list above, try out these step-by-step guide on manually deploying BentoML packaged model to cloud platforms:

Lastly, if you have a DevOps or ML Engineering team who's operating a Kubernetes or OpenShift cluster, use the following guides as references for implementating your deployment strategy:

In [ ]: