BentoML Example: Linear Regression with Paddlepaddle

BentoML makes moving trained ML models to production easy:

  • Package models trained with any ML framework and reproduce them for model serving in production
  • Deploy anywhere for online API serving or offline batch serving
  • High-Performance API model server with adaptive micro-batching support
  • Central hub for managing models and deployment process via Web UI and APIs
  • Modular and flexible design making it adaptable to your infrastrcuture

BentoML is a framework for serving, managing, and deploying machine learning models. It is aiming to bridge the gap between Data Science and DevOps, and enable teams to deliver prediction services in a fast, repeatable, and scalable way.

Before reading this example project, be sure to check out the Getting started guide to learn about the basic concepts in BentoML.

This notebook demonstrates how to use BentoML to turn a paddlepaddle model into a docker image containing a REST API server serving this model, how to use your ML service built with BentoML as a CLI tool, and how to distribute it a pypi package.

The example is based on this tutorial, using dataset from the UCI Machine Learning Repository

In [3]:
%reload_ext autoreload
%autoreload 2
%matplotlib inline
In [4]:
!pip install bentoml

#!pip install -q bentoml 'paddlepaddle>=2.0.0' 'pandas>=1.1.1' 'numpy>=1.8.2'
!pip install paddlepaddle
Collecting git+https://github.com/bentoml/BentoML.git
  Cloning https://github.com/bentoml/BentoML.git to /private/var/folders/kn/xnc9k74x03567n1mx2tfqnpr0000gn/T/pip-req-build-y5l3v5a6
  Running command git clone -q https://github.com/bentoml/BentoML.git /private/var/folders/kn/xnc9k74x03567n1mx2tfqnpr0000gn/T/pip-req-build-y5l3v5a6
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
Requirement already satisfied: humanfriendly in /usr/local/Caskroom/miniconda/base/envs/dev/lib/python3.7/site-packages (from BentoML==0.12.0+15.g7d40998) (9.1)
Requirement already satisfied: urllib3<=1.25.11 in /usr/local/Caskroom/miniconda/base/envs/dev/lib/python3.7/site-packages (from BentoML==0.12.0+15.g7d40998) (1.25.11)
Requirement already satisfied: gunicorn in /usr/local/Caskroom/miniconda/base/envs/dev/lib/python3.7/site-packages (from BentoML==0.12.0+15.g7d40998) (20.0.4)
Requirement already satisfied: configparser in /usr/local/Caskroom/miniconda/base/envs/dev/lib/python3.7/site-packages (from BentoML==0.12.0+15.g7d40998) (5.0.2)
Requirement already satisfied: sqlalchemy<1.4.0,>=1.3.0 in /usr/local/Caskroom/miniconda/base/envs/dev/lib/python3.7/site-packages (from BentoML==0.12.0+15.g7d40998) (1.3.23)
Requirement already satisfied: python-json-logger in /usr/local/Caskroom/miniconda/base/envs/dev/lib/python3.7/site-packages (from BentoML==0.12.0+15.g7d40998) (2.0.1)
Requirement already satisfied: numpy in /usr/local/Caskroom/miniconda/base/envs/dev/lib/python3.7/site-packages (from BentoML==0.12.0+15.g7d40998) (1.20.1)
Requirement already satisfied: docker in /usr/local/Caskroom/miniconda/base/envs/dev/lib/python3.7/site-packages (from BentoML==0.12.0+15.g7d40998) (4.2.2)
Requirement already satisfied: python-dateutil<3.0.0,>=2.7.3 in /usr/local/Caskroom/miniconda/base/envs/dev/lib/python3.7/site-packages (from BentoML==0.12.0+15.g7d40998) (2.8.0)
Requirement already satisfied: ruamel.yaml>=0.15.0 in /usr/local/Caskroom/miniconda/base/envs/dev/lib/python3.7/site-packages (from BentoML==0.12.0+15.g7d40998) (0.16.13)
Requirement already satisfied: dependency-injector<5.0,>=4.0 in /usr/local/Caskroom/miniconda/base/envs/dev/lib/python3.7/site-packages (from BentoML==0.12.0+15.g7d40998) (4.29.2)
Requirement already satisfied: tabulate in /usr/local/Caskroom/miniconda/base/envs/dev/lib/python3.7/site-packages (from BentoML==0.12.0+15.g7d40998) (0.8.9)
Requirement already satisfied: flask in /usr/local/Caskroom/miniconda/base/envs/dev/lib/python3.7/site-packages (from BentoML==0.12.0+15.g7d40998) (1.0.4)
Requirement already satisfied: protobuf>=3.8.0 in /usr/local/Caskroom/miniconda/base/envs/dev/lib/python3.7/site-packages (from BentoML==0.12.0+15.g7d40998) (3.15.6)
Requirement already satisfied: chardet in /usr/local/Caskroom/miniconda/base/envs/dev/lib/python3.7/site-packages (from BentoML==0.12.0+15.g7d40998) (3.0.4)
Requirement already satisfied: cerberus in /usr/local/Caskroom/miniconda/base/envs/dev/lib/python3.7/site-packages (from BentoML==0.12.0+15.g7d40998) (1.3.2)
Requirement already satisfied: certifi in /usr/local/Caskroom/miniconda/base/envs/dev/lib/python3.7/site-packages (from BentoML==0.12.0+15.g7d40998) (2020.12.5)
Requirement already satisfied: packaging in /usr/local/Caskroom/miniconda/base/envs/dev/lib/python3.7/site-packages (from BentoML==0.12.0+15.g7d40998) (20.9)
Requirement already satisfied: alembic in /usr/local/Caskroom/miniconda/base/envs/dev/lib/python3.7/site-packages (from BentoML==0.12.0+15.g7d40998) (1.5.7)
Requirement already satisfied: click>=7.0 in /usr/local/Caskroom/miniconda/base/envs/dev/lib/python3.7/site-packages (from BentoML==0.12.0+15.g7d40998) (7.1.2)
Requirement already satisfied: requests in /usr/local/Caskroom/miniconda/base/envs/dev/lib/python3.7/site-packages (from BentoML==0.12.0+15.g7d40998) (2.22.0)
Requirement already satisfied: boto3 in /usr/local/Caskroom/miniconda/base/envs/dev/lib/python3.7/site-packages (from BentoML==0.12.0+15.g7d40998) (1.17.30)
Requirement already satisfied: prometheus-client in /usr/local/Caskroom/miniconda/base/envs/dev/lib/python3.7/site-packages (from BentoML==0.12.0+15.g7d40998) (0.9.0)
Requirement already satisfied: sqlalchemy-utils<0.36.8 in /usr/local/Caskroom/miniconda/base/envs/dev/lib/python3.7/site-packages (from BentoML==0.12.0+15.g7d40998) (0.36.7)
Requirement already satisfied: deepmerge in /usr/local/Caskroom/miniconda/base/envs/dev/lib/python3.7/site-packages (from BentoML==0.12.0+15.g7d40998) (0.2.1)
Requirement already satisfied: grpcio in /usr/local/Caskroom/miniconda/base/envs/dev/lib/python3.7/site-packages (from BentoML==0.12.0+15.g7d40998) (1.36.1)
Requirement already satisfied: schema in /usr/local/Caskroom/miniconda/base/envs/dev/lib/python3.7/site-packages (from BentoML==0.12.0+15.g7d40998) (0.7.4)
Requirement already satisfied: aiohttp in /usr/local/Caskroom/miniconda/base/envs/dev/lib/python3.7/site-packages (from BentoML==0.12.0+15.g7d40998) (3.7.4.post0)
Requirement already satisfied: psutil in /usr/local/Caskroom/miniconda/base/envs/dev/lib/python3.7/site-packages (from BentoML==0.12.0+15.g7d40998) (5.8.0)
Requirement already satisfied: six<=1.15.0,>=1.7.0 in /usr/local/Caskroom/miniconda/base/envs/dev/lib/python3.7/site-packages (from dependency-injector<5.0,>=4.0->BentoML==0.12.0+15.g7d40998) (1.15.0)
Requirement already satisfied: ruamel.yaml.clib>=0.1.2 in /usr/local/Caskroom/miniconda/base/envs/dev/lib/python3.7/site-packages (from ruamel.yaml>=0.15.0->BentoML==0.12.0+15.g7d40998) (0.2.2)
Requirement already satisfied: attrs>=17.3.0 in /usr/local/Caskroom/miniconda/base/envs/dev/lib/python3.7/site-packages (from aiohttp->BentoML==0.12.0+15.g7d40998) (20.3.0)
Requirement already satisfied: yarl<2.0,>=1.0 in /usr/local/Caskroom/miniconda/base/envs/dev/lib/python3.7/site-packages (from aiohttp->BentoML==0.12.0+15.g7d40998) (1.6.3)
Requirement already satisfied: multidict<7.0,>=4.5 in /usr/local/Caskroom/miniconda/base/envs/dev/lib/python3.7/site-packages (from aiohttp->BentoML==0.12.0+15.g7d40998) (5.1.0)
Requirement already satisfied: typing-extensions>=3.6.5 in /usr/local/Caskroom/miniconda/base/envs/dev/lib/python3.7/site-packages (from aiohttp->BentoML==0.12.0+15.g7d40998) (3.7.4.3)
Requirement already satisfied: async-timeout<4.0,>=3.0 in /usr/local/Caskroom/miniconda/base/envs/dev/lib/python3.7/site-packages (from aiohttp->BentoML==0.12.0+15.g7d40998) (3.0.1)
Requirement already satisfied: idna>=2.0 in /usr/local/Caskroom/miniconda/base/envs/dev/lib/python3.7/site-packages (from yarl<2.0,>=1.0->aiohttp->BentoML==0.12.0+15.g7d40998) (2.8)
Requirement already satisfied: python-editor>=0.3 in /usr/local/Caskroom/miniconda/base/envs/dev/lib/python3.7/site-packages (from alembic->BentoML==0.12.0+15.g7d40998) (1.0.4)
Requirement already satisfied: Mako in /usr/local/Caskroom/miniconda/base/envs/dev/lib/python3.7/site-packages (from alembic->BentoML==0.12.0+15.g7d40998) (1.1.4)
Requirement already satisfied: s3transfer<0.4.0,>=0.3.0 in /usr/local/Caskroom/miniconda/base/envs/dev/lib/python3.7/site-packages (from boto3->BentoML==0.12.0+15.g7d40998) (0.3.4)
Requirement already satisfied: botocore<1.21.0,>=1.20.30 in /usr/local/Caskroom/miniconda/base/envs/dev/lib/python3.7/site-packages (from boto3->BentoML==0.12.0+15.g7d40998) (1.20.30)
Requirement already satisfied: jmespath<1.0.0,>=0.7.1 in /usr/local/Caskroom/miniconda/base/envs/dev/lib/python3.7/site-packages (from boto3->BentoML==0.12.0+15.g7d40998) (0.9.5)
Requirement already satisfied: setuptools in /usr/local/Caskroom/miniconda/base/envs/dev/lib/python3.7/site-packages (from cerberus->BentoML==0.12.0+15.g7d40998) (49.6.0.post20210108)
Requirement already satisfied: websocket-client>=0.32.0 in /usr/local/Caskroom/miniconda/base/envs/dev/lib/python3.7/site-packages (from docker->BentoML==0.12.0+15.g7d40998) (0.58.0)
Requirement already satisfied: Jinja2>=2.10 in /usr/local/Caskroom/miniconda/base/envs/dev/lib/python3.7/site-packages (from flask->BentoML==0.12.0+15.g7d40998) (2.11.3)
Requirement already satisfied: Werkzeug>=0.14 in /usr/local/Caskroom/miniconda/base/envs/dev/lib/python3.7/site-packages (from flask->BentoML==0.12.0+15.g7d40998) (1.0.1)
Requirement already satisfied: itsdangerous>=0.24 in /usr/local/Caskroom/miniconda/base/envs/dev/lib/python3.7/site-packages (from flask->BentoML==0.12.0+15.g7d40998) (1.1.0)
Requirement already satisfied: MarkupSafe>=0.23 in /usr/local/Caskroom/miniconda/base/envs/dev/lib/python3.7/site-packages (from Jinja2>=2.10->flask->BentoML==0.12.0+15.g7d40998) (1.1.1)
Requirement already satisfied: pyparsing>=2.0.2 in /usr/local/Caskroom/miniconda/base/envs/dev/lib/python3.7/site-packages (from packaging->BentoML==0.12.0+15.g7d40998) (2.4.7)
Requirement already satisfied: contextlib2>=0.5.5 in /usr/local/Caskroom/miniconda/base/envs/dev/lib/python3.7/site-packages (from schema->BentoML==0.12.0+15.g7d40998) (0.6.0.post1)
Building wheels for collected packages: BentoML
  Building wheel for BentoML (PEP 517) ... done
  Created wheel for BentoML: filename=BentoML-0.12.0+15.g7d40998-py3-none-any.whl size=674023 sha256=5bd65f7f3eadd7e78aad7142a77ecf2619efe6b6147f1fc8fb0fb7be343a17d5
  Stored in directory: /private/var/folders/kn/xnc9k74x03567n1mx2tfqnpr0000gn/T/pip-ephem-wheel-cache-taoztxg4/wheels/dd/93/68/1a11f763a858794f96abf3c1def8b101125e236de566ebeb25
Successfully built BentoML
Installing collected packages: BentoML
  Attempting uninstall: BentoML
    Found existing installation: BentoML 0.12.0+14.gb6fe25cf
    Uninstalling BentoML-0.12.0+14.gb6fe25cf:
      Successfully uninstalled BentoML-0.12.0+14.gb6fe25cf
Successfully installed BentoML-0.12.0+15.g7d40998
Collecting paddlepaddle
  Downloading paddlepaddle-2.0.1-cp37-cp37m-macosx_10_6_intel.whl (74.9 MB)
     |████████████████████████████████| 74.9 MB 99 kB/s s eta 0:00:01
Requirement already satisfied: numpy>=1.13 in /usr/local/Caskroom/miniconda/base/envs/dev/lib/python3.7/site-packages (from paddlepaddle) (1.20.1)
Requirement already satisfied: Pillow in /usr/local/Caskroom/miniconda/base/envs/dev/lib/python3.7/site-packages (from paddlepaddle) (8.1.2)
Collecting gast>=0.3.3
  Downloading gast-0.4.0-py3-none-any.whl (9.8 kB)
Collecting astor
  Downloading astor-0.8.1-py2.py3-none-any.whl (27 kB)
Requirement already satisfied: six in /usr/local/Caskroom/miniconda/base/envs/dev/lib/python3.7/site-packages (from paddlepaddle) (1.15.0)
Requirement already satisfied: requests>=2.20.0 in /usr/local/Caskroom/miniconda/base/envs/dev/lib/python3.7/site-packages (from paddlepaddle) (2.22.0)
Requirement already satisfied: decorator in /usr/local/Caskroom/miniconda/base/envs/dev/lib/python3.7/site-packages (from paddlepaddle) (5.0.3)
Requirement already satisfied: protobuf>=3.1.0 in /usr/local/Caskroom/miniconda/base/envs/dev/lib/python3.7/site-packages (from paddlepaddle) (3.15.6)
Requirement already satisfied: idna<2.9,>=2.5 in /usr/local/Caskroom/miniconda/base/envs/dev/lib/python3.7/site-packages (from requests>=2.20.0->paddlepaddle) (2.8)
Requirement already satisfied: certifi>=2017.4.17 in /usr/local/Caskroom/miniconda/base/envs/dev/lib/python3.7/site-packages (from requests>=2.20.0->paddlepaddle) (2020.12.5)
Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/Caskroom/miniconda/base/envs/dev/lib/python3.7/site-packages (from requests>=2.20.0->paddlepaddle) (1.25.11)
Requirement already satisfied: chardet<3.1.0,>=3.0.2 in /usr/local/Caskroom/miniconda/base/envs/dev/lib/python3.7/site-packages (from requests>=2.20.0->paddlepaddle) (3.0.4)
Installing collected packages: gast, astor, paddlepaddle
Successfully installed astor-0.8.1 gast-0.4.0 paddlepaddle-2.0.1
In [6]:
import numpy as np
import paddle
import paddle.nn as nn
import paddle.optimizer as opt
import pandas as pd

import bentoml
/usr/local/Caskroom/miniconda/base/envs/dev/lib/python3.7/site-packages/ipykernel/ipkernel.py:283: DeprecationWarning: `should_run_async` will not call `transform_cell` automatically in the future. Please pass the result to `transformed_cell` argument and any exception that happen during thetransform in `preprocessing_exc_tuple` in IPython 7.17 and above.
  and should_run_async(code)

Prepare Dataset

In [7]:
import paddle
import numpy as np
import bentoml
from paddle.static import InputSpec

BATCH_SIZE = 8
BATCH_NUM = 4
EPOCH_NUM = 5

IN_FEATURES = 13
OUT_FEATURES = 1

class LinearNet(paddle.nn.Layer):
    def __init__(self):
        super(LinearNet, self).__init__()
        self._linear = paddle.nn.Linear(IN_FEATURES, OUT_FEATURES)

    @paddle.jit.to_static(input_spec=[InputSpec(shape=[IN_FEATURES], dtype='float32')])
    def forward(self, x):
        return self._linear(x)

    def train(self, loader, loss_fn, opt):
        for epoch_id in range(EPOCH_NUM):
            for batch_id, (image, label) in enumerate(loader()):
                out = self._linear(image)
                loss = loss_fn(out, label)
                loss.backward()
                opt.step()
                opt.clear_grad()
                print("Epoch {} batch {}: loss = {}".format(
                    epoch_id, batch_id, np.mean(loss.numpy())))
/usr/local/Caskroom/miniconda/base/envs/dev/lib/python3.7/site-packages/paddle/fluid/layers/utils.py:77: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated since Python 3.3,and in 3.9 it will stop working
  return (isinstance(seq, collections.Sequence) and

Model Training

In [8]:
model = LinearNet()
loss_fn = paddle.nn.MSELoss()
adam = paddle.optimizer.Adam(parameters=model.parameters())

dataset = paddle.text.datasets.UCIHousing(mode="train")

loader = paddle.io.DataLoader(dataset,
  batch_size=BATCH_SIZE,
  shuffle=True,
  drop_last=True,
  num_workers=2)

model.train(loader, loss_fn, adam)
Cache file /Users/bozhaoyu/.cache/paddle/dataset/uci_housing/housing.data not found, downloading http://paddlemodels.bj.bcebos.com/uci_housing/housing.data 
Begin to download
............
Download finished
/usr/local/Caskroom/miniconda/base/envs/dev/lib/python3.7/site-packages/paddle/fluid/reader.py:352: UserWarning: DataLoader with multi-process mode is not supported on MacOs and Windows currently. Please use signle-process mode with num_workers = 0 instead
  "DataLoader with multi-process mode is not supported on MacOs and Windows currently." \
/usr/local/Caskroom/miniconda/base/envs/dev/lib/python3.7/site-packages/paddle/fluid/dataloader/dataloader_iter.py:89: DeprecationWarning: `np.bool` is a deprecated alias for the builtin `bool`. To silence this warning, use `bool` by itself. Doing this will not modify any behavior and is safe. If you specifically wanted the numpy scalar type, use `np.bool_` here.
Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations
  if isinstance(slot[0], (np.ndarray, np.bool, numbers.Number)):
Epoch 0 batch 0: loss = 435.19635009765625
Epoch 0 batch 1: loss = 421.4579772949219
Epoch 0 batch 2: loss = 459.74871826171875
Epoch 0 batch 3: loss = 680.4847412109375
Epoch 0 batch 4: loss = 858.70458984375
Epoch 0 batch 5: loss = 536.1829833984375
Epoch 0 batch 6: loss = 555.8958740234375
Epoch 0 batch 7: loss = 388.1265563964844
Epoch 0 batch 8: loss = 647.7759399414062
Epoch 0 batch 9: loss = 494.9188232421875
Epoch 0 batch 10: loss = 970.626220703125
Epoch 0 batch 11: loss = 659.16064453125
Epoch 0 batch 12: loss = 983.6190185546875
Epoch 0 batch 13: loss = 936.4552612304688
Epoch 0 batch 14: loss = 1019.9678344726562
Epoch 0 batch 15: loss = 747.4344482421875
Epoch 0 batch 16: loss = 831.9373168945312
Epoch 0 batch 17: loss = 350.56976318359375
Epoch 0 batch 18: loss = 515.817626953125
Epoch 0 batch 19: loss = 713.2019653320312
Epoch 0 batch 20: loss = 610.8594360351562
Epoch 0 batch 21: loss = 594.1142578125
Epoch 0 batch 22: loss = 471.9237060546875
Epoch 0 batch 23: loss = 1203.9696044921875
Epoch 0 batch 24: loss = 600.8118286132812
Epoch 0 batch 25: loss = 372.2288818359375
Epoch 0 batch 26: loss = 596.131103515625
Epoch 0 batch 27: loss = 712.577392578125
Epoch 0 batch 28: loss = 823.0017700195312
Epoch 0 batch 29: loss = 838.0972290039062
Epoch 0 batch 30: loss = 755.33935546875
Epoch 0 batch 31: loss = 414.0630798339844
Epoch 0 batch 32: loss = 482.1142883300781
Epoch 0 batch 33: loss = 559.952880859375
Epoch 0 batch 34: loss = 896.4400634765625
Epoch 0 batch 35: loss = 1178.97607421875
Epoch 0 batch 36: loss = 344.52386474609375
Epoch 0 batch 37: loss = 696.4715576171875
Epoch 0 batch 38: loss = 587.26611328125
Epoch 0 batch 39: loss = 524.2654418945312
Epoch 0 batch 40: loss = 719.9611206054688
Epoch 0 batch 41: loss = 883.761962890625
Epoch 0 batch 42: loss = 605.924560546875
Epoch 0 batch 43: loss = 790.5938720703125
Epoch 0 batch 44: loss = 860.56396484375
Epoch 0 batch 45: loss = 1057.826904296875
Epoch 0 batch 46: loss = 558.65869140625
Epoch 0 batch 47: loss = 657.4581909179688
Epoch 0 batch 48: loss = 440.3964538574219
Epoch 0 batch 49: loss = 741.140625
Epoch 1 batch 0: loss = 503.2971496582031
Epoch 1 batch 1: loss = 574.934326171875
Epoch 1 batch 2: loss = 791.82470703125
Epoch 1 batch 3: loss = 561.6544189453125
Epoch 1 batch 4: loss = 576.545166015625
Epoch 1 batch 5: loss = 824.7929077148438
Epoch 1 batch 6: loss = 514.4303588867188
Epoch 1 batch 7: loss = 480.4468688964844
Epoch 1 batch 8: loss = 910.288818359375
Epoch 1 batch 9: loss = 428.64263916015625
Epoch 1 batch 10: loss = 812.5103759765625
Epoch 1 batch 11: loss = 1199.470703125
Epoch 1 batch 12: loss = 811.8062744140625
Epoch 1 batch 13: loss = 474.69183349609375
Epoch 1 batch 14: loss = 431.4730224609375
Epoch 1 batch 15: loss = 681.200927734375
Epoch 1 batch 16: loss = 543.3580322265625
Epoch 1 batch 17: loss = 643.1007080078125
Epoch 1 batch 18: loss = 621.5284423828125
Epoch 1 batch 19: loss = 760.8117065429688
Epoch 1 batch 20: loss = 499.61346435546875
Epoch 1 batch 21: loss = 910.94970703125
Epoch 1 batch 22: loss = 549.989990234375
Epoch 1 batch 23: loss = 555.82763671875
Epoch 1 batch 24: loss = 453.82476806640625
Epoch 1 batch 25: loss = 445.43994140625
Epoch 1 batch 26: loss = 722.4783325195312
Epoch 1 batch 27: loss = 770.901611328125
Epoch 1 batch 28: loss = 782.9249267578125
Epoch 1 batch 29: loss = 635.4414672851562
Epoch 1 batch 30: loss = 806.8521118164062
Epoch 1 batch 31: loss = 575.8161010742188
Epoch 1 batch 32: loss = 601.47265625
Epoch 1 batch 33: loss = 853.6051025390625
Epoch 1 batch 34: loss = 943.209716796875
Epoch 1 batch 35: loss = 755.8599853515625
Epoch 1 batch 36: loss = 850.079833984375
Epoch 1 batch 37: loss = 812.576416015625
Epoch 1 batch 38: loss = 578.6965942382812
Epoch 1 batch 39: loss = 755.2635498046875
Epoch 1 batch 40: loss = 615.0452880859375
Epoch 1 batch 41: loss = 514.7759399414062
Epoch 1 batch 42: loss = 718.6151733398438
Epoch 1 batch 43: loss = 710.81787109375
Epoch 1 batch 44: loss = 1070.00439453125
Epoch 1 batch 45: loss = 599.828125
Epoch 1 batch 46: loss = 420.0142822265625
Epoch 1 batch 47: loss = 317.08905029296875
Epoch 1 batch 48: loss = 772.1884765625
Epoch 1 batch 49: loss = 581.118896484375
Epoch 2 batch 0: loss = 862.5804443359375
Epoch 2 batch 1: loss = 462.8978271484375
Epoch 2 batch 2: loss = 652.2523803710938
Epoch 2 batch 3: loss = 1217.7396240234375
Epoch 2 batch 4: loss = 628.7020874023438
Epoch 2 batch 5: loss = 531.6495361328125
Epoch 2 batch 6: loss = 330.69866943359375
Epoch 2 batch 7: loss = 770.7513427734375
Epoch 2 batch 8: loss = 401.77020263671875
Epoch 2 batch 9: loss = 863.222412109375
Epoch 2 batch 10: loss = 533.8890380859375
Epoch 2 batch 11: loss = 506.18646240234375
Epoch 2 batch 12: loss = 888.6613159179688
Epoch 2 batch 13: loss = 543.69384765625
Epoch 2 batch 14: loss = 535.5054321289062
Epoch 2 batch 15: loss = 403.19146728515625
Epoch 2 batch 16: loss = 489.041259765625
Epoch 2 batch 17: loss = 581.9658813476562
Epoch 2 batch 18: loss = 388.47894287109375
Epoch 2 batch 19: loss = 1026.50341796875
Epoch 2 batch 20: loss = 588.2946166992188
Epoch 2 batch 21: loss = 901.6898193359375
Epoch 2 batch 22: loss = 576.9415283203125
Epoch 2 batch 23: loss = 816.2774658203125
Epoch 2 batch 24: loss = 671.8216552734375
Epoch 2 batch 25: loss = 806.2049560546875
Epoch 2 batch 26: loss = 747.4718627929688
Epoch 2 batch 27: loss = 565.2826538085938
Epoch 2 batch 28: loss = 653.98193359375
Epoch 2 batch 29: loss = 611.644775390625
Epoch 2 batch 30: loss = 852.8026123046875
Epoch 2 batch 31: loss = 428.0990295410156
Epoch 2 batch 32: loss = 941.274658203125
Epoch 2 batch 33: loss = 452.7620849609375
Epoch 2 batch 34: loss = 897.3431396484375
Epoch 2 batch 35: loss = 737.9261474609375
Epoch 2 batch 36: loss = 919.672607421875
Epoch 2 batch 37: loss = 688.052001953125
Epoch 2 batch 38: loss = 483.1783142089844
Epoch 2 batch 39: loss = 623.8746337890625
Epoch 2 batch 40: loss = 789.4937744140625
Epoch 2 batch 41: loss = 497.65966796875
Epoch 2 batch 42: loss = 529.1936645507812
Epoch 2 batch 43: loss = 874.4013671875
Epoch 2 batch 44: loss = 884.49658203125
Epoch 2 batch 45: loss = 842.5672607421875
Epoch 2 batch 46: loss = 396.98028564453125
Epoch 2 batch 47: loss = 460.8985595703125
Epoch 2 batch 48: loss = 816.37158203125
Epoch 2 batch 49: loss = 712.0831909179688
Epoch 3 batch 0: loss = 649.838134765625
Epoch 3 batch 1: loss = 739.3240356445312
Epoch 3 batch 2: loss = 652.0592041015625
Epoch 3 batch 3: loss = 595.1077880859375
Epoch 3 batch 4: loss = 672.396484375
Epoch 3 batch 5: loss = 769.217041015625
Epoch 3 batch 6: loss = 376.50384521484375
Epoch 3 batch 7: loss = 565.156005859375
Epoch 3 batch 8: loss = 555.4974975585938
Epoch 3 batch 9: loss = 1074.7545166015625
Epoch 3 batch 10: loss = 737.5693969726562
Epoch 3 batch 11: loss = 605.9752197265625
Epoch 3 batch 12: loss = 702.8925170898438
Epoch 3 batch 13: loss = 432.9101257324219
Epoch 3 batch 14: loss = 935.0133056640625
Epoch 3 batch 15: loss = 363.78936767578125
Epoch 3 batch 16: loss = 853.4984741210938
Epoch 3 batch 17: loss = 491.4356384277344
Epoch 3 batch 18: loss = 773.9022216796875
Epoch 3 batch 19: loss = 642.4447021484375
Epoch 3 batch 20: loss = 972.908447265625
Epoch 3 batch 21: loss = 499.309814453125
Epoch 3 batch 22: loss = 643.506103515625
Epoch 3 batch 23: loss = 564.5712280273438
Epoch 3 batch 24: loss = 442.2332763671875
Epoch 3 batch 25: loss = 366.79266357421875
Epoch 3 batch 26: loss = 1001.761474609375
Epoch 3 batch 27: loss = 800.8330078125
Epoch 3 batch 28: loss = 621.2838134765625
Epoch 3 batch 29: loss = 536.276123046875
Epoch 3 batch 30: loss = 516.718505859375
Epoch 3 batch 31: loss = 743.0946655273438
Epoch 3 batch 32: loss = 596.0198974609375
Epoch 3 batch 33: loss = 513.815185546875
Epoch 3 batch 34: loss = 580.443115234375
Epoch 3 batch 35: loss = 885.9213256835938
Epoch 3 batch 36: loss = 462.1646423339844
Epoch 3 batch 37: loss = 764.1090087890625
Epoch 3 batch 38: loss = 836.8423461914062
Epoch 3 batch 39: loss = 641.364501953125
Epoch 3 batch 40: loss = 1067.87646484375
Epoch 3 batch 41: loss = 639.5401611328125
Epoch 3 batch 42: loss = 734.51123046875
Epoch 3 batch 43: loss = 569.3967895507812
Epoch 3 batch 44: loss = 774.8538818359375
Epoch 3 batch 45: loss = 514.9826049804688
Epoch 3 batch 46: loss = 685.5824584960938
Epoch 3 batch 47: loss = 564.9667358398438
Epoch 3 batch 48: loss = 800.9647827148438
Epoch 3 batch 49: loss = 586.7379150390625
Epoch 4 batch 0: loss = 485.76617431640625
Epoch 4 batch 1: loss = 458.2221984863281
Epoch 4 batch 2: loss = 448.7731018066406
Epoch 4 batch 3: loss = 943.52734375
Epoch 4 batch 4: loss = 565.762451171875
Epoch 4 batch 5: loss = 665.5576171875
Epoch 4 batch 6: loss = 672.037841796875
Epoch 4 batch 7: loss = 765.3156127929688
Epoch 4 batch 8: loss = 509.41900634765625
Epoch 4 batch 9: loss = 742.4193725585938
Epoch 4 batch 10: loss = 423.85565185546875
Epoch 4 batch 11: loss = 700.9935913085938
Epoch 4 batch 12: loss = 531.791748046875
Epoch 4 batch 13: loss = 544.2742309570312
Epoch 4 batch 14: loss = 847.1261596679688
Epoch 4 batch 15: loss = 605.2373046875
Epoch 4 batch 16: loss = 652.822509765625
Epoch 4 batch 17: loss = 852.6063232421875
Epoch 4 batch 18: loss = 898.5792846679688
Epoch 4 batch 19: loss = 684.3765258789062
Epoch 4 batch 20: loss = 595.689208984375
Epoch 4 batch 21: loss = 780.4011840820312
Epoch 4 batch 22: loss = 745.6404418945312
Epoch 4 batch 23: loss = 821.27099609375
Epoch 4 batch 24: loss = 729.1907958984375
Epoch 4 batch 25: loss = 726.6057739257812
Epoch 4 batch 26: loss = 653.0880126953125
Epoch 4 batch 27: loss = 551.7437744140625
Epoch 4 batch 28: loss = 525.1512451171875
Epoch 4 batch 29: loss = 748.1895751953125
Epoch 4 batch 30: loss = 487.51812744140625
Epoch 4 batch 31: loss = 626.6509399414062
Epoch 4 batch 32: loss = 609.98486328125
Epoch 4 batch 33: loss = 594.4813232421875
Epoch 4 batch 34: loss = 422.2345886230469
Epoch 4 batch 35: loss = 577.8804321289062
Epoch 4 batch 36: loss = 369.119384765625
Epoch 4 batch 37: loss = 1091.2344970703125
Epoch 4 batch 38: loss = 759.0220336914062
Epoch 4 batch 39: loss = 716.1146240234375
Epoch 4 batch 40: loss = 878.747314453125
Epoch 4 batch 41: loss = 474.2106628417969
Epoch 4 batch 42: loss = 398.3009338378906
Epoch 4 batch 43: loss = 893.1238403320312
Epoch 4 batch 44: loss = 504.133056640625
Epoch 4 batch 45: loss = 743.451904296875
Epoch 4 batch 46: loss = 731.81494140625
Epoch 4 batch 47: loss = 731.53369140625
Epoch 4 batch 48: loss = 713.1758422851562
Epoch 4 batch 49: loss = 680.8133544921875
In [9]:
 test_x = np.array([[-0.0405441 ,  0.06636364, -0.32356227, -0.06916996, -0.03435197,
        0.05563625, -0.03475696,  0.02682186, -0.37171335, -0.21419304,
       -0.33569506,  0.10143217, -0.21172912]]).astype('float32')

df_test_x = pd.DataFrame(test_x)
In [10]:
import csv

with open('test.csv', 'w', newline='') as csvfile:
    spamwriter = csv.writer(csvfile, delimiter=',', quoting=csv.QUOTE_MINIMAL)
    spamwriter.writerow(df_test_x.columns)
    spamwriter.writerow([-0.0405441 ,  0.06636364, -0.32356227, -0.06916996, -0.03435197,
        0.05563625, -0.03475696,  0.02682186, -0.37171335, -0.21419304,
       -0.33569506,  0.10143217, -0.21172912])

Create BentoService for model serving

In [11]:
%%writefile paddle_linear_regression.py
import pandas as pd
import numpy as np

import bentoml
from bentoml import env, artifacts, api, BentoService
from bentoml.adapters import DataframeInput
from bentoml.frameworks.paddle import PaddlePaddleModelArtifact

@env(infer_pip_packages=True)
@artifacts([PaddlePaddleModelArtifact('model')])
class PaddleLinearRegression(bentoml.BentoService):

  @api(input=DataframeInput(), batch=True)
  def predict(self, df: pd.DataFrame):
        input_data = df.to_numpy().astype('float32')

        predictor = self.artifacts.model
        input_names = predictor.get_input_names()
        input_handle = predictor.get_input_handle(input_names[0])

        input_handle.reshape(input_data.shape)
        input_handle.copy_from_cpu(input_data)

        predictor.run()

        output_names = predictor.get_output_names()
        output_handle = predictor.get_output_handle(output_names[0])
        output_data = output_handle.copy_to_cpu()

        return output_data
Writing paddle_linear_regression.py
In [12]:
# 1) import the custom BentoService defined above
from paddle_linear_regression import PaddleLinearRegression

# 2) `pack` it with required artifacts
bento_svc = PaddleLinearRegression()
bento_svc.pack('model', model)

# 3) save your BentoSerivce
saved_path = bento_svc.save()
[2021-04-04 19:01:27,215] WARNING - Using BentoML not from official PyPI release. In order to find the same version of BentoML when deploying your BentoService, you must set the 'core/bentoml_deploy_version' config to a http/git location of your BentoML fork, e.g.: 'bentoml_deploy_version = git+https://github.com/{username}/[email protected]{branch}'
2021-04-04 19:01:28,269 - INFO - Context impl SQLiteImpl.
2021-04-04 19:01:28,270 - INFO - Will assume non-transactional DDL.
[2021-04-04 19:01:28,740] WARNING - Saved BentoService bundle version mismatch: loading BentoService bundle create with BentoML version 0.12.0, but loading from BentoML version 0.12.0+15.g7d40998
[2021-04-04 19:01:28,779] INFO - BentoService bundle 'PaddleLinearRegression:20210404190128_F4577F' saved to: /Users/bozhaoyu/bentoml/repository/PaddleLinearRegression/20210404190128_F4577F

REST API Model Serving

In [ ]:
!bentoml serve PaddleLinearRegression:latest
/usr/local/lib/python3.7/dist-packages/psycopg2/__init__.py:144: UserWarning: The psycopg2 wheel package will be renamed from release 2.8; in order to keep installing from binary please use "pip install psycopg2-binary" instead. For details see: <http://initd.org/psycopg/docs/install.html#binary-install-from-pypi>.
  """)
[2021-04-01 21:29:16,663] INFO - Getting latest version PaddleLinearRegression:20210401212846_F88F15
[2021-04-01 21:29:16,683] INFO - Starting BentoML API proxy in development mode..
[2021-04-01 21:29:16,685] INFO - Starting BentoML API server in development mode..
[2021-04-01 21:29:16,890] WARNING - Using BentoML not from official PyPI release. In order to find the same version of BentoML when deploying your BentoService, you must set the 'core/bentoml_deploy_version' config to a http/git location of your BentoML fork, e.g.: 'bentoml_deploy_version = git+https://github.com/{username}/[email protected]{branch}'
[2021-04-01 21:29:16,911] WARNING - Saved BentoService bundle version mismatch: loading BentoService bundle create with BentoML version 0.12.0, but loading from BentoML version 0.12.0+14.g0fac537
[2021-04-01 21:29:16,914] INFO - Your system nofile limit is 1048576, which means each instance of microbatch service is able to hold this number of connections at same time. You can increase the number of file descriptors for the server process, or launch more microbatch instances to accept more concurrent connection.
======== Running on http://0.0.0.0:5000 ========
(Press CTRL+C to quit)
[2021-04-01 21:29:17,386] WARNING - Using BentoML not from official PyPI release. In order to find the same version of BentoML when deploying your BentoService, you must set the 'core/bentoml_deploy_version' config to a http/git location of your BentoML fork, e.g.: 'bentoml_deploy_version = git+https://github.com/{username}/[email protected]{branch}'
[2021-04-01 21:29:17,405] WARNING - Saved BentoService bundle version mismatch: loading BentoService bundle create with BentoML version 0.12.0, but loading from BentoML version 0.12.0+14.g0fac537
/usr/local/lib/python3.7/dist-packages/paddle/fluid/backward.py:1640: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated since Python 3.3,and in 3.9 it will stop working
  return list(x) if isinstance(x, collections.Sequence) else [x]
 * Serving Flask app "PaddleLinearRegression" (lazy loading)
 * Environment: production
   WARNING: This is a development server. Do not use it in a production deployment.
   Use a production WSGI server instead.
 * Debug mode: off
2021-04-01 21:29:19,971 - INFO -  * Running on http://127.0.0.1:50135/ (Press CTRL+C to quit)


--------------------------------------
C++ Traceback (most recent call last):
--------------------------------------
0   bvar::detail::SamplerCollector::sampling_thread(void*)
1   bvar::detail::SamplerCollector::run()
2   paddle::framework::SignalHandle(char const*, int)
3   paddle::platform::GetCurrentTraceBackString[abi:cxx11]()

----------------------
Error Message Summary:
----------------------
FatalError: `Termination signal` is detected by the operating system.
  [TimeInfo: *** Aborted at 1617312560 (unix time) try "date -d @1617312560" if you are using GNU date ***]
  [SignalInfo: *** SIGTERM (@0x325) received by PID 808 (TID 0x7f191ce54700) from PID 805 ***]

If you are running this notebook from Google Colab, you can start the dev server with --run-with-ngrok option, to gain acccess to the API endpoint via a public endpoint managed by ngrok:

In [ ]:
!bentoml serve PaddleLinearRegression:latest --run-with-ngrok
/usr/local/lib/python3.7/dist-packages/psycopg2/__init__.py:144: UserWarning: The psycopg2 wheel package will be renamed from release 2.8; in order to keep installing from binary please use "pip install psycopg2-binary" instead. For details see: <http://initd.org/psycopg/docs/install.html#binary-install-from-pypi>.
  """)
[2021-04-01 21:30:36,588] INFO - Getting latest version PaddleLinearRegression:20210401212846_F88F15
[2021-04-01 21:30:36,609] INFO - Starting BentoML API proxy in development mode..
[2021-04-01 21:30:36,611] INFO - Starting BentoML API server in development mode..
[2021-04-01 21:30:36,731] WARNING - Using BentoML not from official PyPI release. In order to find the same version of BentoML when deploying your BentoService, you must set the 'core/bentoml_deploy_version' config to a http/git location of your BentoML fork, e.g.: 'bentoml_deploy_version = git+https://github.com/{username}/[email protected]{branch}'
[2021-04-01 21:30:36,750] WARNING - Saved BentoService bundle version mismatch: loading BentoService bundle create with BentoML version 0.12.0, but loading from BentoML version 0.12.0+14.g0fac537
[2021-04-01 21:30:36,751] INFO - Your system nofile limit is 1048576, which means each instance of microbatch service is able to hold this number of connections at same time. You can increase the number of file descriptors for the server process, or launch more microbatch instances to accept more concurrent connection.
======== Running on http://0.0.0.0:5000 ========
(Press CTRL+C to quit)
[2021-04-01 21:30:37,292] WARNING - Using BentoML not from official PyPI release. In order to find the same version of BentoML when deploying your BentoService, you must set the 'core/bentoml_deploy_version' config to a http/git location of your BentoML fork, e.g.: 'bentoml_deploy_version = git+https://github.com/{username}/[email protected]{branch}'
[2021-04-01 21:30:37,309] WARNING - Saved BentoService bundle version mismatch: loading BentoService bundle create with BentoML version 0.12.0, but loading from BentoML version 0.12.0+14.g0fac537
[2021-04-01 21:30:38,624] INFO -  * Running on http://6a59231a38e1.ngrok.io
[2021-04-01 21:30:38,624] INFO -  * Traffic stats available on http://127.0.0.1:4040
/usr/local/lib/python3.7/dist-packages/paddle/fluid/backward.py:1640: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated since Python 3.3,and in 3.9 it will stop working
  return list(x) if isinstance(x, collections.Sequence) else [x]
 * Serving Flask app "PaddleLinearRegression" (lazy loading)
 * Environment: production
   WARNING: This is a development server. Do not use it in a production deployment.
   Use a production WSGI server instead.
 * Debug mode: off
2021-04-01 21:30:39,718 - INFO -  * Running on http://127.0.0.1:36783/ (Press CTRL+C to quit)

Make request to the REST server

After navigating to the location of this notebook, copy and paste the following code to your terminal and run it to make request

In [ ]:
curl -i \
--request POST \
--header "Content-Type: text/csv" \
-d @test.csv \
localhost:5000/predict

Containerize model server with Docker

One common way of distributing this model API server for production deployment, is via Docker containers. And BentoML provides a convenient way to do that.

Note that docker is not available in Google Colab. You will need to download and run this notebook locally to try out this containerization with docker feature.

If you already have docker configured, simply run the follow command to product a docker container serving the PaddleLinearRegression prediction service created above:

In [ ]:
!bentoml containerize PaddleLinearRegression:latest
In [ ]:
!docker run --rm -p 5000:5000 PaddleLinearRegression:20210306050051_766D0A
/bin/bash: docker: command not found

Load Saved Bento Service

In [ ]:
#TODO ADD INPUT
from bentoml import load

svc = load(saved_path)

input = pd.DataFrame([[-0.0405441 ,  0.06636364, -0.32356227, -0.06916996, -0.03435197,
        0.05563625, -0.03475696,  0.02682186, -0.37171335, -0.21419304,
       -0.33569506,  0.10143217, -0.21172912]]).astype('float32')

print(svc.predict(input))
[2021-03-06 05:05:25,666] WARNING - Saved BentoService bundle version mismatch: loading BentoService bundle create with BentoML version 0.7.8, but loading from BentoML version 0.7.8+402.g8c47823
[2021-03-06 05:05:25,669] WARNING - Module `paddle_linear_regression` already loaded, using existing imported module.
[2021-03-06 05:05:25,680] WARNING - pip package requirement pandas already exist
[2021-03-06 05:05:25,681] WARNING - pip package requirement paddlepaddle already exist
[[0.85793805]]

Launch inference job from CLI

In [ ]:
!bentoml run PaddleLinearRegression:latest predict --format csv --input-file test.csv
/usr/local/lib/python3.7/dist-packages/psycopg2/__init__.py:144: UserWarning: The psycopg2 wheel package will be renamed from release 2.8; in order to keep installing from binary please use "pip install psycopg2-binary" instead. For details see: <http://initd.org/psycopg/docs/install.html#binary-install-from-pypi>.
  """)
[2021-04-01 21:29:58,275] INFO - Getting latest version PaddleLinearRegression:20210401212846_F88F15
[2021-04-01 21:29:58,962] WARNING - Using BentoML not from official PyPI release. In order to find the same version of BentoML when deploying your BentoService, you must set the 'core/bentoml_deploy_version' config to a http/git location of your BentoML fork, e.g.: 'bentoml_deploy_version = git+https://github.com/{username}/[email protected]{branch}'
[2021-04-01 21:29:58,978] WARNING - Saved BentoService bundle version mismatch: loading BentoService bundle create with BentoML version 0.12.0, but loading from BentoML version 0.12.0+14.g0fac537
/usr/local/lib/python3.7/dist-packages/paddle/fluid/backward.py:1640: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated since Python 3.3,and in 3.9 it will stop working
  return list(x) if isinstance(x, collections.Sequence) else [x]
I0401 21:30:01.203775   849 analysis_predictor.cc:155] Profiler is deactivated, and no profiling report will be generated.
--- Running analysis [ir_graph_build_pass]
--- Running analysis [ir_graph_clean_pass]
--- Running analysis [ir_analysis_pass]
--- Running IR pass [simplify_with_basic_ops_pass]
--- Running IR pass [attention_lstm_fuse_pass]
--- Running IR pass [seqconv_eltadd_relu_fuse_pass]
--- Running IR pass [seqpool_cvm_concat_fuse_pass]
--- Running IR pass [mul_lstm_fuse_pass]
--- Running IR pass [fc_gru_fuse_pass]
--- Running IR pass [mul_gru_fuse_pass]
--- Running IR pass [seq_concat_fc_fuse_pass]
--- Running IR pass [squeeze2_matmul_fuse_pass]
--- Running IR pass [reshape2_matmul_fuse_pass]
--- Running IR pass [flatten2_matmul_fuse_pass]
--- Running IR pass [map_matmul_to_mul_pass]
I0401 21:30:01.213358   849 graph_pattern_detector.cc:101] ---  detected 1 subgraphs
--- Running IR pass [fc_fuse_pass]
--- Running IR pass [repeated_fc_relu_fuse_pass]
--- Running IR pass [squared_mat_sub_fuse_pass]
--- Running IR pass [conv_bn_fuse_pass]
--- Running IR pass [conv_eltwiseadd_bn_fuse_pass]
--- Running IR pass [conv_transpose_bn_fuse_pass]
--- Running IR pass [conv_transpose_eltwiseadd_bn_fuse_pass]
--- Running IR pass [is_test_pass]
--- Running IR pass [runtime_context_cache_pass]
--- Running analysis [ir_params_sync_among_devices_pass]
--- Running analysis [adjust_cudnn_workspace_size_pass]
--- Running analysis [inference_op_replace_pass]
--- Running analysis [memory_optimize_pass]
I0401 21:30:01.214834   849 memory_optimize_pass.cc:200] Cluster name : linear_1.tmp_2  size: 4
I0401 21:30:01.214855   849 memory_optimize_pass.cc:200] Cluster name : linear_1.tmp_1  size: 4
I0401 21:30:01.214864   849 memory_optimize_pass.cc:200] Cluster name : x  size: 52
I0401 21:30:01.214872   849 memory_optimize_pass.cc:200] Cluster name : x_0  size: 52
--- Running analysis [ir_graph_to_program_pass]
I0401 21:30:01.216336   849 analysis_predictor.cc:598] ======= optimize end =======
I0401 21:30:01.216375   849 naive_executor.cc:107] ---  skip [feed], feed -> x
I0401 21:30:01.216478   849 naive_executor.cc:107] ---  skip [linear_1.tmp_1], fetch -> fetch
2021-04-01 21:30:01.490969: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
[2021-04-01 21:30:04,291] INFO - {'service_name': 'PaddleLinearRegression', 'service_version': '20210401212846_F88F15', 'api': 'predict', 'task': {'data': '0,1,2,3,4,5,6,7,8,9,10,11,12\r\n-0.0405441,0.06636364,-0.32356227,-0.06916996,-0.03435197,0.05563625,-0.03475696,0.02682186,-0.37171335,-0.21419304,-0.33569506,0.10143217,-0.21172912\r\n', 'task_id': '951521eb-16e6-4dbf-8ff9-6c596a2c9645', 'batch': 1, 'cli_args': ('--format', 'csv', '--input-file', 'test.csv'), 'inference_job_args': {}}, 'result': {'data': '[[0.9648699760437012]]', 'http_status': 200, 'http_headers': (('Content-Type', 'application/json'),)}, 'request_id': '951521eb-16e6-4dbf-8ff9-6c596a2c9645'}
[[0.9648699760437012]]

Deployment Options

If you are at a small team with limited engineering or DevOps resources, try out automated deployment with BentoML CLI, currently supporting AWS Lambda, AWS SageMaker, and Azure Functions:

If the cloud platform you are working with is not on the list above, try out these step-by-step guide on manually deploying BentoML packaged model to cloud platforms:

Lastly, if you have a DevOps or ML Engineering team who's operating a Kubernetes or OpenShift cluster, use the following guides as references for implementating your deployment strategy: