1. gallery
  2. fast-ai
  3. salary-range-prediction

BentoML Example: Fast AI with Tabular data

BentoML makes moving trained ML models to production easy:

  • Package models trained with any ML framework and reproduce them for model serving in production
  • Deploy anywhere for online API serving or offline batch serving
  • High-Performance API model server with adaptive micro-batching support
  • Central hub for managing models and deployment process via Web UI and APIs
  • Modular and flexible design making it adaptable to your infrastrcuture

BentoML is a framework for serving, managing, and deploying machine learning models. It is aiming to bridge the gap between Data Science and DevOps, and enable teams to deliver prediction services in a fast, repeatable, and scalable way. Before reading this example project, be sure to check out the Getting started guide to learn about the basic concepts in BentoML.

This notebook is based on fastai v1's cours v3 lesson 4. It will train a model that predict salary range base on the data we provided.

Impression

In [1]:
%reload_ext autoreload
%autoreload 2
%matplotlib inline
In [2]:
!pip install -q -U 'fastai<=1.0.61'

WARNING: You are using pip version 20.2.2; however, version 20.2.3 is available.
You should consider upgrading via the '/usr/local/anaconda3/envs/dev-py3/bin/python -m pip install --upgrade pip' command.
In [3]:
from fastai.tabular import *

Prepare Training Data

In [4]:
path = untar_data(URLs.ADULT_SAMPLE)
df = pd.read_csv(path/'adult.csv')
In [5]:
dep_var = 'salary'
cat_names = ['workclass', 'education', 'marital-status', 'occupation', 'relationship', 'race']
cont_names = ['age', 'fnlwgt', 'education-num']
procs = [FillMissing, Categorify, Normalize]
In [6]:
test = TabularList.from_df(df.iloc[800:1000].copy(), path=path, cat_names=cat_names, cont_names=cont_names)
In [7]:
data = (TabularList.from_df(df, path=path, cat_names=cat_names, cont_names=cont_names, procs=procs)
                           .split_by_idx(list(range(800,1000)))
                           .label_from_df(cols=dep_var)
                           .add_test(test)
                           .databunch())
In [8]:
data.show_batch(rows=10)
workclass education marital-status occupation relationship race education-num_na age fnlwgt education-num target
? HS-grad Married-spouse-absent ? Not-in-family White False -1.3624 -1.3855 -0.4224 <50k
Private Some-college Never-married Adm-clerical Own-child White False -1.4357 1.0365 -0.0312 <50k
Private Bachelors Married-civ-spouse Sales Husband White False 2.4491 -0.3046 1.1422 <50k
Local-gov HS-grad Never-married Transport-moving Not-in-family Asian-Pac-Islander False -0.1896 -0.0845 -0.4224 <50k
Private HS-grad Married-civ-spouse Adm-clerical Own-child White False -0.7760 -1.3352 -0.4224 <50k
? Assoc-voc Married-civ-spouse ? Husband White False -0.8493 0.5664 0.3599 <50k
Private HS-grad Widowed Adm-clerical Unmarried White False 0.8365 -0.4172 -0.4224 <50k
Private HS-grad Never-married Adm-clerical Not-in-family White False -1.2891 -1.3147 -0.4224 <50k
Private Some-college Married-civ-spouse Prof-specialty Husband White False 0.6166 0.7047 -0.0312 >=50k
Private 9th Divorced Machine-op-inspct Not-in-family White False -0.4095 -0.3273 -1.9869 <50k

Model Training

In [9]:
learn = tabular_learner(data, layers=[200,100], metrics=accuracy)
In [10]:
learn.fit(1, 1e-2)
epoch train_loss valid_loss accuracy time
0 0.371307 0.374452 0.845000 00:04
In [11]:
row = df.iloc[0] # sample input date for testing

learn.predict(row)
Out[11]:
(Category tensor(1), tensor(1), tensor([0.4633, 0.5367]))

Create BentoService for model serving

In [12]:
%%writefile tabular_csv.py

from bentoml import env, api, artifacts, BentoService
from bentoml.frameworks.fastai import Fastai1ModelArtifact
from bentoml.adapters import DataframeInput


@env(pip_packages=['fastai'])
@artifacts([Fastai1ModelArtifact('model')])
class FastaiTabularModel(BentoService):
    
    @api(input=DataframeInput(), batch=True)
    def predict(self, df):
        results = []
        for _, row in df.iterrows():       
            prediction = self.artifacts.model.predict(row)
            results.append(prediction[0].obj)
        return results

Overwriting tabular_csv.py

Save BentoService to file archive

In [13]:
# 1) import the custom BentoService defined above
from tabular_csv import FastaiTabularModel

# 2) `pack` it with required artifacts
svc = FastaiTabularModel()
svc.pack('model', learn)

# 3) save your BentoSerivce
saved_path = svc.save()

[2020-10-01 14:45:17,839] WARNING - Using BentoML installed in `editable` model, the local BentoML repository including all code changes will be packaged together with saved bundle created, under the './bundled_pip_dependencies' directory of the saved bundle.
[2020-10-01 14:45:18,011] INFO - Using default docker base image: `None` specified inBentoML config file or env var. User must make sure that the docker base image either has Python 3.7 or conda installed.
[2020-10-01 14:45:18,013] WARNING - BentoML by default does not include spacy and torchvision package when using FastaiModelArtifact. To make sure BentoML bundle those packages if they are required for your model, either import those packages in BentoService definition file or manually add them via `@env(pip_packages=['torchvision'])` when defining a BentoService
[2020-10-01 14:45:18,015] WARNING - Overwriting existing pip package requirement 'fastai==1.0.61' to 'fastai<2.0.0'
[2020-10-01 14:45:18,517] INFO - Detected non-PyPI-released BentoML installed, copying local BentoML modulefiles to target saved bundle path..

warning: no previously-included files matching '*~' found anywhere in distribution
warning: no previously-included files matching '*.pyo' found anywhere in distribution
warning: no previously-included files matching '.git' found anywhere in distribution
warning: no previously-included files matching '.ipynb_checkpoints' found anywhere in distribution
warning: no previously-included files matching '__pycache__' found anywhere in distribution
no previously-included directories found matching 'e2e_tests'
no previously-included directories found matching 'tests'
no previously-included directories found matching 'benchmark'

UPDATING BentoML-0.9.1+1.g0655cf16.dirty/bentoml/_version.py
set BentoML-0.9.1+1.g0655cf16.dirty/bentoml/_version.py to '0.9.1+1.g0655cf16.dirty'
[2020-10-01 14:45:22,277] INFO - BentoService bundle 'FastaiTabularModel:20201001144518_0C60FD' saved to: /Users/bozhaoyu/bentoml/repository/FastaiTabularModel/20201001144518_0C60FD

REST API Model Serving

To start a REST API model server with the BentoService saved above, use the bentoml serve command:

In [15]:
!bentoml serve FastaiTabularModel:latest

[2020-09-22 16:56:10,329] INFO - Getting latest version FastaiTabularModel:20200922163833_30289D
[2020-09-22 16:56:10,330] INFO - Starting BentoML API server in development mode..
[2020-09-22 16:56:10,622] WARNING - Using BentoML installed in `editable` model, the local BentoML repository including all code changes will be packaged together with saved bundle created, under the './bundled_pip_dependencies' directory of the saved bundle.
[2020-09-22 16:56:10,640] WARNING - Saved BentoService bundle version mismatch: loading BentoService bundle create with BentoML version 0.9.0.pre, but loading from BentoML version 0.9.0.pre+3.gcebf2015
[2020-09-22 16:56:11,014] INFO - Using default docker base image: `None` specified inBentoML config file or env var. User must make sure that the docker base image either has Python 3.7 or conda installed.
[2020-09-22 16:56:13,998] WARNING - BentoML by default does not include spacy and torchvision package when using FastaiModelArtifact. To make sure BentoML bundle those packages if they are required for your model, either import those packages in BentoService definition file or manually add them via `@env(pip_packages=['torchvision'])` when defining a BentoService
[2020-09-22 16:56:14,000] WARNING - pip package requirement fastai already exist
 * Serving Flask app "FastaiTabularModel" (lazy loading)
 * Environment: production
   WARNING: This is a development server. Do not use it in a production deployment.
   Use a production WSGI server instead.
 * Debug mode: off
 * Running on http://127.0.0.1:5000/ (Press CTRL+C to quit)
[2020-09-22 16:56:23,936] INFO - {'service_name': 'FastaiTabularModel', 'service_version': '20200922163833_30289D', 'api': 'predict', 'task': {'data': {}, 'task_id': 'd93bf027-f1db-4eef-bff9-c60e96d394ba', 'batch': 1, 'http_headers': (('Host', 'localhost:5000'), ('User-Agent', 'curl/7.65.3'), ('Accept', '*/*'), ('Content-Type', 'application/json'), ('Content-Length', '370'))}, 'result': {'data': '["<50k"]', 'http_status': 200, 'http_headers': (('Content-Type', 'application/json'),)}, 'request_id': 'd93bf027-f1db-4eef-bff9-c60e96d394ba'}
127.0.0.1 - - [22/Sep/2020 16:56:23] "POST /predict HTTP/1.1" 200 -
WARNING: Logging before flag parsing goes to stderr.
I0922 16:56:23.937808 4711075264 _internal.py:122] 127.0.0.1 - - [22/Sep/2020 16:56:23] "POST /predict HTTP/1.1" 200 -
^C

If you are running this notebook from Google Colab, you can start the dev server with --run-with-ngrok option, to gain acccess to the API endpoint via a public endpoint managed by ngrok:

In [ ]:
!bentoml serve FastaiTabularModel:latest --run-with-ngrok

Send prediction requeset to the REST API server

JSON Request

curl -X POST \
  http://localhost:5000/predict \
  -H 'Content-Type: application/json' \
  -d '[{
  "age": 49,
  "workclass": "Private",
  "fnlwgt": 101320,
  "education": "Assoc-acdm",
  "education-num": 12.0,
  "marital-status": "Married-civ-spouse",
  "occupation": "",
  "relationship": "Wift",
  "race": "White",
  "sex": "Female",
  "capital-gain": 0,
  "capital-loss": 1902,
  "hours-per-week": 40,
  "native-country": "United-States",
  "salary": ">=50k"
}]'

CSV Request

curl -X POST "http://127.0.0.1:5000/predict" \
    -H "Content-Type: text/csv" \
    --data-binary @test.csv

Containerize model server with Docker

One common way of distributing this model API server for production deployment, is via Docker containers. And BentoML provides a convenient way to do that.

Note that docker is not available in Google Colab. You will need to download and run this notebook locally to try out this containerization with docker feature.

If you already have docker configured, simply run the follow command to product a docker container serving the IrisClassifier prediction service created above:

In [16]:
!bentoml containerize FastaiTabularModel:latest

[2020-09-22 16:56:43,434] INFO - Getting latest version FastaiTabularModel:20200922163833_30289D
Found Bento: /Users/bozhaoyu/bentoml/repository/FastaiTabularModel/20200922163833_30289D
[2020-09-22 16:56:43,475] WARNING - Using BentoML installed in `editable` model, the local BentoML repository including all code changes will be packaged together with saved bundle created, under the './bundled_pip_dependencies' directory of the saved bundle.
[2020-09-22 16:56:43,496] WARNING - Saved BentoService bundle version mismatch: loading BentoService bundle create with BentoML version 0.9.0.pre, but loading from BentoML version 0.9.0.pre+3.gcebf2015
Tag not specified, using tag parsed from BentoService: 'fastaitabularmodel:20200922163833_30289D'
Building Docker image fastaitabularmodel:20200922163833_30289D from FastaiTabularModel:latest 
-we in here
processed docker file
(None, None)
root in create archive /Users/bozhaoyu/bentoml/repository/FastaiTabularModel/20200922163833_30289D ['Dockerfile', 'FastaiTabularModel', 'FastaiTabularModel/__init__.py', 'FastaiTabularModel/__pycache__', 'FastaiTabularModel/__pycache__/tabular_csv.cpython-37.pyc', 'FastaiTabularModel/artifacts', 'FastaiTabularModel/artifacts/__init__.py', 'FastaiTabularModel/artifacts/model.pkl', 'FastaiTabularModel/bentoml.yml', 'FastaiTabularModel/tabular_csv.py', 'MANIFEST.in', 'README.md', 'bentoml-init.sh', 'bentoml.yml', 'bundled_pip_dependencies', 'bundled_pip_dependencies/BentoML-0.9.0rc0+3.gcebf2015.tar.gz', 'docker-entrypoint.sh', 'environment.yml', 'python_version', 'requirements.txt', 'setup.py']
about to build
about to upgrade params
check each param and update
if use config proxy
if buildargs
if shmsize
if labels
if cache from
if target
if network_mode
if squash
if extra hosts is not None
if platform is not None
if isolcation is not None
if context is not None
setting auth {'Content-Type': 'application/tar'}
-docker build <tempfile._TemporaryFileWrapper object at 0x7fc514d73cf8> {'t': 'fastaitabularmodel:20200922163833_30289D', 'remote': None, 'q': False, 'nocache': False, 'rm': False, 'forcerm': False, 'pull': False, 'dockerfile': (None, None)}
/docker response <Response [200]>
context closes
print responses
Step 1/15 : FROM bentoml/model-server:0.9.0.pre
 ---> a25066aa8b0e
Step 2/15 : ARG EXTRA_PIP_INSTALL_ARGS=
 ---> Using cache
 ---> fc6e47d06522
Step 3/15 : ENV EXTRA_PIP_INSTALL_ARGS $EXTRA_PIP_INSTALL_ARGS
 ---> Using cache
 ---> db8172e98571
Step 4/15 : COPY environment.yml requirements.txt setup.sh* bentoml-init.sh python_version* /bento/
| ---> 55a0a6097230
Step 5/15 : WORKDIR /bento
\ ---> Running in 608a0ecc0656
- ---> 7671d9cbcbe2
Step 6/15 : RUN chmod +x /bento/bentoml-init.sh
 ---> Running in 51de07768042
| ---> d76239495190
Step 7/15 : RUN if [ -f /bento/bentoml-init.sh ]; then bash -c /bento/bentoml-init.sh; fi
 ---> Running in be68f5f82c22
/+++ dirname /bento/bentoml-init.sh

++ cd /bento
++ pwd -P
+ SAVED_BUNDLE_PATH=/bento
+ cd /bento
+ '[' -f ./setup.sh ']'
+ '[' -f ./python_version ']'
++ cat ./python_version

+ PY_VERSION_SAVED=3.7.3
+ DESIRED_PY_VERSION=3.7
++ python -c 'import sys; print(f"{sys.version_info.major}.{sys.version_info.minor}")'

+ CURRENT_PY_VERSION=3.7
+ [[ 3.7 == \3\.\7 ]]
+ echo 'Python Version in docker base image 3.7 matches requirement python=3.7. Skipping.'
+ command -v conda

+ echo 'Updating conda base environment with environment.yml'
+ conda env update -n base -f ./environment.yml

Python Version in docker base image 3.7 matches requirement python=3.7. Skipping.
Updating conda base environment with environment.yml
|Collecting package metadata (repodata.json): ...working... 
|done
Solving environment: ...working... 
\done
-
Downloading and Extracting Packages
openssl-1.1.1h       | 2.1 MB    |            |   0% 
openssl-1.1.1h       | 2.1 MB    |            |   1% 
openssl-1.1.1h       | 2.1 MB    | ###2       |  32% 
openssl-1.1.1h       | 2.1 MB    | ########7  |  88% 
openssl-1.1.1h       | 2.1 MB    | ########## | 100% 

pip-20.2.3           | 1.1 MB    |            |   0% 
pip-20.2.3           | 1.1 MB    | #####2     |  53% 
pip-20.2.3           | 1.1 MB    | ########## | 100% 
pip-20.2.3           | 1.1 MB    | ########## | 100% 

certifi-2020.6.20    | 151 KB    |            |   0% 
certifi-2020.6.20    | 151 KB    | ########## | 100% 

cffi-1.14.3          | 223 KB    |            |   0% 
cffi-1.14.3          | 223 KB    | ########## | 100% 
cffi-1.14.3          | 223 KB    | ########## | 100% 

ca-certificates-2020 | 145 KB    |            |   0% 
ca-certificates-2020 | 145 KB    | ########## | 100% 

python_abi-3.7       | 4 KB      |            |   0% 
python_abi-3.7       | 4 KB      | ########## | 100% 

libffi-3.2.1         | 47 KB     |            |   0% 
libffi-3.2.1         | 47 KB     | ########## | 100% 

Preparing transaction: 
...working... 
-done
Verifying transaction: ...working... 
/done
Executing transaction: 
...working... 
\done
-#
# To activate this environment, use
#
#     $ conda activate base
#
# To deactivate an active environment, use
#
#     $ conda deactivate
-+ pip install -r ./requirements.txt --no-cache-dir

-Requirement already satisfied: bentoml==0.9.0.pre in /opt/conda/lib/python3.7/site-packages (from -r ./requirements.txt (line 1)) (0.9.0rc0)
/Collecting fastai==1.0.61
|  Downloading fastai-1.0.61-py3-none-any.whl (239 kB)
\Collecting pandas==0.24.2
  Downloading pandas-0.24.2-cp37-cp37m-manylinux1_x86_64.whl (10.1 MB)
\Collecting torch==1.6.0
-  Downloading torch-1.6.0-cp37-cp37m-manylinux1_x86_64.whl (748.8 MB)
-Requirement already satisfied: humanfriendly in /opt/conda/lib/python3.7/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (8.2)
Requirement already satisfied: alembic in /opt/conda/lib/python3.7/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (1.4.3)
Requirement already satisfied: prometheus-client in /opt/conda/lib/python3.7/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (0.8.0)
Requirement already satisfied: click>=7.0 in /opt/conda/lib/python3.7/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (7.1.2)
Requirement already satisfied: flask in /opt/conda/lib/python3.7/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (1.1.2)
Requirement already satisfied: cerberus in /opt/conda/lib/python3.7/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (1.3.2)
Requirement already satisfied: gunicorn in /opt/conda/lib/python3.7/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (20.0.4)
Requirement already satisfied: certifi in /opt/conda/lib/python3.7/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (2020.6.20)
Requirement already satisfied: docker in /opt/conda/lib/python3.7/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (4.3.1)
Requirement already satisfied: configparser in /opt/conda/lib/python3.7/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (5.0.0)
Requirement already satisfied: sqlalchemy-utils<0.36.8 in /opt/conda/lib/python3.7/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (0.36.7)
/Requirement already satisfied: protobuf>=3.6.0 in /opt/conda/lib/python3.7/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (3.13.0)
Requirement already satisfied: numpy in /opt/conda/lib/python3.7/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (1.19.2)
Requirement already satisfied: packaging in /opt/conda/lib/python3.7/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (20.4)
Requirement already satisfied: aiohttp in /opt/conda/lib/python3.7/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (3.6.2)
|Requirement already satisfied: py-zipkin in /opt/conda/lib/python3.7/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (0.20.0)
Requirement already satisfied: sqlalchemy>=1.3.0 in /opt/conda/lib/python3.7/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (1.3.19)
Requirement already satisfied: python-dateutil<3.0.0,>=2.7.3 in /opt/conda/lib/python3.7/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (2.8.1)
Requirement already satisfied: ruamel.yaml>=0.15.0 in /opt/conda/lib/python3.7/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (0.15.87)
Requirement already satisfied: psutil in /opt/conda/lib/python3.7/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (5.7.2)
Requirement already satisfied: python-json-logger in /opt/conda/lib/python3.7/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (0.1.11)
Requirement already satisfied: multidict in /opt/conda/lib/python3.7/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (4.7.6)
Requirement already satisfied: grpcio<=1.27.2 in /opt/conda/lib/python3.7/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (1.27.2)
Requirement already satisfied: boto3 in /opt/conda/lib/python3.7/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (1.15.2)
Requirement already satisfied: requests in /opt/conda/lib/python3.7/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (2.24.0)
Requirement already satisfied: tabulate in /opt/conda/lib/python3.7/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (0.8.7)
/Collecting scipy
|  Downloading scipy-1.5.2-cp37-cp37m-manylinux1_x86_64.whl (25.9 MB)
/Collecting beautifulsoup4
  Downloading beautifulsoup4-4.9.1-py3-none-any.whl (115 kB)
|Collecting nvidia-ml-py3
  Downloading nvidia-ml-py3-7.352.0.tar.gz (19 kB)
|Collecting pyyaml
\  Downloading PyYAML-5.3.1.tar.gz (269 kB)
|Collecting matplotlib
  Downloading matplotlib-3.3.2-cp37-cp37m-manylinux1_x86_64.whl (11.6 MB)
\Collecting Pillow
-  Downloading Pillow-7.2.0-cp37-cp37m-manylinux1_x86_64.whl (2.2 MB)
|Collecting torchvision
  Downloading torchvision-0.7.0-cp37-cp37m-manylinux1_x86_64.whl (5.9 MB)
\Collecting fastprogress>=0.2.1
  Downloading fastprogress-1.0.0-py3-none-any.whl (12 kB)
/Collecting spacy>=2.0.18; python_version < "3.8"
|  Downloading spacy-2.3.2-cp37-cp37m-manylinux1_x86_64.whl (9.9 MB)
/Collecting bottleneck
|  Downloading Bottleneck-1.3.2.tar.gz (88 kB)
  Installing build dependencies: started
\  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
/  Getting requirements to build wheel: finished with status 'done'
|    Preparing wheel metadata: started
/    Preparing wheel metadata: finished with status 'done'
\Collecting numexpr
  Downloading numexpr-2.7.1-cp37-cp37m-manylinux1_x86_64.whl (162 kB)
/Collecting pytz>=2011k
  Downloading pytz-2020.1-py2.py3-none-any.whl (510 kB)
\Collecting future
  Downloading future-0.18.2.tar.gz (829 kB)
-Requirement already satisfied: Mako in /opt/conda/lib/python3.7/site-packages (from alembic->bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (1.1.3)
Requirement already satisfied: python-editor>=0.3 in /opt/conda/lib/python3.7/site-packages (from alembic->bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (1.0.4)
Requirement already satisfied: Werkzeug>=0.15 in /opt/conda/lib/python3.7/site-packages (from flask->bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (1.0.1)
Requirement already satisfied: itsdangerous>=0.24 in /opt/conda/lib/python3.7/site-packages (from flask->bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (1.1.0)
Requirement already satisfied: Jinja2>=2.10.1 in /opt/conda/lib/python3.7/site-packages (from flask->bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (2.11.2)
Requirement already satisfied: setuptools in /opt/conda/lib/python3.7/site-packages (from cerberus->bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (49.6.0.post20200814)
Requirement already satisfied: six>=1.4.0 in /opt/conda/lib/python3.7/site-packages (from docker->bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (1.15.0)
Requirement already satisfied: websocket-client>=0.32.0 in /opt/conda/lib/python3.7/site-packages (from docker->bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (0.57.0)
Requirement already satisfied: pyparsing>=2.0.2 in /opt/conda/lib/python3.7/site-packages (from packaging->bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (2.4.7)
Requirement already satisfied: async-timeout<4.0,>=3.0 in /opt/conda/lib/python3.7/site-packages (from aiohttp->bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (3.0.1)
Requirement already satisfied: attrs>=17.3.0 in /opt/conda/lib/python3.7/site-packages (from aiohttp->bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (20.2.0)
/Requirement already satisfied: yarl<2.0,>=1.0 in /opt/conda/lib/python3.7/site-packages (from aiohttp->bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (1.5.1)
Requirement already satisfied: chardet<4.0,>=2.0 in /opt/conda/lib/python3.7/site-packages (from aiohttp->bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (3.0.4)
Requirement already satisfied: thriftpy2>=0.4.0 in /opt/conda/lib/python3.7/site-packages (from py-zipkin->bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (0.4.11)
Requirement already satisfied: s3transfer<0.4.0,>=0.3.0 in /opt/conda/lib/python3.7/site-packages (from boto3->bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (0.3.3)
Requirement already satisfied: botocore<1.19.0,>=1.18.2 in /opt/conda/lib/python3.7/site-packages (from boto3->bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (1.18.2)
Requirement already satisfied: jmespath<1.0.0,>=0.7.1 in /opt/conda/lib/python3.7/site-packages (from boto3->bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (0.10.0)
Requirement already satisfied: idna<3,>=2.5 in /opt/conda/lib/python3.7/site-packages (from requests->bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (2.10)
Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /opt/conda/lib/python3.7/site-packages (from requests->bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (1.25.10)
|Collecting soupsieve>1.2
  Downloading soupsieve-2.0.1-py3-none-any.whl (32 kB)
\Collecting kiwisolver>=1.0.1
  Downloading kiwisolver-1.2.0-cp37-cp37m-manylinux1_x86_64.whl (88 kB)
-Collecting cycler>=0.10
  Downloading cycler-0.10.0-py2.py3-none-any.whl (6.5 kB)
|Collecting preshed<3.1.0,>=3.0.2
  Downloading preshed-3.0.2-cp37-cp37m-manylinux1_x86_64.whl (118 kB)
\Collecting blis<0.5.0,>=0.4.0
  Downloading blis-0.4.1-cp37-cp37m-manylinux1_x86_64.whl (3.7 MB)
/Collecting thinc==7.4.1
  Downloading thinc-7.4.1-cp37-cp37m-manylinux1_x86_64.whl (2.1 MB)
\Collecting catalogue<1.1.0,>=0.0.7
-  Downloading catalogue-1.0.0-py2.py3-none-any.whl (7.7 kB)
Requirement already satisfied: tqdm<5.0.0,>=4.38.0 in /opt/conda/lib/python3.7/site-packages (from spacy>=2.0.18; python_version < "3.8"->fastai==1.0.61->-r ./requirements.txt (line 2)) (4.48.2)
Collecting plac<1.2.0,>=0.9.6
/  Downloading plac-1.1.3-py2.py3-none-any.whl (20 kB)
|Collecting srsly<1.1.0,>=1.0.2
  Downloading srsly-1.0.2-cp37-cp37m-manylinux1_x86_64.whl (185 kB)
-Collecting cymem<2.1.0,>=2.0.2
  Downloading cymem-2.0.3-cp37-cp37m-manylinux1_x86_64.whl (32 kB)
/Collecting wasabi<1.1.0,>=0.4.0
  Downloading wasabi-0.8.0-py3-none-any.whl (23 kB)
\Collecting murmurhash<1.1.0,>=0.28.0
  Downloading murmurhash-1.0.2-cp37-cp37m-manylinux1_x86_64.whl (19 kB)
Requirement already satisfied: MarkupSafe>=0.9.2 in /opt/conda/lib/python3.7/site-packages (from Mako->alembic->bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (1.1.1)
Requirement already satisfied: typing-extensions>=3.7.4; python_version < "3.8" in /opt/conda/lib/python3.7/site-packages (from yarl<2.0,>=1.0->aiohttp->bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (3.7.4.3)
Requirement already satisfied: ply<4.0,>=3.4 in /opt/conda/lib/python3.7/site-packages (from thriftpy2>=0.4.0->py-zipkin->bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (3.11)
-Collecting importlib-metadata>=0.20; python_version < "3.8"
  Downloading importlib_metadata-2.0.0-py2.py3-none-any.whl (31 kB)
/Collecting zipp>=0.5
  Downloading zipp-3.2.0-py3-none-any.whl (5.1 kB)
|Building wheels for collected packages: nvidia-ml-py3, pyyaml, bottleneck, future
  Building wheel for nvidia-ml-py3 (setup.py): started
/  Building wheel for nvidia-ml-py3 (setup.py): finished with status 'done'
  Created wheel for nvidia-ml-py3: filename=nvidia_ml_py3-7.352.0-py3-none-any.whl size=19191 sha256=9133f42be2d5c905262136fe2f11fafeee49d48405dfeccf433b1a1f6880596c
  Stored in directory: /tmp/pip-ephem-wheel-cache-3pc2f4g3/wheels/df/99/da/c34f202dc8fd1dffd35e0ecf1a7d7f8374ca05fbcbaf974b83
  Building wheel for pyyaml (setup.py): started
\  Building wheel for pyyaml (setup.py): finished with status 'done'
  Created wheel for pyyaml: filename=PyYAML-5.3.1-cp37-cp37m-linux_x86_64.whl size=44619 sha256=06b029825047443a06a6d902a0a8aea3043fcda80e5b64aee2d9a3885fdb1686
  Stored in directory: /tmp/pip-ephem-wheel-cache-3pc2f4g3/wheels/5e/03/1e/e1e954795d6f35dfc7b637fe2277bff021303bd9570ecea653
  Building wheel for bottleneck (PEP 517): started
\  Building wheel for bottleneck (PEP 517): finished with status 'done'
  Created wheel for bottleneck: filename=Bottleneck-1.3.2-cp37-cp37m-linux_x86_64.whl size=386286 sha256=ff257a0246aa8e3b327ff8d4bce3f82cbe0fa07b81cd4447c680962e28c27f22
  Stored in directory: /tmp/pip-ephem-wheel-cache-3pc2f4g3/wheels/87/85/9c/a325c89ff0498660ef8a335fb4b3912939c273ea4f094af29f
  Building wheel for future (setup.py): started
/  Building wheel for future (setup.py): finished with status 'done'
  Created wheel for future: filename=future-0.18.2-py3-none-any.whl size=491059 sha256=d614ba77d286ffaa4b10e65903c7aa37d69cc5a21e84864cd70caffd39ef17f8
  Stored in directory: /tmp/pip-ephem-wheel-cache-3pc2f4g3/wheels/56/b0/fe/4410d17b32f1f0c3cf54cdfb2bc04d7b4b8f4ae377e2229ba0
Successfully built nvidia-ml-py3 pyyaml bottleneck future
-Installing collected packages: scipy, soupsieve, beautifulsoup4, nvidia-ml-py3, pyyaml, Pillow, kiwisolver, cycler, matplotlib, future, torch, torchvision, pytz, pandas, fastprogress, cymem, murmurhash, preshed, blis, plac, wasabi, zipp, importlib-metadata, catalogue, srsly, thinc, spacy, bottleneck, numexpr, fastai
\Successfully installed Pillow-7.2.0 beautifulsoup4-4.9.1 blis-0.4.1 bottleneck-1.3.2 catalogue-1.0.0 cycler-0.10.0 cymem-2.0.3 fastai-1.0.61 fastprogress-1.0.0 future-0.18.2 importlib-metadata-2.0.0 kiwisolver-1.2.0 matplotlib-3.3.2 murmurhash-1.0.2 numexpr-2.7.1 nvidia-ml-py3-7.352.0 pandas-0.24.2 plac-1.1.3 preshed-3.0.2 pytz-2020.1 pyyaml-5.3.1 scipy-1.5.2 soupsieve-2.0.1 spacy-2.3.2 srsly-1.0.2 thinc-7.4.1 torch-1.6.0 torchvision-0.7.0 wasabi-0.8.0 zipp-3.2.0
\ ---> d72434920c0a
Step 8/15 : COPY . /bento
| ---> 296bc01ddba4
Step 9/15 : RUN if [ -d /bento/bundled_pip_dependencies ]; then pip install -U bundled_pip_dependencies/* ;fi
\ ---> Running in 63a0655d30a7
|Processing ./bundled_pip_dependencies/BentoML-0.9.0rc0+3.gcebf2015.tar.gz
|  Installing build dependencies: started
/  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
|  Getting requirements to build wheel: finished with status 'done'
    Preparing wheel metadata: started
-    Preparing wheel metadata: finished with status 'done'
|Requirement already satisfied, skipping upgrade: protobuf>=3.6.0 in /opt/conda/lib/python3.7/site-packages (from BentoML==0.9.0rc0+3.gcebf2015) (3.13.0)
Requirement already satisfied, skipping upgrade: alembic in /opt/conda/lib/python3.7/site-packages (from BentoML==0.9.0rc0+3.gcebf2015) (1.4.3)
Requirement already satisfied, skipping upgrade: humanfriendly in /opt/conda/lib/python3.7/site-packages (from BentoML==0.9.0rc0+3.gcebf2015) (8.2)
Requirement already satisfied, skipping upgrade: prometheus-client in /opt/conda/lib/python3.7/site-packages (from BentoML==0.9.0rc0+3.gcebf2015) (0.8.0)
Requirement already satisfied, skipping upgrade: click>=7.0 in /opt/conda/lib/python3.7/site-packages (from BentoML==0.9.0rc0+3.gcebf2015) (7.1.2)
Requirement already satisfied, skipping upgrade: cerberus in /opt/conda/lib/python3.7/site-packages (from BentoML==0.9.0rc0+3.gcebf2015) (1.3.2)
Requirement already satisfied, skipping upgrade: requests in /opt/conda/lib/python3.7/site-packages (from BentoML==0.9.0rc0+3.gcebf2015) (2.24.0)
Requirement already satisfied, skipping upgrade: python-dateutil<3.0.0,>=2.7.3 in /opt/conda/lib/python3.7/site-packages (from BentoML==0.9.0rc0+3.gcebf2015) (2.8.1)
Requirement already satisfied, skipping upgrade: flask in /opt/conda/lib/python3.7/site-packages (from BentoML==0.9.0rc0+3.gcebf2015) (1.1.2)
Requirement already satisfied, skipping upgrade: numpy in /opt/conda/lib/python3.7/site-packages (from BentoML==0.9.0rc0+3.gcebf2015) (1.19.2)
Requirement already satisfied, skipping upgrade: psutil in /opt/conda/lib/python3.7/site-packages (from BentoML==0.9.0rc0+3.gcebf2015) (5.7.2)
\Requirement already satisfied, skipping upgrade: multidict in /opt/conda/lib/python3.7/site-packages (from BentoML==0.9.0rc0+3.gcebf2015) (4.7.6)
Requirement already satisfied, skipping upgrade: py-zipkin in /opt/conda/lib/python3.7/site-packages (from BentoML==0.9.0rc0+3.gcebf2015) (0.20.0)
Requirement already satisfied, skipping upgrade: tabulate in /opt/conda/lib/python3.7/site-packages (from BentoML==0.9.0rc0+3.gcebf2015) (0.8.7)
Requirement already satisfied, skipping upgrade: grpcio<=1.27.2 in /opt/conda/lib/python3.7/site-packages (from BentoML==0.9.0rc0+3.gcebf2015) (1.27.2)
Requirement already satisfied, skipping upgrade: docker in /opt/conda/lib/python3.7/site-packages (from BentoML==0.9.0rc0+3.gcebf2015) (4.3.1)
Requirement already satisfied, skipping upgrade: sqlalchemy>=1.3.0 in /opt/conda/lib/python3.7/site-packages (from BentoML==0.9.0rc0+3.gcebf2015) (1.3.19)
Requirement already satisfied, skipping upgrade: boto3 in /opt/conda/lib/python3.7/site-packages (from BentoML==0.9.0rc0+3.gcebf2015) (1.15.2)
Requirement already satisfied, skipping upgrade: gunicorn in /opt/conda/lib/python3.7/site-packages (from BentoML==0.9.0rc0+3.gcebf2015) (20.0.4)
Requirement already satisfied, skipping upgrade: certifi in /opt/conda/lib/python3.7/site-packages (from BentoML==0.9.0rc0+3.gcebf2015) (2020.6.20)
Requirement already satisfied, skipping upgrade: packaging in /opt/conda/lib/python3.7/site-packages (from BentoML==0.9.0rc0+3.gcebf2015) (20.4)
Requirement already satisfied, skipping upgrade: ruamel.yaml>=0.15.0 in /opt/conda/lib/python3.7/site-packages (from BentoML==0.9.0rc0+3.gcebf2015) (0.15.87)
Requirement already satisfied, skipping upgrade: python-json-logger in /opt/conda/lib/python3.7/site-packages (from BentoML==0.9.0rc0+3.gcebf2015) (0.1.11)
Requirement already satisfied, skipping upgrade: aiohttp in /opt/conda/lib/python3.7/site-packages (from BentoML==0.9.0rc0+3.gcebf2015) (3.6.2)
Requirement already satisfied, skipping upgrade: sqlalchemy-utils<0.36.8 in /opt/conda/lib/python3.7/site-packages (from BentoML==0.9.0rc0+3.gcebf2015) (0.36.7)
-Requirement already satisfied, skipping upgrade: configparser in /opt/conda/lib/python3.7/site-packages (from BentoML==0.9.0rc0+3.gcebf2015) (5.0.0)
Requirement already satisfied, skipping upgrade: setuptools in /opt/conda/lib/python3.7/site-packages (from protobuf>=3.6.0->BentoML==0.9.0rc0+3.gcebf2015) (49.6.0.post20200814)
Requirement already satisfied, skipping upgrade: six>=1.9 in /opt/conda/lib/python3.7/site-packages (from protobuf>=3.6.0->BentoML==0.9.0rc0+3.gcebf2015) (1.15.0)
Requirement already satisfied, skipping upgrade: python-editor>=0.3 in /opt/conda/lib/python3.7/site-packages (from alembic->BentoML==0.9.0rc0+3.gcebf2015) (1.0.4)
Requirement already satisfied, skipping upgrade: Mako in /opt/conda/lib/python3.7/site-packages (from alembic->BentoML==0.9.0rc0+3.gcebf2015) (1.1.3)
Requirement already satisfied, skipping upgrade: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /opt/conda/lib/python3.7/site-packages (from requests->BentoML==0.9.0rc0+3.gcebf2015) (1.25.10)
Requirement already satisfied, skipping upgrade: chardet<4,>=3.0.2 in /opt/conda/lib/python3.7/site-packages (from requests->BentoML==0.9.0rc0+3.gcebf2015) (3.0.4)
/Requirement already satisfied, skipping upgrade: idna<3,>=2.5 in /opt/conda/lib/python3.7/site-packages (from requests->BentoML==0.9.0rc0+3.gcebf2015) (2.10)
Requirement already satisfied, skipping upgrade: itsdangerous>=0.24 in /opt/conda/lib/python3.7/site-packages (from flask->BentoML==0.9.0rc0+3.gcebf2015) (1.1.0)
Requirement already satisfied, skipping upgrade: Jinja2>=2.10.1 in /opt/conda/lib/python3.7/site-packages (from flask->BentoML==0.9.0rc0+3.gcebf2015) (2.11.2)
Requirement already satisfied, skipping upgrade: Werkzeug>=0.15 in /opt/conda/lib/python3.7/site-packages (from flask->BentoML==0.9.0rc0+3.gcebf2015) (1.0.1)
Requirement already satisfied, skipping upgrade: thriftpy2>=0.4.0 in /opt/conda/lib/python3.7/site-packages (from py-zipkin->BentoML==0.9.0rc0+3.gcebf2015) (0.4.11)
Requirement already satisfied, skipping upgrade: websocket-client>=0.32.0 in /opt/conda/lib/python3.7/site-packages (from docker->BentoML==0.9.0rc0+3.gcebf2015) (0.57.0)
Requirement already satisfied, skipping upgrade: s3transfer<0.4.0,>=0.3.0 in /opt/conda/lib/python3.7/site-packages (from boto3->BentoML==0.9.0rc0+3.gcebf2015) (0.3.3)
Requirement already satisfied, skipping upgrade: botocore<1.19.0,>=1.18.2 in /opt/conda/lib/python3.7/site-packages (from boto3->BentoML==0.9.0rc0+3.gcebf2015) (1.18.2)
Requirement already satisfied, skipping upgrade: jmespath<1.0.0,>=0.7.1 in /opt/conda/lib/python3.7/site-packages (from boto3->BentoML==0.9.0rc0+3.gcebf2015) (0.10.0)
Requirement already satisfied, skipping upgrade: pyparsing>=2.0.2 in /opt/conda/lib/python3.7/site-packages (from packaging->BentoML==0.9.0rc0+3.gcebf2015) (2.4.7)
Requirement already satisfied, skipping upgrade: attrs>=17.3.0 in /opt/conda/lib/python3.7/site-packages (from aiohttp->BentoML==0.9.0rc0+3.gcebf2015) (20.2.0)
Requirement already satisfied, skipping upgrade: async-timeout<4.0,>=3.0 in /opt/conda/lib/python3.7/site-packages (from aiohttp->BentoML==0.9.0rc0+3.gcebf2015) (3.0.1)
Requirement already satisfied, skipping upgrade: yarl<2.0,>=1.0 in /opt/conda/lib/python3.7/site-packages (from aiohttp->BentoML==0.9.0rc0+3.gcebf2015) (1.5.1)
Requirement already satisfied, skipping upgrade: MarkupSafe>=0.9.2 in /opt/conda/lib/python3.7/site-packages (from Mako->alembic->BentoML==0.9.0rc0+3.gcebf2015) (1.1.1)
|Requirement already satisfied, skipping upgrade: ply<4.0,>=3.4 in /opt/conda/lib/python3.7/site-packages (from thriftpy2>=0.4.0->py-zipkin->BentoML==0.9.0rc0+3.gcebf2015) (3.11)
Requirement already satisfied, skipping upgrade: typing-extensions>=3.7.4; python_version < "3.8" in /opt/conda/lib/python3.7/site-packages (from yarl<2.0,>=1.0->aiohttp->BentoML==0.9.0rc0+3.gcebf2015) (3.7.4.3)
Building wheels for collected packages: BentoML
  Building wheel for BentoML (PEP 517): started
|  Building wheel for BentoML (PEP 517): finished with status 'done'
  Created wheel for BentoML: filename=BentoML-0.9.0rc0+3.gcebf2015-py3-none-any.whl size=3064091 sha256=762aa6ea85795b1fa82fe6196527c7db0fe6e17da0a48a128cbc2e5b2f846d2d
  Stored in directory: /root/.cache/pip/wheels/a0/45/41/62152db705af4ff47e7a3d6abf6247986eef4aa1b94a58d3b9
Successfully built BentoML
\Installing collected packages: BentoML
  Attempting uninstall: BentoML
    Found existing installation: BentoML 0.9.0rc0
/    Uninstalling BentoML-0.9.0rc0:
\      Successfully uninstalled BentoML-0.9.0rc0
-Successfully installed BentoML-0.9.0rc0+3.gcebf2015
\ ---> e2f758c32fe8
Step 10/15 : ENV PORT 5000
 ---> Running in bcf40c69a9b0
- ---> 3e6f27372f5d
Step 11/15 : EXPOSE $PORT
/ ---> Running in 022d6305af14
| ---> 1466c0426ea9
Step 12/15 : COPY docker-entrypoint.sh /usr/local/bin/
\ ---> cfd5416160a9
Step 13/15 : RUN chmod +x /usr/local/bin/docker-entrypoint.sh
- ---> Running in 1696f568c09a
/ ---> ff0ad7d18c18
Step 14/15 : ENTRYPOINT [ "docker-entrypoint.sh" ]
| ---> Running in 649edeb9207b
\ ---> 5aa2d29a2153
Step 15/15 : CMD ["bentoml", "serve-gunicorn", "/bento"]
 ---> Running in 2863ca14b743
- ---> 031f971273f4
Successfully built 031f971273f4
Successfully tagged fastaitabularmodel:20200922163833_30289D
Finished building fastaitabularmodel:20200922163833_30289D from FastaiTabularModel:latest
In [17]:
!docker run -p 5000:5000 fastaitabularmodel:20200922163833_30289D

[2020-09-23 00:01:08,992] INFO - Starting BentoML API server in production mode..
[2020-09-23 00:01:09,478] INFO - get_gunicorn_num_of_workers: 3, calculated by cpu count
[2020-09-23 00:01:09 +0000] [1] [INFO] Starting gunicorn 20.0.4
[2020-09-23 00:01:09 +0000] [1] [INFO] Listening at: http://0.0.0.0:5000 (1)
[2020-09-23 00:01:09 +0000] [1] [INFO] Using worker: sync
[2020-09-23 00:01:09 +0000] [12] [INFO] Booting worker with pid: 12
[2020-09-23 00:01:09 +0000] [13] [INFO] Booting worker with pid: 13
[2020-09-23 00:01:09 +0000] [14] [INFO] Booting worker with pid: 14
[2020-09-23 00:01:09,734] WARNING - Using BentoML not from official PyPI release. In order to find the same version of BentoML when deploying your BentoService, you must set the 'core/bentoml_deploy_version' config to a http/git location of your BentoML fork, e.g.: 'bentoml_deploy_version = git+https://github.com/{username}/[email protected]{branch}'
[2020-09-23 00:01:09,736] WARNING - Using BentoML not from official PyPI release. In order to find the same version of BentoML when deploying your BentoService, you must set the 'core/bentoml_deploy_version' config to a http/git location of your BentoML fork, e.g.: 'bentoml_deploy_version = git+https://github.com/{username}/[email protected]{branch}'
[2020-09-23 00:01:09,759] WARNING - Saved BentoService bundle version mismatch: loading BentoService bundle create with BentoML version 0.9.0.pre, but loading from BentoML version 0.9.0.pre+3.gcebf2015
[2020-09-23 00:01:09,760] WARNING - Saved BentoService Python version mismatch: loading BentoService bundle created with Python version 3.7.3, but current environment version is 3.7.6.
[2020-09-23 00:01:09,761] WARNING - Saved BentoService bundle version mismatch: loading BentoService bundle create with BentoML version 0.9.0.pre, but loading from BentoML version 0.9.0.pre+3.gcebf2015
[2020-09-23 00:01:09,762] WARNING - Saved BentoService Python version mismatch: loading BentoService bundle created with Python version 3.7.3, but current environment version is 3.7.6.
[2020-09-23 00:01:09,799] WARNING - Using BentoML not from official PyPI release. In order to find the same version of BentoML when deploying your BentoService, you must set the 'core/bentoml_deploy_version' config to a http/git location of your BentoML fork, e.g.: 'bentoml_deploy_version = git+https://github.com/{username}/[email protected]{branch}'
[2020-09-23 00:01:09,822] WARNING - Saved BentoService bundle version mismatch: loading BentoService bundle create with BentoML version 0.9.0.pre, but loading from BentoML version 0.9.0.pre+3.gcebf2015
[2020-09-23 00:01:09,822] WARNING - Saved BentoService Python version mismatch: loading BentoService bundle created with Python version 3.7.3, but current environment version is 3.7.6.
^C
[2020-09-23 00:01:12 +0000] [1] [INFO] Handling signal: int
[2020-09-23 00:01:12 +0000] [14] [INFO] Worker exiting (pid: 14)
[2020-09-23 00:01:12 +0000] [13] [INFO] Worker exiting (pid: 13)
[2020-09-23 00:01:12 +0000] [12] [INFO] Worker exiting (pid: 12)

Load saved BentoService

bentoml.load is the API for loading a BentoML packaged model in python:

In [14]:
from bentoml import load

svc = load(saved_path)
print(svc.predict(df.iloc[0:1]))

[2020-10-01 14:45:29,278] WARNING - Saved BentoService bundle version mismatch: loading BentoService bundle create with BentoML version 0.9.1, but loading from BentoML version 0.9.1+1.g0655cf16.dirty
[2020-10-01 14:45:29,279] WARNING - Module `tabular_csv` already loaded, using existing imported module.
[2020-10-01 14:45:29,286] WARNING - pip package requirement pandas already exist
[2020-10-01 14:45:29,287] WARNING - BentoML by default does not include spacy and torchvision package when using FastaiModelArtifact. To make sure BentoML bundle those packages if they are required for your model, either import those packages in BentoService definition file or manually add them via `@env(pip_packages=['torchvision'])` when defining a BentoService
[2020-10-01 14:45:29,288] WARNING - pip package requirement torch already exist
[2020-10-01 14:45:29,290] WARNING - pip package requirement fastai<2.0.0 already exist
['>=50k']

Launch inference job from CLI

BentoML cli supports loading and running a packaged model from CLI. With the DataframeInput adapter, the CLI command supports reading input Dataframe data from CLI argument or local csv or json files:

In [ ]:
!wget https://raw.githubusercontent.com/bentoml/gallery/master/fast-ai/salary-range-prediction/test.csv
!bentoml run FastaiTabularModel:latest predict \
    --input-file test.csv

Deployment Options

If you are at a small team with limited engineering or DevOps resources, try out automated deployment with BentoML CLI, currently supporting AWS Lambda, AWS SageMaker, and Azure Functions:

If the cloud platform you are working with is not on the list above, try out these step-by-step guide on manually deploying BentoML packaged model to cloud platforms:

Lastly, if you have a DevOps or ML Engineering team who's operating a Kubernetes or OpenShift cluster, use the following guides as references for implementating your deployment strategy:

In [ ]: