Fastai 2 medical imaging

BentoML makes moving trained ML models to production easy:

  • Package models trained with any ML framework and reproduce them for model serving in production
  • Deploy anywhere for online API serving or offline batch serving
  • High-Performance API model server with adaptive micro-batching support
  • Central hub for managing models and deployment process via Web UI and APIs
  • Modular and flexible design making it adaptable to your infrastrcuture

BentoML is a framework for serving, managing, and deploying machine learning models. It is aiming to bridge the gap between Data Science and DevOps, and enable teams to deliver prediction services in a fast, repeatable, and scalable way.

Before reading this example project, be sure to check out the Getting started guide to learn about the basic concepts in BentoML.

This notebook demonstrates using BentoML to deploy Fastai2 medical image classification model as API server.

Impression

In [1]:
%reload_ext autoreload
%autoreload 2
%matplotlib inline
In [23]:
!pip install -U -q fastai>=2.0.13 pydicom>=1.4.0 nbdev>=1.0.0 kornia>=0.4.0 fastcore>=1.0.13
WARNING: You are using pip version 20.2.2; however, version 20.2.3 is available.
You should consider upgrading via the '/usr/local/anaconda3/envs/dev-py3/bin/python -m pip install --upgrade pip' command.
In [2]:
from fastai.data.block import DataBlock, CategoryBlock
from fastai.data.external import untar_data, URLs
from fastai.data.transforms import RandomSplitter
from fastai.metrics import accuracy
from fastai.vision.data import ImageBlock
from fastai.medical.imaging import get_dicom_files, dcmread, PILDicom
from fastai.vision.augment import aug_transforms
from fastai.vision.learner import cnn_learner
from fastai.vision.models import resnet34
# from fastai.basics import *
# from fastai.callback.all import *
# from fastai.vision.all import *
# from fastai.medical.imaging import *


import pydicom

import pandas as pd

import torch
import bentoml.utils.cloudpickle as cloudpickle
In [3]:
pneumothorax_source = untar_data(URLs.SIIM_SMALL)
In [4]:
items = get_dicom_files(pneumothorax_source/f"train/")
trn, val = RandomSplitter()(items)
In [5]:
patient = 3
xray_sample = dcmread(items[patient])
xray_sample.show()
In [6]:
df = pd.read_csv(pneumothorax_source/f"labels.csv")
df.head()
Out[6]:
file label
0 train/No Pneumothorax/000000.dcm No Pneumothorax
1 train/Pneumothorax/000001.dcm Pneumothorax
2 train/No Pneumothorax/000002.dcm No Pneumothorax
3 train/Pneumothorax/000003.dcm Pneumothorax
4 train/Pneumothorax/000004.dcm Pneumothorax
In [9]:
%%writefile datablock_utils.py

from fastai.data.external import untar_data, URLs

pneumothorax_source = untar_data(URLs.SIIM_SMALL)

def get_x(x):
    return pneumothorax_source/f"{x[0]}"

def get_y(x):
    return x[1]
Overwriting datablock_utils.py
In [10]:
from datablock_utils import get_x, get_y

pneumothorax = DataBlock(blocks=(ImageBlock(cls=PILDicom), CategoryBlock),
                   #get_x=lambda x:pneumothorax_source/f"{x[0]}",
                   #get_y=lambda x:x[1],
                   get_x=get_x,
                   get_y=get_y,
                   batch_tfms=aug_transforms(size=224))
In [11]:
dls = pneumothorax.dataloaders(df.values)
dls.show_batch(max_n=16)
In [12]:
learn = cnn_learner(dls, resnet34, metrics=accuracy)
learn.fit_one_cycle(1)
epoch train_loss valid_loss accuracy time
0 1.422205 0.796621 0.680000 01:03
In [13]:
learn.predict(pneumothorax_source/f"train/Pneumothorax/000004.dcm")
Out[13]:
('No Pneumothorax', tensor(0), tensor([0.6964, 0.3036]))
In [14]:
%%writefile medical_image_service.py
from bentoml.frameworks.fastai import FastaiModelArtifact
from bentoml.adapters import FileInput
from fastcore.utils import tuplify, detuplify

import bentoml
import datablock_utils

@bentoml.artifacts([FastaiModelArtifact('learner')])
@bentoml.env(infer_pip_packages=True)
class FastaiMedicalImagingService(bentoml.BentoService):

    @bentoml.api(input=FileInput(), batch=True)
    def predict(self, files):
        files = [i.read() for i in files]
        dl = self.artifacts.learner.dls.test_dl(files, rm_type_tfms=None, num_workers=0)
        inp, preds, _, dec_preds = self.artifacts.learner.get_preds(dl=dl, with_input=True, with_decoded=True)
        i = getattr(self.artifacts.learner.dls, 'n_inp', -1)
        inp = (inp,)
        dec_list = self.artifacts.learner.dls.decode_batch(inp + tuplify(dec_preds))
        res = []
        for dec in dec_list:
            dec_inp, dec_targ = map(detuplify, [dec[:i], dec[i:]])
            res.append(dec_targ)
        return res
Overwriting medical_image_service.py
In [16]:
import bentoml

# from fastcore.utils import remove_patches_path
from medical_image_service import FastaiMedicalImagingService

learn.metrics = []

svc = FastaiMedicalImagingService()
svc.pack('learner', learn)

# Fastcore modify Pathlib module from python, need to remove the patch in order to save
# with remove_patches_path():
saved_path = svc.save()
[2020-10-13 15:34:36,812] WARNING - Using BentoML installed in `editable` model, the local BentoML repository including all code changes will be packaged together with saved bundle created, under the './bundled_pip_dependencies' directory of the saved bundle.
[2020-10-13 15:34:37,365] INFO - Using default docker base image: `None` specified inBentoML config file or env var. User must make sure that the docker base image either has Python 3.7 or conda installed.
[2020-10-13 15:34:37,367] WARNING - BentoML by default does not include spacy and torchvision package when using Fastai2ModelArtifact. To make sure BentoML bundle those packages if they are required for your model, either import those packages in BentoService definition file or manually add them via `@env(pip_packages=['torchvision'])` when defining a BentoService
[2020-10-13 15:34:39,160] INFO - Detected non-PyPI-released BentoML installed, copying local BentoML modulefiles to target saved bundle path..
warning: no previously-included files matching '*~' found anywhere in distribution
warning: no previously-included files matching '*.pyo' found anywhere in distribution
warning: no previously-included files matching '.git' found anywhere in distribution
warning: no previously-included files matching '.ipynb_checkpoints' found anywhere in distribution
warning: no previously-included files matching '__pycache__' found anywhere in distribution
no previously-included directories found matching 'e2e_tests'
no previously-included directories found matching 'tests'
no previously-included directories found matching 'benchmark'
UPDATING BentoML-0.9.1+7.g25020513.dirty/bentoml/_version.py
set BentoML-0.9.1+7.g25020513.dirty/bentoml/_version.py to '0.9.1+7.g25020513.dirty'
[2020-10-13 15:34:43,792] INFO - BentoService bundle 'FastaiMedicalImagingService:20201013153437_51E59E' saved to: /Users/bozhaoyu/bentoml/repository/FastaiMedicalImagingService/20201013153437_51E59E
In [16]:
bento_tag = f'{svc.name}:{svc.version}'

REST API Model Serving

To start a REST API model server with the BentoService saved above, use the bentoml serve command:

In [ ]:
!bentoml serve FastaiMedicalImagingService:latest
[2020-10-01 14:42:51,187] INFO - Getting latest version FastaiMedicalImagingService:20201001144041_3F01EF
[2020-10-01 14:42:51,187] INFO - Starting BentoML API server in development mode..
[2020-10-01 14:42:51,368] WARNING - Using BentoML installed in `editable` model, the local BentoML repository including all code changes will be packaged together with saved bundle created, under the './bundled_pip_dependencies' directory of the saved bundle.
[2020-10-01 14:42:51,383] WARNING - Saved BentoService bundle version mismatch: loading BentoService bundle create with BentoML version 0.9.1, but loading from BentoML version 0.9.1+1.g0655cf16.dirty
[2020-10-01 14:42:53,214] WARNING - pip package requirement `fastai2` not found in current python environment
[2020-10-01 14:42:53,214] INFO - Using default docker base image: `None` specified inBentoML config file or env var. User must make sure that the docker base image either has Python 3.7 or conda installed.
[2020-10-01 14:42:54,038] WARNING - BentoML by default does not include spacy and torchvision package when using Fastai2ModelArtifact. To make sure BentoML bundle those packages if they are required for your model, either import those packages in BentoService definition file or manually add them via `@env(pip_packages=['torchvision'])` when defining a BentoService
[2020-10-01 14:42:54,039] WARNING - pip package requirement torch already exist
 * Serving Flask app "FastaiMedicalImagingService" (lazy loading)
 * Environment: production
   WARNING: This is a development server. Do not use it in a production deployment.
   Use a production WSGI server instead.
 * Debug mode: off
 * Running on http://127.0.0.1:5000/ (Press CTRL+C to quit)
[2020-10-01 14:43:05,347] INFO - {'service_name': 'FastaiMedicalImagingService', 'service_version': '20201001144041_3F01EF', 'api': 'predict', 'task': {'data': {'name': '000004.dcm'}, 'task_id': '4dac05a7-d960-4a6a-b6af-5cecac6aa3f9', 'http_headers': (('Host', 'localhost:5000'), ('User-Agent', 'curl/7.65.3'), ('Accept', '*/*'), ('Content-Length', '129541'), ('Content-Type', 'multipart/form-data; boundary=------------------------d792461a9363a4e5'), ('Expect', '100-continue'))}, 'result': {'data': '"No Pneumothorax"', 'http_status': 200, 'http_headers': (('Content-Type', 'application/json'),)}, 'request_id': '4dac05a7-d960-4a6a-b6af-5cecac6aa3f9'}
127.0.0.1 - - [01/Oct/2020 14:43:05] "POST /predict HTTP/1.1" 200 -
WARNING: Logging before flag parsing goes to stderr.
I1001 14:43:05.348261 4460494272 _internal.py:122] 127.0.0.1 - - [01/Oct/2020 14:43:05] "POST /predict HTTP/1.1" 200 -

If you are running this notebook from Google Colab, you can start the dev server with --run-with-ngrok option, to gain acccess to the API endpoint via a public endpoint managed by ngrok:

In [ ]:
!bentoml serve FastaiMedicalImagingService:atest --run-with-ngrok

Send prediction request to the REST API server

Go visit http://127.0.0.1:5000/ from your browser, click /predict -> Try it out -> Choose File -> Execute to sumbit an image from your computer

or Using curl request:

curl -X POST "http://localhost:5000/predict" \
  -F image=@000004.dcm

Containerize model server with Docker

One common way of distributing this model API server for production deployment, is via Docker containers. And BentoML provides a convenient way to do that.

Note that docker is not available in Google Colab. You will need to download and run this notebook locally to try out this containerization with docker feature.

If you already have docker configured, simply run the follow command to product a docker container serving the IrisClassifier prediction service created above:

In [23]:
!bentoml containerize FastaiMedicalImagingService:latest
[2020-09-22 20:59:55,325] INFO - Getting latest version FastaiMedicalImagingService:20200922205913_3AE93C
Found Bento: /Users/bozhaoyu/bentoml/repository/FastaiMedicalImagingService/20200922205913_3AE93C
[2020-09-22 20:59:55,363] WARNING - Using BentoML installed in `editable` model, the local BentoML repository including all code changes will be packaged together with saved bundle created, under the './bundled_pip_dependencies' directory of the saved bundle.
[2020-09-22 20:59:55,380] WARNING - Saved BentoService bundle version mismatch: loading BentoService bundle create with BentoML version 0.9.0.pre, but loading from BentoML version 0.9.0.pre+3.gcebf2015.dirty
Tag not specified, using tag parsed from BentoService: 'fastaimedicalimagingservice:20200922205913_3AE93C'
Building Docker image fastaimedicalimagingservice:20200922205913_3AE93C from FastaiMedicalImagingService:latest 
-we in here
processed docker file
(None, None)
root in create archive /Users/bozhaoyu/bentoml/repository/FastaiMedicalImagingService/20200922205913_3AE93C ['Dockerfile', 'FastaiMedicalImagingService', 'FastaiMedicalImagingService/__init__.py', 'FastaiMedicalImagingService/__pycache__', 'FastaiMedicalImagingService/__pycache__/datablock_utils.cpython-37.pyc', 'FastaiMedicalImagingService/__pycache__/medical_image_service.cpython-37.pyc', 'FastaiMedicalImagingService/artifacts', 'FastaiMedicalImagingService/artifacts/__init__.py', 'FastaiMedicalImagingService/artifacts/learner.pkl', 'FastaiMedicalImagingService/bentoml.yml', 'FastaiMedicalImagingService/datablock_utils.py', 'FastaiMedicalImagingService/medical_image_service.py', 'MANIFEST.in', 'README.md', 'bentoml-init.sh', 'bentoml.yml', 'bundled_pip_dependencies', 'bundled_pip_dependencies/BentoML-0.9.0rc0+3.gcebf2015.dirty.tar.gz', 'docker-entrypoint.sh', 'environment.yml', 'python_version', 'requirements.txt', 'setup.py']
|about to build
about to upgrade params
check each param and update
if use config proxy
if buildargs
if shmsize
if labels
if cache from
if target
if network_mode
if squash
if extra hosts is not None
if platform is not None
if isolcation is not None
if context is not None
setting auth {'Content-Type': 'application/tar'}
|docker build <tempfile._TemporaryFileWrapper object at 0x7fc33e4b4c88> {'t': 'fastaimedicalimagingservice:20200922205913_3AE93C', 'remote': None, 'q': False, 'nocache': False, 'rm': False, 'forcerm': False, 'pull': False, 'dockerfile': (None, None)}
/docker response <Response [200]>
context closes
print responses
Step 1/15 : FROM bentoml/model-server:0.9.0.pre
 ---> a25066aa8b0e
Step 2/15 : ARG EXTRA_PIP_INSTALL_ARGS=
 ---> Using cache
 ---> 315719b8980e
Step 3/15 : ENV EXTRA_PIP_INSTALL_ARGS $EXTRA_PIP_INSTALL_ARGS
 ---> Using cache
 ---> a3b6c8107d94
Step 4/15 : COPY environment.yml requirements.txt setup.sh* bentoml-init.sh python_version* /bento/
\ ---> db408bdb1479
Step 5/15 : WORKDIR /bento
- ---> Running in 7efd5ba7f606
/ ---> f3679fcf05ef
Step 6/15 : RUN chmod +x /bento/bentoml-init.sh
| ---> Running in 66ba1c2af561
\ ---> 6a3880f24c5c
Step 7/15 : RUN if [ -f /bento/bentoml-init.sh ]; then bash -c /bento/bentoml-init.sh; fi
 ---> Running in 6cadd549b80c
|+++ dirname /bento/bentoml-init.sh

++ cd /bento
++ pwd -P

+ SAVED_BUNDLE_PATH=/bento
+ cd /bento
+ '[' -f ./setup.sh ']'

+ '[' -f ./python_version ']'

++ cat ./python_version

+ PY_VERSION_SAVED=3.7.3
+ DESIRED_PY_VERSION=3.7

++ python -c 'import sys; print(f"{sys.version_info.major}.{sys.version_info.minor}")'

\+ CURRENT_PY_VERSION=3.7
+ [[ 3.7 == \3\.\7 ]]
+ echo 'Python Version in docker base image 3.7 matches requirement python=3.7. Skipping.'
+ command -v conda

Python Version in docker base image 3.7 matches requirement python=3.7. Skipping.
+ echo 'Updating conda base environment with environment.yml'
+ conda env update -n base -f ./environment.yml

Updating conda base environment with environment.yml
|Collecting package metadata (repodata.json): ...working... 
\done
Solving environment: ...working... 
-done
/
Downloading and Extracting Packages
cffi-1.14.3          | 223 KB    |            |   0% 
cffi-1.14.3          | 223 KB    | 7          |   7% 
cffi-1.14.3          | 223 KB    | ########## | 100% 

openssl-1.1.1h       | 2.1 MB    |            |   0% 
openssl-1.1.1h       | 2.1 MB    | ##3        |  24% 
openssl-1.1.1h       | 2.1 MB    | ########## | 100% 
openssl-1.1.1h       | 2.1 MB    | ########## | 100% 

python_abi-3.7       | 4 KB      |            |   0% 
python_abi-3.7       | 4 KB      | ########## | 100% 

libffi-3.2.1         | 47 KB     |            |   0% 
libffi-3.2.1         | 47 KB     | ########## | 100% 

pip-20.2.3           | 1.1 MB    |            |   0% 
pip-20.2.3           | 1.1 MB    | ########## | 100% 
pip-20.2.3           | 1.1 MB    | ########## | 100% 

ca-certificates-2020 | 145 KB    |            |   0% 
ca-certificates-2020 | 145 KB    | ########## | 100% 

certifi-2020.6.20    | 151 KB    |            |   0% 
certifi-2020.6.20    | 151 KB    | ########## | 100% 

Preparing transaction: 
...working... 
-done
Verifying transaction: 
...working... 
/done
Executing transaction: 
...working... 
|done
-#
# To activate this environment, use
#
#     $ conda activate base
#
# To deactivate an active environment, use
#
#     $ conda deactivate
/+ pip install -r ./requirements.txt --no-cache-dir

/Requirement already satisfied: bentoml==0.9.0.pre in /opt/conda/lib/python3.7/site-packages (from -r ./requirements.txt (line 1)) (0.9.0rc0)
-Collecting torch==1.6.0
/  Downloading torch-1.6.0-cp37-cp37m-manylinux1_x86_64.whl (748.8 MB)
\Collecting fastai2
-  Downloading fastai2-0.0.30-py3-none-any.whl (179 kB)
/Collecting fastcore==1.0.13
  Downloading fastcore-1.0.13-py3-none-any.whl (40 kB)
|Collecting fastai>=2.0.0
\  Downloading fastai-2.0.13-py3-none-any.whl (355 kB)
Requirement already satisfied: aiohttp in /opt/conda/lib/python3.7/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (3.6.2)
Requirement already satisfied: multidict in /opt/conda/lib/python3.7/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (4.7.6)
Requirement already satisfied: py-zipkin in /opt/conda/lib/python3.7/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (0.20.0)
Requirement already satisfied: click>=7.0 in /opt/conda/lib/python3.7/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (7.1.2)
Requirement already satisfied: boto3 in /opt/conda/lib/python3.7/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (1.15.2)
Requirement already satisfied: python-json-logger in /opt/conda/lib/python3.7/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (0.1.11)
Requirement already satisfied: sqlalchemy>=1.3.0 in /opt/conda/lib/python3.7/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (1.3.19)
-Requirement already satisfied: flask in /opt/conda/lib/python3.7/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (1.1.2)
Requirement already satisfied: configparser in /opt/conda/lib/python3.7/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (5.0.0)
Requirement already satisfied: certifi in /opt/conda/lib/python3.7/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (2020.6.20)
Requirement already satisfied: packaging in /opt/conda/lib/python3.7/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (20.4)
Requirement already satisfied: humanfriendly in /opt/conda/lib/python3.7/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (8.2)
Requirement already satisfied: python-dateutil<3.0.0,>=2.7.3 in /opt/conda/lib/python3.7/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (2.8.1)
Requirement already satisfied: grpcio<=1.27.2 in /opt/conda/lib/python3.7/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (1.27.2)
Requirement already satisfied: cerberus in /opt/conda/lib/python3.7/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (1.3.2)
Requirement already satisfied: ruamel.yaml>=0.15.0 in /opt/conda/lib/python3.7/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (0.15.87)
Requirement already satisfied: prometheus-client in /opt/conda/lib/python3.7/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (0.8.0)
Requirement already satisfied: psutil in /opt/conda/lib/python3.7/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (5.7.2)
Requirement already satisfied: tabulate in /opt/conda/lib/python3.7/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (0.8.7)
Requirement already satisfied: sqlalchemy-utils<0.36.8 in /opt/conda/lib/python3.7/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (0.36.7)
/Requirement already satisfied: requests in /opt/conda/lib/python3.7/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (2.24.0)
Requirement already satisfied: docker in /opt/conda/lib/python3.7/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (4.3.1)
Requirement already satisfied: numpy in /opt/conda/lib/python3.7/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (1.19.2)
Requirement already satisfied: alembic in /opt/conda/lib/python3.7/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (1.4.3)
Requirement already satisfied: gunicorn in /opt/conda/lib/python3.7/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (20.0.4)
|Requirement already satisfied: protobuf>=3.6.0 in /opt/conda/lib/python3.7/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (3.13.0)
Collecting future
  Downloading future-0.18.2.tar.gz (829 kB)
-Collecting torchvision>=0.7
/  Downloading torchvision-0.7.0-cp37-cp37m-manylinux1_x86_64.whl (5.9 MB)
/Collecting scikit-learn
  Downloading scikit_learn-0.23.2-cp37-cp37m-manylinux1_x86_64.whl (6.8 MB)
\Requirement already satisfied: pip in /opt/conda/lib/python3.7/site-packages (from fastai2->-r ./requirements.txt (line 3)) (20.2.3)
/Collecting spacy
|  Downloading spacy-2.3.2-cp37-cp37m-manylinux1_x86_64.whl (9.9 MB)
/Collecting pandas
  Downloading pandas-1.1.2-cp37-cp37m-manylinux1_x86_64.whl (10.5 MB)
/Collecting fastprogress>=0.2.4
  Downloading fastprogress-1.0.0-py3-none-any.whl (12 kB)
-Collecting matplotlib
  Downloading matplotlib-3.3.2-cp37-cp37m-manylinux1_x86_64.whl (11.6 MB)
/Collecting pyyaml
  Downloading PyYAML-5.3.1.tar.gz (269 kB)
\Collecting pillow
  Downloading Pillow-7.2.0-cp37-cp37m-manylinux1_x86_64.whl (2.2 MB)
\Collecting scipy
-  Downloading scipy-1.5.2-cp37-cp37m-manylinux1_x86_64.whl (25.9 MB)
\Requirement already satisfied: yarl<2.0,>=1.0 in /opt/conda/lib/python3.7/site-packages (from aiohttp->bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (1.5.1)
Requirement already satisfied: chardet<4.0,>=2.0 in /opt/conda/lib/python3.7/site-packages (from aiohttp->bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (3.0.4)
Requirement already satisfied: async-timeout<4.0,>=3.0 in /opt/conda/lib/python3.7/site-packages (from aiohttp->bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (3.0.1)
Requirement already satisfied: attrs>=17.3.0 in /opt/conda/lib/python3.7/site-packages (from aiohttp->bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (20.2.0)
Requirement already satisfied: thriftpy2>=0.4.0 in /opt/conda/lib/python3.7/site-packages (from py-zipkin->bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (0.4.11)
-Requirement already satisfied: six in /opt/conda/lib/python3.7/site-packages (from py-zipkin->bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (1.15.0)
Requirement already satisfied: s3transfer<0.4.0,>=0.3.0 in /opt/conda/lib/python3.7/site-packages (from boto3->bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (0.3.3)
Requirement already satisfied: jmespath<1.0.0,>=0.7.1 in /opt/conda/lib/python3.7/site-packages (from boto3->bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (0.10.0)
Requirement already satisfied: botocore<1.19.0,>=1.18.2 in /opt/conda/lib/python3.7/site-packages (from boto3->bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (1.18.2)
Requirement already satisfied: Jinja2>=2.10.1 in /opt/conda/lib/python3.7/site-packages (from flask->bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (2.11.2)
Requirement already satisfied: itsdangerous>=0.24 in /opt/conda/lib/python3.7/site-packages (from flask->bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (1.1.0)
Requirement already satisfied: Werkzeug>=0.15 in /opt/conda/lib/python3.7/site-packages (from flask->bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (1.0.1)
Requirement already satisfied: pyparsing>=2.0.2 in /opt/conda/lib/python3.7/site-packages (from packaging->bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (2.4.7)
Requirement already satisfied: setuptools in /opt/conda/lib/python3.7/site-packages (from cerberus->bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (49.6.0.post20200814)
Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /opt/conda/lib/python3.7/site-packages (from requests->bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (1.25.10)
Requirement already satisfied: idna<3,>=2.5 in /opt/conda/lib/python3.7/site-packages (from requests->bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (2.10)
Requirement already satisfied: websocket-client>=0.32.0 in /opt/conda/lib/python3.7/site-packages (from docker->bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (0.57.0)
Requirement already satisfied: Mako in /opt/conda/lib/python3.7/site-packages (from alembic->bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (1.1.3)
Requirement already satisfied: python-editor>=0.3 in /opt/conda/lib/python3.7/site-packages (from alembic->bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (1.0.4)
/Collecting joblib>=0.11
  Downloading joblib-0.16.0-py3-none-any.whl (300 kB)
|Collecting threadpoolctl>=2.0.0
  Downloading threadpoolctl-2.1.0-py3-none-any.whl (12 kB)
\Requirement already satisfied: tqdm<5.0.0,>=4.38.0 in /opt/conda/lib/python3.7/site-packages (from spacy->fastai2->-r ./requirements.txt (line 3)) (4.48.2)
Collecting blis<0.5.0,>=0.4.0
-  Downloading blis-0.4.1-cp37-cp37m-manylinux1_x86_64.whl (3.7 MB)
|Collecting preshed<3.1.0,>=3.0.2
  Downloading preshed-3.0.2-cp37-cp37m-manylinux1_x86_64.whl (118 kB)
\Collecting cymem<2.1.0,>=2.0.2
  Downloading cymem-2.0.3-cp37-cp37m-manylinux1_x86_64.whl (32 kB)
Collecting plac<1.2.0,>=0.9.6
-  Downloading plac-1.1.3-py2.py3-none-any.whl (20 kB)
|Collecting thinc==7.4.1
\  Downloading thinc-7.4.1-cp37-cp37m-manylinux1_x86_64.whl (2.1 MB)
/Collecting wasabi<1.1.0,>=0.4.0
  Downloading wasabi-0.8.0-py3-none-any.whl (23 kB)
|Collecting murmurhash<1.1.0,>=0.28.0
  Downloading murmurhash-1.0.2-cp37-cp37m-manylinux1_x86_64.whl (19 kB)
Collecting catalogue<1.1.0,>=0.0.7
\  Downloading catalogue-1.0.0-py2.py3-none-any.whl (7.7 kB)
-Collecting srsly<1.1.0,>=1.0.2
  Downloading srsly-1.0.2-cp37-cp37m-manylinux1_x86_64.whl (185 kB)
|Collecting pytz>=2017.2
\  Downloading pytz-2020.1-py2.py3-none-any.whl (510 kB)
-Collecting kiwisolver>=1.0.1
  Downloading kiwisolver-1.2.0-cp37-cp37m-manylinux1_x86_64.whl (88 kB)
Collecting cycler>=0.10
/  Downloading cycler-0.10.0-py2.py3-none-any.whl (6.5 kB)
Requirement already satisfied: typing-extensions>=3.7.4; python_version < "3.8" in /opt/conda/lib/python3.7/site-packages (from yarl<2.0,>=1.0->aiohttp->bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (3.7.4.3)
Requirement already satisfied: ply<4.0,>=3.4 in /opt/conda/lib/python3.7/site-packages (from thriftpy2>=0.4.0->py-zipkin->bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (3.11)
Requirement already satisfied: MarkupSafe>=0.23 in /opt/conda/lib/python3.7/site-packages (from Jinja2>=2.10.1->flask->bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (1.1.1)
|Collecting importlib-metadata>=0.20; python_version < "3.8"
  Downloading importlib_metadata-2.0.0-py2.py3-none-any.whl (31 kB)
\Collecting zipp>=0.5
  Downloading zipp-3.2.0-py3-none-any.whl (5.1 kB)
-Building wheels for collected packages: future, pyyaml
  Building wheel for future (setup.py): started
|  Building wheel for future (setup.py): finished with status 'done'
  Created wheel for future: filename=future-0.18.2-py3-none-any.whl size=491059 sha256=dec071dcb8bdef442079d7ff5c364ef85fa9de80326216327406a08c4762eef6
  Stored in directory: /tmp/pip-ephem-wheel-cache-mcjg0e8a/wheels/56/b0/fe/4410d17b32f1f0c3cf54cdfb2bc04d7b4b8f4ae377e2229ba0
  Building wheel for pyyaml (setup.py): started
-  Building wheel for pyyaml (setup.py): finished with status 'done'
  Created wheel for pyyaml: filename=PyYAML-5.3.1-cp37-cp37m-linux_x86_64.whl size=44619 sha256=90ca2ee45095ac0999e90fe96a3ea9b6136ae85e5b867a13b810d7e120613ed8
  Stored in directory: /tmp/pip-ephem-wheel-cache-mcjg0e8a/wheels/5e/03/1e/e1e954795d6f35dfc7b637fe2277bff021303bd9570ecea653
Successfully built future pyyaml
\Installing collected packages: future, torch, pillow, torchvision, joblib, scipy, threadpoolctl, scikit-learn, blis, murmurhash, cymem, preshed, plac, srsly, wasabi, zipp, importlib-metadata, catalogue, thinc, spacy, pytz, pandas, fastprogress, fastcore, kiwisolver, cycler, matplotlib, pyyaml, fastai2, fastai
-Successfully installed blis-0.4.1 catalogue-1.0.0 cycler-0.10.0 cymem-2.0.3 fastai-2.0.13 fastai2-0.0.30 fastcore-1.0.13 fastprogress-1.0.0 future-0.18.2 importlib-metadata-2.0.0 joblib-0.16.0 kiwisolver-1.2.0 matplotlib-3.3.2 murmurhash-1.0.2 pandas-1.1.2 pillow-7.2.0 plac-1.1.3 preshed-3.0.2 pytz-2020.1 pyyaml-5.3.1 scikit-learn-0.23.2 scipy-1.5.2 spacy-2.3.2 srsly-1.0.2 thinc-7.4.1 threadpoolctl-2.1.0 torch-1.6.0 torchvision-0.7.0 wasabi-0.8.0 zipp-3.2.0
\ ---> b86ba0536d5d
Step 8/15 : COPY . /bento
\ ---> 6429ecdb105b
Step 9/15 : RUN if [ -d /bento/bundled_pip_dependencies ]; then pip install -U bundled_pip_dependencies/* ;fi
- ---> Running in 21329d21a713
\Processing ./bundled_pip_dependencies/BentoML-0.9.0rc0+3.gcebf2015.dirty.tar.gz
\  Installing build dependencies: started
-  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
/  Getting requirements to build wheel: finished with status 'done'
    Preparing wheel metadata: started
\    Preparing wheel metadata: finished with status 'done'
/Requirement already satisfied, skipping upgrade: ruamel.yaml>=0.15.0 in /opt/conda/lib/python3.7/site-packages (from BentoML==0.9.0rc0+3.gcebf2015.dirty) (0.15.87)
Requirement already satisfied, skipping upgrade: humanfriendly in /opt/conda/lib/python3.7/site-packages (from BentoML==0.9.0rc0+3.gcebf2015.dirty) (8.2)
Requirement already satisfied, skipping upgrade: numpy in /opt/conda/lib/python3.7/site-packages (from BentoML==0.9.0rc0+3.gcebf2015.dirty) (1.19.2)
Requirement already satisfied, skipping upgrade: python-json-logger in /opt/conda/lib/python3.7/site-packages (from BentoML==0.9.0rc0+3.gcebf2015.dirty) (0.1.11)
Requirement already satisfied, skipping upgrade: grpcio<=1.27.2 in /opt/conda/lib/python3.7/site-packages (from BentoML==0.9.0rc0+3.gcebf2015.dirty) (1.27.2)
Requirement already satisfied, skipping upgrade: python-dateutil<3.0.0,>=2.7.3 in /opt/conda/lib/python3.7/site-packages (from BentoML==0.9.0rc0+3.gcebf2015.dirty) (2.8.1)
Requirement already satisfied, skipping upgrade: click>=7.0 in /opt/conda/lib/python3.7/site-packages (from BentoML==0.9.0rc0+3.gcebf2015.dirty) (7.1.2)
Requirement already satisfied, skipping upgrade: multidict in /opt/conda/lib/python3.7/site-packages (from BentoML==0.9.0rc0+3.gcebf2015.dirty) (4.7.6)
Requirement already satisfied, skipping upgrade: psutil in /opt/conda/lib/python3.7/site-packages (from BentoML==0.9.0rc0+3.gcebf2015.dirty) (5.7.2)
Requirement already satisfied, skipping upgrade: protobuf>=3.6.0 in /opt/conda/lib/python3.7/site-packages (from BentoML==0.9.0rc0+3.gcebf2015.dirty) (3.13.0)
Requirement already satisfied, skipping upgrade: gunicorn in /opt/conda/lib/python3.7/site-packages (from BentoML==0.9.0rc0+3.gcebf2015.dirty) (20.0.4)
Requirement already satisfied, skipping upgrade: sqlalchemy>=1.3.0 in /opt/conda/lib/python3.7/site-packages (from BentoML==0.9.0rc0+3.gcebf2015.dirty) (1.3.19)
Requirement already satisfied, skipping upgrade: docker in /opt/conda/lib/python3.7/site-packages (from BentoML==0.9.0rc0+3.gcebf2015.dirty) (4.3.1)
Requirement already satisfied, skipping upgrade: sqlalchemy-utils<0.36.8 in /opt/conda/lib/python3.7/site-packages (from BentoML==0.9.0rc0+3.gcebf2015.dirty) (0.36.7)
|Requirement already satisfied, skipping upgrade: configparser in /opt/conda/lib/python3.7/site-packages (from BentoML==0.9.0rc0+3.gcebf2015.dirty) (5.0.0)
Requirement already satisfied, skipping upgrade: aiohttp in /opt/conda/lib/python3.7/site-packages (from BentoML==0.9.0rc0+3.gcebf2015.dirty) (3.6.2)
\Requirement already satisfied, skipping upgrade: py-zipkin in /opt/conda/lib/python3.7/site-packages (from BentoML==0.9.0rc0+3.gcebf2015.dirty) (0.20.0)
Requirement already satisfied, skipping upgrade: packaging in /opt/conda/lib/python3.7/site-packages (from BentoML==0.9.0rc0+3.gcebf2015.dirty) (20.4)
Requirement already satisfied, skipping upgrade: flask in /opt/conda/lib/python3.7/site-packages (from BentoML==0.9.0rc0+3.gcebf2015.dirty) (1.1.2)
Requirement already satisfied, skipping upgrade: tabulate in /opt/conda/lib/python3.7/site-packages (from BentoML==0.9.0rc0+3.gcebf2015.dirty) (0.8.7)
Requirement already satisfied, skipping upgrade: alembic in /opt/conda/lib/python3.7/site-packages (from BentoML==0.9.0rc0+3.gcebf2015.dirty) (1.4.3)
Requirement already satisfied, skipping upgrade: prometheus-client in /opt/conda/lib/python3.7/site-packages (from BentoML==0.9.0rc0+3.gcebf2015.dirty) (0.8.0)
Requirement already satisfied, skipping upgrade: cerberus in /opt/conda/lib/python3.7/site-packages (from BentoML==0.9.0rc0+3.gcebf2015.dirty) (1.3.2)
Requirement already satisfied, skipping upgrade: certifi in /opt/conda/lib/python3.7/site-packages (from BentoML==0.9.0rc0+3.gcebf2015.dirty) (2020.6.20)
Requirement already satisfied, skipping upgrade: requests in /opt/conda/lib/python3.7/site-packages (from BentoML==0.9.0rc0+3.gcebf2015.dirty) (2.24.0)
Requirement already satisfied, skipping upgrade: boto3 in /opt/conda/lib/python3.7/site-packages (from BentoML==0.9.0rc0+3.gcebf2015.dirty) (1.15.2)
Requirement already satisfied, skipping upgrade: six>=1.5.2 in /opt/conda/lib/python3.7/site-packages (from grpcio<=1.27.2->BentoML==0.9.0rc0+3.gcebf2015.dirty) (1.15.0)
Requirement already satisfied, skipping upgrade: setuptools in /opt/conda/lib/python3.7/site-packages (from protobuf>=3.6.0->BentoML==0.9.0rc0+3.gcebf2015.dirty) (49.6.0.post20200814)
Requirement already satisfied, skipping upgrade: websocket-client>=0.32.0 in /opt/conda/lib/python3.7/site-packages (from docker->BentoML==0.9.0rc0+3.gcebf2015.dirty) (0.57.0)
Requirement already satisfied, skipping upgrade: async-timeout<4.0,>=3.0 in /opt/conda/lib/python3.7/site-packages (from aiohttp->BentoML==0.9.0rc0+3.gcebf2015.dirty) (3.0.1)
Requirement already satisfied, skipping upgrade: attrs>=17.3.0 in /opt/conda/lib/python3.7/site-packages (from aiohttp->BentoML==0.9.0rc0+3.gcebf2015.dirty) (20.2.0)
-Requirement already satisfied, skipping upgrade: yarl<2.0,>=1.0 in /opt/conda/lib/python3.7/site-packages (from aiohttp->BentoML==0.9.0rc0+3.gcebf2015.dirty) (1.5.1)
Requirement already satisfied, skipping upgrade: chardet<4.0,>=2.0 in /opt/conda/lib/python3.7/site-packages (from aiohttp->BentoML==0.9.0rc0+3.gcebf2015.dirty) (3.0.4)
Requirement already satisfied, skipping upgrade: thriftpy2>=0.4.0 in /opt/conda/lib/python3.7/site-packages (from py-zipkin->BentoML==0.9.0rc0+3.gcebf2015.dirty) (0.4.11)
Requirement already satisfied, skipping upgrade: pyparsing>=2.0.2 in /opt/conda/lib/python3.7/site-packages (from packaging->BentoML==0.9.0rc0+3.gcebf2015.dirty) (2.4.7)
Requirement already satisfied, skipping upgrade: Jinja2>=2.10.1 in /opt/conda/lib/python3.7/site-packages (from flask->BentoML==0.9.0rc0+3.gcebf2015.dirty) (2.11.2)
Requirement already satisfied, skipping upgrade: Werkzeug>=0.15 in /opt/conda/lib/python3.7/site-packages (from flask->BentoML==0.9.0rc0+3.gcebf2015.dirty) (1.0.1)
Requirement already satisfied, skipping upgrade: itsdangerous>=0.24 in /opt/conda/lib/python3.7/site-packages (from flask->BentoML==0.9.0rc0+3.gcebf2015.dirty) (1.1.0)
Requirement already satisfied, skipping upgrade: python-editor>=0.3 in /opt/conda/lib/python3.7/site-packages (from alembic->BentoML==0.9.0rc0+3.gcebf2015.dirty) (1.0.4)
Requirement already satisfied, skipping upgrade: Mako in /opt/conda/lib/python3.7/site-packages (from alembic->BentoML==0.9.0rc0+3.gcebf2015.dirty) (1.1.3)
Requirement already satisfied, skipping upgrade: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /opt/conda/lib/python3.7/site-packages (from requests->BentoML==0.9.0rc0+3.gcebf2015.dirty) (1.25.10)
/Requirement already satisfied, skipping upgrade: idna<3,>=2.5 in /opt/conda/lib/python3.7/site-packages (from requests->BentoML==0.9.0rc0+3.gcebf2015.dirty) (2.10)
Requirement already satisfied, skipping upgrade: botocore<1.19.0,>=1.18.2 in /opt/conda/lib/python3.7/site-packages (from boto3->BentoML==0.9.0rc0+3.gcebf2015.dirty) (1.18.2)
Requirement already satisfied, skipping upgrade: jmespath<1.0.0,>=0.7.1 in /opt/conda/lib/python3.7/site-packages (from boto3->BentoML==0.9.0rc0+3.gcebf2015.dirty) (0.10.0)
Requirement already satisfied, skipping upgrade: s3transfer<0.4.0,>=0.3.0 in /opt/conda/lib/python3.7/site-packages (from boto3->BentoML==0.9.0rc0+3.gcebf2015.dirty) (0.3.3)
Requirement already satisfied, skipping upgrade: typing-extensions>=3.7.4; python_version < "3.8" in /opt/conda/lib/python3.7/site-packages (from yarl<2.0,>=1.0->aiohttp->BentoML==0.9.0rc0+3.gcebf2015.dirty) (3.7.4.3)
Requirement already satisfied, skipping upgrade: ply<4.0,>=3.4 in /opt/conda/lib/python3.7/site-packages (from thriftpy2>=0.4.0->py-zipkin->BentoML==0.9.0rc0+3.gcebf2015.dirty) (3.11)
Requirement already satisfied, skipping upgrade: MarkupSafe>=0.23 in /opt/conda/lib/python3.7/site-packages (from Jinja2>=2.10.1->flask->BentoML==0.9.0rc0+3.gcebf2015.dirty) (1.1.1)
Building wheels for collected packages: BentoML
  Building wheel for BentoML (PEP 517): started
/  Building wheel for BentoML (PEP 517): finished with status 'done'
|  Created wheel for BentoML: filename=BentoML-0.9.0rc0+3.gcebf2015.dirty-py3-none-any.whl size=3064184 sha256=57f58ae44fcc4beff647947a6578f54e9e06fdff40c74e69e78f9a12a039345b
  Stored in directory: /root/.cache/pip/wheels/09/ed/45/f83b3e5c126046ae57f1c3767cd5bc58206dd4a099a8d9abac
Successfully built BentoML
\Installing collected packages: BentoML
  Attempting uninstall: BentoML
    Found existing installation: BentoML 0.9.0rc0
|    Uninstalling BentoML-0.9.0rc0:
/      Successfully uninstalled BentoML-0.9.0rc0
|Successfully installed BentoML-0.9.0rc0+3.gcebf2015.dirty
\ ---> e988028f7339
Step 10/15 : ENV PORT 5000
- ---> Running in 90641f72c254
/ ---> f93b8e55a16f
Step 11/15 : EXPOSE $PORT
 ---> Running in 479b42121c48
| ---> 10bd9c19e66a
Step 12/15 : COPY docker-entrypoint.sh /usr/local/bin/
- ---> 76669a6f7936
Step 13/15 : RUN chmod +x /usr/local/bin/docker-entrypoint.sh
 ---> Running in 84ca9ef61b7b
| ---> 59722cd589c9
Step 14/15 : ENTRYPOINT [ "docker-entrypoint.sh" ]
\ ---> Running in b3c070b2a3b1
- ---> 51a45e793832
Step 15/15 : CMD ["bentoml", "serve-gunicorn", "/bento"]
 ---> Running in 20dbe5c95e29
/ ---> b128509666c8
|Successfully built b128509666c8
Successfully tagged fastaimedicalimagingservice:20200922205913_3AE93C
Finished building fastaimedicalimagingservice:20200922205913_3AE93C from FastaiMedicalImagingService:latest
In [24]:
!docker run --rm -p 5000:5000 fastaimedicalimagingservice:20200922205913_3AE93C
[2020-09-23 04:04:10,232] INFO - Starting BentoML API server in production mode..
[2020-09-23 04:04:10,654] INFO - get_gunicorn_num_of_workers: 3, calculated by cpu count
[2020-09-23 04:04:10 +0000] [1] [INFO] Starting gunicorn 20.0.4
[2020-09-23 04:04:10 +0000] [1] [INFO] Listening at: http://0.0.0.0:5000 (1)
[2020-09-23 04:04:10 +0000] [1] [INFO] Using worker: sync
[2020-09-23 04:04:10 +0000] [11] [INFO] Booting worker with pid: 11
[2020-09-23 04:04:10 +0000] [12] [INFO] Booting worker with pid: 12
[2020-09-23 04:04:10 +0000] [13] [INFO] Booting worker with pid: 13
[2020-09-23 04:04:10,875] WARNING - Using BentoML installed in `editable` model, the local BentoML repository including all code changes will be packaged together with saved bundle created, under the './bundled_pip_dependencies' directory of the saved bundle.
[2020-09-23 04:04:10,893] WARNING - Using BentoML installed in `editable` model, the local BentoML repository including all code changes will be packaged together with saved bundle created, under the './bundled_pip_dependencies' directory of the saved bundle.
[2020-09-23 04:04:10,895] WARNING - Saved BentoService bundle version mismatch: loading BentoService bundle create with BentoML version 0.9.0.pre, but loading from BentoML version 0.9.0.pre+3.gcebf2015.dirty
[2020-09-23 04:04:10,896] WARNING - Saved BentoService Python version mismatch: loading BentoService bundle created with Python version 3.7.3, but current environment version is 3.7.6.
[2020-09-23 04:04:10,914] WARNING - Saved BentoService bundle version mismatch: loading BentoService bundle create with BentoML version 0.9.0.pre, but loading from BentoML version 0.9.0.pre+3.gcebf2015.dirty
[2020-09-23 04:04:10,915] WARNING - Saved BentoService Python version mismatch: loading BentoService bundle created with Python version 3.7.3, but current environment version is 3.7.6.
[2020-09-23 04:04:11,005] WARNING - Using BentoML installed in `editable` model, the local BentoML repository including all code changes will be packaged together with saved bundle created, under the './bundled_pip_dependencies' directory of the saved bundle.
[2020-09-23 04:04:11,028] WARNING - Saved BentoService bundle version mismatch: loading BentoService bundle create with BentoML version 0.9.0.pre, but loading from BentoML version 0.9.0.pre+3.gcebf2015.dirty
[2020-09-23 04:04:11,028] WARNING - Saved BentoService Python version mismatch: loading BentoService bundle created with Python version 3.7.3, but current environment version is 3.7.6.
^C
[2020-09-23 04:04:12 +0000] [1] [INFO] Handling signal: int
[2020-09-23 04:04:12 +0000] [11] [INFO] Worker exiting (pid: 11)
[2020-09-23 04:04:12 +0000] [13] [INFO] Worker exiting (pid: 13)
[2020-09-23 04:04:12 +0000] [12] [INFO] Worker exiting (pid: 12)

Launch inference job from CLI

BentoML cli supports loading and running a packaged model from CLI. With the DataframeInput adapter, the CLI command supports reading input Dataframe data from CLI argument or local csv or json files:

In [ ]:
!bentoml run FastaiMedicalImagingService:latest predict \
    --input-file /Users/bozhaoyu/.fastai/data/siim_small/train/Pneumothorax/000004.dcm

Load saved BentoService

bentoml.load is the API for loading a BentoML packaged model in python:

In [ ]:
from bentoml import load


loaded_svc = load(saved_path)
print(loaded_svc.predict(pneumothorax_source/f"train/Pneumothorax/000004.dcm"))

Deployment Options

If you are at a small team with limited engineering or DevOps resources, try out automated deployment with BentoML CLI, currently supporting AWS Lambda, AWS SageMaker, and Azure Functions:

If the cloud platform you are working with is not on the list above, try out these step-by-step guide on manually deploying BentoML packaged model to cloud platforms:

Lastly, if you have a DevOps or ML Engineering team who's operating a Kubernetes or OpenShift cluster, use the following guides as references for implementating your deployment strategy: