BentoML makes moving trained ML models to production easy:
Package models trained with any ML framework and reproduce them for model serving in production
Deploy anywhere for online API serving or offline batch serving
High-Performance API model server with adaptive micro-batching support
Central hub for managing models and deployment process via Web UI and APIs
Modular and flexible design making it adaptable to your infrastrcuture
BentoML is a framework for serving, managing, and deploying machine learning models. It is aiming to bridge the gap between Data Science and DevOps, and enable teams to deliver prediction services in a fast, repeatable, and scalable way. Before reading this example project, be sure to check out the Getting started guide to learn about the basic concepts in BentoML.
This notebook demonstrates how to serve your PyTorch model with BentoML, building a Docker Images that has GPU supports. Please refers to GPU Serving guides for more information.
This is an extension of PyTorch's text_sentiment_ngrams_tutorial
%reload_ext autoreload
%autoreload 2
!pip install -q bentoml torch==1.8.1+cu111 torchtext==0.9.1 -f https://download.pytorch.org/whl/torch_stable.html
We are building a simple news classification model with PyTorch, using the dataset AG_NEWS provided by torchtext
library
import os
import time
from collections import Counter
import torch
import torch.nn as nn
import torch.optim as optim
from torch.utils.data import DataLoader
from torch.utils.data.dataset import random_split
from torchtext.datasets import AG_NEWS
from torchtext.data.utils import get_tokenizer
from torchtext.vocab import Vocab
from bentoml import BentoService, api, artifacts, env
from bentoml.adapters import JsonInput, JsonOutput
from bentoml.frameworks.pytorch import PytorchModelArtifact
from bentoml.service.artifacts.pickle import PickleArtifact
device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
print(f"CUDA: {torch.cuda.is_available()} -> {device}")
CUDA: True -> cuda:0
Please refers to torchtext API
def get_tokenizer_vocab(dataset=AG_NEWS, tokenizer_fn='basic_english', root_data_dir='dataset'):
print('Getting tokenizer and vocab...')
tokenizer = get_tokenizer(tokenizer_fn)
train_ = dataset(root=root_data_dir, split='train')
counter = Counter()
for (label, line) in train_:
counter.update(tokenizer(line))
vocab = Vocab(counter, min_freq=1)
return tokenizer, vocab
tokenizer, vocab = get_tokenizer_vocab()
Getting tokenizer and vocab...
def get_pipeline(tokenizer, vocab):
print('Setup pipeline...')
text_pipeline = lambda x: [vocab[token] for token in tokenizer(x)]
label_pipeline = lambda x: int(x) - 1
return text_pipeline, label_pipeline
text_pipeline, label_pipeline = get_pipeline(tokenizer, vocab)
Setup pipeline...
def get_train_valid_split(train_iter):
train_dataset = list(train_iter)
num_train = int(len(train_dataset) * 0.95)
split_train_, split_valid_ = random_split(train_dataset, [num_train, len(train_dataset) - num_train])
return split_train_, split_valid_
We are going to use torch.utils.data.DataLoader.
def collate_batch(batch):
label_list, text_list, offsets = [], [], [0]
for (_label, _text) in batch:
label_list.append(label_pipeline(_label))
processed_text = torch.tensor(text_pipeline(_text), dtype=torch.int64)
text_list.append(processed_text)
offsets.append(processed_text.size(0))
label_list = torch.tensor(label_list, dtype=torch.int64)
offsets = torch.tensor(offsets[:-1]).cumsum(dim=0)
text_list = torch.cat(text_list)
return label_list.to(device), text_list.to(device), offsets.to(device)
The model is composed of the nn.EmbeddingBag
layer plus a linear layer for the classification purpose.
nn.EmbeddingBag
with the default mode of “mean” computes the mean value of a “bag” of embeddings. Although the text entries here have different lengths, nn.EmbeddingBag
module requires no padding here since the text lengths are saved in offsets.
class TextClassificationModel(nn.Module):
def __init__(self, vocab_size, embed_dim, num_class):
super().__init__()
self.embedding = nn.EmbeddingBag(vocab_size, embed_dim, sparse=True)
self.fc = nn.Linear(embed_dim, num_class)
self.init_weights()
def init_weights(self):
init_range = 0.5
self.embedding.weight.data.uniform_(-init_range, init_range)
self.fc.weight.data.uniform_(-init_range, init_range)
self.fc.bias.data.zero_()
def forward(self, text, offsets=None):
embedded = self.embedding(text, offsets=offsets)
return self.fc(embedded)
def summary(model):
count_params = sum(p.numel() for p in model.parameters() if p.requires_grad)
print(f'\nThe model has {count_params:,} trainable parameters')
print(f"Model summary:\n{model}\nDetails:")
for n, p in model.named_parameters():
print(f'name: {n}, shape: {p.shape}')
# Hyperparameters
EPOCHS = 10 # epoch
LR = 5 # learning rate
BATCH_SIZE = 64 # batch size for training
EMBEDDING_SIZE = 64 # embedding size
train_iter = AG_NEWS(root='dataset', split='train')
num_class = len(set([label for (label, text) in train_iter]))
vocab_size = len(vocab)
model = TextClassificationModel(vocab_size, EMBEDDING_SIZE, num_class).to(device)
summary(model)
The model has 6,132,228 trainable parameters Model summary: TextClassificationModel( (embedding): EmbeddingBag(95812, 64, mode=mean) (fc): Linear(in_features=64, out_features=4, bias=True) ) Details: name: embedding.weight, shape: torch.Size([95812, 64]) name: fc.weight, shape: torch.Size([4, 64]) name: fc.bias, shape: torch.Size([4])
def train(model, data_loader, optimizer, criterion, epoch):
model.train()
total_acc, total_count = 0, 0
log_interval = 500
for idx, (label, text, offsets) in enumerate(data_loader):
optimizer.zero_grad()
predicted = model(text, offsets=offsets)
loss = criterion(predicted, label)
loss.backward()
torch.nn.utils.clip_grad_norm_(model.parameters(), 0.1)
optimizer.step()
total_acc += (predicted.argmax(1) == label).sum().item()
total_count += label.size(0)
if idx % log_interval == 0 and idx > 0:
print(f'| epoch {epoch:3d} | {idx:5d}/{len(data_loader):5d} batches | accuracy {(total_acc / total_count):5.3f}')
total_acc, total_count = 0, 0
def evaluate(model, data_loader, criterion):
model.eval()
total_acc, total_count = 0, 0
with torch.no_grad():
for idx, (label, text, offsets) in enumerate(data_loader):
predited_label = model(text, offsets)
loss = criterion(predited_label, label)
total_acc += (predited_label.argmax(1) == label).sum().item()
total_count += label.size(0)
return total_acc / total_count
criterion = nn.CrossEntropyLoss()
optimizer = optim.SGD(model.parameters(), lr=LR)
scheduler = optim.lr_scheduler.StepLR(optimizer, 1.0, gamma=0.1)
total_accu = None
train_iter, test_iter = AG_NEWS(root='dataset')
test_dataset = list(test_iter)
split_train_, split_valid_ = get_train_valid_split(train_iter)
train_data_loader = DataLoader(split_train_, batch_size=BATCH_SIZE, shuffle=True, collate_fn=collate_batch)
valid_data_loader = DataLoader(split_valid_, batch_size=BATCH_SIZE, shuffle=True, collate_fn=collate_batch)
test_data_loader = DataLoader(test_dataset, batch_size=BATCH_SIZE, shuffle=True, collate_fn=collate_batch)
for epoch in range(1, EPOCHS + 1):
epoch_start_time = time.time()
train(model, train_data_loader, optimizer, criterion, epoch)
accu_val = evaluate(model, valid_data_loader, criterion)
if total_accu is not None and total_accu > accu_val:
scheduler.step()
else:
total_accu = accu_val
torch.save(model.state_dict(), 'model/pytorch_model.pt')
print('-' * 59)
print(f'| end of epoch {epoch:1d} | time: {time.time() - epoch_start_time:5.2f}s | valid accuracy {accu_val:8.3f}')
print('-' * 59)
print('Checking the results of test dataset.')
accu_test = evaluate(model, test_data_loader, criterion)
print('test accuracy {:8.3f}'.format(accu_test))
| epoch 1 | 500/ 1782 batches | accuracy 0.689 | epoch 1 | 1000/ 1782 batches | accuracy 0.856 | epoch 1 | 1500/ 1782 batches | accuracy 0.876 ----------------------------------------------------------- | end of epoch 1 | time: 8.45s | valid accuracy 0.889 ----------------------------------------------------------- | epoch 2 | 500/ 1782 batches | accuracy 0.896 | epoch 2 | 1000/ 1782 batches | accuracy 0.901 | epoch 2 | 1500/ 1782 batches | accuracy 0.904 ----------------------------------------------------------- | end of epoch 2 | time: 7.70s | valid accuracy 0.898 ----------------------------------------------------------- | epoch 3 | 500/ 1782 batches | accuracy 0.919 | epoch 3 | 1000/ 1782 batches | accuracy 0.913 | epoch 3 | 1500/ 1782 batches | accuracy 0.915 ----------------------------------------------------------- | end of epoch 3 | time: 7.74s | valid accuracy 0.904 ----------------------------------------------------------- | epoch 4 | 500/ 1782 batches | accuracy 0.924 | epoch 4 | 1000/ 1782 batches | accuracy 0.921 | epoch 4 | 1500/ 1782 batches | accuracy 0.922 ----------------------------------------------------------- | end of epoch 4 | time: 7.77s | valid accuracy 0.905 ----------------------------------------------------------- | epoch 5 | 500/ 1782 batches | accuracy 0.932 | epoch 5 | 1000/ 1782 batches | accuracy 0.927 | epoch 5 | 1500/ 1782 batches | accuracy 0.929 ----------------------------------------------------------- | end of epoch 5 | time: 7.16s | valid accuracy 0.894 ----------------------------------------------------------- | epoch 6 | 500/ 1782 batches | accuracy 0.940 | epoch 6 | 1000/ 1782 batches | accuracy 0.942 | epoch 6 | 1500/ 1782 batches | accuracy 0.943 ----------------------------------------------------------- | end of epoch 6 | time: 7.77s | valid accuracy 0.911 ----------------------------------------------------------- | epoch 7 | 500/ 1782 batches | accuracy 0.944 | epoch 7 | 1000/ 1782 batches | accuracy 0.945 | epoch 7 | 1500/ 1782 batches | accuracy 0.944 ----------------------------------------------------------- | end of epoch 7 | time: 8.14s | valid accuracy 0.914 ----------------------------------------------------------- | epoch 8 | 500/ 1782 batches | accuracy 0.944 | epoch 8 | 1000/ 1782 batches | accuracy 0.945 | epoch 8 | 1500/ 1782 batches | accuracy 0.944 ----------------------------------------------------------- | end of epoch 8 | time: 8.92s | valid accuracy 0.913 ----------------------------------------------------------- | epoch 9 | 500/ 1782 batches | accuracy 0.947 | epoch 9 | 1000/ 1782 batches | accuracy 0.946 | epoch 9 | 1500/ 1782 batches | accuracy 0.946 ----------------------------------------------------------- | end of epoch 9 | time: 8.10s | valid accuracy 0.913 ----------------------------------------------------------- | epoch 10 | 500/ 1782 batches | accuracy 0.945 | epoch 10 | 1000/ 1782 batches | accuracy 0.948 | epoch 10 | 1500/ 1782 batches | accuracy 0.947 ----------------------------------------------------------- | end of epoch 10 | time: 9.61s | valid accuracy 0.913 ----------------------------------------------------------- Checking the results of test dataset. test accuracy 0.911
Please refers to our GPU Serving guide to setup your environment correctly.
We will be using Docker images provided by BentoML : bentoml/model-server:0.12.1-py38-gpu
to prepare our CUDA-enabled images.
%%writefile bento_svc.py
from bentoml import BentoService, api, artifacts, env
from bentoml.adapters import JsonInput, JsonOutput
from bentoml.frameworks.pytorch import PytorchModelArtifact
from bentoml.service.artifacts.pickle import PickleArtifact
from train import get_pipeline
import torch
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
@env(conda_dependencies=['pytorch','torchtext', 'cudatoolkit=11.1'], conda_channels=['pytorch', 'nvidia'], docker_base_image="bentoml/model-server:0.12.1-py38-gpu")
@artifacts([PytorchModelArtifact("model"), PickleArtifact("tokenizer"), PickleArtifact("vocab")])
class PytorchService(BentoService):
def __init__(self):
super().__init__()
self.news_label = {1: 'World',
2: 'Sports',
3: 'Business',
4: 'Sci/Tec'}
def classify_categories(self, sentence):
text_pipeline, _ = get_pipeline(self.artifacts.tokenizer, self.artifacts.vocab)
with torch.no_grad():
text = torch.tensor(text_pipeline(sentence)).to(device)
offsets = torch.tensor([0]).to(device)
output = self.artifacts.model(text, offsets=offsets)
return output.argmax(1).item() + 1
@api(input=JsonInput(), output=JsonOutput())
def predict(self, parsed_json):
label = self.classify_categories(parsed_json.get("text"))
return {'categories': self.news_label[label]}
Overwriting bento_svc.py
tokenizer, vocab = get_tokenizer_vocab()
train_iter = AG_NEWS(root='dataset', split='train')
num_class = len(set([label for (label, text) in train_iter]))
vocab_size = len(vocab)
model = TextClassificationModel(vocab_size, EMBEDDING_SIZE, num_class).to(device)
model.load_state_dict(torch.load("model/pytorch_model.pt"))
model.eval()
Getting tokenizer and vocab...
TextClassificationModel( (embedding): EmbeddingBag(95812, 64, mode=mean) (fc): Linear(in_features=64, out_features=4, bias=True) )
# 1) import the custom BentoService defined above
from bento_svc import PytorchService
bento_svc = PytorchService()
bento_svc.pack("model", model)
bento_svc.pack("tokenizer", tokenizer)
bento_svc.pack("vocab", vocab)
saved_path = bento_svc.save()
[2021-06-04 09:59:40,285] WARNING - Using BentoML installed in `editable` model, the local BentoML repository including all code changes will be packaged together with saved bundle created, under the './bundled_pip_dependencies' directory of the saved bundle. [2021-06-04 09:59:40,334] INFO - Using user specified docker base image: `bentoml/model-server:0.12.1-py38-gpu`, usermust make sure that the base image either has Python 3.8 or conda installed. [2021-06-04 09:59:40,335] WARNING - BentoML by default does not include spacy and torchvision package when using PytorchModelArtifact. To make sure BentoML bundle those packages if they are required for your model, either import those packages in BentoService definition file or manually add them via `@env(pip_packages=['torchvision'])` when defining a BentoService [2021-06-04 09:59:43,136] INFO - Detected non-PyPI-released BentoML installed, copying local BentoML modulefiles to target saved bundle path..
/home/aarnphm/.pyenv/versions/3.8.8/lib/python3.8/site-packages/setuptools/distutils_patch.py:25: UserWarning: Distutils was imported before Setuptools. This usage is discouraged and may exhibit undesirable behaviors or errors. Please use Setuptools' objects directly or at least import Setuptools first. warnings.warn( warning: no previously-included files matching '*~' found anywhere in distribution warning: no previously-included files matching '*.pyo' found anywhere in distribution warning: no previously-included files matching '.git' found anywhere in distribution warning: no previously-included files matching '.ipynb_checkpoints' found anywhere in distribution warning: no previously-included files matching '__pycache__' found anywhere in distribution no previously-included directories found matching 'e2e_tests' no previously-included directories found matching 'tests' no previously-included directories found matching 'benchmark'
UPDATING BentoML-0.12.1+52.g55c7bfb/bentoml/_version.py set BentoML-0.12.1+52.g55c7bfb/bentoml/_version.py to '0.12.1+52.g55c7bfb' [2021-06-04 09:59:48,513] WARNING - Saved BentoService bundle version mismatch: loading BentoService bundle create with BentoML version 0.12.1, but loading from BentoML version 0.12.1+52.g55c7bfb [2021-06-04 09:59:48,580] INFO - BentoService bundle 'PytorchService:20210604095940_7515CC' saved to: /home/aarnphm/bentoml/repository/PytorchService/20210604095940_7515CC
To start a REST API model server with the BentoService save above, use the serve
command:
!bentoml serve PytorchService:latest
[2021-06-04 09:59:49,906] INFO - Getting latest version PytorchService:20210604095940_7515CC [2021-06-04 09:59:49,912] INFO - Starting BentoML API proxy in development mode.. [2021-06-04 09:59:49,913] INFO - Starting BentoML API server in development mode.. [2021-06-04 09:59:49,944] WARNING - Using BentoML installed in `editable` model, the local BentoML repository including all code changes will be packaged together with saved bundle created, under the './bundled_pip_dependencies' directory of the saved bundle. [2021-06-04 09:59:49,944] WARNING - Using BentoML installed in `editable` model, the local BentoML repository including all code changes will be packaged together with saved bundle created, under the './bundled_pip_dependencies' directory of the saved bundle. [2021-06-04 09:59:49,965] WARNING - Saved BentoService bundle version mismatch: loading BentoService bundle create with BentoML version 0.12.1, but loading from BentoML version 0.12.1+52.g55c7bfb [2021-06-04 09:59:49,965] WARNING - Saved BentoService bundle version mismatch: loading BentoService bundle create with BentoML version 0.12.1, but loading from BentoML version 0.12.1+52.g55c7bfb [2021-06-04 09:59:49,967] INFO - Your system nofile limit is 4096, which means each instance of microbatch service is able to hold this number of connections at same time. You can increase the number of file descriptors for the server process, or launch more microbatch instances to accept more concurrent connection. ======== Running on http://0.0.0.0:5000 ======== (Press CTRL+C to quit) [2021-06-04 09:59:50,658] INFO - Using user specified docker base image: `bentoml/model-server:0.12.1-py38-gpu`, usermust make sure that the base image either has Python 3.8 or conda installed. [2021-06-04 09:59:52,782] WARNING - BentoML by default does not include spacy and torchvision package when using PytorchModelArtifact. To make sure BentoML bundle those packages if they are required for your model, either import those packages in BentoService definition file or manually add them via `@env(pip_packages=['torchvision'])` when defining a BentoService * Serving Flask app 'PytorchService' (lazy loading) * Environment: production WARNING: This is a development server. Do not use it in a production deployment. Use a production WSGI server instead. * Debug mode: off INFO:werkzeug: * Running on http://127.0.0.1:34823/ (Press CTRL+C to quit) INFO:werkzeug:127.0.0.1 - - [04/Jun/2021 09:59:53] "GET / HTTP/1.1" 200 - INFO:werkzeug:127.0.0.1 - - [04/Jun/2021 09:59:53] "GET /static_content/main.css HTTP/1.1" 304 - INFO:werkzeug:127.0.0.1 - - [04/Jun/2021 09:59:53] "GET /static_content/readme.css HTTP/1.1" 304 - INFO:werkzeug:127.0.0.1 - - [04/Jun/2021 09:59:53] "GET /static_content/swagger-ui-bundle.js HTTP/1.1" 304 - INFO:werkzeug:127.0.0.1 - - [04/Jun/2021 09:59:53] "GET /static_content/swagger-ui.css HTTP/1.1" 304 - INFO:werkzeug:127.0.0.1 - - [04/Jun/2021 09:59:53] "GET /static_content/marked.min.js HTTP/1.1" 304 - INFO:werkzeug:127.0.0.1 - - [04/Jun/2021 09:59:54] "GET /docs.json HTTP/1.1" 200 - Setup pipeline... [2021-06-04 10:00:16,167] INFO - {'service_name': 'PytorchService', 'service_version': '20210604095940_7515CC', 'api': 'predict', 'task': {'data': '{"text":"WASHINGTON — President Biden offered a series of concessions to try to secure a $1 trillion infrastructure deal with Senate Republicans in an Oval Office meeting this week, narrowing both his spending and tax proposals as negotiations barreled into the final days of what could be an improbable agreement or a blame game that escalates quickly."}', 'task_id': 'a8019712-49a5-4163-96a8-d0cd12286f97', 'http_headers': (('Host', 'localhost:5000'), ('User-Agent', 'Mozilla/5.0 (X11; Linux x86_64; rv:88.0) Gecko/20100101 Firefox/88.0'), ('Accept', '*/*'), ('Accept-Language', 'en-US,en;q=0.5'), ('Accept-Encoding', 'gzip, deflate'), ('Referer', 'http://localhost:5000/'), ('Content-Type', 'application/json'), ('Origin', 'http://localhost:5000'), ('Content-Length', '357'), ('Dnt', '1'), ('Connection', 'keep-alive'), ('Cookie', 'username-localhost-8888="2|1:0|10:1622774823|23:username-localhost-8888|44:NzFiOWVlZjA2YmI3NGI0NmJlMmExNDc3YTU0MDE1MGM=|cb3ea22ef6f3140a1224188b484f48ed62e14b0d50082b1a236251c3a980df86"; _xsrf=2|33d87053|2d1249d56e8ad5d63c884ea9a243cc2a|1622740800'))}, 'result': {'data': '{"categories": "Business"}', 'http_status': 200, 'http_headers': (('Content-Type', 'application/json'),)}, 'request_id': 'a8019712-49a5-4163-96a8-d0cd12286f97'} INFO:werkzeug:127.0.0.1 - - [04/Jun/2021 10:00:16] "POST /predict HTTP/1.1" 200 - ^C
Check if BentoService
is running on GPU
!nvidia-smi
Fri Jun 4 10:01:07 2021 +-----------------------------------------------------------------------------+ | NVIDIA-SMI 465.31 Driver Version: 465.31 CUDA Version: 11.3 | |-------------------------------+----------------------+----------------------+ | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. | | | | MIG M. | |===============================+======================+======================| | 0 NVIDIA GeForce ... Off | 00000000:01:00.0 Off | N/A | | N/A 71C P8 7W / N/A | 849MiB / 6078MiB | 0% Default | | | | N/A | +-------------------------------+----------------------+----------------------+ +-----------------------------------------------------------------------------+ | Processes: | | GPU GI CI PID Type Process name GPU Memory | | ID ID Usage | |=============================================================================| | 0 N/A N/A 1191 G /usr/lib/Xorg 4MiB | | 0 N/A N/A 405631 C ...sions/3.8.8/bin/python3.8 841MiB | +-----------------------------------------------------------------------------+
If you are running this notebook from Google Colab, start the dev server with --run-with-ngrok
option to gain access to the API endpoint via a public endpoint managed by ngrok:
!bentoml serve PyTorchFashionClassifier:latest --run-with-ngrok
One common way of distributing this model API server for production deployment, is via Docker containers. And BentoML provides a convenient way to do that.
Note that docker is not available in Google Colab. You will need to download and run this notebook locally to try out this containerization with docker feature.
If you already have docker configured, simply run the follow command to product a docker container serving the PytorchService with GPU prediction service created above:
!bentoml containerize PytorchService:latest -t pytorch-service-gpu:latest
[2021-06-04 10:08:54,054] INFO - Getting latest version PytorchService:20210604095940_7515CC Found Bento: /home/aarnphm/bentoml/repository/PytorchService/20210604095940_7515CC [2021-06-04 10:08:54,079] WARNING - Using BentoML installed in `editable` model, the local BentoML repository including all code changes will be packaged together with saved bundle created, under the './bundled_pip_dependencies' directory of the saved bundle. [2021-06-04 10:08:54,094] WARNING - Saved BentoService bundle version mismatch: loading BentoService bundle create with BentoML version 0.12.1, but loading from BentoML version 0.12.1+52.g55c7bfb Containerizing PytorchService:20210604095940_7515CC with local YataiService and docker daemon from local environment|^C
!docker run --gpus all --device /dev/nvidia0 --device /dev/nvidiactl --device /dev/nvidia-modeset --device /dev/nvidia-uvm --device /dev/nvidia-uvm-tools -p 5000:5000 pytorch-service-gpu
[2021-06-04 03:48:28,799] INFO - Starting BentoML proxy in production mode.. [2021-06-04 03:48:28,801] INFO - Starting BentoML API server in production mode.. [2021-06-04 03:48:28,833] INFO - Running micro batch service on :5000 [2021-06-04 03:48:28 +0000] [8] [INFO] Starting gunicorn 20.1.0 [2021-06-04 03:48:28 +0000] [8] [INFO] Listening at: http://0.0.0.0:52545 (8) [2021-06-04 03:48:28 +0000] [8] [INFO] Using worker: sync [2021-06-04 03:48:28 +0000] [9] [INFO] Booting worker with pid: 9 [2021-06-04 03:48:28,884] WARNING - Using BentoML not from official PyPI release. In order to find the same version of BentoML when deploying your BentoService, you must set the 'core/bentoml_deploy_version' config to a http/git location of your BentoML fork, e.g.: 'bentoml_deploy_version = git+https://github.com/{username}/bentoml.git@{branch}' [2021-06-04 03:48:28,914] WARNING - Saved BentoService bundle version mismatch: loading BentoService bundle create with BentoML version 0.12.1, but loading from BentoML version 0.12.1+52.g55c7bfb [2021-06-04 03:48:28 +0000] [1] [INFO] Starting gunicorn 20.1.0 [2021-06-04 03:48:29 +0000] [1] [INFO] Listening at: http://0.0.0.0:5000 (1) [2021-06-04 03:48:29 +0000] [1] [INFO] Using worker: aiohttp.worker.GunicornWebWorker [2021-06-04 03:48:29 +0000] [10] [INFO] Booting worker with pid: 10 [2021-06-04 03:48:29,012] WARNING - Using BentoML not from official PyPI release. In order to find the same version of BentoML when deploying your BentoService, you must set the 'core/bentoml_deploy_version' config to a http/git location of your BentoML fork, e.g.: 'bentoml_deploy_version = git+https://github.com/{username}/bentoml.git@{branch}' [2021-06-04 03:48:29,044] WARNING - Saved BentoService bundle version mismatch: loading BentoService bundle create with BentoML version 0.12.1, but loading from BentoML version 0.12.1+52.g55c7bfb [2021-06-04 03:48:29,047] INFO - Your system nofile limit is 1048576, which means each instance of microbatch service is able to hold this number of connections at same time. You can increase the number of file descriptors for the server process, or launch more microbatch instances to accept more concurrent connection. [2021-06-04 03:48:29,801] INFO - Using user specified docker base image: `bentoml/model-server:0.12.1-py38-gpu`, usermust make sure that the base image either has Python 3.8 or conda installed. [2021-06-04 03:48:32,065] WARNING - BentoML by default does not include spacy and torchvision package when using PytorchModelArtifact. To make sure BentoML bundle those packages if they are required for your model, either import those packages in BentoService definition file or manually add them via `@env(pip_packages=['torchvision'])` when defining a BentoService Setup pipeline... [2021-06-04 03:48:54,455] INFO - {'service_name': 'PytorchService', 'service_version': '20210604095940_7515CC', 'api': 'predict', 'task': {'data': '{"text":"WASHINGTON — President Biden offered a series of concessions to try to secure a $1 trillion infrastructure deal with Senate Republicans in an Oval Office meeting this week, narrowing both his spending and tax proposals as negotiations barreled into the final days of what could be an improbable agreement or a blame game that escalates quickly."}', 'task_id': 'e723d038-0bb3-41be-912d-5d605192918e', 'http_headers': (('Host', 'localhost:5000'), ('User-Agent', 'Mozilla/5.0 (X11; Linux x86_64; rv:88.0) Gecko/20100101 Firefox/88.0'), ('Accept', '*/*'), ('Accept-Language', 'en-US,en;q=0.5'), ('Accept-Encoding', 'gzip, deflate'), ('Referer', 'http://localhost:5000/'), ('Content-Type', 'application/json'), ('Origin', 'http://localhost:5000'), ('Content-Length', '357'), ('Dnt', '1'), ('Connection', 'keep-alive'), ('Cookie', 'username-localhost-8888="2|1:0|10:1622774823|23:username-localhost-8888|44:NzFiOWVlZjA2YmI3NGI0NmJlMmExNDc3YTU0MDE1MGM=|cb3ea22ef6f3140a1224188b484f48ed62e14b0d50082b1a236251c3a980df86"; _xsrf=2|33d87053|2d1249d56e8ad5d63c884ea9a243cc2a|1622740800'))}, 'result': {'data': '{"categories": "Business"}', 'http_status': 200, 'http_headers': (('Content-Type', 'application/json'),)}, 'request_id': 'e723d038-0bb3-41be-912d-5d605192918e'} ^C [2021-06-04 03:56:05 +0000] [1] [INFO] Handling signal: int [2021-06-04 03:56:05 +0000] [10] [INFO] Worker exiting (pid: 10) [2021-06-04 03:56:05 +0000] [8] [INFO] Handling signal: term [2021-06-04 03:56:05 +0000] [9] [INFO] Worker exiting (pid: 9)
If you are at a small team with limited engineering or DevOps resources, try out automated deployment with BentoML CLI, currently supporting AWS Lambda, AWS SageMaker, and Azure Functions:
If the cloud platform you are working with is not on the list above, try out these step-by-step guide on manually deploying BentoML packaged model to cloud platforms:
Lastly, if you have a DevOps or ML Engineering team who's operating a Kubernetes or OpenShift cluster, use the following guides as references for implementating your deployment strategy: