BentoML Example: PyTorch Fashion MNIST Classification

BentoML is an open source platform for machine learning model serving and deployment. In this project we will use BentoML to package the image classifier model, and build a containerized REST API model server.

This notebook demonstrates how to use BentoML to turn a PyTorch model into a docker image containing a REST API server serving this model, how to use your ML service built with BentoML as a CLI tool, and how to distribute it a pypi package.

This example was built based on https://github.com/baldassarreFe/zalando-pytorch/blob/master/notebooks/4.0-fb-autoencoder.ipynb, if you are familiar with this, jump start to Model Serving using BentoML

Impression

In [1]:
%reload_ext autoreload
%autoreload 2
%matplotlib inline
In [ ]:
!pip install -q bentoml
!pip install -q torch torchvision sklearn pillow pandas numpy
In [1]:
import bentoml
import matplotlib.pyplot as plt

import torch
import torch.nn as nn
import torch.optim as optim

from torchvision import transforms
from torch.autograd import Variable

from sklearn.manifold import TSNE
from sklearn.metrics import accuracy_score

print("Torch version: ", torch.__version__)
print("CUDA: ", torch.cuda.is_available())
Torch version:  1.4.0
CUDA:  True

Prepare Dataset

PyTorch supports FashionMNIST now, so we can import it directly.

In [2]:
from torchvision.datasets import FashionMNIST
FASHION_MNIST_CLASSES = ['T-shirt/top', 'Trouser', 'Pullover', 'Dress', 'Coat', 'Sandal', 'Shirt', 'Sneaker', 'Bag', 'Ankle boot']

Load train and test set in batches of 1000.

The 28x28 images are scaled up to 29x29 so that combining convolutions and transposed convolutions would not chop off pixels from the reconstructed images.

In [3]:
batch_size = 1000

train_dataset = FashionMNIST(
    '../data', train=True, download=True, 
    transform=transforms.Compose([transforms.CenterCrop((29, 29)), transforms.ToTensor()]))
train_loader = torch.utils.data.DataLoader(train_dataset, batch_size=batch_size, shuffle=True)

test_dataset = FashionMNIST(
    '../data', train=False, download=True, 
    transform=transforms.Compose([transforms.CenterCrop((29, 29)), transforms.ToTensor()]))
test_loader = torch.utils.data.DataLoader(test_dataset, batch_size=batch_size, shuffle=True)

Unsupervised reconstruction

Note that in this section we'll never use the image labels, the whole training is unsupervised.

Autoencoder

The two components of the autoencoder are defined subclassing nn.Module, that gives more flexibility than nn.Sequential.

Encoder

A series of convolutions with kernel_size=5 and stride=2 is used to squeeze the images into a volume of 40x1x1, then a fully connected layer turns this vector in a vector of size embedding_size, that can be specified externally.

Decoder

The decoder takes up from where the encoder left, first transforming back the embedding of size embedding_size into a volume of size 40x1x1, then applying a series of Transposed Convolutions to yield an image of the same size of the original input.

At this time we can show some images in this Dataloader.

In [4]:
class Encoder(nn.Module):
    def __init__(self, embedding_size):
        super(Encoder, self).__init__()
        self.conv1 = nn.Conv2d(1, 10, kernel_size=5, stride=2)
        self.conv2 = nn.Conv2d(10, 20, kernel_size=5, stride=2)
        self.conv3 = nn.Conv2d(20, 40, kernel_size=5, stride=2)
        self.fully = nn.Linear(40, embedding_size)

    def forward(self, x):
        # 1x29x29
        x = torch.relu(self.conv1(x))
        # 10x13x13
        x = torch.relu(self.conv2(x))
        # 20x5x5
        x = torch.relu(self.conv3(x))
        # 40x1x1
        x = x.view(x.data.shape[0], 40)
        # 40
        x = self.fully(x)
        # output_size
        return x

class Decoder(nn.Module):
    def __init__(self, input_size):
        super(Decoder, self).__init__()
        self.fully = nn.Linear(input_size, 40)
        self.conv1 = nn.ConvTranspose2d(40, 20, kernel_size=5, stride=2)
        self.conv2 = nn.ConvTranspose2d(20, 10, kernel_size=5, stride=2)
        self.conv3 = nn.ConvTranspose2d(10, 1, kernel_size=5, stride=2)
    
    def forward(self, x):
        x = self.fully(x)
        x = x.view(x.data.shape[0], 40, 1, 1)
        x = torch.relu(self.conv1(x))
        x = torch.relu(self.conv2(x))
        x = torch.sigmoid(self.conv3(x))
        return x

We are going to use an embedding size of 20, this number has no particular reason, except that it is in the same range of the number of classes. Naively, the network could learn to encode coarse-grained information (i.e. the kind of dress) in half of the embedding vector and then use the other half for fine-grained information.

In [5]:
embedding_size = 20
encoder = Encoder(embedding_size)
decoder = Decoder(embedding_size)

autoencoder = nn.Sequential(encoder, decoder)

Sanity check

A 29x29 black and white image passed through the autoencoder should give the same output dimension

In [6]:
x = Variable(torch.ones(1, 1, 29, 29))
e = encoder(x)
d = decoder(e)

print('Input\t ', list(x.data.shape))
print('Embedding', list(e.data.shape))
print('Output\t ', list(d.data.shape))
Input	  [1, 1, 29, 29]
Embedding [1, 20]
Output	  [1, 1, 29, 29]

Training

In [7]:
autoencoder.train()

loss_fn = nn.MSELoss()
optimizer = optim.Adam(autoencoder.parameters())
epoch_loss = []

for epoch in range(5):
    batch_loss = []
    for batch_num, (data, _) in enumerate(train_loader):
        data = Variable(data)
        optimizer.zero_grad()
        output = autoencoder(data)
        loss = loss_fn(output, data)
        loss.backward()
        optimizer.step()
        batch_loss.append(loss.item())
    epoch_loss.append(sum(batch_loss) / len(batch_loss))
    print('Epoch {}:\tloss {:.4f}'.format(epoch, epoch_loss[-1]))
Epoch 0:	loss 0.1322
Epoch 1:	loss 0.0824
Epoch 2:	loss 0.0532
Epoch 3:	loss 0.0383
Epoch 4:	loss 0.0324
In [8]:
plt.plot(epoch_loss)
plt.title('Final value {:.4f}'.format(epoch_loss[-1]))
plt.xlabel('Epoch')
plt.grid(True)

Evaluation

Reconsruction evaluation on a single batch

In [9]:
autoencoder.eval()
data, targets = next(test_loader.__iter__())
encodings = encoder(Variable(data))
outputs = decoder(encodings)

print('Test loss: {:.4f}'.format(loss_fn(outputs, Variable(data)).item()))
Test loss: 0.0289
In [10]:
fig, axes = plt.subplots(8, 8, figsize=(16, 16))
axes = axes.ravel()

zip_these = axes[::2], axes[1::2], data.numpy().squeeze(), outputs.data.numpy().squeeze(), targets
for ax1, ax2, original, reconstructed, target in zip(*zip_these):
    ax1.imshow(original, cmap='gray')
    ax1.axis('off')
    ax1.set_title(FASHION_MNIST_CLASSES[target])
    ax2.imshow(reconstructed, cmap='gray')
    ax2.axis('off')

Embeddings

The embeddings are 20-dimensional, t-SNE is used to visualize them as clusters in 2D space.

Even though the autoencoder learned the embeddings in a completely unsupervised way we can observe the emergence of clusters:

  • shoes (sandals, sneakers and ankle boot) are clustered together
  • bags form a group on their own (they are the only images with a clear squared-rectangular shape)
  • same goes for trousers, that form their own group
  • all the others are quite mixed together, meaning that the network has learned the concept of clothes for the upper body, but is not able to tell a coat from a pullover
In [11]:
pca = TSNE(n_components=2)
encodings_2 = pca.fit_transform(encodings.data.numpy())
plt.figure(figsize=(10, 10))
for k in range(len(FASHION_MNIST_CLASSES)):
    class_indexes = (targets.numpy() == k)
    plt.scatter(encodings_2[class_indexes, 0], encodings_2[class_indexes, 1], label=FASHION_MNIST_CLASSES[k])
plt.legend();

Supervised classification

Once trained in an unsupervised fashion, the encoder module can be used to generate fashion embeddings (see what I did here?), that can then be used to train a simple classifier on the original labels.

Model

The weights of the encoder are freezed, so only the classifier will be trained.

(later on, when the classifier starts performing decently, we could unfreeze them and do some fine-tuning)

In [12]:
for param in encoder.parameters():
    param.requires_grad = False

classifier = nn.Sequential(
    encoder, 
    nn.Linear(embedding_size, 15),
    nn.ReLU(),
    nn.Linear(15, len(FASHION_MNIST_CLASSES)),
    nn.LogSoftmax()
)

Training

In [13]:
classifier.train()

loss_fn = nn.NLLLoss()
optimizer = optim.Adam([p for p in classifier.parameters() if p.requires_grad])
epoch_loss = []

for epoch in range(5):
    batch_loss = []
    for batch_num, (data, targets) in enumerate(train_loader):
        data, targets = Variable(data), Variable(targets)
        optimizer.zero_grad()
        output = classifier(data)
        loss = loss_fn(output, targets)
        loss.backward()
        optimizer.step()
        batch_loss.append(loss.item())
    epoch_loss.append(sum(batch_loss) / len(batch_loss))
    accuracy = accuracy_score(targets.data.numpy(), output.data.numpy().argmax(axis=1))
    print('Epoch {}:\tloss {:.4f}\taccuracy {:.2%}'.format(epoch, epoch_loss[-1], accuracy))
/opt/anaconda3/envs/bentoml-dev-py36/lib/python3.6/site-packages/torch/nn/modules/container.py:100: UserWarning: Implicit dimension choice for log_softmax has been deprecated. Change the call to include dim=X as an argument.
  input = module(input)
Epoch 0:	loss 2.7911	accuracy 33.90%
Epoch 1:	loss 1.4918	accuracy 57.20%
Epoch 2:	loss 1.1795	accuracy 65.70%
Epoch 3:	loss 1.0348	accuracy 66.40%
Epoch 4:	loss 0.9637	accuracy 68.30%
In [14]:
plt.plot(epoch_loss)
plt.title('Final value {:.4f}'.format(epoch_loss[-1]))
plt.xlabel('Epoch')
plt.grid(True)

Evaluation

Reconsruction evaluation on a single batch

In [15]:
classifier.eval()
data, targets = next(test_loader.__iter__())
outputs = classifier(Variable(data))
log_probs, output_classes = outputs.max(dim=1)

accuracy = accuracy_score(targets.numpy(), output_classes.data.numpy())
print('Accuracy: {:.2%}'.format(accuracy))
Accuracy: 66.20%
/opt/anaconda3/envs/bentoml-dev-py36/lib/python3.6/site-packages/torch/nn/modules/container.py:100: UserWarning: Implicit dimension choice for log_softmax has been deprecated. Change the call to include dim=X as an argument.
  input = module(input)
In [ ]:
 
In [16]:
fig, axex = plt.subplots(8, 8, figsize=(16, 16))

zip_these = axex.ravel(), log_probs.data.exp(), output_classes.data, targets, data.numpy().squeeze()
for ax, prob, output_class, target, img in zip(*zip_these):
    ax.imshow(img, cmap='gray' if output_class == target else 'autumn')
    ax.axis('off')
    ax.set_title('{} {:.1%}'.format(FASHION_MNIST_CLASSES[output_class], prob))

Define BentoService for model serving

In [107]:
%%writefile pytorch_fashion_mnist.py

import bentoml
from PIL import Image
import torch
from torchvision import transforms

from bentoml.artifact import PytorchModelArtifact
from bentoml.adapters import FileInput


FASHION_MNIST_CLASSES = ['T-shirt/top', 'Trouser', 'Pullover', 'Dress', 'Coat',
                         'Sandal', 'Shirt', 'Sneaker', 'Bag', 'Ankle boot']


@bentoml.env(pip_dependencies=['torch', 'numpy', 'torchvision', 'scikit-learn'])
@bentoml.artifacts([PytorchModelArtifact('classifier')])
class PyTorchFashionClassifier(bentoml.BentoService):
    
    @bentoml.utils.cached_property  # reuse transformer
    def transform(self):
        return transforms.Compose([transforms.CenterCrop((29, 29)), transforms.ToTensor()])

    @bentoml.api(input=FileInput())
    def predict(self, file_streams):
        img_tensors = []
        for fs in file_streams:
            img = Image.open(fs).convert(mode="L").resize((28, 28))
            img_tensors.append(self.transform(img))
        outputs = self.artifacts.classifier(torch.stack(img_tensors))
        _, output_classes = outputs.max(dim=1)
        
        return [FASHION_MNIST_CLASSES[output_class] for output_class in output_classes]
Overwriting pytorch_fashion_mnist.py

Save BentoService

In [106]:
# 1) import the custom BentoService defined above
from pytorch_fashion_mnist import PyTorchFashionClassifier

# 2) `pack` it with required artifacts
bento_svc = PyTorchFashionClassifier()
bento_svc.pack('classifier', classifier)

# 3) save your BentoSerivce
saved_path = bento_svc.save()
[2020-08-03 17:16:02,891] WARNING - BentoML by default does not include spacy and torchvision package when using PytorchModelArtifact. To make sure BentoML bundle those packages if they are required for your model, either import those packages in BentoService definition file or manually add them via `@env(pip_dependencies=['torchvision'])` when defining a BentoService
[2020-08-03 17:16:13,359] INFO - Detect BentoML installed in development model, copying local BentoML module file to target saved bundle path
running sdist
running egg_info
writing BentoML.egg-info/PKG-INFO
writing dependency_links to BentoML.egg-info/dependency_links.txt
writing entry points to BentoML.egg-info/entry_points.txt
writing requirements to BentoML.egg-info/requires.txt
writing top-level names to BentoML.egg-info/top_level.txt
reading manifest file 'BentoML.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
warning: no previously-included files matching '*~' found anywhere in distribution
warning: no previously-included files matching '*.pyo' found anywhere in distribution
warning: no previously-included files matching '.git' found anywhere in distribution
warning: no previously-included files matching '.ipynb_checkpoints' found anywhere in distribution
warning: no previously-included files matching '__pycache__' found anywhere in distribution
warning: no directories found matching 'bentoml/server/static'
warning: no directories found matching 'bentoml/yatai/web/dist'
no previously-included directories found matching 'e2e_tests'
no previously-included directories found matching 'tests'
no previously-included directories found matching 'benchmark'
writing manifest file 'BentoML.egg-info/SOURCES.txt'
running check
creating BentoML-0.8.3+44.gadf1adc.dirty
creating BentoML-0.8.3+44.gadf1adc.dirty/BentoML.egg-info
creating BentoML-0.8.3+44.gadf1adc.dirty/bentoml
creating BentoML-0.8.3+44.gadf1adc.dirty/bentoml/adapters
creating BentoML-0.8.3+44.gadf1adc.dirty/bentoml/artifact
creating BentoML-0.8.3+44.gadf1adc.dirty/bentoml/cli
creating BentoML-0.8.3+44.gadf1adc.dirty/bentoml/clipper
creating BentoML-0.8.3+44.gadf1adc.dirty/bentoml/configuration
creating BentoML-0.8.3+44.gadf1adc.dirty/bentoml/configuration/__pycache__
creating BentoML-0.8.3+44.gadf1adc.dirty/bentoml/handlers
creating BentoML-0.8.3+44.gadf1adc.dirty/bentoml/marshal
creating BentoML-0.8.3+44.gadf1adc.dirty/bentoml/saved_bundle
creating BentoML-0.8.3+44.gadf1adc.dirty/bentoml/server
creating BentoML-0.8.3+44.gadf1adc.dirty/bentoml/utils
creating BentoML-0.8.3+44.gadf1adc.dirty/bentoml/yatai
creating BentoML-0.8.3+44.gadf1adc.dirty/bentoml/yatai/client
creating BentoML-0.8.3+44.gadf1adc.dirty/bentoml/yatai/deployment
creating BentoML-0.8.3+44.gadf1adc.dirty/bentoml/yatai/deployment/aws_lambda
creating BentoML-0.8.3+44.gadf1adc.dirty/bentoml/yatai/deployment/azure_functions
creating BentoML-0.8.3+44.gadf1adc.dirty/bentoml/yatai/deployment/sagemaker
creating BentoML-0.8.3+44.gadf1adc.dirty/bentoml/yatai/migrations
creating BentoML-0.8.3+44.gadf1adc.dirty/bentoml/yatai/migrations/__pycache__
creating BentoML-0.8.3+44.gadf1adc.dirty/bentoml/yatai/migrations/versions
creating BentoML-0.8.3+44.gadf1adc.dirty/bentoml/yatai/migrations/versions/__pycache__
creating BentoML-0.8.3+44.gadf1adc.dirty/bentoml/yatai/proto
creating BentoML-0.8.3+44.gadf1adc.dirty/bentoml/yatai/repository
creating BentoML-0.8.3+44.gadf1adc.dirty/bentoml/yatai/validator
copying files to BentoML-0.8.3+44.gadf1adc.dirty...
copying LICENSE -> BentoML-0.8.3+44.gadf1adc.dirty
copying MANIFEST.in -> BentoML-0.8.3+44.gadf1adc.dirty
copying README.md -> BentoML-0.8.3+44.gadf1adc.dirty
copying pyproject.toml -> BentoML-0.8.3+44.gadf1adc.dirty
copying setup.cfg -> BentoML-0.8.3+44.gadf1adc.dirty
copying setup.py -> BentoML-0.8.3+44.gadf1adc.dirty
copying versioneer.py -> BentoML-0.8.3+44.gadf1adc.dirty
copying BentoML.egg-info/PKG-INFO -> BentoML-0.8.3+44.gadf1adc.dirty/BentoML.egg-info
copying BentoML.egg-info/SOURCES.txt -> BentoML-0.8.3+44.gadf1adc.dirty/BentoML.egg-info
copying BentoML.egg-info/dependency_links.txt -> BentoML-0.8.3+44.gadf1adc.dirty/BentoML.egg-info
copying BentoML.egg-info/entry_points.txt -> BentoML-0.8.3+44.gadf1adc.dirty/BentoML.egg-info
copying BentoML.egg-info/requires.txt -> BentoML-0.8.3+44.gadf1adc.dirty/BentoML.egg-info
copying BentoML.egg-info/top_level.txt -> BentoML-0.8.3+44.gadf1adc.dirty/BentoML.egg-info
copying bentoml/__init__.py -> BentoML-0.8.3+44.gadf1adc.dirty/bentoml
copying bentoml/_version.py -> BentoML-0.8.3+44.gadf1adc.dirty/bentoml
copying bentoml/exceptions.py -> BentoML-0.8.3+44.gadf1adc.dirty/bentoml
copying bentoml/service.py -> BentoML-0.8.3+44.gadf1adc.dirty/bentoml
copying bentoml/service_env.py -> BentoML-0.8.3+44.gadf1adc.dirty/bentoml
copying bentoml/adapters/__init__.py -> BentoML-0.8.3+44.gadf1adc.dirty/bentoml/adapters
copying bentoml/adapters/base_input.py -> BentoML-0.8.3+44.gadf1adc.dirty/bentoml/adapters
copying bentoml/adapters/base_output.py -> BentoML-0.8.3+44.gadf1adc.dirty/bentoml/adapters
copying bentoml/adapters/clipper_input.py -> BentoML-0.8.3+44.gadf1adc.dirty/bentoml/adapters
copying bentoml/adapters/dataframe_input.py -> BentoML-0.8.3+44.gadf1adc.dirty/bentoml/adapters
copying bentoml/adapters/dataframe_output.py -> BentoML-0.8.3+44.gadf1adc.dirty/bentoml/adapters
copying bentoml/adapters/default_output.py -> BentoML-0.8.3+44.gadf1adc.dirty/bentoml/adapters
copying bentoml/adapters/fastai_image_input.py -> BentoML-0.8.3+44.gadf1adc.dirty/bentoml/adapters
copying bentoml/adapters/file_input.py -> BentoML-0.8.3+44.gadf1adc.dirty/bentoml/adapters
copying bentoml/adapters/image_input.py -> BentoML-0.8.3+44.gadf1adc.dirty/bentoml/adapters
copying bentoml/adapters/json_input.py -> BentoML-0.8.3+44.gadf1adc.dirty/bentoml/adapters
copying bentoml/adapters/json_output.py -> BentoML-0.8.3+44.gadf1adc.dirty/bentoml/adapters
copying bentoml/adapters/legacy_image_input.py -> BentoML-0.8.3+44.gadf1adc.dirty/bentoml/adapters
copying bentoml/adapters/legacy_json_input.py -> BentoML-0.8.3+44.gadf1adc.dirty/bentoml/adapters
copying bentoml/adapters/multi_image_input.py -> BentoML-0.8.3+44.gadf1adc.dirty/bentoml/adapters
copying bentoml/adapters/pytorch_tensor_input.py -> BentoML-0.8.3+44.gadf1adc.dirty/bentoml/adapters
copying bentoml/adapters/tensorflow_tensor_input.py -> BentoML-0.8.3+44.gadf1adc.dirty/bentoml/adapters
copying bentoml/adapters/tensorflow_tensor_output.py -> BentoML-0.8.3+44.gadf1adc.dirty/bentoml/adapters
copying bentoml/adapters/utils.py -> BentoML-0.8.3+44.gadf1adc.dirty/bentoml/adapters
copying bentoml/artifact/__init__.py -> BentoML-0.8.3+44.gadf1adc.dirty/bentoml/artifact
copying bentoml/artifact/artifact.py -> BentoML-0.8.3+44.gadf1adc.dirty/bentoml/artifact
copying bentoml/artifact/fastai2_model_artifact.py -> BentoML-0.8.3+44.gadf1adc.dirty/bentoml/artifact
copying bentoml/artifact/fastai_model_artifact.py -> BentoML-0.8.3+44.gadf1adc.dirty/bentoml/artifact
copying bentoml/artifact/fasttext_model_artifact.py -> BentoML-0.8.3+44.gadf1adc.dirty/bentoml/artifact
copying bentoml/artifact/h2o_model_artifact.py -> BentoML-0.8.3+44.gadf1adc.dirty/bentoml/artifact
copying bentoml/artifact/json_artifact.py -> BentoML-0.8.3+44.gadf1adc.dirty/bentoml/artifact
copying bentoml/artifact/keras_model_artifact.py -> BentoML-0.8.3+44.gadf1adc.dirty/bentoml/artifact
copying bentoml/artifact/lightgbm_model_artifact.py -> BentoML-0.8.3+44.gadf1adc.dirty/bentoml/artifact
copying bentoml/artifact/onnx_model_artifact.py -> BentoML-0.8.3+44.gadf1adc.dirty/bentoml/artifact
copying bentoml/artifact/pickle_artifact.py -> BentoML-0.8.3+44.gadf1adc.dirty/bentoml/artifact
copying bentoml/artifact/pytorch_model_artifact.py -> BentoML-0.8.3+44.gadf1adc.dirty/bentoml/artifact
copying bentoml/artifact/sklearn_model_artifact.py -> BentoML-0.8.3+44.gadf1adc.dirty/bentoml/artifact
copying bentoml/artifact/spacy_model_artifact.py -> BentoML-0.8.3+44.gadf1adc.dirty/bentoml/artifact
copying bentoml/artifact/text_file_artifact.py -> BentoML-0.8.3+44.gadf1adc.dirty/bentoml/artifact
copying bentoml/artifact/tf_savedmodel_artifact.py -> BentoML-0.8.3+44.gadf1adc.dirty/bentoml/artifact
copying bentoml/artifact/xgboost_model_artifact.py -> BentoML-0.8.3+44.gadf1adc.dirty/bentoml/artifact
copying bentoml/cli/__init__.py -> BentoML-0.8.3+44.gadf1adc.dirty/bentoml/cli
copying bentoml/cli/aws_lambda.py -> BentoML-0.8.3+44.gadf1adc.dirty/bentoml/cli
copying bentoml/cli/aws_sagemaker.py -> BentoML-0.8.3+44.gadf1adc.dirty/bentoml/cli
copying bentoml/cli/azure_functions.py -> BentoML-0.8.3+44.gadf1adc.dirty/bentoml/cli
copying bentoml/cli/bento_management.py -> BentoML-0.8.3+44.gadf1adc.dirty/bentoml/cli
copying bentoml/cli/bento_service.py -> BentoML-0.8.3+44.gadf1adc.dirty/bentoml/cli
copying bentoml/cli/click_utils.py -> BentoML-0.8.3+44.gadf1adc.dirty/bentoml/cli
copying bentoml/cli/config.py -> BentoML-0.8.3+44.gadf1adc.dirty/bentoml/cli
copying bentoml/cli/deployment.py -> BentoML-0.8.3+44.gadf1adc.dirty/bentoml/cli
copying bentoml/cli/utils.py -> BentoML-0.8.3+44.gadf1adc.dirty/bentoml/cli
copying bentoml/cli/yatai_service.py -> BentoML-0.8.3+44.gadf1adc.dirty/bentoml/cli
copying bentoml/clipper/__init__.py -> BentoML-0.8.3+44.gadf1adc.dirty/bentoml/clipper
copying bentoml/configuration/__init__.py -> BentoML-0.8.3+44.gadf1adc.dirty/bentoml/configuration
copying bentoml/configuration/configparser.py -> BentoML-0.8.3+44.gadf1adc.dirty/bentoml/configuration
copying bentoml/configuration/default_bentoml.cfg -> BentoML-0.8.3+44.gadf1adc.dirty/bentoml/configuration
copying bentoml/configuration/__pycache__/__init__.cpython-36.pyc -> BentoML-0.8.3+44.gadf1adc.dirty/bentoml/configuration/__pycache__
copying bentoml/configuration/__pycache__/__init__.cpython-37.pyc -> BentoML-0.8.3+44.gadf1adc.dirty/bentoml/configuration/__pycache__
copying bentoml/configuration/__pycache__/__init__.cpython-38.pyc -> BentoML-0.8.3+44.gadf1adc.dirty/bentoml/configuration/__pycache__
copying bentoml/configuration/__pycache__/configparser.cpython-36.pyc -> BentoML-0.8.3+44.gadf1adc.dirty/bentoml/configuration/__pycache__
copying bentoml/configuration/__pycache__/configparser.cpython-37.pyc -> BentoML-0.8.3+44.gadf1adc.dirty/bentoml/configuration/__pycache__
copying bentoml/configuration/__pycache__/configparser.cpython-38.pyc -> BentoML-0.8.3+44.gadf1adc.dirty/bentoml/configuration/__pycache__
copying bentoml/handlers/__init__.py -> BentoML-0.8.3+44.gadf1adc.dirty/bentoml/handlers
copying bentoml/marshal/__init__.py -> BentoML-0.8.3+44.gadf1adc.dirty/bentoml/marshal
copying bentoml/marshal/dispatcher.py -> BentoML-0.8.3+44.gadf1adc.dirty/bentoml/marshal
copying bentoml/marshal/marshal.py -> BentoML-0.8.3+44.gadf1adc.dirty/bentoml/marshal
copying bentoml/marshal/utils.py -> BentoML-0.8.3+44.gadf1adc.dirty/bentoml/marshal
copying bentoml/saved_bundle/__init__.py -> BentoML-0.8.3+44.gadf1adc.dirty/bentoml/saved_bundle
copying bentoml/saved_bundle/bentoml-init.sh -> BentoML-0.8.3+44.gadf1adc.dirty/bentoml/saved_bundle
copying bentoml/saved_bundle/bundler.py -> BentoML-0.8.3+44.gadf1adc.dirty/bentoml/saved_bundle
copying bentoml/saved_bundle/config.py -> BentoML-0.8.3+44.gadf1adc.dirty/bentoml/saved_bundle
copying bentoml/saved_bundle/docker-entrypoint.sh -> BentoML-0.8.3+44.gadf1adc.dirty/bentoml/saved_bundle
copying bentoml/saved_bundle/loader.py -> BentoML-0.8.3+44.gadf1adc.dirty/bentoml/saved_bundle
copying bentoml/saved_bundle/pip_pkg.py -> BentoML-0.8.3+44.gadf1adc.dirty/bentoml/saved_bundle
copying bentoml/saved_bundle/py_module_utils.py -> BentoML-0.8.3+44.gadf1adc.dirty/bentoml/saved_bundle
copying bentoml/saved_bundle/templates.py -> BentoML-0.8.3+44.gadf1adc.dirty/bentoml/saved_bundle
copying bentoml/server/__init__.py -> BentoML-0.8.3+44.gadf1adc.dirty/bentoml/server
copying bentoml/server/api_server.py -> BentoML-0.8.3+44.gadf1adc.dirty/bentoml/server
copying bentoml/server/gunicorn_config.py -> BentoML-0.8.3+44.gadf1adc.dirty/bentoml/server
copying bentoml/server/gunicorn_server.py -> BentoML-0.8.3+44.gadf1adc.dirty/bentoml/server
copying bentoml/server/instruments.py -> BentoML-0.8.3+44.gadf1adc.dirty/bentoml/server
copying bentoml/server/marshal_server.py -> BentoML-0.8.3+44.gadf1adc.dirty/bentoml/server
copying bentoml/server/open_api.py -> BentoML-0.8.3+44.gadf1adc.dirty/bentoml/server
copying bentoml/server/trace.py -> BentoML-0.8.3+44.gadf1adc.dirty/bentoml/server
copying bentoml/server/utils.py -> BentoML-0.8.3+44.gadf1adc.dirty/bentoml/server
copying bentoml/utils/__init__.py -> BentoML-0.8.3+44.gadf1adc.dirty/bentoml/utils
copying bentoml/utils/alg.py -> BentoML-0.8.3+44.gadf1adc.dirty/bentoml/utils
copying bentoml/utils/benchmark.py -> BentoML-0.8.3+44.gadf1adc.dirty/bentoml/utils
copying bentoml/utils/cloudpickle.py -> BentoML-0.8.3+44.gadf1adc.dirty/bentoml/utils
copying bentoml/utils/dataframe_util.py -> BentoML-0.8.3+44.gadf1adc.dirty/bentoml/utils
copying bentoml/utils/flask_ngrok.py -> BentoML-0.8.3+44.gadf1adc.dirty/bentoml/utils
copying bentoml/utils/hybridmethod.py -> BentoML-0.8.3+44.gadf1adc.dirty/bentoml/utils
copying bentoml/utils/lazy_loader.py -> BentoML-0.8.3+44.gadf1adc.dirty/bentoml/utils
copying bentoml/utils/log.py -> BentoML-0.8.3+44.gadf1adc.dirty/bentoml/utils
copying bentoml/utils/s3.py -> BentoML-0.8.3+44.gadf1adc.dirty/bentoml/utils
copying bentoml/utils/tempdir.py -> BentoML-0.8.3+44.gadf1adc.dirty/bentoml/utils
copying bentoml/utils/usage_stats.py -> BentoML-0.8.3+44.gadf1adc.dirty/bentoml/utils
copying bentoml/yatai/__init__.py -> BentoML-0.8.3+44.gadf1adc.dirty/bentoml/yatai
copying bentoml/yatai/alembic.ini -> BentoML-0.8.3+44.gadf1adc.dirty/bentoml/yatai
copying bentoml/yatai/db.py -> BentoML-0.8.3+44.gadf1adc.dirty/bentoml/yatai
copying bentoml/yatai/deployment_utils.py -> BentoML-0.8.3+44.gadf1adc.dirty/bentoml/yatai
copying bentoml/yatai/status.py -> BentoML-0.8.3+44.gadf1adc.dirty/bentoml/yatai
copying bentoml/yatai/utils.py -> BentoML-0.8.3+44.gadf1adc.dirty/bentoml/yatai
copying bentoml/yatai/yatai_service.py -> BentoML-0.8.3+44.gadf1adc.dirty/bentoml/yatai
copying bentoml/yatai/yatai_service_impl.py -> BentoML-0.8.3+44.gadf1adc.dirty/bentoml/yatai
copying bentoml/yatai/client/__init__.py -> BentoML-0.8.3+44.gadf1adc.dirty/bentoml/yatai/client
copying bentoml/yatai/client/bento_repository_api.py -> BentoML-0.8.3+44.gadf1adc.dirty/bentoml/yatai/client
copying bentoml/yatai/client/deployment_api.py -> BentoML-0.8.3+44.gadf1adc.dirty/bentoml/yatai/client
copying bentoml/yatai/deployment/__init__.py -> BentoML-0.8.3+44.gadf1adc.dirty/bentoml/yatai/deployment
copying bentoml/yatai/deployment/operator.py -> BentoML-0.8.3+44.gadf1adc.dirty/bentoml/yatai/deployment
copying bentoml/yatai/deployment/store.py -> BentoML-0.8.3+44.gadf1adc.dirty/bentoml/yatai/deployment
copying bentoml/yatai/deployment/utils.py -> BentoML-0.8.3+44.gadf1adc.dirty/bentoml/yatai/deployment
copying bentoml/yatai/deployment/aws_lambda/__init__.py -> BentoML-0.8.3+44.gadf1adc.dirty/bentoml/yatai/deployment/aws_lambda
copying bentoml/yatai/deployment/aws_lambda/download_extra_resources.py -> BentoML-0.8.3+44.gadf1adc.dirty/bentoml/yatai/deployment/aws_lambda
copying bentoml/yatai/deployment/aws_lambda/lambda_app.py -> BentoML-0.8.3+44.gadf1adc.dirty/bentoml/yatai/deployment/aws_lambda
copying bentoml/yatai/deployment/aws_lambda/operator.py -> BentoML-0.8.3+44.gadf1adc.dirty/bentoml/yatai/deployment/aws_lambda
copying bentoml/yatai/deployment/aws_lambda/utils.py -> BentoML-0.8.3+44.gadf1adc.dirty/bentoml/yatai/deployment/aws_lambda
copying bentoml/yatai/deployment/azure_functions/Dockerfile -> BentoML-0.8.3+44.gadf1adc.dirty/bentoml/yatai/deployment/azure_functions
copying bentoml/yatai/deployment/azure_functions/__init__.py -> BentoML-0.8.3+44.gadf1adc.dirty/bentoml/yatai/deployment/azure_functions
copying bentoml/yatai/deployment/azure_functions/app_init.py -> BentoML-0.8.3+44.gadf1adc.dirty/bentoml/yatai/deployment/azure_functions
copying bentoml/yatai/deployment/azure_functions/constants.py -> BentoML-0.8.3+44.gadf1adc.dirty/bentoml/yatai/deployment/azure_functions
copying bentoml/yatai/deployment/azure_functions/host.json -> BentoML-0.8.3+44.gadf1adc.dirty/bentoml/yatai/deployment/azure_functions
copying bentoml/yatai/deployment/azure_functions/local.settings.json -> BentoML-0.8.3+44.gadf1adc.dirty/bentoml/yatai/deployment/azure_functions
copying bentoml/yatai/deployment/azure_functions/operator.py -> BentoML-0.8.3+44.gadf1adc.dirty/bentoml/yatai/deployment/azure_functions
copying bentoml/yatai/deployment/azure_functions/templates.py -> BentoML-0.8.3+44.gadf1adc.dirty/bentoml/yatai/deployment/azure_functions
copying bentoml/yatai/deployment/sagemaker/__init__.py -> BentoML-0.8.3+44.gadf1adc.dirty/bentoml/yatai/deployment/sagemaker
copying bentoml/yatai/deployment/sagemaker/model_server.py -> BentoML-0.8.3+44.gadf1adc.dirty/bentoml/yatai/deployment/sagemaker
copying bentoml/yatai/deployment/sagemaker/nginx.conf -> BentoML-0.8.3+44.gadf1adc.dirty/bentoml/yatai/deployment/sagemaker
copying bentoml/yatai/deployment/sagemaker/operator.py -> BentoML-0.8.3+44.gadf1adc.dirty/bentoml/yatai/deployment/sagemaker
copying bentoml/yatai/deployment/sagemaker/serve -> BentoML-0.8.3+44.gadf1adc.dirty/bentoml/yatai/deployment/sagemaker
copying bentoml/yatai/deployment/sagemaker/wsgi.py -> BentoML-0.8.3+44.gadf1adc.dirty/bentoml/yatai/deployment/sagemaker
copying bentoml/yatai/migrations/README -> BentoML-0.8.3+44.gadf1adc.dirty/bentoml/yatai/migrations
copying bentoml/yatai/migrations/__init__.py -> BentoML-0.8.3+44.gadf1adc.dirty/bentoml/yatai/migrations
copying bentoml/yatai/migrations/env.py -> BentoML-0.8.3+44.gadf1adc.dirty/bentoml/yatai/migrations
copying bentoml/yatai/migrations/script.py.mako -> BentoML-0.8.3+44.gadf1adc.dirty/bentoml/yatai/migrations
copying bentoml/yatai/migrations/__pycache__/env.cpython-36.pyc -> BentoML-0.8.3+44.gadf1adc.dirty/bentoml/yatai/migrations/__pycache__
copying bentoml/yatai/migrations/versions/__init__.py -> BentoML-0.8.3+44.gadf1adc.dirty/bentoml/yatai/migrations/versions
copying bentoml/yatai/migrations/versions/a6b00ae45279_add_last_updated_at_for_deployments.py -> BentoML-0.8.3+44.gadf1adc.dirty/bentoml/yatai/migrations/versions
copying bentoml/yatai/migrations/versions/__pycache__/a6b00ae45279_add_last_updated_at_for_deployments.cpython-36.pyc -> BentoML-0.8.3+44.gadf1adc.dirty/bentoml/yatai/migrations/versions/__pycache__
copying bentoml/yatai/proto/__init__.py -> BentoML-0.8.3+44.gadf1adc.dirty/bentoml/yatai/proto
copying bentoml/yatai/proto/deployment_pb2.py -> BentoML-0.8.3+44.gadf1adc.dirty/bentoml/yatai/proto
copying bentoml/yatai/proto/repository_pb2.py -> BentoML-0.8.3+44.gadf1adc.dirty/bentoml/yatai/proto
copying bentoml/yatai/proto/status_pb2.py -> BentoML-0.8.3+44.gadf1adc.dirty/bentoml/yatai/proto
copying bentoml/yatai/proto/yatai_service_pb2.py -> BentoML-0.8.3+44.gadf1adc.dirty/bentoml/yatai/proto
copying bentoml/yatai/proto/yatai_service_pb2_grpc.py -> BentoML-0.8.3+44.gadf1adc.dirty/bentoml/yatai/proto
copying bentoml/yatai/repository/__init__.py -> BentoML-0.8.3+44.gadf1adc.dirty/bentoml/yatai/repository
copying bentoml/yatai/repository/base_repository.py -> BentoML-0.8.3+44.gadf1adc.dirty/bentoml/yatai/repository
copying bentoml/yatai/repository/local_repository.py -> BentoML-0.8.3+44.gadf1adc.dirty/bentoml/yatai/repository
copying bentoml/yatai/repository/metadata_store.py -> BentoML-0.8.3+44.gadf1adc.dirty/bentoml/yatai/repository
copying bentoml/yatai/repository/repository.py -> BentoML-0.8.3+44.gadf1adc.dirty/bentoml/yatai/repository
copying bentoml/yatai/repository/s3_repository.py -> BentoML-0.8.3+44.gadf1adc.dirty/bentoml/yatai/repository
copying bentoml/yatai/validator/__init__.py -> BentoML-0.8.3+44.gadf1adc.dirty/bentoml/yatai/validator
copying bentoml/yatai/validator/deployment_pb_validator.py -> BentoML-0.8.3+44.gadf1adc.dirty/bentoml/yatai/validator
Writing BentoML-0.8.3+44.gadf1adc.dirty/setup.cfg
UPDATING BentoML-0.8.3+44.gadf1adc.dirty/bentoml/_version.py
set BentoML-0.8.3+44.gadf1adc.dirty/bentoml/_version.py to '0.8.3+44.gadf1adc.dirty'
Creating tar archive
removing 'BentoML-0.8.3+44.gadf1adc.dirty' (and everything under it)
[2020-08-03 17:16:14,390] INFO - BentoService bundle 'PyTorchFashionClassifier:20200803171602_556423' saved to: /home/bentoml/bentoml/repository/PyTorchFashionClassifier/20200803171602_556423

Use bentoml get <BentoService_Name> to get list of service versions

In [ ]:
!bentoml get PyTorchFashionClassifier

With version info, bentoml get display additional information and metadata

In [85]:
!bentoml get PyTorchFashionClassifier:latest
[2020-08-03 16:51:16,675] INFO - Getting latest version PyTorchFashionClassifier:20200803164836_307D6C
{
  "name": "PyTorchFashionClassifier",
  "version": "20200803164836_307D6C",
  "uri": {
    "type": "LOCAL",
    "uri": "/home/bentoml/bentoml/repository/PyTorchFashionClassifier/20200803164836_307D6C"
  },
  "bentoServiceMetadata": {
    "name": "PyTorchFashionClassifier",
    "version": "20200803164836_307D6C",
    "createdAt": "2020-08-03T08:48:47.580674Z",
    "env": {
      "condaEnv": "name: bentoml-PyTorchFashionClassifier\nchannels:\n- defaults\ndependencies:\n- python=3.6.10\n- pip\n",
      "pipDependencies": "torch\nbentoml==0.8.3\nnumpy\ntorchvision\nscikit-learn",
      "pythonVersion": "3.6.10",
      "dockerBaseImage": "bentoml/model-server:0.8.3"
    },
    "artifacts": [
      {
        "name": "classifier",
        "artifactType": "PytorchModelArtifact"
      }
    ],
    "apis": [
      {
        "name": "predict",
        "inputType": "FileInput",
        "docs": "BentoService inference API 'predict', input: 'FileInput', output: 'DefaultOutput'",
        "outputConfig": {
          "cors": "*"
        },
        "outputType": "DefaultOutput",
        "mbMaxLatency": 10000,
        "mbMaxBatchSize": 2000
      }
    ]
  }
}

Test and validate BentoService with bentoml run

In [108]:
!bentoml run PyTorchFashionClassifier:latest predict --input sample_image.png
[2020-08-03 17:16:39,504] INFO - Getting latest version PyTorchFashionClassifier:20200803171602_556423
[2020-08-03 17:16:40,531] WARNING - Using BentoML installed in `editable` model, the local BentoML repository including all code changes will be packaged together with saved bundle created, under the './bundled_pip_dependencies' directory of the saved bundle.
[2020-08-03 17:16:40,566] WARNING - Saved BentoService bundle version mismatch: loading BentoService bundle create with BentoML version 0.8.3, but loading from BentoML version 0.8.3+44.gadf1adc.dirty
[2020-08-03 17:16:41,319] WARNING - BentoML by default does not include spacy and torchvision package when using PytorchModelArtifact. To make sure BentoML bundle those packages if they are required for your model, either import those packages in BentoService definition file or manually add them via `@env(pip_dependencies=['torchvision'])` when defining a BentoService
/opt/anaconda3/envs/bentoml-dev-py36/lib/python3.6/site-packages/torch/nn/modules/container.py:100: UserWarning: Implicit dimension choice for log_softmax has been deprecated. Change the call to include dim=X as an argument.
  input = module(input)
Ankle boot

Model Serving via REST API

In your termnial, run the following command to start the REST API server:### Run REST API server

In [91]:
!bentoml serve PyTorchFashionClassifier:latest
[2020-08-03 16:54:03,858] INFO - Getting latest version PyTorchFashionClassifier:20200803164836_307D6C
[2020-08-03 16:54:03,859] INFO - Starting BentoML API server in development mode..
[2020-08-03 16:54:05,152] WARNING - Using BentoML installed in `editable` model, the local BentoML repository including all code changes will be packaged together with saved bundle created, under the './bundled_pip_dependencies' directory of the saved bundle.
[2020-08-03 16:54:05,185] WARNING - Saved BentoService bundle version mismatch: loading BentoService bundle create with BentoML version 0.8.3, but loading from BentoML version 0.8.3+44.gadf1adc.dirty
[2020-08-03 16:54:05,864] WARNING - BentoML by default does not include spacy and torchvision package when using PytorchModelArtifact. To make sure BentoML bundle those packages if they are required for your model, either import those packages in BentoService definition file or manually add them via `@env(pip_dependencies=['torchvision'])` when defining a BentoService
 * Serving Flask app "PyTorchFashionClassifier" (lazy loading)
 * Environment: production
   WARNING: This is a development server. Do not use it in a production deployment.
   Use a production WSGI server instead.
 * Debug mode: off
 * Running on http://127.0.0.1:5000/ (Press CTRL+C to quit)
^C
In [ ]:
!bentoml serve-gunicorn PyTorchFashionClassifier:latest --enable-microbatch --workers 1 
[2020-08-03 17:16:51,155] INFO - Getting latest version PyTorchFashionClassifier:20200803171602_556423
[2020-08-03 17:16:51,155] INFO - Starting BentoML API server in production mode..
[2020-08-03 17:16:51,371] INFO - Running micro batch service on :5000
[2020-08-03 17:16:51 +0800] [166176] [INFO] Starting gunicorn 20.0.4
[2020-08-03 17:16:51 +0800] [166161] [INFO] Starting gunicorn 20.0.4
[2020-08-03 17:16:51 +0800] [166176] [INFO] Listening at: http://0.0.0.0:5000 (166176)
[2020-08-03 17:16:51 +0800] [166161] [INFO] Listening at: http://0.0.0.0:44115 (166161)
[2020-08-03 17:16:51 +0800] [166176] [INFO] Using worker: aiohttp.worker.GunicornWebWorker
[2020-08-03 17:16:51 +0800] [166161] [INFO] Using worker: sync
[2020-08-03 17:16:51 +0800] [166177] [INFO] Booting worker with pid: 166177
[2020-08-03 17:16:51 +0800] [166178] [INFO] Booting worker with pid: 166178
[2020-08-03 17:16:51,423] WARNING - Using BentoML installed in `editable` model, the local BentoML repository including all code changes will be packaged together with saved bundle created, under the './bundled_pip_dependencies' directory of the saved bundle.
[2020-08-03 17:16:51,440] WARNING - Saved BentoService bundle version mismatch: loading BentoService bundle create with BentoML version 0.8.3, but loading from BentoML version 0.8.3+44.gadf1adc.dirty
[2020-08-03 17:16:51,662] INFO - Micro batch enabled for API `predict`
[2020-08-03 17:16:51,662] INFO - Your system nofile limit is 10000, which means each instance of microbatch service is able to hold this number of connections at same time. You can increase the number of file descriptors for the server process, or launch more microbatch instances to accept more concurrent connection.
[2020-08-03 17:16:52,413] WARNING - Using BentoML installed in `editable` model, the local BentoML repository including all code changes will be packaged together with saved bundle created, under the './bundled_pip_dependencies' directory of the saved bundle.
[2020-08-03 17:16:52,440] WARNING - Saved BentoService bundle version mismatch: loading BentoService bundle create with BentoML version 0.8.3, but loading from BentoML version 0.8.3+44.gadf1adc.dirty
[2020-08-03 17:16:53,133] WARNING - BentoML by default does not include spacy and torchvision package when using PytorchModelArtifact. To make sure BentoML bundle those packages if they are required for your model, either import those packages in BentoService definition file or manually add them via `@env(pip_dependencies=['torchvision'])` when defining a BentoService
/opt/anaconda3/envs/bentoml-dev-py36/lib/python3.6/site-packages/torch/nn/modules/container.py:100: UserWarning: Implicit dimension choice for log_softmax has been deprecated. Change the call to include dim=X as an argument.
  input = module(input)
/opt/anaconda3/envs/bentoml-dev-py36/lib/python3.6/site-packages/torch/nn/modules/container.py:100: UserWarning: Implicit dimension choice for log_softmax has been deprecated. Change the call to include dim=X as an argument.
  input = module(input)

Call REST API from a client

Sending POST request from termnial:

curl -X POST "http://127.0.0.1:5000/predict" -F image=@sample_image.png
curl -X POST "http://127.0.0.1:5000/predict" -H "Content-Type: image/png" --data-binary @sample_image.png

Go visit http://127.0.0.1:5000/ from your browser, click /predict -> Try it out -> Choose File -> Execute to sumbit an image from your computer

image.png

Use BentoService as PyPI package

In [101]:
!pip install --quiet {saved_path}
In [95]:
!PyTorchFashionClassifier info
{
  "name": "PyTorchFashionClassifier",
  "version": "20200803164836_307D6C",
  "created_at": "2020-08-03T08:48:46.586364Z",
  "env": {
    "conda_env": "name: bentoml-PyTorchFashionClassifier\nchannels:\n- defaults\ndependencies:\n- python=3.6.10\n- pip\n",
    "pip_dependencies": "torch\nbentoml==0.8.3\nnumpy\ntorchvision\nscikit-learn",
    "python_version": "3.6.10",
    "docker_base_image": "bentoml/model-server:0.8.3"
  },
  "artifacts": [
    {
      "name": "classifier",
      "artifact_type": "PytorchModelArtifact"
    }
  ],
  "apis": [
    {
      "name": "predict",
      "input_type": "FileInput",
      "docs": "BentoService inference API 'predict', input: 'FileInput', output: 'DefaultOutput'",
      "output_config": {
        "cors": "*"
      },
      "output_type": "DefaultOutput",
      "mb_max_latency": 10000,
      "mb_max_batch_size": 2000
    }
  ]
}
In [97]:
!PyTorchFashionClassifier run predict --input sample_image.png
/opt/anaconda3/envs/bentoml-dev-py36/lib/python3.6/site-packages/torch/nn/modules/container.py:100: UserWarning: Implicit dimension choice for log_softmax has been deprecated. Change the call to include dim=X as an argument.
  input = module(input)
Ankle boot

Containerize REST API server with Docker

** Make sure you have docker installed, note that it is not available when running in Google Colaboratory

In [103]:
!docker build {saved_path} --quiet -t pytorch-fashion-mnist
sha256:6c1269967800249676fb0e54b675ddf9647aac3320ab772b32f30cfd7a025545
In [104]:
!docker run -p 5000:5000 pytorch-fashion-mnist
[2020-08-03 09:13:26,158] INFO - Starting BentoML API server in production mode..
[2020-08-03 09:13:26,634] INFO - get_gunicorn_num_of_workers: 3, calculated by cpu count
[2020-08-03 09:13:26 +0000] [1] [INFO] Starting gunicorn 20.0.4
[2020-08-03 09:13:26 +0000] [1] [INFO] Listening at: http://0.0.0.0:5000 (1)
[2020-08-03 09:13:26 +0000] [1] [INFO] Using worker: sync
[2020-08-03 09:13:26 +0000] [12] [INFO] Booting worker with pid: 12
[2020-08-03 09:13:26 +0000] [13] [INFO] Booting worker with pid: 13
[2020-08-03 09:13:26 +0000] [14] [INFO] Booting worker with pid: 14
[2020-08-03 09:13:27,638] WARNING - Using BentoML installed in `editable` model, the local BentoML repository including all code changes will be packaged together with saved bundle created, under the './bundled_pip_dependencies' directory of the saved bundle.
[2020-08-03 09:13:27,652] WARNING - Using BentoML installed in `editable` model, the local BentoML repository including all code changes will be packaged together with saved bundle created, under the './bundled_pip_dependencies' directory of the saved bundle.
[2020-08-03 09:13:27,658] WARNING - Saved BentoService bundle version mismatch: loading BentoService bundle create with BentoML version 0.8.3, but loading from BentoML version 0.8.3+44.gadf1adc.dirty
[2020-08-03 09:13:27,673] WARNING - Saved BentoService bundle version mismatch: loading BentoService bundle create with BentoML version 0.8.3, but loading from BentoML version 0.8.3+44.gadf1adc.dirty
[2020-08-03 09:13:27,696] WARNING - Using BentoML installed in `editable` model, the local BentoML repository including all code changes will be packaged together with saved bundle created, under the './bundled_pip_dependencies' directory of the saved bundle.
[2020-08-03 09:13:27,739] WARNING - Saved BentoService bundle version mismatch: loading BentoService bundle create with BentoML version 0.8.3, but loading from BentoML version 0.8.3+44.gadf1adc.dirty
[2020-08-03 09:13:28,170] WARNING - BentoML by default does not include spacy and torchvision package when using PytorchModelArtifact. To make sure BentoML bundle those packages if they are required for your model, either import those packages in BentoService definition file or manually add them via `@env(pip_dependencies=['torchvision'])` when defining a BentoService
[2020-08-03 09:13:28,211] WARNING - BentoML by default does not include spacy and torchvision package when using PytorchModelArtifact. To make sure BentoML bundle those packages if they are required for your model, either import those packages in BentoService definition file or manually add them via `@env(pip_dependencies=['torchvision'])` when defining a BentoService
[2020-08-03 09:13:28,228] WARNING - BentoML by default does not include spacy and torchvision package when using PytorchModelArtifact. To make sure BentoML bundle those packages if they are required for your model, either import those packages in BentoService definition file or manually add them via `@env(pip_dependencies=['torchvision'])` when defining a BentoService
^C
[2020-08-03 09:13:52 +0000] [1] [INFO] Handling signal: int
/opt/conda/lib/python3.6/site-packages/torch/nn/modules/container.py:117: UserWarning: Implicit dimension choice for log_softmax has been deprecated. Change the call to include dim=X as an argument.
  input = module(input)
[2020-08-03 09:13:53 +0000] [12] [INFO] Worker exiting (pid: 12)
/opt/conda/lib/python3.6/site-packages/torch/nn/modules/container.py:117: UserWarning: Implicit dimension choice for log_softmax has been deprecated. Change the call to include dim=X as an argument.
  input = module(input)
[2020-08-03 09:13:53 +0000] [13] [INFO] Worker exiting (pid: 13)
/opt/conda/lib/python3.6/site-packages/torch/nn/modules/container.py:117: UserWarning: Implicit dimension choice for log_softmax has been deprecated. Change the call to include dim=X as an argument.
  input = module(input)
[2020-08-03 09:13:53 +0000] [14] [INFO] Worker exiting (pid: 14)

Deploy BentoService as REST API server to the cloud

BentoML support deployment to multiply cloud provider services, such as AWS Lambda, AWS Sagemaker, Google Cloudrun and etc. You can find the full list and guide on the documentation site at https://docs.bentoml.org/en/latest/deployment/index.html

For this project, we are going to deploy to AWS Sagemaker

Deploy to Sagemaker with single command bentoml sagemaker deploy

In [30]:
!bentoml sagemaker deploy pytorch-fashion -b PyTorchFashionClassifier:20200213142355_A51D6E --api-name predict
Deploying Sagemaker deployment \[2020-02-13 14:31:00,094] INFO - Step 1/11 : FROM continuumio/miniconda3:4.7.12
[2020-02-13 14:31:00,095] INFO - 

[2020-02-13 14:31:00,095] INFO -  ---> 406f2b43ea59

[2020-02-13 14:31:00,095] INFO - Step 2/11 : EXPOSE 8080
[2020-02-13 14:31:00,095] INFO - 

[2020-02-13 14:31:00,096] INFO -  ---> Using cache

[2020-02-13 14:31:00,096] INFO -  ---> 85b4bb5fff81

[2020-02-13 14:31:00,096] INFO - Step 3/11 : RUN set -x      && apt-get update      && apt-get install --no-install-recommends --no-install-suggests -y libpq-dev build-essential     && apt-get install -y nginx      && rm -rf /var/lib/apt/lists/*
[2020-02-13 14:31:00,096] INFO - 

[2020-02-13 14:31:00,096] INFO -  ---> Using cache

[2020-02-13 14:31:00,096] INFO -  ---> 66280bc479d1

[2020-02-13 14:31:00,096] INFO - Step 4/11 : RUN conda install pip numpy scipy       && pip install gunicorn gevent
[2020-02-13 14:31:00,097] INFO - 

[2020-02-13 14:31:00,097] INFO -  ---> Using cache

[2020-02-13 14:31:00,097] INFO -  ---> 1dbfc408f11d

[2020-02-13 14:31:00,097] INFO - Step 5/11 : COPY . /opt/program
[2020-02-13 14:31:00,097] INFO - 

\[2020-02-13 14:31:00,551] INFO -  ---> 3c6e215c3dc8

[2020-02-13 14:31:00,551] INFO - Step 6/11 : WORKDIR /opt/program
[2020-02-13 14:31:00,552] INFO - 

/[2020-02-13 14:31:00,735] INFO -  ---> Running in 52d615890214

-[2020-02-13 14:31:01,027] INFO -  ---> 5e43ae14fe08

[2020-02-13 14:31:01,027] INFO - Step 7/11 : RUN conda env update -n base -f /opt/program/environment.yml
[2020-02-13 14:31:01,027] INFO - 

/[2020-02-13 14:31:01,154] INFO -  ---> Running in c06ec52df534

|[2020-02-13 14:31:02,890] INFO - Collecting package metadata (repodata.json): ...working... 
\[2020-02-13 14:31:08,267] INFO - done
Solving environment: ...working... 
|[2020-02-13 14:31:13,049] INFO - done

[2020-02-13 14:31:13,134] INFO - 
Downloading and Extracting Packages
python-3.7.3         | 32.1 MB   |            |   0% 
python-3.7.3         | 32.1 MB   |            |   0% 
python-3.7.3         | 32.1 MB   | 1          |   1% 
python-3.7.3         | 32.1 MB   | 5          |   6% 
python-3.7.3         | 32.1 MB   | 7          |   8% 
python-3.7.3         | 32.1 MB   | #          |  10% 
python-3.7.3         | 32.1 MB   | #1         |  12% 
python-3.7.3         | 32.1 MB   | #3         |  14% 
python-3.7.3         | 32.1 MB   | #5         |  15% 
python-3.7.3         | 32.1 MB   | #7         |  18% 
python-3.7.3         | 32.1 MB   | #9         |  19% 
python-3.7.3         | 32.1 MB   | ##1        |  21% 
python-3.7.3         | 32.1 MB   | ##2        |  23% 
python-3.7.3         | 32.1 MB   | ##3        |  24% 
python-3.7.3         | 32.1 MB   | ##5        |  25% 
python-3.7.3         | 32.1 MB   | ##6        |  26% 
python-3.7.3         | 32.1 MB   | ##8        |  28% 
python-3.7.3         | 32.1 MB   | ###        |  30% 
python-3.7.3         | 32.1 MB   | ###1       |  32% 
python-3.7.3         | 32.1 MB   | ###3       |  33% 
python-3.7.3         | 32.1 MB   | ###5       |  35% 
python-3.7.3         | 32.1 MB   | ###7       |  38% 
python-3.7.3         | 32.1 MB   | ###9       |  39% 
python-3.7.3         | 32.1 MB   | ####1      |  41% 
python-3.7.3         | 32.1 MB   | ####2      |  42% 
python-3.7.3         | 32.1 MB   | ####3      |  44% 
python-3.7.3         | 32.1 MB   | ####5      |  45% 
python-3.7.3         | 32.1 MB   | ####6      |  47% 
python-3.7.3         | 32.1 MB   | ####8      |  48% 
python-3.7.3         | 32.1 MB   | ####9      |  50% 
python-3.7.3         | 32.1 MB   | #####1     |  52% 
python-3.7.3         | 32.1 MB   | #####3     |  53% 
python-3.7.3         | 32.1 MB   | #####5     |  55% 
python-3.7.3         | 32.1 MB   | #####7     |  57% 
python-3.7.3         | 32.1 MB   | #####9     |  59% 
python-3.7.3         | 32.1 MB   | ######1    |  61% 
python-3.7.3         | 32.1 MB   | ######3    |  63% 
python-3.7.3         | 32.1 MB   | ######5    |  65% 
python-3.7.3         | 32.1 MB   | ######7    |  67% 
python-3.7.3         | 32.1 MB   | ######8    |  69% 
python-3.7.3         | 32.1 MB   | #######    |  71% 
python-3.7.3         | 32.1 MB   | #######2   |  72% 
python-3.7.3         | 32.1 MB   | #######4   |  74% 
python-3.7.3         | 32.1 MB   | #######6   |  76% 
python-3.7.3         | 32.1 MB   | #######7   |  78% 
python-3.7.3         | 32.1 MB   | #######9   |  80% 
python-3.7.3         | 32.1 MB   | ########1  |  82% 
python-3.7.3         | 32.1 MB   | ########3  |  83% 
python-3.7.3         | 32.1 MB   | ########4  |  85% 
python-3.7.3         | 32.1 MB   | ########7  |  87% 
python-3.7.3         | 32.1 MB   | ########8  |  89% 
python-3.7.3         | 32.1 MB   | #########  |  91% 
python-3.7.3         | 32.1 MB   | #########2 |  92% 
python-3.7.3         | 32.1 MB   | #########4 |  95% 
python-3.7.3         | 32.1 MB   | #########6 |  96% 
python-3.7.3         | 32.1 MB   | #########7 |  97% 
python-3.7.3         | 32.1 MB   | #########8 |  98% 
python-3.7.3         | 32.1 MB   | #########9 | 100% 
python-3.7.3         | 32.1 MB   | ########## | 100% 
[2020-02-13 14:31:21,815] INFO - 
Preparing transaction: 
[2020-02-13 14:31:21,815] INFO - ...working... 
/[2020-02-13 14:31:21,963] INFO - done

[2020-02-13 14:31:21,963] INFO - Verifying transaction: 
[2020-02-13 14:31:21,964] INFO - ...working... 
-[2020-02-13 14:31:22,731] INFO - done
Executing transaction: ...working... 
\[2020-02-13 14:31:26,670] INFO - done

/[2020-02-13 14:31:27,326] INFO - #
# To activate this environment, use
#
#     $ conda activate base
#
# To deactivate an active environment, use
#
#     $ conda deactivate


-[2020-02-13 14:31:30,473] INFO -  ---> 190cac8cf4e2

[2020-02-13 14:31:30,474] INFO - Step 8/11 : RUN pip install -r /opt/program/requirements.txt
[2020-02-13 14:31:30,474] INFO - 

|[2020-02-13 14:31:30,681] INFO -  ---> Running in 83227fbbb2e5

-[2020-02-13 14:31:32,106] INFO - Collecting bentoml==0.6.2

/[2020-02-13 14:31:32,189] INFO -   Downloading BentoML-0.6.2-py3-none-any.whl (554 kB)

/[2020-02-13 14:31:32,648] INFO - Collecting torch

[2020-02-13 14:31:32,675] INFO -   Downloading torch-1.4.0-cp37-cp37m-manylinux1_x86_64.whl (753.4 MB)

\[2020-02-13 14:34:37,513] INFO - Requirement already satisfied: numpy in /opt/conda/lib/python3.7/site-packages (from -r /opt/program/requirements.txt (line 3)) (1.18.1)

|[2020-02-13 14:34:39,002] INFO - Collecting torchvision

[2020-02-13 14:34:39,035] INFO -   Downloading torchvision-0.5.0-cp37-cp37m-manylinux1_x86_64.whl (4.0 MB)

|[2020-02-13 14:34:40,280] INFO - Collecting scikit-learn

[2020-02-13 14:34:40,294] INFO -   Downloading scikit_learn-0.22.1-cp37-cp37m-manylinux1_x86_64.whl (7.0 MB)

|[2020-02-13 14:34:42,684] INFO - Collecting imageio

[2020-02-13 14:34:42,694] INFO -   Downloading imageio-2.6.1-py3-none-any.whl (3.3 MB)

-[2020-02-13 14:34:43,307] INFO - Collecting flask

[2020-02-13 14:34:43,321] INFO -   Downloading Flask-1.1.1-py2.py3-none-any.whl (94 kB)

/[2020-02-13 14:34:43,399] INFO - Collecting cerberus

[2020-02-13 14:34:43,410] INFO -   Downloading Cerberus-1.3.2.tar.gz (52 kB)

-[2020-02-13 14:34:44,088] INFO - Collecting packaging

[2020-02-13 14:34:44,101] INFO -   Downloading packaging-20.1-py2.py3-none-any.whl (36 kB)

[2020-02-13 14:34:44,124] INFO - Requirement already satisfied: requests in /opt/conda/lib/python3.7/site-packages (from bentoml==0.6.2->-r /opt/program/requirements.txt (line 1)) (2.22.0)

[2020-02-13 14:34:44,147] INFO - Requirement already satisfied: gunicorn in /opt/conda/lib/python3.7/site-packages (from bentoml==0.6.2->-r /opt/program/requirements.txt (line 1)) (20.0.4)

/[2020-02-13 14:34:44,242] INFO - Collecting humanfriendly

[2020-02-13 14:34:44,253] INFO -   Downloading humanfriendly-6.1-py2.py3-none-any.whl (75 kB)

/[2020-02-13 14:34:45,446] INFO - Collecting pandas

[2020-02-13 14:34:45,475] INFO -   Downloading pandas-1.0.1-cp37-cp37m-manylinux1_x86_64.whl (10.1 MB)

/[2020-02-13 14:34:47,966] INFO - Collecting docker

[2020-02-13 14:34:47,978] INFO -   Downloading docker-4.2.0-py2.py3-none-any.whl (143 kB)

|[2020-02-13 14:34:48,071] INFO - Collecting configparser

[2020-02-13 14:34:48,080] INFO -   Downloading configparser-4.0.2-py2.py3-none-any.whl (22 kB)

/[2020-02-13 14:34:48,724] INFO - Collecting boto3

[2020-02-13 14:34:48,752] INFO -   Downloading boto3-1.11.17-py2.py3-none-any.whl (128 kB)

|[2020-02-13 14:34:48,826] INFO - Collecting prometheus-client

[2020-02-13 14:34:48,839] INFO -   Downloading prometheus_client-0.7.1.tar.gz (38 kB)

/[2020-02-13 14:34:49,162] INFO - Collecting python-dateutil<2.8.1,>=2.1

[2020-02-13 14:34:49,173] INFO -   Downloading python_dateutil-2.8.0-py2.py3-none-any.whl (226 kB)

|[2020-02-13 14:34:49,319] INFO - Collecting alembic

\[2020-02-13 14:34:49,338] INFO -   Downloading alembic-1.4.0.tar.gz (1.1 MB)

/[2020-02-13 14:34:50,365] INFO - Collecting protobuf>=3.6.0

[2020-02-13 14:34:50,376] INFO -   Downloading protobuf-3.11.3-cp37-cp37m-manylinux1_x86_64.whl (1.3 MB)

/[2020-02-13 14:34:50,784] INFO - Collecting click>=7.0

[2020-02-13 14:34:50,797] INFO -   Downloading Click-7.0-py2.py3-none-any.whl (81 kB)

/[2020-02-13 14:34:51,682] INFO - Collecting grpcio

|[2020-02-13 14:34:51,700] INFO -   Downloading grpcio-1.27.1-cp37-cp37m-manylinux2010_x86_64.whl (2.7 MB)

\[2020-02-13 14:34:53,118] INFO - Collecting ruamel.yaml>=0.15.0

-[2020-02-13 14:34:53,133] INFO -   Downloading ruamel.yaml-0.16.10-py2.py3-none-any.whl (111 kB)

|[2020-02-13 14:34:53,376] INFO - Collecting sqlalchemy>=1.3.0

[2020-02-13 14:34:53,385] INFO -   Downloading SQLAlchemy-1.3.13.tar.gz (6.0 MB)

|[2020-02-13 14:34:56,215] INFO - Collecting python-json-logger

[2020-02-13 14:34:56,226] INFO -   Downloading python-json-logger-0.1.11.tar.gz (6.0 kB)

-[2020-02-13 14:34:56,501] INFO - Collecting tabulate

/[2020-02-13 14:34:56,512] INFO -   Downloading tabulate-0.8.6.tar.gz (45 kB)

/[2020-02-13 14:34:57,339] INFO - Collecting pillow>=4.1.1

[2020-02-13 14:34:57,358] INFO -   Downloading Pillow-7.0.0-cp37-cp37m-manylinux1_x86_64.whl (2.1 MB)

|[2020-02-13 14:34:57,879] INFO - Requirement already satisfied: six in /opt/conda/lib/python3.7/site-packages (from torchvision->-r /opt/program/requirements.txt (line 4)) (1.12.0)

\[2020-02-13 14:34:58,009] INFO - Collecting joblib>=0.11

[2020-02-13 14:34:58,035] INFO -   Downloading joblib-0.14.1-py2.py3-none-any.whl (294 kB)

|[2020-02-13 14:34:58,328] INFO - Requirement already satisfied: scipy>=0.17.0 in /opt/conda/lib/python3.7/site-packages (from scikit-learn->-r /opt/program/requirements.txt (line 5)) (1.4.1)

\[2020-02-13 14:34:58,385] INFO - Collecting Jinja2>=2.10.1

[2020-02-13 14:34:58,407] INFO -   Downloading Jinja2-2.11.1-py2.py3-none-any.whl (126 kB)

-[2020-02-13 14:34:58,513] INFO - Collecting itsdangerous>=0.24

[2020-02-13 14:34:58,525] INFO -   Downloading itsdangerous-1.1.0-py2.py3-none-any.whl (16 kB)

/[2020-02-13 14:34:58,626] INFO - Collecting Werkzeug>=0.15

[2020-02-13 14:34:58,637] INFO -   Downloading Werkzeug-1.0.0-py2.py3-none-any.whl (298 kB)

\[2020-02-13 14:34:58,771] INFO - Requirement already satisfied: setuptools in /opt/conda/lib/python3.7/site-packages (from cerberus->bentoml==0.6.2->-r /opt/program/requirements.txt (line 1)) (41.4.0)

-[2020-02-13 14:34:58,960] INFO - Collecting pyparsing>=2.0.2

/[2020-02-13 14:34:58,974] INFO -   Downloading pyparsing-2.4.6-py2.py3-none-any.whl (67 kB)

[2020-02-13 14:34:59,013] INFO - Requirement already satisfied: idna<2.9,>=2.5 in /opt/conda/lib/python3.7/site-packages (from requests->bentoml==0.6.2->-r /opt/program/requirements.txt (line 1)) (2.8)

[2020-02-13 14:34:59,016] INFO - Requirement already satisfied: certifi>=2017.4.17 in /opt/conda/lib/python3.7/site-packages (from requests->bentoml==0.6.2->-r /opt/program/requirements.txt (line 1)) (2019.11.28)

[2020-02-13 14:34:59,019] INFO - Requirement already satisfied: chardet<3.1.0,>=3.0.2 in /opt/conda/lib/python3.7/site-packages (from requests->bentoml==0.6.2->-r /opt/program/requirements.txt (line 1)) (3.0.4)

[2020-02-13 14:34:59,023] INFO - Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /opt/conda/lib/python3.7/site-packages (from requests->bentoml==0.6.2->-r /opt/program/requirements.txt (line 1)) (1.24.2)

\[2020-02-13 14:34:59,246] INFO - Collecting pytz>=2017.2

[2020-02-13 14:34:59,258] INFO -   Downloading pytz-2019.3-py2.py3-none-any.whl (509 kB)

\[2020-02-13 14:34:59,590] INFO - Collecting websocket-client>=0.32.0

[2020-02-13 14:34:59,601] INFO -   Downloading websocket_client-0.57.0-py2.py3-none-any.whl (200 kB)

-[2020-02-13 14:34:59,701] INFO - Collecting jmespath<1.0.0,>=0.7.1

[2020-02-13 14:34:59,714] INFO -   Downloading jmespath-0.9.4-py2.py3-none-any.whl (24 kB)

|[2020-02-13 14:35:00,299] INFO - Collecting botocore<1.15.0,>=1.14.17

[2020-02-13 14:35:00,318] INFO -   Downloading botocore-1.14.17-py2.py3-none-any.whl (5.9 MB)

\[2020-02-13 14:35:02,120] INFO - Collecting s3transfer<0.4.0,>=0.3.0

[2020-02-13 14:35:02,134] INFO -   Downloading s3transfer-0.3.3-py2.py3-none-any.whl (69 kB)

-[2020-02-13 14:35:02,218] INFO - Collecting Mako

[2020-02-13 14:35:02,234] INFO -   Downloading Mako-1.1.1.tar.gz (468 kB)

/[2020-02-13 14:35:02,722] INFO - Collecting python-editor>=0.3

[2020-02-13 14:35:02,737] INFO -   Downloading python_editor-1.0.4-py3-none-any.whl (4.9 kB)

|[2020-02-13 14:35:02,802] INFO - Collecting ruamel.yaml.clib>=0.1.2; platform_python_implementation == "CPython" and python_version < "3.9"

[2020-02-13 14:35:02,824] INFO -   Downloading ruamel.yaml.clib-0.2.0-cp37-cp37m-manylinux1_x86_64.whl (547 kB)

/[2020-02-13 14:35:03,072] INFO - Collecting MarkupSafe>=0.23

[2020-02-13 14:35:03,088] INFO -   Downloading MarkupSafe-1.1.1-cp37-cp37m-manylinux1_x86_64.whl (27 kB)

[2020-02-13 14:35:03,143] INFO - Collecting docutils<0.16,>=0.10

[2020-02-13 14:35:03,157] INFO -   Downloading docutils-0.15.2-py3-none-any.whl (547 kB)

\[2020-02-13 14:35:03,333] INFO - Building wheels for collected packages: cerberus, prometheus-client, alembic, sqlalchemy, python-json-logger, tabulate, Mako

[2020-02-13 14:35:03,334] INFO -   Building wheel for cerberus (setup.py): started

|[2020-02-13 14:35:03,654] INFO -   Building wheel for cerberus (setup.py): finished with status 'done'

[2020-02-13 14:35:03,661] INFO -   Created wheel for cerberus: filename=Cerberus-1.3.2-py3-none-any.whl size=54335 sha256=f702d0a95aaa3d389b27ba5e86ac0b8354fbaee4399da2804e2a45b6f3eaf3e3
  Stored in directory: /root/.cache/pip/wheels/17/3a/0d/e2fc48cf85cb858f5e65f1baa36180ebb5dce6397c35c4cfcb

[2020-02-13 14:35:03,662] INFO -   Building wheel for prometheus-client (setup.py): started

/[2020-02-13 14:35:03,956] INFO -   Building wheel for prometheus-client (setup.py): finished with status 'done'

[2020-02-13 14:35:03,956] INFO -   Created wheel for prometheus-client: filename=prometheus_client-0.7.1-py3-none-any.whl size=41402 sha256=86fb50e126740340c291fffd58a91a6be7a6051c2d750a0a3b40055e2a2fd480
  Stored in directory: /root/.cache/pip/wheels/30/0c/26/59ba285bf65dc79d195e9b25e2ddde4c61070422729b0cd914

[2020-02-13 14:35:03,959] INFO -   Building wheel for alembic (setup.py): started

/[2020-02-13 14:35:04,339] INFO -   Building wheel for alembic (setup.py): finished with status 'done'

[2020-02-13 14:35:04,340] INFO -   Created wheel for alembic: filename=alembic-1.4.0-py2.py3-none-any.whl size=157563 sha256=37426f5a1a6b0bce2282f78ae0b2e5ed5054f982bc02d83b6b993749d8c83a47
  Stored in directory: /root/.cache/pip/wheels/33/a9/f9/a53f885636269db5b76cf7afa3a1ab86d9d2fe96610d09274e

[2020-02-13 14:35:04,342] INFO -   Building wheel for sqlalchemy (setup.py): started

/[2020-02-13 14:35:05,974] INFO -   Building wheel for sqlalchemy (setup.py): finished with status 'done'

[2020-02-13 14:35:05,980] INFO -   Created wheel for sqlalchemy: filename=SQLAlchemy-1.3.13-cp37-cp37m-linux_x86_64.whl size=1223686 sha256=4e8babce2bbdccc02b8496265d9d815c3079deb3bd524f536cd7a864f31c3f9b
  Stored in directory: /root/.cache/pip/wheels/b9/ba/77/163f10f14bd489351530603e750c195b0ceceed2f3be2b32f1

[2020-02-13 14:35:05,981] INFO -   Building wheel for python-json-logger (setup.py): started

-[2020-02-13 14:35:06,247] INFO -   Building wheel for python-json-logger (setup.py): finished with status 'done'

[2020-02-13 14:35:06,248] INFO -   Created wheel for python-json-logger: filename=python_json_logger-0.1.11-py2.py3-none-any.whl size=5076 sha256=cf4c02fd3273e7cfbef2b94d0308f84d2f74aa80636d5250d07edfffe48bb46e
  Stored in directory: /root/.cache/pip/wheels/fa/7f/fd/92ccdbb9d1a65486406e0363d2ba5b4ce52f400a915f602ecb

[2020-02-13 14:35:06,250] INFO -   Building wheel for tabulate (setup.py): started

\[2020-02-13 14:35:06,535] INFO -   Building wheel for tabulate (setup.py): finished with status 'done'

[2020-02-13 14:35:06,536] INFO -   Created wheel for tabulate: filename=tabulate-0.8.6-py3-none-any.whl size=23273 sha256=f69d7a2c0734c7d2338d2b2268becb25e996e68aaa64880714173fba6439b652
  Stored in directory: /root/.cache/pip/wheels/09/b6/7e/08b4ee715a1239453e89a59081f0ac369a9036f232e013ecd8

[2020-02-13 14:35:06,538] INFO -   Building wheel for Mako (setup.py): started

|[2020-02-13 14:35:06,871] INFO -   Building wheel for Mako (setup.py): finished with status 'done'

[2020-02-13 14:35:06,872] INFO -   Created wheel for Mako: filename=Mako-1.1.1-py3-none-any.whl size=75409 sha256=390d1d9ec24fb4b66662491e70dbef3c9593c163a0cfdf90f6355611539b5113

[2020-02-13 14:35:06,872] INFO -   Stored in directory: /root/.cache/pip/wheels/11/fe/fa/3693b62cf5ec2b2784b6496734f0ee3e2321eb66d66607e5f9

[2020-02-13 14:35:06,874] INFO - Successfully built cerberus prometheus-client alembic sqlalchemy python-json-logger tabulate Mako

\[2020-02-13 14:35:07,387] INFO - Installing collected packages: MarkupSafe, Jinja2, click, itsdangerous, Werkzeug, flask, cerberus, pyparsing, packaging, humanfriendly, python-dateutil, pytz, pandas, websocket-client, docker, configparser, jmespath, docutils, botocore, s3transfer, boto3, prometheus-client, sqlalchemy, Mako, python-editor, alembic, protobuf, grpcio, ruamel.yaml.clib, ruamel.yaml, python-json-logger, tabulate, bentoml, torch, pillow, torchvision, joblib, scikit-learn, imageio

|[2020-02-13 14:35:45,338] INFO - Successfully installed Jinja2-2.11.1 Mako-1.1.1 MarkupSafe-1.1.1 Werkzeug-1.0.0 alembic-1.4.0 bentoml-0.6.2 boto3-1.11.17 botocore-1.14.17 cerberus-1.3.2 click-7.0 configparser-4.0.2 docker-4.2.0 docutils-0.15.2 flask-1.1.1 grpcio-1.27.1 humanfriendly-6.1 imageio-2.6.1 itsdangerous-1.1.0 jmespath-0.9.4 joblib-0.14.1 packaging-20.1 pandas-1.0.1 pillow-7.0.0 prometheus-client-0.7.1 protobuf-3.11.3 pyparsing-2.4.6 python-dateutil-2.8.0 python-editor-1.0.4 python-json-logger-0.1.11 pytz-2019.3 ruamel.yaml-0.16.10 ruamel.yaml.clib-0.2.0 s3transfer-0.3.3 scikit-learn-0.22.1 sqlalchemy-1.3.13 tabulate-0.8.6 torch-1.4.0 torchvision-0.5.0 websocket-client-0.57.0

/[2020-02-13 14:36:17,983] INFO -  ---> 7bcaa2552bdd

[2020-02-13 14:36:17,983] INFO - Step 9/11 : RUN if [ -f /bento/bentoml_init.sh ]; then /bin/bash -c /bento/bentoml_init.sh; fi
[2020-02-13 14:36:17,985] INFO - 

\[2020-02-13 14:36:18,136] INFO -  ---> Running in 56b0bfa6b9c4

/[2020-02-13 14:36:19,559] INFO -  ---> 9939648103b2

[2020-02-13 14:36:19,560] INFO - Step 10/11 : RUN if [ -f /opt/program/setup.sh ]; then /bin/bash -c /opt/program/setup.sh; fi
[2020-02-13 14:36:19,560] INFO - 

|[2020-02-13 14:36:19,737] INFO -  ---> Running in a271178c15c3

/[2020-02-13 14:36:21,218] INFO -  ---> 6a8c3da0635f

[2020-02-13 14:36:21,218] INFO - Step 11/11 : ENV PATH="/opt/program:${PATH}"
[2020-02-13 14:36:21,219] INFO - 

\[2020-02-13 14:36:21,404] INFO -  ---> Running in bd09032aff38

/[2020-02-13 14:36:21,658] INFO -  ---> da1cac6c729f

[2020-02-13 14:36:21,685] INFO - Successfully built da1cac6c729f

[2020-02-13 14:36:21,690] INFO - Successfully tagged 192023623294.dkr.ecr.us-west-2.amazonaws.com/pytorchfashionclassifier-sagemaker:20200213142355_A51D6E

\[2020-02-13 14:44:24,873] INFO - ApplyDeployment (pytorch-fashion, namespace bobo) succeeded
Successfully created AWS Sagemaker deployment pytorch-fashion
{
  "namespace": "bobo",
  "name": "pytorch-fashion",
  "spec": {
    "bentoName": "PyTorchFashionClassifier",
    "bentoVersion": "20200213142355_A51D6E",
    "operator": "AWS_SAGEMAKER",
    "sagemakerOperatorConfig": {
      "region": "us-west-2",
      "instanceType": "ml.m4.xlarge",
      "instanceCount": 1,
      "apiName": "predict"
    }
  },
  "state": {
    "state": "RUNNING",
    "infoJson": {
      "EndpointName": "bobo-pytorch-fashion",
      "EndpointArn": "arn:aws:sagemaker:us-west-2:192023623294:endpoint/bobo-pytorch-fashion",
      "EndpointConfigName": "bobo-pytorch-fash-PyTorchFashionClassi-20200213142355-A51D6E",
      "ProductionVariants": [
        {
          "VariantName": "bobo-pytorch-fash-PyTorchFashionClassi-20200213142355-A51D6E",
          "DeployedImages": [
            {
              "SpecifiedImage": "192023623294.dkr.ecr.us-west-2.amazonaws.com/pytorchfashionclassifier-sagemaker:20200213142355_A51D6E",
              "ResolvedImage": "192023623294.dkr.ecr.us-west-2.amazonaws.com/[email protected]:6b69da9461ea2a0c8304b53a77629cac2e67ba98c9a8ab98c9f7623183a27d34",
              "ResolutionTime": "2020-02-13 14:44:28.351000-08:00"
            }
          ],
          "CurrentWeight": 1.0,
          "DesiredWeight": 1.0,
          "CurrentInstanceCount": 1,
          "DesiredInstanceCount": 1
        }
      ],
      "EndpointStatus": "InService",
      "CreationTime": "2020-02-13 14:44:24.788000-08:00",
      "LastModifiedTime": "2020-02-13 14:53:35.549000-08:00",
      "ResponseMetadata": {
        "RequestId": "4f042573-3809-43cb-bb58-9c7008fe6640",
        "HTTPStatusCode": 200,
        "HTTPHeaders": {
          "x-amzn-requestid": "4f042573-3809-43cb-bb58-9c7008fe6640",
          "content-type": "application/x-amz-json-1.1",
          "content-length": "845",
          "date": "Thu, 13 Feb 2020 22:53:40 GMT"
        },
        "RetryAttempts": 0
      }
    },
    "timestamp": "2020-02-13T22:53:40.538570Z"
  },
  "createdAt": "2020-02-13T22:30:58.546725Z",
  "lastUpdatedAt": "2020-02-13T22:30:58.546756Z"
}

bentoml sagemaker list lists all Sagemaker deployments

In [31]:
!bentoml sagemaker list
NAME             NAMESPACE    PLATFORM       BENTO_SERVICE                                   STATUS    AGE
pytorch-fashion  bobo         aws-sagemaker  PyTorchFashionClassifier:20200213142355_A51D6E  running   22 minutes and 57.12 seconds
In [34]:
!bentoml sagemaker get pytorch-fashion
{
  "namespace": "bobo",
  "name": "pytorch-fashion",
  "spec": {
    "bentoName": "PyTorchFashionClassifier",
    "bentoVersion": "20200213142355_A51D6E",
    "operator": "AWS_SAGEMAKER",
    "sagemakerOperatorConfig": {
      "region": "us-west-2",
      "instanceType": "ml.m4.xlarge",
      "instanceCount": 1,
      "apiName": "predict"
    }
  },
  "state": {
    "state": "RUNNING",
    "infoJson": {
      "EndpointName": "bobo-pytorch-fashion",
      "EndpointArn": "arn:aws:sagemaker:us-west-2:192023623294:endpoint/bobo-pytorch-fashion",
      "EndpointConfigName": "bobo-pytorch-fash-PyTorchFashionClassi-20200213142355-A51D6E",
      "ProductionVariants": [
        {
          "VariantName": "bobo-pytorch-fash-PyTorchFashionClassi-20200213142355-A51D6E",
          "DeployedImages": [
            {
              "SpecifiedImage": "192023623294.dkr.ecr.us-west-2.amazonaws.com/pytorchfashionclassifier-sagemaker:20200213142355_A51D6E",
              "ResolvedImage": "192023623294.dkr.ecr.us-west-2.amazonaws.com/[email protected]:6b69da9461ea2a0c8304b53a77629cac2e67ba98c9a8ab98c9f7623183a27d34",
              "ResolutionTime": "2020-02-13 14:44:28.351000-08:00"
            }
          ],
          "CurrentWeight": 1.0,
          "DesiredWeight": 1.0,
          "CurrentInstanceCount": 1,
          "DesiredInstanceCount": 1
        }
      ],
      "EndpointStatus": "InService",
      "CreationTime": "2020-02-13 14:44:24.788000-08:00",
      "LastModifiedTime": "2020-02-13 14:53:35.549000-08:00",
      "ResponseMetadata": {
        "RequestId": "ada1ef58-47b4-4b65-a1ac-cfc02c0eb969",
        "HTTPStatusCode": 200,
        "HTTPHeaders": {
          "x-amzn-requestid": "ada1ef58-47b4-4b65-a1ac-cfc02c0eb969",
          "content-type": "application/x-amz-json-1.1",
          "content-length": "845",
          "date": "Thu, 13 Feb 2020 22:54:43 GMT"
        },
        "RetryAttempts": 0
      }
    },
    "timestamp": "2020-02-13T22:54:43.436462Z"
  },
  "createdAt": "2020-02-13T22:30:58.546725Z",
  "lastUpdatedAt": "2020-02-13T22:30:58.546756Z"
}

Test and validate Sagemaker deployment with aws sagemaker-runtime invoke-endpoint command

In [35]:
!aws sagemaker-runtime invoke-endpoint --endpoint-name bobo-pytorch-fashion \
--body fileb:///Users/bozhaoyu/src/bento_gallery/pytorch/fashion-mnist/sample_image.png \
--content-type image/png output.json && cat output.json
{
    "ContentType": "application/json",
    "InvokedProductionVariant": "bobo-pytorch-fash-PyTorchFashionClassi-20200213142355-A51D6E"
}
"Bag"

Clean up Sagemaker deployment with bentoml sagemaker delete

In [36]:
!bentoml sagemaker delete pytorch-fashion
Successfully deleted AWS Sagemaker deployment "pytorch-fashion"
In [ ]: