Use BentoML with ONNX model zoo(resnet50)

BentoML makes moving trained ML models to production easy:

  • Package models trained with any ML framework and reproduce them for model serving in production
  • Deploy anywhere for online API serving or offline batch serving
  • High-Performance API model server with adaptive micro-batching support
  • Central hub for managing models and deployment process via Web UI and APIs
  • Modular and flexible design making it adaptable to your infrastrcuture

BentoML is a framework for serving, managing, and deploying machine learning models. It is aiming to bridge the gap between Data Science and DevOps, and enable teams to deliver prediction services in a fast, repeatable, and scalable way.

Before reading this example project, be sure to check out the Getting started guide to learn about the basic concepts in BentoML.

This example notebook demonstrates how to use ONNX model zoo with BentoML. It defines a BentoService with resnet50 model and deploys it to AWS sagemaker as an API endpoint.

original notebook: https://github.com/onnx/onnx-docker/blob/master/onnx-ecosystem/inference_demos/resnet50_modelzoo_onnxruntime_inference.ipynb

Impression

In [1]:
%reload_ext autoreload
%autoreload 2
%matplotlib inline
In [3]:
!pip install -q bentoml onnx>=1.7.0 onnxruntime>=1.4.0
Requirement already satisfied: onnx in /opt/anaconda3/lib/python3.7/site-packages (1.7.0)
Requirement already satisfied: onnxruntime in /opt/anaconda3/lib/python3.7/site-packages (1.4.0)
Requirement already satisfied: typing-extensions>=3.6.2.1 in /opt/anaconda3/lib/python3.7/site-packages (from onnx) (3.7.4.2)
Requirement already satisfied: six in /opt/anaconda3/lib/python3.7/site-packages (from onnx) (1.15.0)
Requirement already satisfied: protobuf in /opt/anaconda3/lib/python3.7/site-packages (from onnx) (3.11.3)
Requirement already satisfied: numpy in /opt/anaconda3/lib/python3.7/site-packages (from onnx) (1.18.5)
Requirement already satisfied: setuptools in /opt/anaconda3/lib/python3.7/site-packages (from protobuf->onnx) (49.1.0.post20200710)
In [5]:
import numpy as np    # we're going to use numpy to process input and output data
import onnxruntime    # to inference ONNX models, we use the ONNX Runtime
import onnx
from onnx import numpy_helper
import urllib.request
import json
import time

# display images in notebook
import matplotlib.pyplot as plt
from PIL import Image, ImageDraw, ImageFont

%matplotlib inline
In [3]:
onnx_model_url = "https://s3.amazonaws.com/onnx-model-zoo/resnet/resnet50v2/resnet50v2.tar.gz"
imagenet_labels_url = "https://raw.githubusercontent.com/anishathalye/imagenet-simple-labels/master/imagenet-simple-labels.json"

# retrieve our model from the ONNX model zoo
urllib.request.urlretrieve(onnx_model_url, filename="resnet50v2.tar.gz")
urllib.request.urlretrieve(imagenet_labels_url, filename="imagenet-simple-labels.json")

!curl https://raw.githubusercontent.com/onnx/onnx-docker/master/onnx-ecosystem/inference_demos/images/dog.jpg -o dog.jpg
!tar xvzf resnet50v2.tar.gz
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 21240  100 21240    0     0  59830      0 --:--:-- --:--:-- --:--:-- 59662
x resnet50v2/
x resnet50v2/resnet50v2.onnx
x resnet50v2/test_data_set_0/
x resnet50v2/test_data_set_1/
x resnet50v2/test_data_set_2/
x resnet50v2/test_data_set_2/input_0.pb
x resnet50v2/test_data_set_2/output_0.pb
x resnet50v2/test_data_set_1/input_0.pb
x resnet50v2/test_data_set_1/output_0.pb
x resnet50v2/test_data_set_0/input_0.pb
x resnet50v2/test_data_set_0/output_0.pb

Load sample outputs and inputs

In [4]:
test_data_dir = 'resnet50v2/test_data_set'
test_data_num = 3
In [5]:
import glob
import os

# Load inputs
inputs = []
for i in range(test_data_num):
    input_file = os.path.join(test_data_dir + '_{}'.format(i), 'input_0.pb')
    tensor = onnx.TensorProto()
    with open(input_file, 'rb') as f:
        tensor.ParseFromString(f.read())
        inputs.append(numpy_helper.to_array(tensor))

print('Loaded {} inputs successfully.'.format(test_data_num))
        
# Load reference outputs

ref_outputs = []
for i in range(test_data_num):
    output_file = os.path.join(test_data_dir + '_{}'.format(i), 'output_0.pb')
    tensor = onnx.TensorProto()
    with open(output_file, 'rb') as f:
        tensor.ParseFromString(f.read())    
        ref_outputs.append(numpy_helper.to_array(tensor))
        
print('Loaded {} reference outputs successfully.'.format(test_data_num))
Loaded 3 inputs successfully.
Loaded 3 reference outputs successfully.
In [6]:
def load_labels(path):
    with open(path) as f:
        data = json.load(f)
    return np.asarray(data)

labels = load_labels('imagenet-simple-labels.json')
labels
Out[6]:
array(['tench', 'goldfish', 'great white shark', 'tiger shark',
       'hammerhead shark', 'electric ray', 'stingray', 'cock', 'hen',
       'ostrich', 'brambling', 'goldfinch', 'house finch', 'junco',
       'indigo bunting', 'American robin', 'bulbul', 'jay', 'magpie',
       'chickadee', 'American dipper', 'kite', 'bald eagle', 'vulture',
       'great grey owl', 'fire salamander', 'smooth newt', 'newt',
       'spotted salamander', 'axolotl', 'American bullfrog', 'tree frog',
       'tailed frog', 'loggerhead sea turtle', 'leatherback sea turtle',
       'mud turtle', 'terrapin', 'box turtle', 'banded gecko',
       'green iguana', 'Carolina anole',
       'desert grassland whiptail lizard', 'agama',
       'frilled-necked lizard', 'alligator lizard', 'Gila monster',
       'European green lizard', 'chameleon', 'Komodo dragon',
       'Nile crocodile', 'American alligator', 'triceratops',
       'worm snake', 'ring-necked snake', 'eastern hog-nosed snake',
       'smooth green snake', 'kingsnake', 'garter snake', 'water snake',
       'vine snake', 'night snake', 'boa constrictor',
       'African rock python', 'Indian cobra', 'green mamba', 'sea snake',
       'Saharan horned viper', 'eastern diamondback rattlesnake',
       'sidewinder', 'trilobite', 'harvestman', 'scorpion',
       'yellow garden spider', 'barn spider', 'European garden spider',
       'southern black widow', 'tarantula', 'wolf spider', 'tick',
       'centipede', 'black grouse', 'ptarmigan', 'ruffed grouse',
       'prairie grouse', 'peacock', 'quail', 'partridge', 'grey parrot',
       'macaw', 'sulphur-crested cockatoo', 'lorikeet', 'coucal',
       'bee eater', 'hornbill', 'hummingbird', 'jacamar', 'toucan',
       'duck', 'red-breasted merganser', 'goose', 'black swan', 'tusker',
       'echidna', 'platypus', 'wallaby', 'koala', 'wombat', 'jellyfish',
       'sea anemone', 'brain coral', 'flatworm', 'nematode', 'conch',
       'snail', 'slug', 'sea slug', 'chiton', 'chambered nautilus',
       'Dungeness crab', 'rock crab', 'fiddler crab', 'red king crab',
       'American lobster', 'spiny lobster', 'crayfish', 'hermit crab',
       'isopod', 'white stork', 'black stork', 'spoonbill', 'flamingo',
       'little blue heron', 'great egret', 'bittern', 'crane (bird)',
       'limpkin', 'common gallinule', 'American coot', 'bustard',
       'ruddy turnstone', 'dunlin', 'common redshank', 'dowitcher',
       'oystercatcher', 'pelican', 'king penguin', 'albatross',
       'grey whale', 'killer whale', 'dugong', 'sea lion', 'Chihuahua',
       'Japanese Chin', 'Maltese', 'Pekingese', 'Shih Tzu',
       'King Charles Spaniel', 'Papillon', 'toy terrier',
       'Rhodesian Ridgeback', 'Afghan Hound', 'Basset Hound', 'Beagle',
       'Bloodhound', 'Bluetick Coonhound', 'Black and Tan Coonhound',
       'Treeing Walker Coonhound', 'English foxhound',
       'Redbone Coonhound', 'borzoi', 'Irish Wolfhound',
       'Italian Greyhound', 'Whippet', 'Ibizan Hound',
       'Norwegian Elkhound', 'Otterhound', 'Saluki', 'Scottish Deerhound',
       'Weimaraner', 'Staffordshire Bull Terrier',
       'American Staffordshire Terrier', 'Bedlington Terrier',
       'Border Terrier', 'Kerry Blue Terrier', 'Irish Terrier',
       'Norfolk Terrier', 'Norwich Terrier', 'Yorkshire Terrier',
       'Wire Fox Terrier', 'Lakeland Terrier', 'Sealyham Terrier',
       'Airedale Terrier', 'Cairn Terrier', 'Australian Terrier',
       'Dandie Dinmont Terrier', 'Boston Terrier', 'Miniature Schnauzer',
       'Giant Schnauzer', 'Standard Schnauzer', 'Scottish Terrier',
       'Tibetan Terrier', 'Australian Silky Terrier',
       'Soft-coated Wheaten Terrier', 'West Highland White Terrier',
       'Lhasa Apso', 'Flat-Coated Retriever', 'Curly-coated Retriever',
       'Golden Retriever', 'Labrador Retriever',
       'Chesapeake Bay Retriever', 'German Shorthaired Pointer', 'Vizsla',
       'English Setter', 'Irish Setter', 'Gordon Setter', 'Brittany',
       'Clumber Spaniel', 'English Springer Spaniel',
       'Welsh Springer Spaniel', 'Cocker Spaniels', 'Sussex Spaniel',
       'Irish Water Spaniel', 'Kuvasz', 'Schipperke', 'Groenendael',
       'Malinois', 'Briard', 'Australian Kelpie', 'Komondor',
       'Old English Sheepdog', 'Shetland Sheepdog', 'collie',
       'Border Collie', 'Bouvier des Flandres', 'Rottweiler',
       'German Shepherd Dog', 'Dobermann', 'Miniature Pinscher',
       'Greater Swiss Mountain Dog', 'Bernese Mountain Dog',
       'Appenzeller Sennenhund', 'Entlebucher Sennenhund', 'Boxer',
       'Bullmastiff', 'Tibetan Mastiff', 'French Bulldog', 'Great Dane',
       'St. Bernard', 'husky', 'Alaskan Malamute', 'Siberian Husky',
       'Dalmatian', 'Affenpinscher', 'Basenji', 'pug', 'Leonberger',
       'Newfoundland', 'Pyrenean Mountain Dog', 'Samoyed', 'Pomeranian',
       'Chow Chow', 'Keeshond', 'Griffon Bruxellois',
       'Pembroke Welsh Corgi', 'Cardigan Welsh Corgi', 'Toy Poodle',
       'Miniature Poodle', 'Standard Poodle', 'Mexican hairless dog',
       'grey wolf', 'Alaskan tundra wolf', 'red wolf', 'coyote', 'dingo',
       'dhole', 'African wild dog', 'hyena', 'red fox', 'kit fox',
       'Arctic fox', 'grey fox', 'tabby cat', 'tiger cat', 'Persian cat',
       'Siamese cat', 'Egyptian Mau', 'cougar', 'lynx', 'leopard',
       'snow leopard', 'jaguar', 'lion', 'tiger', 'cheetah', 'brown bear',
       'American black bear', 'polar bear', 'sloth bear', 'mongoose',
       'meerkat', 'tiger beetle', 'ladybug', 'ground beetle',
       'longhorn beetle', 'leaf beetle', 'dung beetle',
       'rhinoceros beetle', 'weevil', 'fly', 'bee', 'ant', 'grasshopper',
       'cricket', 'stick insect', 'cockroach', 'mantis', 'cicada',
       'leafhopper', 'lacewing', 'dragonfly', 'damselfly', 'red admiral',
       'ringlet', 'monarch butterfly', 'small white', 'sulphur butterfly',
       'gossamer-winged butterfly', 'starfish', 'sea urchin',
       'sea cucumber', 'cottontail rabbit', 'hare', 'Angora rabbit',
       'hamster', 'porcupine', 'fox squirrel', 'marmot', 'beaver',
       'guinea pig', 'common sorrel', 'zebra', 'pig', 'wild boar',
       'warthog', 'hippopotamus', 'ox', 'water buffalo', 'bison', 'ram',
       'bighorn sheep', 'Alpine ibex', 'hartebeest', 'impala', 'gazelle',
       'dromedary', 'llama', 'weasel', 'mink', 'European polecat',
       'black-footed ferret', 'otter', 'skunk', 'badger', 'armadillo',
       'three-toed sloth', 'orangutan', 'gorilla', 'chimpanzee', 'gibbon',
       'siamang', 'guenon', 'patas monkey', 'baboon', 'macaque', 'langur',
       'black-and-white colobus', 'proboscis monkey', 'marmoset',
       'white-headed capuchin', 'howler monkey', 'titi',
       "Geoffroy's spider monkey", 'common squirrel monkey',
       'ring-tailed lemur', 'indri', 'Asian elephant',
       'African bush elephant', 'red panda', 'giant panda', 'snoek',
       'eel', 'coho salmon', 'rock beauty', 'clownfish', 'sturgeon',
       'garfish', 'lionfish', 'pufferfish', 'abacus', 'abaya',
       'academic gown', 'accordion', 'acoustic guitar',
       'aircraft carrier', 'airliner', 'airship', 'altar', 'ambulance',
       'amphibious vehicle', 'analog clock', 'apiary', 'apron',
       'waste container', 'assault rifle', 'backpack', 'bakery',
       'balance beam', 'balloon', 'ballpoint pen', 'Band-Aid', 'banjo',
       'baluster', 'barbell', 'barber chair', 'barbershop', 'barn',
       'barometer', 'barrel', 'wheelbarrow', 'baseball', 'basketball',
       'bassinet', 'bassoon', 'swimming cap', 'bath towel', 'bathtub',
       'station wagon', 'lighthouse', 'beaker', 'military cap',
       'beer bottle', 'beer glass', 'bell-cot', 'bib', 'tandem bicycle',
       'bikini', 'ring binder', 'binoculars', 'birdhouse', 'boathouse',
       'bobsleigh', 'bolo tie', 'poke bonnet', 'bookcase', 'bookstore',
       'bottle cap', 'bow', 'bow tie', 'brass', 'bra', 'breakwater',
       'breastplate', 'broom', 'bucket', 'buckle', 'bulletproof vest',
       'high-speed train', 'butcher shop', 'taxicab', 'cauldron',
       'candle', 'cannon', 'canoe', 'can opener', 'cardigan',
       'car mirror', 'carousel', 'tool kit', 'carton', 'car wheel',
       'automated teller machine', 'cassette', 'cassette player',
       'castle', 'catamaran', 'CD player', 'cello', 'mobile phone',
       'chain', 'chain-link fence', 'chain mail', 'chainsaw', 'chest',
       'chiffonier', 'chime', 'china cabinet', 'Christmas stocking',
       'church', 'movie theater', 'cleaver', 'cliff dwelling', 'cloak',
       'clogs', 'cocktail shaker', 'coffee mug', 'coffeemaker', 'coil',
       'combination lock', 'computer keyboard', 'confectionery store',
       'container ship', 'convertible', 'corkscrew', 'cornet',
       'cowboy boot', 'cowboy hat', 'cradle', 'crane (machine)',
       'crash helmet', 'crate', 'infant bed', 'Crock Pot', 'croquet ball',
       'crutch', 'cuirass', 'dam', 'desk', 'desktop computer',
       'rotary dial telephone', 'diaper', 'digital clock',
       'digital watch', 'dining table', 'dishcloth', 'dishwasher',
       'disc brake', 'dock', 'dog sled', 'dome', 'doormat',
       'drilling rig', 'drum', 'drumstick', 'dumbbell', 'Dutch oven',
       'electric fan', 'electric guitar', 'electric locomotive',
       'entertainment center', 'envelope', 'espresso machine',
       'face powder', 'feather boa', 'filing cabinet', 'fireboat',
       'fire engine', 'fire screen sheet', 'flagpole', 'flute',
       'folding chair', 'football helmet', 'forklift', 'fountain',
       'fountain pen', 'four-poster bed', 'freight car', 'French horn',
       'frying pan', 'fur coat', 'garbage truck', 'gas mask', 'gas pump',
       'goblet', 'go-kart', 'golf ball', 'golf cart', 'gondola', 'gong',
       'gown', 'grand piano', 'greenhouse', 'grille', 'grocery store',
       'guillotine', 'barrette', 'hair spray', 'half-track', 'hammer',
       'hamper', 'hair dryer', 'hand-held computer', 'handkerchief',
       'hard disk drive', 'harmonica', 'harp', 'harvester', 'hatchet',
       'holster', 'home theater', 'honeycomb', 'hook', 'hoop skirt',
       'horizontal bar', 'horse-drawn vehicle', 'hourglass', 'iPod',
       'clothes iron', "jack-o'-lantern", 'jeans', 'jeep', 'T-shirt',
       'jigsaw puzzle', 'pulled rickshaw', 'joystick', 'kimono',
       'knee pad', 'knot', 'lab coat', 'ladle', 'lampshade',
       'laptop computer', 'lawn mower', 'lens cap', 'paper knife',
       'library', 'lifeboat', 'lighter', 'limousine', 'ocean liner',
       'lipstick', 'slip-on shoe', 'lotion', 'speaker', 'loupe',
       'sawmill', 'magnetic compass', 'mail bag', 'mailbox', 'tights',
       'tank suit', 'manhole cover', 'maraca', 'marimba', 'mask', 'match',
       'maypole', 'maze', 'measuring cup', 'medicine chest', 'megalith',
       'microphone', 'microwave oven', 'military uniform', 'milk can',
       'minibus', 'miniskirt', 'minivan', 'missile', 'mitten',
       'mixing bowl', 'mobile home', 'Model T', 'modem', 'monastery',
       'monitor', 'moped', 'mortar', 'square academic cap', 'mosque',
       'mosquito net', 'scooter', 'mountain bike', 'tent',
       'computer mouse', 'mousetrap', 'moving van', 'muzzle', 'nail',
       'neck brace', 'necklace', 'nipple', 'notebook computer', 'obelisk',
       'oboe', 'ocarina', 'odometer', 'oil filter', 'organ',
       'oscilloscope', 'overskirt', 'bullock cart', 'oxygen mask',
       'packet', 'paddle', 'paddle wheel', 'padlock', 'paintbrush',
       'pajamas', 'palace', 'pan flute', 'paper towel', 'parachute',
       'parallel bars', 'park bench', 'parking meter', 'passenger car',
       'patio', 'payphone', 'pedestal', 'pencil case', 'pencil sharpener',
       'perfume', 'Petri dish', 'photocopier', 'plectrum', 'Pickelhaube',
       'picket fence', 'pickup truck', 'pier', 'piggy bank',
       'pill bottle', 'pillow', 'ping-pong ball', 'pinwheel',
       'pirate ship', 'pitcher', 'hand plane', 'planetarium',
       'plastic bag', 'plate rack', 'plow', 'plunger', 'Polaroid camera',
       'pole', 'police van', 'poncho', 'billiard table', 'soda bottle',
       'pot', "potter's wheel", 'power drill', 'prayer rug', 'printer',
       'prison', 'projectile', 'projector', 'hockey puck', 'punching bag',
       'purse', 'quill', 'quilt', 'race car', 'racket', 'radiator',
       'radio', 'radio telescope', 'rain barrel', 'recreational vehicle',
       'reel', 'reflex camera', 'refrigerator', 'remote control',
       'restaurant', 'revolver', 'rifle', 'rocking chair', 'rotisserie',
       'eraser', 'rugby ball', 'ruler', 'running shoe', 'safe',
       'safety pin', 'salt shaker', 'sandal', 'sarong', 'saxophone',
       'scabbard', 'weighing scale', 'school bus', 'schooner',
       'scoreboard', 'CRT screen', 'screw', 'screwdriver', 'seat belt',
       'sewing machine', 'shield', 'shoe store', 'shoji',
       'shopping basket', 'shopping cart', 'shovel', 'shower cap',
       'shower curtain', 'ski', 'ski mask', 'sleeping bag', 'slide rule',
       'sliding door', 'slot machine', 'snorkel', 'snowmobile',
       'snowplow', 'soap dispenser', 'soccer ball', 'sock',
       'solar thermal collector', 'sombrero', 'soup bowl', 'space bar',
       'space heater', 'space shuttle', 'spatula', 'motorboat',
       'spider web', 'spindle', 'sports car', 'spotlight', 'stage',
       'steam locomotive', 'through arch bridge', 'steel drum',
       'stethoscope', 'scarf', 'stone wall', 'stopwatch', 'stove',
       'strainer', 'tram', 'stretcher', 'couch', 'stupa', 'submarine',
       'suit', 'sundial', 'sunglass', 'sunglasses', 'sunscreen',
       'suspension bridge', 'mop', 'sweatshirt', 'swimsuit', 'swing',
       'switch', 'syringe', 'table lamp', 'tank', 'tape player', 'teapot',
       'teddy bear', 'television', 'tennis ball', 'thatched roof',
       'front curtain', 'thimble', 'threshing machine', 'throne',
       'tile roof', 'toaster', 'tobacco shop', 'toilet seat', 'torch',
       'totem pole', 'tow truck', 'toy store', 'tractor',
       'semi-trailer truck', 'tray', 'trench coat', 'tricycle',
       'trimaran', 'tripod', 'triumphal arch', 'trolleybus', 'trombone',
       'tub', 'turnstile', 'typewriter keyboard', 'umbrella', 'unicycle',
       'upright piano', 'vacuum cleaner', 'vase', 'vault', 'velvet',
       'vending machine', 'vestment', 'viaduct', 'violin', 'volleyball',
       'waffle iron', 'wall clock', 'wallet', 'wardrobe',
       'military aircraft', 'sink', 'washing machine', 'water bottle',
       'water jug', 'water tower', 'whiskey jug', 'whistle', 'wig',
       'window screen', 'window shade', 'Windsor tie', 'wine bottle',
       'wing', 'wok', 'wooden spoon', 'wool', 'split-rail fence',
       'shipwreck', 'yawl', 'yurt', 'website', 'comic book', 'crossword',
       'traffic sign', 'traffic light', 'dust jacket', 'menu', 'plate',
       'guacamole', 'consomme', 'hot pot', 'trifle', 'ice cream',
       'ice pop', 'baguette', 'bagel', 'pretzel', 'cheeseburger',
       'hot dog', 'mashed potato', 'cabbage', 'broccoli', 'cauliflower',
       'zucchini', 'spaghetti squash', 'acorn squash', 'butternut squash',
       'cucumber', 'artichoke', 'bell pepper', 'cardoon', 'mushroom',
       'Granny Smith', 'strawberry', 'orange', 'lemon', 'fig',
       'pineapple', 'banana', 'jackfruit', 'custard apple', 'pomegranate',
       'hay', 'carbonara', 'chocolate syrup', 'dough', 'meatloaf',
       'pizza', 'pot pie', 'burrito', 'red wine', 'espresso', 'cup',
       'eggnog', 'alp', 'bubble', 'cliff', 'coral reef', 'geyser',
       'lakeshore', 'promontory', 'shoal', 'seashore', 'valley',
       'volcano', 'baseball player', 'bridegroom', 'scuba diver',
       'rapeseed', 'daisy', "yellow lady's slipper", 'corn', 'acorn',
       'rose hip', 'horse chestnut seed', 'coral fungus', 'agaric',
       'gyromitra', 'stinkhorn mushroom', 'earth star',
       'hen-of-the-woods', 'bolete', 'ear', 'toilet paper'], dtype='<U32')
In [9]:
%%writefile onnx_resnet50.py
from typing import List

import numpy as np

import bentoml
from bentoml.frameworks.onnx import OnnxModelArtifact
from bentoml.service.artifacts.common import PickleArtifact
from bentoml.adapters import ImageInput


@bentoml.env(infer_pip_packages=True)
@bentoml.artifacts([OnnxModelArtifact('model'), PickleArtifact('labels')])
class OnnxResnet50(bentoml.BentoService):
    def preprocess(self, input_data):
        # convert the input data into the float32 input
        img_data = np.stack(input_data).transpose(0, 3, 1, 2)

        #normalize
        mean_vec = np.array([0.485, 0.456, 0.406])
        stddev_vec = np.array([0.229, 0.224, 0.225])


        norm_img_data = np.zeros(img_data.shape).astype('float32')


        for i in range(img_data.shape[0]):
            for j in range(img_data.shape[1]):
                norm_img_data[i,j,:,:] = (img_data[i,j,:,:]/255 - mean_vec[j]) / stddev_vec[j]

        #add batch channel
        norm_img_data = norm_img_data.reshape(-1, 3, 224, 224).astype('float32')
        return norm_img_data

    def softmax(self, x):
        x = x.reshape(-1)
        e_x = np.exp(x - np.max(x))
        return e_x / e_x.sum(axis=0)
    
    def post_process(self, raw_result):
        return self.softmax(np.array(raw_result)).tolist()

    @bentoml.api(input=ImageInput(), batch=True)
    def predict(self, image_ndarrays: List[np.ndarray]) -> List[str]:
        input_datas = self.preprocess(image_ndarrays)
        input_name = self.artifacts.model.get_inputs()[0].name
        
        outputs = []
        for i in range(input_datas.shape[0]):
            raw_result = self.artifacts.model.run([], {input_name: input_datas[i:i+1]})
            result = self.post_process(raw_result)
            idx = np.argmax(result)
            sort_idx = np.flip(np.squeeze(np.argsort(result)))

            # return top 5 labels
            outputs.append(self.artifacts.labels[sort_idx[:5]])
        return outputs
        
Overwriting onnx_resnet50.py
In [10]:
from onnx_resnet50 import OnnxResnet50

svc = OnnxResnet50()
svc.pack('labels', labels)
svc.pack('model', 'resnet50v2/resnet50v2.onnx')

saved_path = svc.save()
saved_path
[2020-09-22 12:44:00,993] WARNING - Using BentoML installed in `editable` model, the local BentoML repository including all code changes will be packaged together with saved bundle created, under the './bundled_pip_dependencies' directory of the saved bundle.
[2020-09-22 12:44:01,431] INFO - Using default docker base image: `None` specified inBentoML config file or env var. User must make sure that the docker base image either has Python 3.7 or conda installed.
[2020-09-22 12:44:03,187] INFO - Detected non-PyPI-released BentoML installed, copying local BentoML modulefiles to target saved bundle path..
/usr/local/anaconda3/envs/dev-py3/lib/python3.7/site-packages/setuptools/dist.py:476: UserWarning: Normalizing '0.9.0.pre+3.gcebf2015' to '0.9.0rc0+3.gcebf2015'
  normalized_version,
warning: no previously-included files matching '*~' found anywhere in distribution
warning: no previously-included files matching '*.pyo' found anywhere in distribution
warning: no previously-included files matching '.git' found anywhere in distribution
warning: no previously-included files matching '.ipynb_checkpoints' found anywhere in distribution
warning: no previously-included files matching '__pycache__' found anywhere in distribution
no previously-included directories found matching 'e2e_tests'
no previously-included directories found matching 'tests'
no previously-included directories found matching 'benchmark'
UPDATING BentoML-0.9.0rc0+3.gcebf2015/bentoml/_version.py
set BentoML-0.9.0rc0+3.gcebf2015/bentoml/_version.py to '0.9.0.pre+3.gcebf2015'
[2020-09-22 12:44:08,076] INFO - BentoService bundle 'OnnxResnet50:20200922124402_4DE94A' saved to: /Users/bozhaoyu/bentoml/repository/OnnxResnet50/20200922124402_4DE94A
Out[10]:
'/Users/bozhaoyu/bentoml/repository/OnnxResnet50/20200922124402_4DE94A'

REST API Model Serving

To start a REST API model server with the BentoService saved above, use the bentoml serve command:

In [ ]:
!bentoml serve OnnxResnet50:latest
[2020-09-22 14:29:26,500] INFO - Getting latest version OnnxResnet50:20200922124402_4DE94A
[2020-09-22 14:29:26,501] INFO - Starting BentoML API server in development mode..
[2020-09-22 14:29:26,680] WARNING - Using BentoML installed in `editable` model, the local BentoML repository including all code changes will be packaged together with saved bundle created, under the './bundled_pip_dependencies' directory of the saved bundle.
[2020-09-22 14:29:26,696] WARNING - Saved BentoService bundle version mismatch: loading BentoService bundle create with BentoML version 0.9.0.pre, but loading from BentoML version 0.9.0.pre+3.gcebf2015
[2020-09-22 14:29:26,858] INFO - Using default docker base image: `None` specified inBentoML config file or env var. User must make sure that the docker base image either has Python 3.7 or conda installed.
 * Serving Flask app "OnnxResnet50" (lazy loading)
 * Environment: production
   WARNING: This is a development server. Do not use it in a production deployment.
   Use a production WSGI server instead.
 * Debug mode: off
 * Running on http://127.0.0.1:5001/ (Press CTRL+C to quit)
[2020-09-22 14:29:47,228] INFO - Initializing onnxruntime InferenceSession from onnx file:'/Users/bozhaoyu/bentoml/repository/OnnxResnet50/20200922124402_4DE94A/OnnxResnet50/artifacts/model.onnx'
[2020-09-22 14:29:52,696] INFO - {'service_name': 'OnnxResnet50', 'service_version': '20200922124402_4DE94A', 'api': 'predict', 'task': {'data': {'name': 'dog.jpg'}, 'task_id': '78788565-41ac-49ca-9d2e-c09bf8d9cd15', 'http_headers': (('Host', '127.0.0.1:5001'), ('User-Agent', 'curl/7.65.3'), ('Accept', '*/*'), ('Content-Length', '21426'), ('Content-Type', 'multipart/form-data; boundary=------------------------c405a7b3410476a3'), ('Expect', '100-continue'))}, 'result': {'data': '["Golden Retriever", "Labrador Retriever", "Sussex Spaniel", "Vizsla", "Otterhound"]', 'http_status': 200, 'http_headers': (('Content-Type', 'application/json'),)}, 'request_id': '78788565-41ac-49ca-9d2e-c09bf8d9cd15'}
127.0.0.1 - - [22/Sep/2020 14:29:52] "POST /predict HTTP/1.1" 200 -
WARNING: Logging before flag parsing goes to stderr.
I0922 14:29:52.697955 4778073536 _internal.py:122] 127.0.0.1 - - [22/Sep/2020 14:29:52] "POST /predict HTTP/1.1" 200 -

If you are running this notebook from Google Colab, you can start the dev server with --run-with-ngrok option, to gain acccess to the API endpoint via a public endpoint managed by ngrok:

In [ ]:
!bentoml serve OnnxResnet50:latest --run-with-ngrok

Sending POST request from termnial:

curl -X POST "http://127.0.0.1:5000/predict" -F image=@dog.jpg
curl -X POST "http://127.0.0.1:5000/predict" -H "Content-Type: image/png" --data-binary @dog.jpg

Go visit http://127.0.0.1:5000/ from your browser, click /predict -> Try it out -> Choose File -> Execute to sumbit an image from your computer

Containerize model server with Docker

One common way of distributing this model API server for production deployment, is via Docker containers. And BentoML provides a convenient way to do that.

Note that docker is not available in Google Colab. You will need to download and run this notebook locally to try out this containerization with docker feature.

If you already have docker configured, simply run the follow command to product a docker container serving the IrisClassifier prediction service created above:

In [1]:
!bentoml containerize OnnxResnet50:latest
[2020-09-22 14:31:36,307] INFO - Getting latest version OnnxResnet50:20200922124402_4DE94A
Found Bento: /Users/bozhaoyu/bentoml/repository/OnnxResnet50/20200922124402_4DE94A
[2020-09-22 14:31:36,347] WARNING - Using BentoML installed in `editable` model, the local BentoML repository including all code changes will be packaged together with saved bundle created, under the './bundled_pip_dependencies' directory of the saved bundle.
[2020-09-22 14:31:36,364] WARNING - Saved BentoService bundle version mismatch: loading BentoService bundle create with BentoML version 0.9.0.pre, but loading from BentoML version 0.9.0.pre+3.gcebf2015
Tag not specified, using tag parsed from BentoService: 'onnxresnet50:20200922124402_4DE94A'
Building Docker image onnxresnet50:20200922124402_4DE94A from OnnxResnet50:latest 
-we in here
processed docker file
(None, None)
root in create archive /Users/bozhaoyu/bentoml/repository/OnnxResnet50/20200922124402_4DE94A ['Dockerfile', 'MANIFEST.in', 'OnnxResnet50', 'OnnxResnet50/__init__.py', 'OnnxResnet50/__pycache__', 'OnnxResnet50/__pycache__/onnx_resnet50.cpython-37.pyc', 'OnnxResnet50/artifacts', 'OnnxResnet50/artifacts/__init__.py', 'OnnxResnet50/artifacts/labels.pkl', 'OnnxResnet50/artifacts/model.onnx', 'OnnxResnet50/bentoml.yml', 'OnnxResnet50/onnx_resnet50.py', 'README.md', 'bentoml-init.sh', 'bentoml.yml', 'bundled_pip_dependencies', 'bundled_pip_dependencies/BentoML-0.9.0rc0+3.gcebf2015.tar.gz', 'docker-entrypoint.sh', 'environment.yml', 'python_version', 'requirements.txt', 'setup.py']
\about to build
about to upgrade params
check each param and update
if use config proxy
if buildargs
if shmsize
if labels
if cache from
if target
if network_mode
if squash
if extra hosts is not None
if platform is not None
if isolcation is not None
if context is not None
setting auth {'Content-Type': 'application/tar'}
\docker build <tempfile._TemporaryFileWrapper object at 0x7f9717575d68> {'t': 'onnxresnet50:20200922124402_4DE94A', 'remote': None, 'q': False, 'nocache': False, 'rm': False, 'forcerm': False, 'pull': False, 'dockerfile': (None, None)}
|docker response <Response [200]>
context closes
\print responses
Step 1/15 : FROM bentoml/model-server:0.9.0.pre
 ---> a25066aa8b0e
Step 2/15 : ARG EXTRA_PIP_INSTALL_ARGS=
 ---> Using cache
 ---> fc6e47d06522
Step 3/15 : ENV EXTRA_PIP_INSTALL_ARGS $EXTRA_PIP_INSTALL_ARGS
 ---> Using cache
 ---> db8172e98571
Step 4/15 : COPY environment.yml requirements.txt setup.sh* bentoml-init.sh python_version* /bento/
| ---> fafb86f1ecf0
Step 5/15 : WORKDIR /bento
\ ---> Running in a9cb10db03c3
/ ---> 64823e6bac4d
Step 6/15 : RUN chmod +x /bento/bentoml-init.sh
 ---> Running in 5ccb5f255208
\ ---> 40bfb0f94734
Step 7/15 : RUN if [ -f /bento/bentoml-init.sh ]; then bash -c /bento/bentoml-init.sh; fi
 ---> Running in ec371ba7c47d
|+++ dirname /bento/bentoml-init.sh

++ cd /bento
++ pwd -P
+ SAVED_BUNDLE_PATH=/bento
+ cd /bento
+ '[' -f ./setup.sh ']'
+ '[' -f ./python_version ']'

++ cat ./python_version

+ PY_VERSION_SAVED=3.7.3
+ DESIRED_PY_VERSION=3.7

++ python -c 'import sys; print(f"{sys.version_info.major}.{sys.version_info.minor}")'

+ CURRENT_PY_VERSION=3.7
+ [[ 3.7 == \3\.\7 ]]
+ echo 'Python Version in docker base image 3.7 matches requirement python=3.7. Skipping.'

Python Version in docker base image 3.7 matches requirement python=3.7. Skipping.
Updating conda base environment with environment.yml
+ command -v conda
+ echo 'Updating conda base environment with environment.yml'
+ conda env update -n base -f ./environment.yml

-Collecting package metadata (repodata.json): ...working... 
|done
Solving environment: 
...working... 
-done
/
Downloading and Extracting Packages
python_abi-3.7       | 4 KB      |            |   0% 
python_abi-3.7       | 4 KB      | ########## | 100% 
python_abi-3.7       | 4 KB      | ########## | 100% 

ca-certificates-2020 | 145 KB    |            |   0% 
ca-certificates-2020 | 145 KB    | ########## | 100% 

pip-20.2.3           | 1.1 MB    |            |   0% 
pip-20.2.3           | 1.1 MB    | ##9        |  29% 
pip-20.2.3           | 1.1 MB    | ########## | 100% 
pip-20.2.3           | 1.1 MB    | ########## | 100% 

python-3.7.9         | 45.3 MB   |            |   0% 
python-3.7.9         | 45.3 MB   |            |   0% 
python-3.7.9         | 45.3 MB   | 1          |   1% 
python-3.7.9         | 45.3 MB   | 3          |   3% 
python-3.7.9         | 45.3 MB   | 6          |   7% 
python-3.7.9         | 45.3 MB   | 8          |   9% 
python-3.7.9         | 45.3 MB   | #1         |  11% 
python-3.7.9         | 45.3 MB   | #3         |  14% 
python-3.7.9         | 45.3 MB   | #6         |  17% 
python-3.7.9         | 45.3 MB   | ##         |  20% 
python-3.7.9         | 45.3 MB   | ##4        |  24% 
python-3.7.9         | 45.3 MB   | ##9        |  30% 
python-3.7.9         | 45.3 MB   | ###4       |  34% 
python-3.7.9         | 45.3 MB   | ###8       |  39% 
python-3.7.9         | 45.3 MB   | ####2      |  43% 
python-3.7.9         | 45.3 MB   | ####7      |  47% 
python-3.7.9         | 45.3 MB   | #####1     |  51% 
python-3.7.9         | 45.3 MB   | #####5     |  55% 
python-3.7.9         | 45.3 MB   | ######     |  60% 
python-3.7.9         | 45.3 MB   | ######4    |  65% 
python-3.7.9         | 45.3 MB   | ######9    |  69% 
python-3.7.9         | 45.3 MB   | #######3   |  73% 
python-3.7.9         | 45.3 MB   | #######7   |  77% 
python-3.7.9         | 45.3 MB   | ########2  |  82% 
python-3.7.9         | 45.3 MB   | ########6  |  87% 
python-3.7.9         | 45.3 MB   | #########2 |  93% 
python-3.7.9         | 45.3 MB   | #########6 |  97% 
python-3.7.9         | 45.3 MB   | ########## | 100% 

openssl-1.1.1h       | 2.1 MB    |            |   0% 
openssl-1.1.1h       | 2.1 MB    | ###4       |  35% 
openssl-1.1.1h       | 2.1 MB    | ########## | 100% 
openssl-1.1.1h       | 2.1 MB    | ########## | 100% 

readline-8.0         | 281 KB    |            |   0% 
readline-8.0         | 281 KB    | ########## | 100% 
readline-8.0         | 281 KB    | ########## | 100% 

certifi-2020.6.20    | 151 KB    |            |   0% 
certifi-2020.6.20    | 151 KB    | ########## | 100% 

Preparing transaction: ...working... 
\done
Verifying transaction: 
...working... 
-done
Executing transaction: ...working... 
/done
\#
# To activate this environment, use
#
#     $ conda activate base
#
# To deactivate an active environment, use
#
#     $ conda deactivate
/+ pip install -r ./requirements.txt --no-cache-dir

|Requirement already satisfied: bentoml==0.9.0.pre in /opt/conda/lib/python3.7/site-packages (from -r ./requirements.txt (line 1)) (0.9.0rc0)
|Collecting imageio==2.5.0
\  Downloading imageio-2.5.0-py3-none-any.whl (3.3 MB)
-Collecting onnxruntime==1.3.0
/  Downloading onnxruntime-1.3.0-cp37-cp37m-manylinux1_x86_64.whl (3.9 MB)
/Collecting numpy==1.18.4
  Downloading numpy-1.18.4-cp37-cp37m-manylinux1_x86_64.whl (20.2 MB)
/Requirement already satisfied: sqlalchemy>=1.3.0 in /opt/conda/lib/python3.7/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (1.3.19)
Requirement already satisfied: configparser in /opt/conda/lib/python3.7/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (5.0.0)
Requirement already satisfied: packaging in /opt/conda/lib/python3.7/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (20.4)
Requirement already satisfied: sqlalchemy-utils<0.36.8 in /opt/conda/lib/python3.7/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (0.36.7)
|Requirement already satisfied: cerberus in /opt/conda/lib/python3.7/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (1.3.2)
Requirement already satisfied: multidict in /opt/conda/lib/python3.7/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (4.7.6)
Requirement already satisfied: protobuf>=3.6.0 in /opt/conda/lib/python3.7/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (3.13.0)
Requirement already satisfied: alembic in /opt/conda/lib/python3.7/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (1.4.3)
Requirement already satisfied: certifi in /opt/conda/lib/python3.7/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (2020.6.20)
Requirement already satisfied: flask in /opt/conda/lib/python3.7/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (1.1.2)
\Requirement already satisfied: py-zipkin in /opt/conda/lib/python3.7/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (0.20.0)
Requirement already satisfied: grpcio<=1.27.2 in /opt/conda/lib/python3.7/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (1.27.2)
Requirement already satisfied: psutil in /opt/conda/lib/python3.7/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (5.7.2)
Requirement already satisfied: python-dateutil<3.0.0,>=2.7.3 in /opt/conda/lib/python3.7/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (2.8.1)
Requirement already satisfied: requests in /opt/conda/lib/python3.7/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (2.24.0)
Requirement already satisfied: aiohttp in /opt/conda/lib/python3.7/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (3.6.2)
Requirement already satisfied: ruamel.yaml>=0.15.0 in /opt/conda/lib/python3.7/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (0.15.87)
Requirement already satisfied: boto3 in /opt/conda/lib/python3.7/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (1.15.2)
-Requirement already satisfied: prometheus-client in /opt/conda/lib/python3.7/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (0.8.0)
Requirement already satisfied: gunicorn in /opt/conda/lib/python3.7/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (20.0.4)
Requirement already satisfied: python-json-logger in /opt/conda/lib/python3.7/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (0.1.11)
Requirement already satisfied: docker in /opt/conda/lib/python3.7/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (4.3.1)
Requirement already satisfied: humanfriendly in /opt/conda/lib/python3.7/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (8.2)
Requirement already satisfied: tabulate in /opt/conda/lib/python3.7/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (0.8.7)
Requirement already satisfied: click>=7.0 in /opt/conda/lib/python3.7/site-packages (from bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (7.1.2)
|Collecting pillow
  Downloading Pillow-7.2.0-cp37-cp37m-manylinux1_x86_64.whl (2.2 MB)
/Collecting onnx>=1.2.3
  Downloading onnx-1.7.0-cp37-cp37m-manylinux1_x86_64.whl (7.4 MB)
|Requirement already satisfied: pyparsing>=2.0.2 in /opt/conda/lib/python3.7/site-packages (from packaging->bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (2.4.7)
Requirement already satisfied: six in /opt/conda/lib/python3.7/site-packages (from packaging->bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (1.15.0)
Requirement already satisfied: setuptools in /opt/conda/lib/python3.7/site-packages (from cerberus->bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (49.6.0.post20200814)
Requirement already satisfied: Mako in /opt/conda/lib/python3.7/site-packages (from alembic->bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (1.1.3)
Requirement already satisfied: python-editor>=0.3 in /opt/conda/lib/python3.7/site-packages (from alembic->bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (1.0.4)
Requirement already satisfied: itsdangerous>=0.24 in /opt/conda/lib/python3.7/site-packages (from flask->bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (1.1.0)
Requirement already satisfied: Jinja2>=2.10.1 in /opt/conda/lib/python3.7/site-packages (from flask->bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (2.11.2)
Requirement already satisfied: Werkzeug>=0.15 in /opt/conda/lib/python3.7/site-packages (from flask->bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (1.0.1)
\Requirement already satisfied: thriftpy2>=0.4.0 in /opt/conda/lib/python3.7/site-packages (from py-zipkin->bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (0.4.11)
Requirement already satisfied: chardet<4,>=3.0.2 in /opt/conda/lib/python3.7/site-packages (from requests->bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (3.0.4)
Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /opt/conda/lib/python3.7/site-packages (from requests->bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (1.25.10)
Requirement already satisfied: idna<3,>=2.5 in /opt/conda/lib/python3.7/site-packages (from requests->bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (2.10)
Requirement already satisfied: attrs>=17.3.0 in /opt/conda/lib/python3.7/site-packages (from aiohttp->bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (20.2.0)
Requirement already satisfied: yarl<2.0,>=1.0 in /opt/conda/lib/python3.7/site-packages (from aiohttp->bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (1.5.1)
Requirement already satisfied: async-timeout<4.0,>=3.0 in /opt/conda/lib/python3.7/site-packages (from aiohttp->bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (3.0.1)
Requirement already satisfied: jmespath<1.0.0,>=0.7.1 in /opt/conda/lib/python3.7/site-packages (from boto3->bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (0.10.0)
Requirement already satisfied: s3transfer<0.4.0,>=0.3.0 in /opt/conda/lib/python3.7/site-packages (from boto3->bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (0.3.3)
Requirement already satisfied: botocore<1.19.0,>=1.18.2 in /opt/conda/lib/python3.7/site-packages (from boto3->bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (1.18.2)
Requirement already satisfied: websocket-client>=0.32.0 in /opt/conda/lib/python3.7/site-packages (from docker->bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (0.57.0)
Requirement already satisfied: typing-extensions>=3.6.2.1 in /opt/conda/lib/python3.7/site-packages (from onnx>=1.2.3->onnxruntime==1.3.0->-r ./requirements.txt (line 3)) (3.7.4.3)
Requirement already satisfied: MarkupSafe>=0.9.2 in /opt/conda/lib/python3.7/site-packages (from Mako->alembic->bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (1.1.1)
-Requirement already satisfied: ply<4.0,>=3.4 in /opt/conda/lib/python3.7/site-packages (from thriftpy2>=0.4.0->py-zipkin->bentoml==0.9.0.pre->-r ./requirements.txt (line 1)) (3.11)
/Installing collected packages: pillow, numpy, imageio, onnx, onnxruntime
-  Attempting uninstall: numpy
    Found existing installation: numpy 1.19.2
|    Uninstalling numpy-1.19.2:
/      Successfully uninstalled numpy-1.19.2
/Successfully installed imageio-2.5.0 numpy-1.18.4 onnx-1.7.0 onnxruntime-1.3.0 pillow-7.2.0
\ ---> 88481f3057fc
Step 8/15 : COPY . /bento
| ---> 45ca074a1da1
Step 9/15 : RUN if [ -d /bento/bundled_pip_dependencies ]; then pip install -U bundled_pip_dependencies/* ;fi
 ---> Running in 9bf9674c4d65
\Processing ./bundled_pip_dependencies/BentoML-0.9.0rc0+3.gcebf2015.tar.gz
\  Installing build dependencies: started
-  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
/  Getting requirements to build wheel: finished with status 'done'
    Preparing wheel metadata: started
\    Preparing wheel metadata: finished with status 'done'
/Requirement already satisfied, skipping upgrade: gunicorn in /opt/conda/lib/python3.7/site-packages (from BentoML==0.9.0rc0+3.gcebf2015) (20.0.4)
Requirement already satisfied, skipping upgrade: py-zipkin in /opt/conda/lib/python3.7/site-packages (from BentoML==0.9.0rc0+3.gcebf2015) (0.20.0)
Requirement already satisfied, skipping upgrade: boto3 in /opt/conda/lib/python3.7/site-packages (from BentoML==0.9.0rc0+3.gcebf2015) (1.15.2)
Requirement already satisfied, skipping upgrade: grpcio<=1.27.2 in /opt/conda/lib/python3.7/site-packages (from BentoML==0.9.0rc0+3.gcebf2015) (1.27.2)
Requirement already satisfied, skipping upgrade: click>=7.0 in /opt/conda/lib/python3.7/site-packages (from BentoML==0.9.0rc0+3.gcebf2015) (7.1.2)
Requirement already satisfied, skipping upgrade: python-json-logger in /opt/conda/lib/python3.7/site-packages (from BentoML==0.9.0rc0+3.gcebf2015) (0.1.11)
Requirement already satisfied, skipping upgrade: tabulate in /opt/conda/lib/python3.7/site-packages (from BentoML==0.9.0rc0+3.gcebf2015) (0.8.7)
Requirement already satisfied, skipping upgrade: python-dateutil<3.0.0,>=2.7.3 in /opt/conda/lib/python3.7/site-packages (from BentoML==0.9.0rc0+3.gcebf2015) (2.8.1)
Requirement already satisfied, skipping upgrade: ruamel.yaml>=0.15.0 in /opt/conda/lib/python3.7/site-packages (from BentoML==0.9.0rc0+3.gcebf2015) (0.15.87)
Requirement already satisfied, skipping upgrade: flask in /opt/conda/lib/python3.7/site-packages (from BentoML==0.9.0rc0+3.gcebf2015) (1.1.2)
Requirement already satisfied, skipping upgrade: configparser in /opt/conda/lib/python3.7/site-packages (from BentoML==0.9.0rc0+3.gcebf2015) (5.0.0)
Requirement already satisfied, skipping upgrade: psutil in /opt/conda/lib/python3.7/site-packages (from BentoML==0.9.0rc0+3.gcebf2015) (5.7.2)
Requirement already satisfied, skipping upgrade: multidict in /opt/conda/lib/python3.7/site-packages (from BentoML==0.9.0rc0+3.gcebf2015) (4.7.6)
Requirement already satisfied, skipping upgrade: prometheus-client in /opt/conda/lib/python3.7/site-packages (from BentoML==0.9.0rc0+3.gcebf2015) (0.8.0)
Requirement already satisfied, skipping upgrade: docker in /opt/conda/lib/python3.7/site-packages (from BentoML==0.9.0rc0+3.gcebf2015) (4.3.1)
|Requirement already satisfied, skipping upgrade: sqlalchemy-utils<0.36.8 in /opt/conda/lib/python3.7/site-packages (from BentoML==0.9.0rc0+3.gcebf2015) (0.36.7)
Requirement already satisfied, skipping upgrade: packaging in /opt/conda/lib/python3.7/site-packages (from BentoML==0.9.0rc0+3.gcebf2015) (20.4)
Requirement already satisfied, skipping upgrade: cerberus in /opt/conda/lib/python3.7/site-packages (from BentoML==0.9.0rc0+3.gcebf2015) (1.3.2)
Requirement already satisfied, skipping upgrade: sqlalchemy>=1.3.0 in /opt/conda/lib/python3.7/site-packages (from BentoML==0.9.0rc0+3.gcebf2015) (1.3.19)
\Requirement already satisfied, skipping upgrade: humanfriendly in /opt/conda/lib/python3.7/site-packages (from BentoML==0.9.0rc0+3.gcebf2015) (8.2)
Requirement already satisfied, skipping upgrade: certifi in /opt/conda/lib/python3.7/site-packages (from BentoML==0.9.0rc0+3.gcebf2015) (2020.6.20)
Requirement already satisfied, skipping upgrade: requests in /opt/conda/lib/python3.7/site-packages (from BentoML==0.9.0rc0+3.gcebf2015) (2.24.0)
Requirement already satisfied, skipping upgrade: protobuf>=3.6.0 in /opt/conda/lib/python3.7/site-packages (from BentoML==0.9.0rc0+3.gcebf2015) (3.13.0)
Requirement already satisfied, skipping upgrade: numpy in /opt/conda/lib/python3.7/site-packages (from BentoML==0.9.0rc0+3.gcebf2015) (1.18.4)
Requirement already satisfied, skipping upgrade: alembic in /opt/conda/lib/python3.7/site-packages (from BentoML==0.9.0rc0+3.gcebf2015) (1.4.3)
Requirement already satisfied, skipping upgrade: aiohttp in /opt/conda/lib/python3.7/site-packages (from BentoML==0.9.0rc0+3.gcebf2015) (3.6.2)
Requirement already satisfied, skipping upgrade: setuptools>=3.0 in /opt/conda/lib/python3.7/site-packages (from gunicorn->BentoML==0.9.0rc0+3.gcebf2015) (49.6.0.post20200814)
Requirement already satisfied, skipping upgrade: thriftpy2>=0.4.0 in /opt/conda/lib/python3.7/site-packages (from py-zipkin->BentoML==0.9.0rc0+3.gcebf2015) (0.4.11)
Requirement already satisfied, skipping upgrade: six in /opt/conda/lib/python3.7/site-packages (from py-zipkin->BentoML==0.9.0rc0+3.gcebf2015) (1.15.0)
Requirement already satisfied, skipping upgrade: jmespath<1.0.0,>=0.7.1 in /opt/conda/lib/python3.7/site-packages (from boto3->BentoML==0.9.0rc0+3.gcebf2015) (0.10.0)
Requirement already satisfied, skipping upgrade: s3transfer<0.4.0,>=0.3.0 in /opt/conda/lib/python3.7/site-packages (from boto3->BentoML==0.9.0rc0+3.gcebf2015) (0.3.3)
Requirement already satisfied, skipping upgrade: botocore<1.19.0,>=1.18.2 in /opt/conda/lib/python3.7/site-packages (from boto3->BentoML==0.9.0rc0+3.gcebf2015) (1.18.2)
Requirement already satisfied, skipping upgrade: itsdangerous>=0.24 in /opt/conda/lib/python3.7/site-packages (from flask->BentoML==0.9.0rc0+3.gcebf2015) (1.1.0)
Requirement already satisfied, skipping upgrade: Jinja2>=2.10.1 in /opt/conda/lib/python3.7/site-packages (from flask->BentoML==0.9.0rc0+3.gcebf2015) (2.11.2)
Requirement already satisfied, skipping upgrade: Werkzeug>=0.15 in /opt/conda/lib/python3.7/site-packages (from flask->BentoML==0.9.0rc0+3.gcebf2015) (1.0.1)
-Requirement already satisfied, skipping upgrade: websocket-client>=0.32.0 in /opt/conda/lib/python3.7/site-packages (from docker->BentoML==0.9.0rc0+3.gcebf2015) (0.57.0)
Requirement already satisfied, skipping upgrade: pyparsing>=2.0.2 in /opt/conda/lib/python3.7/site-packages (from packaging->BentoML==0.9.0rc0+3.gcebf2015) (2.4.7)
Requirement already satisfied, skipping upgrade: chardet<4,>=3.0.2 in /opt/conda/lib/python3.7/site-packages (from requests->BentoML==0.9.0rc0+3.gcebf2015) (3.0.4)
Requirement already satisfied, skipping upgrade: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /opt/conda/lib/python3.7/site-packages (from requests->BentoML==0.9.0rc0+3.gcebf2015) (1.25.10)
Requirement already satisfied, skipping upgrade: idna<3,>=2.5 in /opt/conda/lib/python3.7/site-packages (from requests->BentoML==0.9.0rc0+3.gcebf2015) (2.10)
Requirement already satisfied, skipping upgrade: Mako in /opt/conda/lib/python3.7/site-packages (from alembic->BentoML==0.9.0rc0+3.gcebf2015) (1.1.3)
Requirement already satisfied, skipping upgrade: python-editor>=0.3 in /opt/conda/lib/python3.7/site-packages (from alembic->BentoML==0.9.0rc0+3.gcebf2015) (1.0.4)
Requirement already satisfied, skipping upgrade: async-timeout<4.0,>=3.0 in /opt/conda/lib/python3.7/site-packages (from aiohttp->BentoML==0.9.0rc0+3.gcebf2015) (3.0.1)
Requirement already satisfied, skipping upgrade: yarl<2.0,>=1.0 in /opt/conda/lib/python3.7/site-packages (from aiohttp->BentoML==0.9.0rc0+3.gcebf2015) (1.5.1)
Requirement already satisfied, skipping upgrade: attrs>=17.3.0 in /opt/conda/lib/python3.7/site-packages (from aiohttp->BentoML==0.9.0rc0+3.gcebf2015) (20.2.0)
Requirement already satisfied, skipping upgrade: ply<4.0,>=3.4 in /opt/conda/lib/python3.7/site-packages (from thriftpy2>=0.4.0->py-zipkin->BentoML==0.9.0rc0+3.gcebf2015) (3.11)
Requirement already satisfied, skipping upgrade: MarkupSafe>=0.23 in /opt/conda/lib/python3.7/site-packages (from Jinja2>=2.10.1->flask->BentoML==0.9.0rc0+3.gcebf2015) (1.1.1)
Requirement already satisfied, skipping upgrade: typing-extensions>=3.7.4; python_version < "3.8" in /opt/conda/lib/python3.7/site-packages (from yarl<2.0,>=1.0->aiohttp->BentoML==0.9.0rc0+3.gcebf2015) (3.7.4.3)
Building wheels for collected packages: BentoML
  Building wheel for BentoML (PEP 517): started
/  Building wheel for BentoML (PEP 517): finished with status 'done'
  Created wheel for BentoML: filename=BentoML-0.9.0rc0+3.gcebf2015-py3-none-any.whl size=3064091 sha256=d62cee806abd33902c47491b27f45eba7c38cfdb2a82beaf8c7917bddb910d5f
  Stored in directory: /root/.cache/pip/wheels/a0/45/41/62152db705af4ff47e7a3d6abf6247986eef4aa1b94a58d3b9
Successfully built BentoML
/Installing collected packages: BentoML
  Attempting uninstall: BentoML
    Found existing installation: BentoML 0.9.0rc0
\    Uninstalling BentoML-0.9.0rc0:
\      Successfully uninstalled BentoML-0.9.0rc0
-Successfully installed BentoML-0.9.0rc0+3.gcebf2015
/ ---> e8f9cf1be143
Step 10/15 : ENV PORT 5000
| ---> Running in 2fa4645bceef
\ ---> 478de8907ddb
Step 11/15 : EXPOSE $PORT
 ---> Running in 68b3a375d4e5
- ---> 936de5a30f54
Step 12/15 : COPY docker-entrypoint.sh /usr/local/bin/
| ---> ca3a4a441847
Step 13/15 : RUN chmod +x /usr/local/bin/docker-entrypoint.sh
 ---> Running in 9b1de5009288
- ---> ca3c583728e0
Step 14/15 : ENTRYPOINT [ "docker-entrypoint.sh" ]
 ---> Running in c715bc5a83c7
/ ---> 6e8c7877de08
Step 15/15 : CMD ["bentoml", "serve-gunicorn", "/bento"]
| ---> Running in e29ad2eed981
\ ---> a27604aa4b60
Successfully built a27604aa4b60
Successfully tagged onnxresnet50:20200922124402_4DE94A
Finished building onnxresnet50:20200922124402_4DE94A from OnnxResnet50:latest
In [2]:
!docker run --rm -p 5000:5000 onnxresnet50:20200922124402_4DE94A
[2020-09-22 21:35:25,206] INFO - Starting BentoML API server in production mode..
[2020-09-22 21:35:25,646] INFO - get_gunicorn_num_of_workers: 3, calculated by cpu count
[2020-09-22 21:35:25 +0000] [1] [INFO] Starting gunicorn 20.0.4
[2020-09-22 21:35:25 +0000] [1] [INFO] Listening at: http://0.0.0.0:5000 (1)
[2020-09-22 21:35:25 +0000] [1] [INFO] Using worker: sync
[2020-09-22 21:35:25 +0000] [12] [INFO] Booting worker with pid: 12
[2020-09-22 21:35:25 +0000] [13] [INFO] Booting worker with pid: 13
[2020-09-22 21:35:25 +0000] [14] [INFO] Booting worker with pid: 14
[2020-09-22 21:35:25,935] WARNING - Using BentoML not from official PyPI release. In order to find the same version of BentoML when deploying your BentoService, you must set the 'core/bentoml_deploy_version' config to a http/git location of your BentoML fork, e.g.: 'bentoml_deploy_version = git+https://github.com/{username}/[email protected]{branch}'
[2020-09-22 21:35:25,935] WARNING - Using BentoML not from official PyPI release. In order to find the same version of BentoML when deploying your BentoService, you must set the 'core/bentoml_deploy_version' config to a http/git location of your BentoML fork, e.g.: 'bentoml_deploy_version = git+https://github.com/{username}/[email protected]{branch}'
[2020-09-22 21:35:25,959] WARNING - Saved BentoService bundle version mismatch: loading BentoService bundle create with BentoML version 0.9.0.pre, but loading from BentoML version 0.9.0.pre+3.gcebf2015
[2020-09-22 21:35:25,959] WARNING - Saved BentoService bundle version mismatch: loading BentoService bundle create with BentoML version 0.9.0.pre, but loading from BentoML version 0.9.0.pre+3.gcebf2015
[2020-09-22 21:35:25,959] WARNING - Saved BentoService Python version mismatch: loading BentoService bundle created with Python version 3.7.3, but current environment version is 3.7.9.
[2020-09-22 21:35:25,960] WARNING - Saved BentoService Python version mismatch: loading BentoService bundle created with Python version 3.7.3, but current environment version is 3.7.9.
[2020-09-22 21:35:25,988] WARNING - Using BentoML not from official PyPI release. In order to find the same version of BentoML when deploying your BentoService, you must set the 'core/bentoml_deploy_version' config to a http/git location of your BentoML fork, e.g.: 'bentoml_deploy_version = git+https://github.com/{username}/[email protected]{branch}'
[2020-09-22 21:35:26,013] WARNING - Saved BentoService bundle version mismatch: loading BentoService bundle create with BentoML version 0.9.0.pre, but loading from BentoML version 0.9.0.pre+3.gcebf2015
[2020-09-22 21:35:26,013] WARNING - Saved BentoService Python version mismatch: loading BentoService bundle created with Python version 3.7.3, but current environment version is 3.7.9.
^C
[2020-09-22 21:35:29 +0000] [1] [INFO] Handling signal: int
[2020-09-22 21:35:29 +0000] [14] [INFO] Worker exiting (pid: 14)
[2020-09-22 21:35:29 +0000] [13] [INFO] Worker exiting (pid: 13)
[2020-09-22 21:35:29 +0000] [12] [INFO] Worker exiting (pid: 12)

Launch inference job from CLI

BentoML cli supports loading and running a packaged model from CLI. With the DataframeInput adapter, the CLI command supports reading input Dataframe data from CLI argument or local csv or json files:

In [ ]:
!bentoml run OnnxResnet50:latest predict --input dog.jpg

Load saved BentoService

bentoml.load is the API for loading a BentoML packaged model in python:

In [6]:
image = Image.open('dog.jpg')
image
Out[6]:
In [11]:
from bentoml import load

loaded_svc = load(saved_path)


image_data = np.array(image)
np.array(loaded_svc.predict([image_data] * 4))
[2020-09-22 14:37:28,411] WARNING - Using BentoML installed in `editable` model, the local BentoML repository including all code changes will be packaged together with saved bundle created, under the './bundled_pip_dependencies' directory of the saved bundle.
[2020-09-22 14:37:28,430] WARNING - Saved BentoService bundle version mismatch: loading BentoService bundle create with BentoML version 0.9.0.pre, but loading from BentoML version 0.9.0.pre+3.gcebf2015
[2020-09-22 14:37:28,838] INFO - Using default docker base image: `None` specified inBentoML config file or env var. User must make sure that the docker base image either has Python 3.7 or conda installed.
[2020-09-22 14:37:28,866] INFO - Initializing onnxruntime InferenceSession from onnx file:'/Users/bozhaoyu/bentoml/repository/OnnxResnet50/20200922124402_4DE94A/OnnxResnet50/artifacts/model.onnx'
Out[11]:
array([['Golden Retriever', 'Labrador Retriever', 'Sussex Spaniel',
        'Vizsla', 'Otterhound'],
       ['Golden Retriever', 'Labrador Retriever', 'Sussex Spaniel',
        'Vizsla', 'Otterhound'],
       ['Golden Retriever', 'Labrador Retriever', 'Sussex Spaniel',
        'Vizsla', 'Otterhound'],
       ['Golden Retriever', 'Labrador Retriever', 'Sussex Spaniel',
        'Vizsla', 'Otterhound']], dtype='<U32')

Deployment Options

If you are at a small team with limited engineering or DevOps resources, try out automated deployment with BentoML CLI, currently supporting AWS Lambda, AWS SageMaker, and Azure Functions:

If the cloud platform you are working with is not on the list above, try out these step-by-step guide on manually deploying BentoML packaged model to cloud platforms:

Lastly, if you have a DevOps or ML Engineering team who's operating a Kubernetes or OpenShift cluster, use the following guides as references for implementating your deployment strategy:

In [ ]: