In this example, we will convert a pretrained TensorFlow 2 model created with the tf.keras
API. For more details on all the different ways to convert TensorFlow 2 models, see the coremltools TensorFlow 2 documentation. For TensorFlow 1.x, see the coremltools TensorFlow 1 documentation.
For this example, we will load MobileNetV2 with pretrained weights from ImageNet. Keras assumes the input image has a been rescaled and offset to a range of [-1, 1], which coremltools can do during the conversion process. If more complex input preparation is required, those layers should be included in the model before conversion, or the target application should do the required data preparation.
import tensorflow as tf
input_shape = (192, 192, 3)
tf_model = tf.keras.applications.MobileNetV2(
input_shape=input_shape,
include_top=True,
weights="imagenet",
classes=1000,
)
We also want the class labels to convert the model predictions to English category names.
# Download class labels (from a separate file)
import urllib
label_url = 'https://storage.googleapis.com/download.tensorflow.org/data/ImageNetLabels.txt'
class_labels = urllib.request.urlopen(label_url).read().decode("utf-8").splitlines()
class_labels = class_labels[1:] # remove the first class which is background
assert len(class_labels) == 1000
As a quick check, let's load an image of a dog (source, CC BY-SA 4.0) and run it through the Keras model to ensure it is working.
import PIL
img = PIL.Image.open('Dog.jpg').resize(input_shape[:2])
img
import numpy as np
# prepare with appropriate dimensionality (batch size of 1), scale and offset
img_np = (np.array(img)[np.newaxis,:,:,:]/127.0) - 1
out = tf_model.predict(img_np)[0]
top_5_indices = np.flip(np.argsort(out))[:5]
for i in top_5_indices:
print(class_labels[i], out[i])
The convert() function in coremltools will take a TensorFlow or PyTorch model in one of many forms (in this case, we have a tf.keras.Model
object), analyze it, and generate a CoreML model wrapped in a ct.models.MLModel
Python object.
The API has two optional arguments which we will use:
inputs
: List of type descriptions for the input layers. The types can be inferred from the input model in some cases, but here we want to interpret the (192, 192, 3) input tensor as an image. This will allow us to specify a scale and offset ("bias") for the pixel values, as required by the model, and also describe the color channel ordering. This color layout metadata is included in the CoreML model itself and can be used by the consuming application of the model to ensure correct pixel ordering.
classifier_config
: Since this is a classifier model, we include the class labels here so that the user of the CoreML model has that data as well.
The converter will translate the TensorFlow model into a Model Intermediate Language representation, optimize it for performance, and finally emit the equivalent CoreML operations.
import coremltools as ct
cml_model = ct.convert(tf_model,
inputs=[ct.ImageType(color_layout="RGB", scale=1/127.0, bias=[-1, -1, -1], )],
classifier_config=ct.ClassifierConfig(class_labels)
)
We can see the metadata describing this model by looking at the string representation:
cml_model
Finally, we can write the model to disk in the CoreML format.
cml_model.save('MobileNet_v2_tf.mlmodel')
There are two ways to use the trained CoreML model for inference:
Load the model in Python using coremltools and call the predict() method. This requires macOS as the model is executed by the CoreML framework. The OS will use whatever specialized hardware may be available to accelerate execution.
Load the model in a compiled macOS or iOS app using the CoreML APIs.
If you are running this notebook on a Mac (not available on Binder), you can test the model directly
import sys
IS_MACOS = sys.platform == 'darwin'
if IS_MACOS:
loaded_model = ct.models.MLModel('MobileNet_v2_tf.mlmodel')
prediction = loaded_model.predict({'input_1': img})
else:
prediction = 'Skipping prediction on non-macOS system'
prediction
If you are running this notebook remotely and browsing on an iOS device, you can download the model you just trained and test it out with the CoreMLCompare app (source code) and your device camera. (Note: the author of CoreMLCompare is not affiliated with coremltools or Apple. It is just a handy app for loading and comparing CoreML image classifiers.)
Steps:
Note that this model is using ImageNet categories which might not correspond to the types of objects you have nearby. A pen on a white piece of paper provides a good test case. It will likely be classified as "Ballpoint" or "Fountain Pen".