CS 20 : TensorFlow for Deep Learning Research¶

Lecture 04 : Eager execution¶

Eager execution¶

You no longer need to worry about...

placeholders
sessions
control dependencies
"lazy loading"
{name, variable, op} scopes

Reference
- https://www.tensorflow.org/tutorials/eager/eager_basics?hl=ko

Notice¶

Main changes in TF 2.0¶

The API for Variables will then change in the following ways for TF 2.0:

*tf.Variable will become an abstract base class with a well-defined interface and a scoped factory to construct instances*
- *users will be able to implement their own variable-like objects by subclassing tf.Variable and adding a scoped factory function to use those variables*
*variable_scope and get_variable will be removed*
- the tf 1.0 version of variable_scope and get_variable will be left in tf.compat.v1
- *to control variable naming users can use tf.name_scope + tf.Variable*
- whether a variable is shared across sessions / processes will be controlled by a constructor argument to tf.Variable; no other type of scope reuse will be done in the framework
- scoped partitioning will be implemented as a factory function at first
- libraries and users are encouraged to reuse variables by reusing their objects, like Keras layers do
- custom_getters will have the following API: variable_creator_scope
the default implementation of the tf.Variable interface will be ResourceVariable
- RefVariable will be kept in tf.compat.v1 and will be the default implementation for tf.compat.v1.Variable
- tf.compat.v1.Variable will have a use_resource argument to control whether a resource variable or a ref variable will be created
symbols like tf.assign* will be removed in favor of methods in tf.Variable
- in tf.compat.v1 these symbols will be marked as deprecated and will call the corresponding methods in the Variable object instead

https://github.com/tensorflow/community/blob/master/rfcs/20180817-variables-20.md

Setup¶

In [1]:

from __future__ import absolute_import, division, print_function
import numpy as np
import tensorflow as tf
tf.enable_eager_execution()

print(tf.__version__)

1.12.0

Boilerplate¶

x = tf.placeholder(dtype = tf.float32, shape = [1, 1])
m = tf.matmul(x, x)

print(m)
with tf.Session() as sess:
    m_out = sess.run(m, feed_dict = {x : [[2.]]})
    print(m_out, m_out.shape)

Tensor("mul:0", shape=(1, 1), dtype=float32)
[[4.]] (1, 1)

When using tf.enable_eager_execution(), Bolierplate changes as belows

In [2]:

x = [[2.]]
m = tf.matmul(x, x)
print(m) # No sessions()!
print(tf.get_default_graph().get_operations()) # No graphs!

tf.Tensor([[4.]], shape=(1, 1), dtype=float32)
[]

Lazy Loading¶

Each iteration adds nodes to the graph

x = tf.constant(value = [[1,2],[3,4]], dtype = tf.int32)

with tf.Session() as sess:
    for i in range(x.shape[0]):
        for j in range(x.shape[1]):
            print(sess.run(x[i, j]))

When using tf.enable_eager_execution(), not graph

In [3]:

x = tf.constant(value = [[1,2],[3,4]], dtype = tf.int32)

for i in range(x.shape[0]):
    for j in range(x.shape[1]):
        print(x[i, j])
        
print(tf.get_default_graph().get_operations())

tf.Tensor(1, shape=(), dtype=int32)
tf.Tensor(2, shape=(), dtype=int32)
tf.Tensor(3, shape=(), dtype=int32)
tf.Tensor(4, shape=(), dtype=int32)
[]

Tensors act like numpy arrays¶

The most obvious differences between NumPy arrays and TensorFlow Tensors are:

Tensors can be backed by accelerator memory (like GPU, TPU)
Tensors are immutable

In [4]:

# Tensors are backed by NumPy arrays
# Tensors are compatible with NumPy functions
x = tf.constant(value = [[1.,2.,3]])
assert type(x.numpy()) == np.ndarray
squared = np.square(x)
print(squared)

# Tensors are iterable!
for i in x[0]:
    print(i)

[[1. 4. 9.]]
tf.Tensor(1.0, shape=(), dtype=float32)
tf.Tensor(2.0, shape=(), dtype=float32)
tf.Tensor(3.0, shape=(), dtype=float32)

In [5]:

print(tf.add(1, 2))
print(tf.add([1, 2], [3, 4]))
print(tf.square(5))
print(tf.reduce_sum([1, 2, 3]))
print(tf.encode_base64("hello world"))

# Operator overloading is also supported
print(tf.square(2) + tf.square(3))

tf.Tensor(3, shape=(), dtype=int32)
tf.Tensor([4 6], shape=(2,), dtype=int32)
tf.Tensor(25, shape=(), dtype=int32)
tf.Tensor(6, shape=(), dtype=int32)
tf.Tensor(b'aGVsbG8gd29ybGQ', shape=(), dtype=string)
tf.Tensor(13, shape=(), dtype=int32)

In [6]:

# Each Tensor has a shape and a datatype
x = tf.matmul([[1]], [[2, 3]])
print(x.shape)
print(x.dtype)

(1, 2)
<dtype: 'int32'>

Numpy Compatibility¶

TensorsFlow operations automatically convert NumPy ndarrays to Tensors.
NumPy operations automatically convert Tensors to NumPy ndarrays.

*Tensors can be explicitly converted to NumPy ndarrays by invoking the .numpy() method on them.* These conversions are typically cheap as the array and Tensor share the underlying memory representation if possible. *However, sharing the underlying representation isn't always possible since the Tensor may be hosted in GPU memory while NumPy arrays are always backed by host memory, and the conversion will thus involve a copy from GPU to host memory.*

In [7]:

ndarray = np.ones([3,3], dtype = np.float32)

print("TensorFlow operations convert numpy arrays to Tensors automatically")
tensor = tf.multiply(ndarray, 42)
print(tensor)

print("And NumPy operations convert Tensors to numpy arrays automatically")
print(np.add(tensor, 1))

print("The .numpy() method explicitly converts a Tensor to a numpy array")
print(tensor.numpy())

TensorFlow operations convert numpy arrays to Tensors automatically
tf.Tensor(
[[42. 42. 42.]
 [42. 42. 42.]
 [42. 42. 42.]], shape=(3, 3), dtype=float32)
And NumPy operations convert Tensors to numpy arrays automatically
[[43. 43. 43.]
 [43. 43. 43.]
 [43. 43. 43.]]
The .numpy() method explicitly converts a Tensor to a numpy array
[[42. 42. 42.]
 [42. 42. 42.]
 [42. 42. 42.]]

GPU acceleration¶

In [8]:

x = tf.random_uniform(shape = [3, 3])

print("Is there a GPU available: "),
print(tf.test.is_gpu_available())

print("Is the Tensor on GPU #0:  "),
print(x.device.endswith('GPU:0'))

Is there a GPU available: 
False
Is the Tensor on GPU #0:  
False

Device Names¶

In [9]:

x.device

Out[9]:

'/job:localhost/replica:0/task:0/device:CPU:0'

Explicit Device Placement¶

In [10]:

with tf.device("CPU:0"):
    y = tf.ones([1,1])
    print(y.device)

/job:localhost/replica:0/task:0/device:CPU:0

In [11]:

print(x.device)
z = x.cpu()
print(z.device)

/job:localhost/replica:0/task:0/device:CPU:0
/job:localhost/replica:0/task:0/device:CPU:0

In [12]:

print(x, '\n',z)

tf.Tensor(
[[0.5488272  0.9693705  0.47811544]
 [0.13793623 0.53724563 0.9553573 ]
 [0.9873563  0.27607608 0.21941674]], shape=(3, 3), dtype=float32) 
 tf.Tensor(
[[0.5488272  0.9693705  0.47811544]
 [0.13793623 0.53724563 0.9553573 ]
 [0.9873563  0.27607608 0.21941674]], shape=(3, 3), dtype=float32)

In [13]:

tf.equal(x, z)

Out[13]:

<tf.Tensor: id=98, shape=(3, 3), dtype=bool, numpy=
array([[ True,  True,  True],
       [ True,  True,  True],
       [ True,  True,  True]])>

Datasets¶

We recommend using the Datasets API for building performant, complex input pipelines from simple, re-usable pieces that will feed your model's training or evaluation loops.

If you're familiar with TensorFlow graphs, the API for constructing the Dataset object remains exactly the same when eager execution is enabled, but the process of iterating over elements of the dataset is slightly simpler. *You can use Python iteration over the tf.data.Dataset object and do not need to explicitly create an tf.data.Iterator object.* As a result, the discussion on iterators in the TensorFlow Guide is not relevant when eager execution is enabled.

In [14]:

tensors = tf.data.Dataset.from_tensor_slices([1, 2, 3, 4, 5, 6])
tensors = tensors.map(np.square) # Numpy Compatibility magic!
tensors = tensors.shuffle(2).batch(2)

In [15]:

for mb_tensor in tensors:
    print(mb_tensor)

tf.Tensor([4 1], shape=(2,), dtype=int32)
tf.Tensor([16  9], shape=(2,), dtype=int32)
tf.Tensor([25 36], shape=(2,), dtype=int32)