### Tensorflow - Notebook -1¶

#### Index :¶

1. Sys config checks .
2. What are Sessions.
3. What are Variables.
4. What are Tensors. What are Tensors.
5. ....

$Geometry Symbols$

In [2]:
import sys
print(sys.executable)
print(sys.version)
print(sys.version_info)

/home/dhankar/anaconda2/envs/tensor/bin/python
3.6.2 |Continuum Analytics, Inc.| (default, Jul 20 2017, 13:51:32)
[GCC 4.4.7 20120313 (Red Hat 4.4.7-1)]
sys.version_info(major=3, minor=6, micro=2, releaselevel='final', serial=0)

In [2]:
import tensorflow as tf

• Create a Constant op == Operation == NODE of the Computational Graph. The op is added as a node to the default graph.

• The value returned by the constructor represents the output of the Constant op.

• The most basic Computational unit is a Constant - it is a value that will not change during evaluation / computation of the graph.

• The Constant is always a Tensor it can be a One Dimensional SCALAR or a STRING or higher dimensional tensors.
In [ ]:
hello = tf.constant('Hello, TensorFlow!')

In [5]:
# Start tf session
sess = tf.Session()

In [6]:
# Run graph
print(sess.run(hello))

b'Hello, TensorFlow!'

In [17]:
# Basic constant operations
# The value returned by the constructor represents the output
# of the Constant op.
a = tf.constant(21)
b = tf.constant(13)
#
print(type(a))
# <class 'tensorflow.python.framework.ops.Tensor'>

<class 'tensorflow.python.framework.ops.Tensor'>


The SCALAR Tensor's == a = tf.constant(21)

• a rank 0 tensor; a scalar with shape [],
In [19]:
# Launch the default graph.
with tf.Session() as sess:
print("a: sess.run(a) b:_______TYPE_______",type(sess.run(b)))
print("b: sess.run(b) b:__________________",sess.run(b))
print("Multiplication with constants:_____",sess.run(a*b))
print("Subtraction with constants_________",sess.run(a-b))

print(type(sess))   ## <class 'tensorflow.python.client.session.Session'>

a: sess.run(a) b:_______TYPE_______ <class 'numpy.int32'>
a: sess.run(a) b:__________________ 13
Addition with constants:___________ 34
Multiplication with constants:_____ 273
Subtraction with constants_________ 8
<class 'tensorflow.python.client.session.Session'>

In [21]:
# Add , Divide - basic Op's

y = tf.div(a, b, name="divide")
print(type(x))
print(type(y))

# x and y are   == # <class 'tensorflow.python.framework.ops.Tensor'>
# same as the - #b = tf.constant(13) ...
#

<class 'tensorflow.python.framework.ops.Tensor'>
<class 'tensorflow.python.framework.ops.Tensor'>

In [20]:
# Closing the session.
sess.close()
print(type(sess))

<class 'tensorflow.python.client.session.Session'>

In [10]:
# Create Graphs over and above the DEFAULT Graph

graph1 = tf.Graph()
print(type(graph1))

graph2 = tf.Graph()
print(type(graph2))

<class 'tensorflow.python.framework.ops.Graph'>
<class 'tensorflow.python.framework.ops.Graph'>


#### Official Tutorials for TF¶

In [3]:
from __future__ import absolute_import
from __future__ import division
from __future__ import print_function

import numpy as np
import tensorflow as tf


### Tensor Values¶

• The central unit of data in TensorFlow is the tensor.
• A tensor consists of a set of primitive values shaped into an array of any number of dimensions.
• A tensor's rank is its number of dimensions, while its shape is a tuple of integers specifying the array's length along each dimension.
• TensorFlow uses numpy arrays to represent tensor values.

Here are some examples of tensor values:

3. # a rank 0 tensor; a scalar with shape [],
[1., 2., 3.] # a rank 1 tensor; a vector with shape [3]
[[1., 2., 3.], [4., 5., 6.]] # a rank 2 tensor; a matrix with shape [2, 3]
[[[1., 2., 3.]], [[7., 8., 9.]]] # a rank 3 tensor with shape [2, 1, 3]


### Creating a Variable¶

• The best way to create a variable is to call the tf.get_variable function. This function requires you to specify the Variable's name.
• This name will be used by other replicas to access the same variable, as well as to name this variable's value when checkpointing and exporting models.
• tf.get_variable also allows you to reuse a previously created variable of the same name, making it easy to define models which reuse layers.

To create a variable with tf.get_variable, simply provide the name and shape..

• When you train a model, you use variables to hold and update parameters.
• Variables are in-memory buffers containing tensors.
• They must be explicitly initialized and can be saved to disk during and after training.
• You can later restore saved values to exercise or analyze the model.

• A tf.Variable represents a tensor whose value can be changed by running ops on it.

• Unlike tf.Tensor objects, a tf.Variable exists outside the context of a single session.run call.

• Internally, a tf.Variable stores a persistent tensor.

• Specific ops allow you to read and modify the values of this tensor.
• These modifications are visible across multiple tf.Sessions, so multiple workers can see the same values for a tf.Variable.

Current Version - r1.6 - https://www.tensorflow.org/programmers_guide/variables

Older version - r1.0 - https://www.tensorflow.org/versions/r1.0/programmers_guide/variables

In [4]:
my_variable = tf.get_variable("my_variable", [1, 2, 3])
print(type(my_variable))

<class 'tensorflow.python.ops.variables.Variable'>

• This creates a variable named "my_variable" which is a three-dimensional tensor with shape [1, 2, 3].
• This variable will, by default, have the dtype tf.float32 and its initial value will be randomized via tf.glorot_uniform_initializer.
• You may optionally specify the dtype and initializer to tf.get_variable.

For example:

#### Pedagogy ---¶

• Seen below the "traceback" when we run the Code Chunk twice or more .
• We have already created a named variable - the code chunk re-run tries to create the same variable again.
• This is checked and Error thrown by - /anaconda2/envs/tensor/lib/python3.6/site-packages/tensorflow/python/ops/variable_scope.py in _get_single_variable

cool

In [6]:
#my_int_variable = tf.get_variable("my_int_variable", [1, 2, 3], dtype=tf.int32,initializer=tf.zeros_initializer)

#print(type(my_int_variable))

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-6-90d7e657800c> in <module>()
1 my_int_variable = tf.get_variable("my_int_variable", [1, 2, 3], dtype=tf.int32,
----> 2   initializer=tf.zeros_initializer)
3 print(type(my_int_variable))

~/anaconda2/envs/tensor/lib/python3.6/site-packages/tensorflow/python/ops/variable_scope.py in get_variable(name, shape, dtype, initializer, regularizer, trainable, collections, caching_device, partitioner, validate_shape, use_resource, custom_getter, constraint)
1260       partitioner=partitioner, validate_shape=validate_shape,
1261       use_resource=use_resource, custom_getter=custom_getter,
-> 1262       constraint=constraint)
1263 get_variable_or_local_docstring = (
1264     """%s

~/anaconda2/envs/tensor/lib/python3.6/site-packages/tensorflow/python/ops/variable_scope.py in get_variable(self, var_store, name, shape, dtype, initializer, regularizer, reuse, trainable, collections, caching_device, partitioner, validate_shape, use_resource, custom_getter, constraint)
1095           partitioner=partitioner, validate_shape=validate_shape,
1096           use_resource=use_resource, custom_getter=custom_getter,
-> 1097           constraint=constraint)
1098
1099   def _get_partitioned_variable(self,

~/anaconda2/envs/tensor/lib/python3.6/site-packages/tensorflow/python/ops/variable_scope.py in get_variable(self, name, shape, dtype, initializer, regularizer, reuse, trainable, collections, caching_device, partitioner, validate_shape, use_resource, custom_getter, constraint)
433           caching_device=caching_device, partitioner=partitioner,
434           validate_shape=validate_shape, use_resource=use_resource,
--> 435           constraint=constraint)
436
437   def _get_partitioned_variable(

~/anaconda2/envs/tensor/lib/python3.6/site-packages/tensorflow/python/ops/variable_scope.py in _true_getter(name, shape, dtype, initializer, regularizer, reuse, trainable, collections, caching_device, partitioner, validate_shape, use_resource, constraint)
402           trainable=trainable, collections=collections,
403           caching_device=caching_device, validate_shape=validate_shape,
--> 404           use_resource=use_resource, constraint=constraint)
405
406     if custom_getter is not None:

~/anaconda2/envs/tensor/lib/python3.6/site-packages/tensorflow/python/ops/variable_scope.py in _get_single_variable(self, name, shape, dtype, initializer, regularizer, partition_info, reuse, trainable, collections, caching_device, validate_shape, use_resource, constraint)
741                          "reuse=tf.AUTO_REUSE in VarScope? "
742                          "Originally defined at:\n\n%s" % (
--> 743                              name, "".join(traceback.format_list(tb))))
744       found_var = self._vars[name]
745       if not shape.is_compatible_with(found_var.get_shape()):

ValueError: Variable my_int_variable already exists, disallowed. Did you mean to set reuse=True or reuse=tf.AUTO_REUSE in VarScope? Originally defined at:

File "<ipython-input-5-5cbd14b8ee95>", line 2, in <module>
initializer=tf.zeros_initializer)
File "/home/dhankar/anaconda2/envs/tensor/lib/python3.6/site-packages/IPython/core/interactiveshell.py", line 2910, in run_code
exec(code_obj, self.user_global_ns, self.user_ns)
File "/home/dhankar/anaconda2/envs/tensor/lib/python3.6/site-packages/IPython/core/interactiveshell.py", line 2850, in run_ast_nodes
if self.run_code(code, result):

In [17]:
my_int_var1 = tf.get_variable("my_int_var1", [1, 2, 3], dtype=tf.int32,initializer=tf.zeros_initializer)

with tf.Session() as sess:
sess.run(tf.initialize_all_variables())
print(my_int_var1.eval())

print("  "*90)
print(type(my_int_var1))

[[[0 0 0]
[0 0 0]]]
<class 'tensorflow.python.ops.variables.Variable'>

In [18]:
my_int_var2 = tf.get_variable("my_int_var2", [1, 3, 9], dtype=tf.int32,initializer=tf.zeros_initializer)

with tf.Session() as sess:
sess.run(tf.initialize_all_variables())
print(my_int_var2.eval())

print("  "*90)
print(type(my_int_var2))

[[[0 0 0 0 0 0 0 0 0]
[0 0 0 0 0 0 0 0 0]
[0 0 0 0 0 0 0 0 0]]]

<class 'tensorflow.python.ops.variables.Variable'>

In [19]:
my_int_var3 = tf.get_variable("my_int_var3", [3, 9, 12], dtype=tf.int32,initializer=tf.zeros_initializer)

with tf.Session() as sess:
sess.run(tf.initialize_all_variables())
print(my_int_var3.eval())

print("  "*90)
print(type(my_int_var3))

[[[0 0 0 0 0 0 0 0 0 0 0 0]
[0 0 0 0 0 0 0 0 0 0 0 0]
[0 0 0 0 0 0 0 0 0 0 0 0]
[0 0 0 0 0 0 0 0 0 0 0 0]
[0 0 0 0 0 0 0 0 0 0 0 0]
[0 0 0 0 0 0 0 0 0 0 0 0]
[0 0 0 0 0 0 0 0 0 0 0 0]
[0 0 0 0 0 0 0 0 0 0 0 0]
[0 0 0 0 0 0 0 0 0 0 0 0]]

[[0 0 0 0 0 0 0 0 0 0 0 0]
[0 0 0 0 0 0 0 0 0 0 0 0]
[0 0 0 0 0 0 0 0 0 0 0 0]
[0 0 0 0 0 0 0 0 0 0 0 0]
[0 0 0 0 0 0 0 0 0 0 0 0]
[0 0 0 0 0 0 0 0 0 0 0 0]
[0 0 0 0 0 0 0 0 0 0 0 0]
[0 0 0 0 0 0 0 0 0 0 0 0]
[0 0 0 0 0 0 0 0 0 0 0 0]]

[[0 0 0 0 0 0 0 0 0 0 0 0]
[0 0 0 0 0 0 0 0 0 0 0 0]
[0 0 0 0 0 0 0 0 0 0 0 0]
[0 0 0 0 0 0 0 0 0 0 0 0]
[0 0 0 0 0 0 0 0 0 0 0 0]
[0 0 0 0 0 0 0 0 0 0 0 0]
[0 0 0 0 0 0 0 0 0 0 0 0]
[0 0 0 0 0 0 0 0 0 0 0 0]
[0 0 0 0 0 0 0 0 0 0 0 0]]]

<class 'tensorflow.python.ops.variables.Variable'>


#### Variables vs. Tensors¶

Further notes to understand Variables and Tensors - also the Difference between the two

https://stackoverflow.com/questions/40866675/implementation-difference-between-tensorflow-variable-and-tensorflow-tensor

• Quoting Verbatim - the tf.tensor ==
In the Python API, a tf.Tensor object represents the symbolic result of a TensorFlow operation. For example, in the expression t = tf.matmul(x, y), t is a tf.Tensor object representing the result of multiplying x and y (which may themselves be symbolic results of other operations, concrete values such as NumPy arrays, or variables).

In this context, a "symbolic result" is more complicated than a pointer to the result of an operation. It is more analogous to a function object that, when called (i.e. passed to tf.Session.run()) will run the necessary computation to produce the result of that operation, and return it to you as a concrete value (e.g. a NumPy array).

Your assumption most closely matches the definition of a (C++) tensorflow::Tensor object. The (Python) tf.Tensor object is more complicated because it refers to a function for computing a value, rather than the value itself.


The (Python) tf.Tensor object is more complicated because it refers to a function for computing a value, rather than the value itself.

• Distinction of Variables ==
A similar distinction exists for variables. In the Python API, a tf.Variable is the symbolic representation of a variable, which has methods for creating operations that read the current value of the variable, and assign values to it. In the C++ implementation, a tensorflow::Var object is a wrapper around a shared, mutable tensorflow::Tensor object.

• In Memory Buffer ==
An in-memory buffer is simply a contiguous region of memory that has been allocated with a TensorFlow allocator. tensorflow::Tensor objects contain a pointer to an in-memory buffer, which holds the values of that tensor. The buffer could be in host memory (i.e. accessible from the CPU) or device memory (e.g. accessible only from a GPU), and TensorFlow has operations to move data between these memory spaces.

• Whats a "handle" ==
The TensorFlow codebase uses "handle" to refer to a name for a stateful object (like a tf.FIFOQueue or tf.TensorArray) that can be passed around without copying all of the values (i.e. call-by-reference).

• Internal implementation Tensor vs. Variable ==
In C++, a tensorflow::Tensor and tensorflow::Var are very similar; the only different is that tensorflow::Var also has a mutex that can be used to lock the variable when it is being updated.

In Python, the essential difference is that a tf.Tensor is implemented as a dataflow graph, and it is read-only (i.e. by calling tf.Session.run()). A tf.Variable can be both read (i.e. by evaluating its read operation) and written (e.g. by running an assign operation).

• Tensors and variables serve different purposes.
• Tensors (tf.Tensor objects) can represent complex compositions of mathematical expressions, like loss functions in a neural network, or symbolic gradients.

• Variables represent state that is updated over time, like weight matrices and convolutional filters during training.

• While in principle you could represent the evolving state of a model without variables, you would end up with a very large (and repetetive) mathematical expression, so variables provide a convenient way to materialize the state of the model, and—for example—share it with other machines for parallel training.

• TensorFlow provides many convenient initializers. Alternatively, you may initialize a tf.Variable to have the value of a tf.Tensor.

• Note that when the initializer is a tf.Tensor you should not specify the variable's shape, as the shape of the initializer tensor will be used.

• Seen below - the initializer tensor == tf.constant

• Seen below - initializer=tf.constant([23, 42]) , here the shape of the initializer tensor == 23,42

In [9]:
other_variable = tf.get_variable("other_variable", dtype=tf.int32,initializer=tf.constant([23, 42]))
print(type(other_variable))

#<class 'tensorflow.python.ops.variables.Variable'>

<class 'tensorflow.python.ops.variables.Variable'>

In [12]:
a = tf.Variable(tf.constant(5, shape=[2, 2], dtype=tf.float32))
b = tf.Variable(tf.random_uniform(shape=[2, 2]))

c = a + b

with tf.Session() as session:
#session.run(tf.initialize_all_variables())
#global_variables_initializer
session.run(tf.global_variables_initializer())

print(c.eval())

[[5.947631 5.226041]
[5.261098 5.890769]]

dhankar@dhankar-VPCEB44EN:/media/dhankar/Dhankar_1/a1_18/a1____Tensor_Mar18/Tensor$source activate tensor (tensor) dhankar@dhankar-VPCEB44EN:/media/dhankar/Dhankar_1/a1_18/a1____Tensor_Mar18/Tensor$ pip show tensorflow
Name: tensorflow
Version: 1.5.0
Summary: TensorFlow helps the tensors flow
Home-page: https://www.tensorflow.org/
Location: /home/dhankar/anaconda2/envs/tensor/lib/python3.6/site-packages
Requires: wheel, numpy, protobuf, six, tensorflow-tensorboard, absl-py
(tensor) dhankar@dhankar-VPCEB44EN:/media/dhankar/Dhankar_1/a1_18/a1____Tensor_Mar18/Tensor$(tensor) dhankar@dhankar-VPCEB44EN:/media/dhankar/Dhankar_1/a1_18/a1____Tensor_Mar18/Tensor$ pip show numpy
Name: numpy
Version: 1.14.1
Summary: NumPy: array processing for numbers, strings, records, and objects.
Home-page: http://www.numpy.org
Author: NumPy Developers
Author-email: numpy-discussion@python.org
Location: /home/dhankar/anaconda2/envs/tensor/lib/python3.6/site-packages
Requires:
(tensor) dhankar@dhankar-VPCEB44EN:/media/dhankar/Dhankar_1/a1_18/a1____Tensor_Mar18/Tensor\$

In [8]:
print("EVERY TIME we run this Chunk - weights_3:0 and biases_3:0 --- increments to weights_4:0 , weights_5:0 etc etc ... Why ? ")
print(" "*90)

## Create two variables.
weights = tf.Variable(tf.random_normal([784, 200], stddev=0.35),name="weights")
biases = tf.Variable(tf.zeros([200]), name="biases")

print(type(biases))

print(biases)
print(weights)

EVERY TIME we run this Chunk - weights_3:0 and biases_3:0 --- inceremnts to weights_4:0 , weights_5:0 etc etc ... Why ?

<class 'tensorflow.python.ops.variables.Variable'>
<tf.Variable 'biases_4:0' shape=(200,) dtype=float32_ref>
<tf.Variable 'weights_4:0' shape=(784, 200) dtype=float32_ref>

In [ ]:
# from above == https://github.com/tensorflow/tensorflow/issues/14788
# Need to understand the Difference between --- dtype=int32_ref and dtype=int32
# is the _ref === REFRENCE ?

"""
In [19]: v = tf.Variable([1])
In [20]: v
Out[20]: <tf.Variable 'Variable_2:0' shape=(1,) dtype=int32_ref>

In [21]: v3 = tf.identity(v)
In [22]: v3
Out[22]: <tf.Tensor 'Identity_1:0' shape=(1,) dtype=int32>
"""

In [32]:
# A VARIABLE is NOT A TENSOR --- Variables are in-memory buffers containing tensors.
#

from tensorflow.python.framework import ops

## Defining Variables [email protected] == https://github.com/tensorflow/tensorflow/issues/14788
#
#I can see how the docstring is confusing if you think a variable is a tensor (it's not).
#A variable is a thing which can be converted to a tensor containing its current value.
#It also can be assigned to, partitioned, etc, which tensors can't

# Create three variables with some default values.
print("EVERY TIME we run this Chunk - weights_3:0 inceremnts to weights_4:0 , weights_5:0 etc etc ... Why ? ")
print(" "*90)

weights = tf.Variable(tf.random_normal([2, 3], stddev=0.1), name="weights")
print(weights)
print(type(weights))
print(" "*90)

biases = tf.Variable(tf.zeros([3]), name="biases")
print(biases)
print(type(biases))
print(" "*90)

custom_variable = tf.Variable(tf.zeros([3]), name="custom")
print(custom_variable)
print(type(custom_variable))
print(" "*90)

# Get all the variables' tensors and store them in a list.
all_variables_list = ops.get_collection(ops.GraphKeys.GLOBAL_VARIABLES)

EVERY TIME we run this Chunk - weights_3:0 inceremnts to weights_4:0 , weights_5:0 etc etc ... Why ?

<tf.Variable 'weights_5:0' shape=(2, 3) dtype=float32_ref>
<class 'tensorflow.python.ops.variables.Variable'>

<tf.Variable 'biases_5:0' shape=(3,) dtype=float32_ref>
<class 'tensorflow.python.ops.variables.Variable'>

<tf.Variable 'custom_5:0' shape=(3,) dtype=float32_ref>
<class 'tensorflow.python.ops.variables.Variable'>


In [26]:
# Graphs

a = tf.constant(3.0, dtype=tf.float32)
print(type(a)) ## <class 'tensorflow.python.framework.ops.Tensor'>

#b = tf.constant(4.0,66,77) # also tf.float32 implicitly
# TypeError: Cannot convert 66 to a dtype. data type not understood

b = tf.constant(66) #
print(type(b)) ## <class 'tensorflow.python.framework.ops.Tensor'>

total = a + b  ## Seems we cant add Floats and INT's ??
print(a)
print(b)
print(total)

<class 'tensorflow.python.framework.ops.Tensor'>
<class 'tensorflow.python.framework.ops.Tensor'>

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
12
13
---> 14 total = a + b  ## Seems we cant add Floats and INT's ??
15 print(a)
16 print(b)

~/anaconda2/envs/tensor/lib/python3.6/site-packages/tensorflow/python/ops/math_ops.py in binary_op_wrapper(x, y)
896       if not isinstance(y, sparse_tensor.SparseTensor):
897         try:
--> 898           y = ops.convert_to_tensor(y, dtype=x.dtype.base_dtype, name="y")
899         except TypeError:
900           # If the RHS is not a tensor, it might be a tensor aware object

~/anaconda2/envs/tensor/lib/python3.6/site-packages/tensorflow/python/framework/ops.py in convert_to_tensor(value, dtype, name, preferred_dtype)
930       name=name,
931       preferred_dtype=preferred_dtype,
--> 932       as_ref=False)
933
934

~/anaconda2/envs/tensor/lib/python3.6/site-packages/tensorflow/python/framework/ops.py in internal_convert_to_tensor(value, dtype, name, as_ref, preferred_dtype, ctx)
1020
1021     if ret is None:
-> 1022       ret = conversion_func(value, dtype=dtype, name=name, as_ref=as_ref)
1023
1024     if ret is NotImplemented:

~/anaconda2/envs/tensor/lib/python3.6/site-packages/tensorflow/python/framework/ops.py in _TensorTensorConversionFunction(t, dtype, name, as_ref)
864     raise ValueError(
865         "Tensor conversion requested dtype %s for Tensor with dtype %s: %r" %
--> 866         (dtype.name, t.dtype.name, str(t)))
867   return t
868

ValueError: Tensor conversion requested dtype float32 for Tensor with dtype int32: 'Tensor("Const_17:0", shape=(), dtype=int32)'