Tensorflow - Notebook -1

Index :

  1. Sys config checks .
  2. What are Sessions.
  3. What are Variables.
  4. What are Tensors. What are Tensors.
  5. ....

$ Geometry Symbols $


In [2]:
import sys
print(sys.executable)
print(sys.version)
print(sys.version_info)
/home/dhankar/anaconda2/envs/tensor/bin/python
3.6.2 |Continuum Analytics, Inc.| (default, Jul 20 2017, 13:51:32) 
[GCC 4.4.7 20120313 (Red Hat 4.4.7-1)]
sys.version_info(major=3, minor=6, micro=2, releaselevel='final', serial=0)
In [2]:
import tensorflow as tf
  • Create a Constant op == Operation == NODE of the Computational Graph. The op is added as a node to the default graph.

  • The value returned by the constructor represents the output of the Constant op.

  • The most basic Computational unit is a Constant - it is a value that will not change during evaluation / computation of the graph.

  • The Constant is always a Tensor it can be a One Dimensional SCALAR or a STRING or higher dimensional tensors.
In [ ]:
hello = tf.constant('Hello, TensorFlow!')
In [5]:
# Start tf session
sess = tf.Session()
In [6]:
# Run graph
print(sess.run(hello))
b'Hello, TensorFlow!'
In [17]:
# Basic constant operations
# The value returned by the constructor represents the output
# of the Constant op.
a = tf.constant(21)
b = tf.constant(13)
#
print(type(a))
# <class 'tensorflow.python.framework.ops.Tensor'>
<class 'tensorflow.python.framework.ops.Tensor'>

The SCALAR Tensor's == a = tf.constant(21)

  • a rank 0 tensor; a scalar with shape [],
In [19]:
# Launch the default graph.
with tf.Session() as sess:
    print("a: sess.run(a) b:_______TYPE_______",type(sess.run(b)))
    print("b: sess.run(b) b:__________________",sess.run(b))
    print("Addition with constants:___________",sess.run(a+b))
    print("Multiplication with constants:_____",sess.run(a*b))
    print("Subtraction with constants_________",sess.run(a-b))

print(type(sess))   ## <class 'tensorflow.python.client.session.Session'> 
a: sess.run(a) b:_______TYPE_______ <class 'numpy.int32'>
a: sess.run(a) b:__________________ 13
Addition with constants:___________ 34
Multiplication with constants:_____ 273
Subtraction with constants_________ 8
<class 'tensorflow.python.client.session.Session'>
In [21]:
# Add , Divide - basic Op's 

x = tf.add(a, b, name="add")
y = tf.div(a, b, name="divide")
print(type(x))
print(type(y))


# x and y are   == # <class 'tensorflow.python.framework.ops.Tensor'>
# same as the - #b = tf.constant(13) ...
#
<class 'tensorflow.python.framework.ops.Tensor'>
<class 'tensorflow.python.framework.ops.Tensor'>
In [20]:
# Closing the session.
sess.close()
print(type(sess))
<class 'tensorflow.python.client.session.Session'>
In [10]:
# Create Graphs over and above the DEFAULT Graph 

graph1 = tf.Graph()
print(type(graph1))

graph2 = tf.Graph()
print(type(graph2))
<class 'tensorflow.python.framework.ops.Graph'>
<class 'tensorflow.python.framework.ops.Graph'>

Official Tutorials for TF


In [3]:
from __future__ import absolute_import
from __future__ import division
from __future__ import print_function

import numpy as np
import tensorflow as tf

Tensor Values

  • The central unit of data in TensorFlow is the tensor.
  • A tensor consists of a set of primitive values shaped into an array of any number of dimensions.
  • A tensor's rank is its number of dimensions, while its shape is a tuple of integers specifying the array's length along each dimension.
  • TensorFlow uses numpy arrays to represent tensor values.

Here are some examples of tensor values:

3. # a rank 0 tensor; a scalar with shape [],
[1., 2., 3.] # a rank 1 tensor; a vector with shape [3]
[[1., 2., 3.], [4., 5., 6.]] # a rank 2 tensor; a matrix with shape [2, 3]
[[[1., 2., 3.]], [[7., 8., 9.]]] # a rank 3 tensor with shape [2, 1, 3]

Creating a Variable


  • The best way to create a variable is to call the tf.get_variable function. This function requires you to specify the Variable's name.
  • This name will be used by other replicas to access the same variable, as well as to name this variable's value when checkpointing and exporting models.
  • tf.get_variable also allows you to reuse a previously created variable of the same name, making it easy to define models which reuse layers.

To create a variable with tf.get_variable, simply provide the name and shape..

  • When you train a model, you use variables to hold and update parameters.
  • Variables are in-memory buffers containing tensors.
  • They must be explicitly initialized and can be saved to disk during and after training.
  • You can later restore saved values to exercise or analyze the model.

  • A tf.Variable represents a tensor whose value can be changed by running ops on it.

  • Unlike tf.Tensor objects, a tf.Variable exists outside the context of a single session.run call.

  • Internally, a tf.Variable stores a persistent tensor.

  • Specific ops allow you to read and modify the values of this tensor.
  • These modifications are visible across multiple tf.Sessions, so multiple workers can see the same values for a tf.Variable.

Current Version - r1.6 - https://www.tensorflow.org/programmers_guide/variables

Older version - r1.0 - https://www.tensorflow.org/versions/r1.0/programmers_guide/variables

In [4]:
my_variable = tf.get_variable("my_variable", [1, 2, 3])
print(type(my_variable))
<class 'tensorflow.python.ops.variables.Variable'>
  • This creates a variable named "my_variable" which is a three-dimensional tensor with shape [1, 2, 3].
  • This variable will, by default, have the dtype tf.float32 and its initial value will be randomized via tf.glorot_uniform_initializer.
  • You may optionally specify the dtype and initializer to tf.get_variable.

For example:

Pedagogy ---

  • Seen below the "traceback" when we run the Code Chunk twice or more .
  • We have already created a named variable - the code chunk re-run tries to create the same variable again.
  • This is checked and Error thrown by - /anaconda2/envs/tensor/lib/python3.6/site-packages/tensorflow/python/ops/variable_scope.py in _get_single_variable

cool

In [6]:
#my_int_variable = tf.get_variable("my_int_variable", [1, 2, 3], dtype=tf.int32,initializer=tf.zeros_initializer)

#print(type(my_int_variable))
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-6-90d7e657800c> in <module>()
      1 my_int_variable = tf.get_variable("my_int_variable", [1, 2, 3], dtype=tf.int32,
----> 2   initializer=tf.zeros_initializer)
      3 print(type(my_int_variable))

~/anaconda2/envs/tensor/lib/python3.6/site-packages/tensorflow/python/ops/variable_scope.py in get_variable(name, shape, dtype, initializer, regularizer, trainable, collections, caching_device, partitioner, validate_shape, use_resource, custom_getter, constraint)
   1260       partitioner=partitioner, validate_shape=validate_shape,
   1261       use_resource=use_resource, custom_getter=custom_getter,
-> 1262       constraint=constraint)
   1263 get_variable_or_local_docstring = (
   1264     """%s

~/anaconda2/envs/tensor/lib/python3.6/site-packages/tensorflow/python/ops/variable_scope.py in get_variable(self, var_store, name, shape, dtype, initializer, regularizer, reuse, trainable, collections, caching_device, partitioner, validate_shape, use_resource, custom_getter, constraint)
   1095           partitioner=partitioner, validate_shape=validate_shape,
   1096           use_resource=use_resource, custom_getter=custom_getter,
-> 1097           constraint=constraint)
   1098 
   1099   def _get_partitioned_variable(self,

~/anaconda2/envs/tensor/lib/python3.6/site-packages/tensorflow/python/ops/variable_scope.py in get_variable(self, name, shape, dtype, initializer, regularizer, reuse, trainable, collections, caching_device, partitioner, validate_shape, use_resource, custom_getter, constraint)
    433           caching_device=caching_device, partitioner=partitioner,
    434           validate_shape=validate_shape, use_resource=use_resource,
--> 435           constraint=constraint)
    436 
    437   def _get_partitioned_variable(

~/anaconda2/envs/tensor/lib/python3.6/site-packages/tensorflow/python/ops/variable_scope.py in _true_getter(name, shape, dtype, initializer, regularizer, reuse, trainable, collections, caching_device, partitioner, validate_shape, use_resource, constraint)
    402           trainable=trainable, collections=collections,
    403           caching_device=caching_device, validate_shape=validate_shape,
--> 404           use_resource=use_resource, constraint=constraint)
    405 
    406     if custom_getter is not None:

~/anaconda2/envs/tensor/lib/python3.6/site-packages/tensorflow/python/ops/variable_scope.py in _get_single_variable(self, name, shape, dtype, initializer, regularizer, partition_info, reuse, trainable, collections, caching_device, validate_shape, use_resource, constraint)
    741                          "reuse=tf.AUTO_REUSE in VarScope? "
    742                          "Originally defined at:\n\n%s" % (
--> 743                              name, "".join(traceback.format_list(tb))))
    744       found_var = self._vars[name]
    745       if not shape.is_compatible_with(found_var.get_shape()):

ValueError: Variable my_int_variable already exists, disallowed. Did you mean to set reuse=True or reuse=tf.AUTO_REUSE in VarScope? Originally defined at:

  File "<ipython-input-5-5cbd14b8ee95>", line 2, in <module>
    initializer=tf.zeros_initializer)
  File "/home/dhankar/anaconda2/envs/tensor/lib/python3.6/site-packages/IPython/core/interactiveshell.py", line 2910, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "/home/dhankar/anaconda2/envs/tensor/lib/python3.6/site-packages/IPython/core/interactiveshell.py", line 2850, in run_ast_nodes
    if self.run_code(code, result):
In [17]:
my_int_var1 = tf.get_variable("my_int_var1", [1, 2, 3], dtype=tf.int32,initializer=tf.zeros_initializer)

with tf.Session() as sess:
    sess.run(tf.initialize_all_variables())
    print(my_int_var1.eval())

print("  "*90)    
print(type(my_int_var1))
[[[0 0 0]
  [0 0 0]]]
<class 'tensorflow.python.ops.variables.Variable'>
In [18]:
my_int_var2 = tf.get_variable("my_int_var2", [1, 3, 9], dtype=tf.int32,initializer=tf.zeros_initializer)

with tf.Session() as sess:
    sess.run(tf.initialize_all_variables())
    print(my_int_var2.eval())

print("  "*90)    
print(type(my_int_var2))
[[[0 0 0 0 0 0 0 0 0]
  [0 0 0 0 0 0 0 0 0]
  [0 0 0 0 0 0 0 0 0]]]
                                                                                                                                                                                    
<class 'tensorflow.python.ops.variables.Variable'>
In [19]:
my_int_var3 = tf.get_variable("my_int_var3", [3, 9, 12], dtype=tf.int32,initializer=tf.zeros_initializer)

with tf.Session() as sess:
    sess.run(tf.initialize_all_variables())
    print(my_int_var3.eval())

print("  "*90)    
print(type(my_int_var3))
[[[0 0 0 0 0 0 0 0 0 0 0 0]
  [0 0 0 0 0 0 0 0 0 0 0 0]
  [0 0 0 0 0 0 0 0 0 0 0 0]
  [0 0 0 0 0 0 0 0 0 0 0 0]
  [0 0 0 0 0 0 0 0 0 0 0 0]
  [0 0 0 0 0 0 0 0 0 0 0 0]
  [0 0 0 0 0 0 0 0 0 0 0 0]
  [0 0 0 0 0 0 0 0 0 0 0 0]
  [0 0 0 0 0 0 0 0 0 0 0 0]]

 [[0 0 0 0 0 0 0 0 0 0 0 0]
  [0 0 0 0 0 0 0 0 0 0 0 0]
  [0 0 0 0 0 0 0 0 0 0 0 0]
  [0 0 0 0 0 0 0 0 0 0 0 0]
  [0 0 0 0 0 0 0 0 0 0 0 0]
  [0 0 0 0 0 0 0 0 0 0 0 0]
  [0 0 0 0 0 0 0 0 0 0 0 0]
  [0 0 0 0 0 0 0 0 0 0 0 0]
  [0 0 0 0 0 0 0 0 0 0 0 0]]

 [[0 0 0 0 0 0 0 0 0 0 0 0]
  [0 0 0 0 0 0 0 0 0 0 0 0]
  [0 0 0 0 0 0 0 0 0 0 0 0]
  [0 0 0 0 0 0 0 0 0 0 0 0]
  [0 0 0 0 0 0 0 0 0 0 0 0]
  [0 0 0 0 0 0 0 0 0 0 0 0]
  [0 0 0 0 0 0 0 0 0 0 0 0]
  [0 0 0 0 0 0 0 0 0 0 0 0]
  [0 0 0 0 0 0 0 0 0 0 0 0]]]
                                                                                                                                                                                    
<class 'tensorflow.python.ops.variables.Variable'>



Variables vs. Tensors

Further notes to understand Variables and Tensors - also the Difference between the two


https://stackoverflow.com/questions/40866675/implementation-difference-between-tensorflow-variable-and-tensorflow-tensor


  • Quoting Verbatim - the tf.tensor ==
In the Python API, a tf.Tensor object represents the symbolic result of a TensorFlow operation. For example, in the expression t = tf.matmul(x, y), t is a tf.Tensor object representing the result of multiplying x and y (which may themselves be symbolic results of other operations, concrete values such as NumPy arrays, or variables).

In this context, a "symbolic result" is more complicated than a pointer to the result of an operation. It is more analogous to a function object that, when called (i.e. passed to tf.Session.run()) will run the necessary computation to produce the result of that operation, and return it to you as a concrete value (e.g. a NumPy array).

Your assumption most closely matches the definition of a (C++) tensorflow::Tensor object. The (Python) tf.Tensor object is more complicated because it refers to a function for computing a value, rather than the value itself.

The (Python) tf.Tensor object is more complicated because it refers to a function for computing a value, rather than the value itself.


  • Distinction of Variables ==
A similar distinction exists for variables. In the Python API, a tf.Variable is the symbolic representation of a variable, which has methods for creating operations that read the current value of the variable, and assign values to it. In the C++ implementation, a tensorflow::Var object is a wrapper around a shared, mutable tensorflow::Tensor object.
  • In Memory Buffer ==
An in-memory buffer is simply a contiguous region of memory that has been allocated with a TensorFlow allocator. tensorflow::Tensor objects contain a pointer to an in-memory buffer, which holds the values of that tensor. The buffer could be in host memory (i.e. accessible from the CPU) or device memory (e.g. accessible only from a GPU), and TensorFlow has operations to move data between these memory spaces.
  • Whats a "handle" ==
The TensorFlow codebase uses "handle" to refer to a name for a stateful object (like a tf.FIFOQueue or tf.TensorArray) that can be passed around without copying all of the values (i.e. call-by-reference).
  • Internal implementation Tensor vs. Variable ==
In C++, a tensorflow::Tensor and tensorflow::Var are very similar; the only different is that tensorflow::Var also has a mutex that can be used to lock the variable when it is being updated.

In Python, the essential difference is that a tf.Tensor is implemented as a dataflow graph, and it is read-only (i.e. by calling tf.Session.run()). A tf.Variable can be both read (i.e. by evaluating its read operation) and written (e.g. by running an assign operation).
  • Tensors and variables serve different purposes.
  • Tensors (tf.Tensor objects) can represent complex compositions of mathematical expressions, like loss functions in a neural network, or symbolic gradients.

  • Variables represent state that is updated over time, like weight matrices and convolutional filters during training.

  • While in principle you could represent the evolving state of a model without variables, you would end up with a very large (and repetetive) mathematical expression, so variables provide a convenient way to materialize the state of the model, and—for example—share it with other machines for parallel training.


Also read --

https://stackoverflow.com/questions/38556078/in-tensorflow-what-is-the-difference-between-a-variable-and-a-tensor https://github.com/tensorflow/tensorflow/issues/10855


  • TensorFlow provides many convenient initializers. Alternatively, you may initialize a tf.Variable to have the value of a tf.Tensor.

  • Note that when the initializer is a tf.Tensor you should not specify the variable's shape, as the shape of the initializer tensor will be used.

  • Seen below - the initializer tensor == tf.constant

  • Seen below - initializer=tf.constant([23, 42]) , here the shape of the initializer tensor == 23,42

  • Also Read --- https://stackoverflow.com/questions/38805599/why-tf-constant-initializer-does-not-take-a-constant-tensor


In [9]:
other_variable = tf.get_variable("other_variable", dtype=tf.int32,initializer=tf.constant([23, 42]))
print(type(other_variable))

#<class 'tensorflow.python.ops.variables.Variable'>
<class 'tensorflow.python.ops.variables.Variable'>
In [12]:
a = tf.Variable(tf.constant(5, shape=[2, 2], dtype=tf.float32))
b = tf.Variable(tf.random_uniform(shape=[2, 2]))

c = a + b

with tf.Session() as session:
    #session.run(tf.initialize_all_variables())
    #global_variables_initializer
    session.run(tf.global_variables_initializer())


    print(c.eval())
[[5.947631 5.226041]
 [5.261098 5.890769]]
dhankar@dhankar-VPCEB44EN:/media/dhankar/Dhankar_1/a1_18/a1____Tensor_Mar18/Tensor$ source activate tensor
(tensor) dhankar@dhankar-VPCEB44EN:/media/dhankar/Dhankar_1/a1_18/a1____Tensor_Mar18/Tensor$ pip show tensorflow
Name: tensorflow
Version: 1.5.0
Summary: TensorFlow helps the tensors flow
Home-page: https://www.tensorflow.org/
Author: Google Inc.
Author-email: opensource@google.com
License: Apache 2.0
Location: /home/dhankar/anaconda2/envs/tensor/lib/python3.6/site-packages
Requires: wheel, numpy, protobuf, six, tensorflow-tensorboard, absl-py
(tensor) dhankar@dhankar-VPCEB44EN:/media/dhankar/Dhankar_1/a1_18/a1____Tensor_Mar18/Tensor$ 
(tensor) dhankar@dhankar-VPCEB44EN:/media/dhankar/Dhankar_1/a1_18/a1____Tensor_Mar18/Tensor$ pip show numpy
Name: numpy
Version: 1.14.1
Summary: NumPy: array processing for numbers, strings, records, and objects.
Home-page: http://www.numpy.org
Author: NumPy Developers
Author-email: numpy-discussion@python.org
License: BSD
Location: /home/dhankar/anaconda2/envs/tensor/lib/python3.6/site-packages
Requires: 
(tensor) dhankar@dhankar-VPCEB44EN:/media/dhankar/Dhankar_1/a1_18/a1____Tensor_Mar18/Tensor$
In [8]:
print("EVERY TIME we run this Chunk - weights_3:0 and biases_3:0 --- increments to weights_4:0 , weights_5:0 etc etc ... Why ? ")
print(" "*90)


## Create two variables.
weights = tf.Variable(tf.random_normal([784, 200], stddev=0.35),name="weights")
biases = tf.Variable(tf.zeros([200]), name="biases")

print(type(biases))

print(biases)
print(weights)
EVERY TIME we run this Chunk - weights_3:0 and biases_3:0 --- inceremnts to weights_4:0 , weights_5:0 etc etc ... Why ? 
                                                                                          
<class 'tensorflow.python.ops.variables.Variable'>
<tf.Variable 'biases_4:0' shape=(200,) dtype=float32_ref>
<tf.Variable 'weights_4:0' shape=(784, 200) dtype=float32_ref>
In [ ]:
# from above == https://github.com/tensorflow/tensorflow/issues/14788
# Need to understand the Difference between --- dtype=int32_ref and dtype=int32
# is the _ref === REFRENCE ? 

"""
In [19]: v = tf.Variable([1])
In [20]: v
Out[20]: <tf.Variable 'Variable_2:0' shape=(1,) dtype=int32_ref>

In [21]: v3 = tf.identity(v)
In [22]: v3
Out[22]: <tf.Tensor 'Identity_1:0' shape=(1,) dtype=int32>
"""
In [32]:
# A VARIABLE is NOT A TENSOR --- Variables are in-memory buffers containing tensors. 
#

from tensorflow.python.framework import ops

## Defining Variables [email protected] == https://github.com/tensorflow/tensorflow/issues/14788
#
#I can see how the docstring is confusing if you think a variable is a tensor (it's not). 
#A variable is a thing which can be converted to a tensor containing its current value. 
#It also can be assigned to, partitioned, etc, which tensors can't


# Create three variables with some default values.
print("EVERY TIME we run this Chunk - weights_3:0 inceremnts to weights_4:0 , weights_5:0 etc etc ... Why ? ")
print(" "*90)

weights = tf.Variable(tf.random_normal([2, 3], stddev=0.1), name="weights")
print(weights)
print(type(weights))
print(" "*90)

biases = tf.Variable(tf.zeros([3]), name="biases")
print(biases)
print(type(biases))
print(" "*90)

custom_variable = tf.Variable(tf.zeros([3]), name="custom")
print(custom_variable)
print(type(custom_variable))
print(" "*90)


# Get all the variables' tensors and store them in a list.
all_variables_list = ops.get_collection(ops.GraphKeys.GLOBAL_VARIABLES)
EVERY TIME we run this Chunk - weights_3:0 inceremnts to weights_4:0 , weights_5:0 etc etc ... Why ? 
                                                                                          
<tf.Variable 'weights_5:0' shape=(2, 3) dtype=float32_ref>
<class 'tensorflow.python.ops.variables.Variable'>
                                                                                          
<tf.Variable 'biases_5:0' shape=(3,) dtype=float32_ref>
<class 'tensorflow.python.ops.variables.Variable'>
                                                                                          
<tf.Variable 'custom_5:0' shape=(3,) dtype=float32_ref>
<class 'tensorflow.python.ops.variables.Variable'>
                                                                                          
In [26]:
# Graphs 

a = tf.constant(3.0, dtype=tf.float32)
print(type(a)) ## <class 'tensorflow.python.framework.ops.Tensor'>

#b = tf.constant(4.0,66,77) # also tf.float32 implicitly
# TypeError: Cannot convert 66 to a dtype. data type not understood


b = tf.constant(66) # 
print(type(b)) ## <class 'tensorflow.python.framework.ops.Tensor'>


total = a + b  ## Seems we cant add Floats and INT's ??
print(a)
print(b)
print(total)
<class 'tensorflow.python.framework.ops.Tensor'>
<class 'tensorflow.python.framework.ops.Tensor'>
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-26-0ad9c4461e85> in <module>()
     12 
     13 
---> 14 total = a + b  ## Seems we cant add Floats and INT's ??
     15 print(a)
     16 print(b)

~/anaconda2/envs/tensor/lib/python3.6/site-packages/tensorflow/python/ops/math_ops.py in binary_op_wrapper(x, y)
    896       if not isinstance(y, sparse_tensor.SparseTensor):
    897         try:
--> 898           y = ops.convert_to_tensor(y, dtype=x.dtype.base_dtype, name="y")
    899         except TypeError:
    900           # If the RHS is not a tensor, it might be a tensor aware object

~/anaconda2/envs/tensor/lib/python3.6/site-packages/tensorflow/python/framework/ops.py in convert_to_tensor(value, dtype, name, preferred_dtype)
    930       name=name,
    931       preferred_dtype=preferred_dtype,
--> 932       as_ref=False)
    933 
    934 

~/anaconda2/envs/tensor/lib/python3.6/site-packages/tensorflow/python/framework/ops.py in internal_convert_to_tensor(value, dtype, name, as_ref, preferred_dtype, ctx)
   1020 
   1021     if ret is None:
-> 1022       ret = conversion_func(value, dtype=dtype, name=name, as_ref=as_ref)
   1023 
   1024     if ret is NotImplemented:

~/anaconda2/envs/tensor/lib/python3.6/site-packages/tensorflow/python/framework/ops.py in _TensorTensorConversionFunction(t, dtype, name, as_ref)
    864     raise ValueError(
    865         "Tensor conversion requested dtype %s for Tensor with dtype %s: %r" %
--> 866         (dtype.name, t.dtype.name, str(t)))
    867   return t
    868 

ValueError: Tensor conversion requested dtype float32 for Tensor with dtype int32: 'Tensor("Const_17:0", shape=(), dtype=int32)'