tf.Variable and tf.Tensor

Posted on |

tf.Variable

Create variable

tf.get_variable

  • easy to define models which reuse layers
1
2
my_int_variable = tf.get_variable("my_int_variable", [1, 2, 3], dtype=tf.int32, 
  initializer=tf.zeros_initializer)
1
2
other_variable = tf.get_variable("other_variable", dtype=tf.int32, 
  initializer=tf.constant([23, 42]))

Variable collections

collections: name lists of tensors or other objects, such as tf.Variable instances.

Predefined collections

By default every tf.Variable gets placed in the following two collections:

  • tf.GraphKeys.GLOBAL_VARIABLES
    • variables that can be shared across multiple devices
  • tf.GraphKeys.TRAINABLE_VARIABLE
    • variables for which TensorFlow will calculate gradients

If you don’t want a variable to be trainable, add it to the tf.GraphKeys.LOCAL_VARIABLES collection instead.
The method is:

1
2
my_local = tf.get_variable("my_local", shape=(), 
							collections=[tf.GraphKeys.LOCAL_VARIABLES])

or

1
my_non_trainable = tf.get_variable("my_non_trainable", shape=(), trainable=False)

Custom collections

There is no need to explicitly create a collection.
To add a variable (or any other object) to a collection:

1
tf.add_to_collection("my_collection_name", my_local)

To retrieve a list of all the variables (or other objects) you’ve placed in a collection:

1
tf.get_collection("my_collection_name")

Variable placement

Placement method:

1
2
with tf.device("/gpu:1"):
  v = tf.get_variable("v", [1])

It is particularly important for variables to be in the correct device in distributed settings.
For this reason we provide tf.train.replica_device_setter, which can automatically place variables in parameter servers.

1
2
3
4
5
6
7
cluster_spec = {
    "ps": ["ps0:2222", "ps1:2222"],
    "worker": ["worker0:2222", "worker1:2222", "worker2:2222"]}
with tf.device(tf.train.replica_device_setter(cluster=cluster_spec)):
  v = tf.get_variable("v", shape=[20, 20])  # this variable is placed 
                                            # in the parameter server
                                            # by the replica_device_setter

Initializing variables

In the low-level TensorFlow API, must explicitly initialize the variables by ourselves.

  • that is, you are explicitly creating your own graphs and sessions)

Most high-level frameworks automatically initialize variables for you

  • tf.contrib.slim, tf.estimator.Estimator and Keras

Explicit initialization is otherwise useful because it

  • allows you not to rerun potentially expensive initializers when reloading a model from a checkpoint ?
  • as well as allowing determinism when randomly-initialized variables are shared in a distributed setting.?

tf.global_variables_initializer()

  • To initialize all trainable variables
  • To be called before training starts
1
2
session.run(tf.global_variables_initializer())
# Now all variables are initialized.

To initialize variables yourself:

1
session.run(my_variable.initializer)

ask which variables have still not been initialized:

1
print(session.run(tf.report_uninitialized_variables()))

Notice

by default tf.global_variables_initializer does not specify the order in which variables are initialized.
If you use a variable’s value while initializing another variable, use variable.initialized_value() instead of variable

1
2
v = tf.get_variable("v", shape=(), initializer=tf.zeros_initializer())
w = tf.get_variable("w", initializer=v.initialized_value() + 1)

Using variables

To use the value of a tf.Variable in a TensorFlow graph, simply treat it like a normal tf.Tensor.
To assign a value to a variable, use the methods such as assign, assign_add in the tf.Variable class.
To force a re-read of the value of a variable after something has happened, you can use tf.Variable.read_value.

1
2
3
4
5
6
7
8
v = tf.get_variable("v", shape=(), initializer=tf.zeros_initializer())
assignment = v.assign_add(1)
assignment2 = v.assign_add(1)
with tf.control_dependencies([assignment, assignment2]):
  w = v.read_value()  # w is guaranteed to reflect v's value after the
                      # assign_add operation.
# v: 0
# w: 2

tf.train.Optimizer

https://www.tensorflow.org/api_docs/python/tf/train/Optimizer

Saving and Restoring

https://www.tensorflow.org/programmers_guide/variables#saving_and_restoring

Sharing variables

https://www.tensorflow.org/programmers_guide/variables#sharing_variables

Tensors

https://www.tensorflow.org/programmers_guide/tensors