Pfthb

I'm using a tf.layers.batch_normalization layer in my network. As you may know, batch normalization employs trainable parameters gamma and beta to each unit u_i in this layer, to choose its own standard deviation and mean across u_i(x) for various inputs x. Typically gamma is initialized to 1 and beta to 0.

I'm interested in peeking at the values of gamma and beta that are being learned at various units, to collect statistics about where they tend to end up after the network trains. How can I peek at their current values during each training instance?

You could get all the variables inside the scope of the batch normalization layer and print them. Example:

import tensorflow as tf

tf.reset_default_graph()
x = tf.constant(3.0, shape=(3,))
x = tf.layers.batch_normalization(x)

print(x.name) # batch_normalization/batchnorm/add_1:0

variables = tf.get_collection(tf.GraphKeys.GLOBAL_VARIABLES,
 scope='batch_normalization')
print(variables)

#[<tf.Variable 'batch_normalization/gamma:0' shape=(3,) dtype=float32_ref>,
# <tf.Variable 'batch_normalization/beta:0' shape=(3,) dtype=float32_ref>,
# <tf.Variable 'batch_normalization/moving_mean:0' shape=(3,) dtype=float32_ref>,
# <tf.Variable 'batch_normalization/moving_variance:0' shape=(3,) dtype=float32_ref>]

with tf.Session() as sess:
 sess.run(tf.global_variables_initializer())
 gamma = sess.run(variables[0])
 print(gamma) # [1. 1. 1.]