How to do weight initialization by Xavier rule in Tensorflow 2.0?
In tensorflow 2.0 you have a package tf.initializer
with all the Keras-like initializers you need.
The Xavier initializer is the same as the Glorot Uniform initializer. Thus, to create a (3,3)
variable with values sampled from that initializer you can just:
shape = (3,3)
initializer = tf.initializers.GlorotUniform()
var = tf.Variable(initializer(shape=shape))
Just use glorot uniform initializer
which is the same as xavier initializer
.
Source: https://www.tensorflow.org/api_docs/python/tf/glorot_uniform_initializer
Also here is an example to prove that they are the same:
tf.reset_default_graph()
tf.set_random_seed(42)
xavier_var = tf.get_variable("w_xavier", shape=[3, 3], initializer=tf.contrib.layers.xavier_initializer())
sess = tf.Session()
sess.run(tf.global_variables_initializer())
print(sess.run(xavier_var))
# [[ 0.27579927 -0.6790426 -0.6128938 ]
# [-0.49439836 -0.36137486 -0.7235348 ]
# [-0.23143482 -0.3394227 -0.34756017]]
tf.reset_default_graph()
tf.set_random_seed(42)
glorot_var = tf.get_variable("w_glorot", shape=[3, 3], initializer=tf.glorot_uniform_initializer())
sess = tf.Session()
sess.run(tf.global_variables_initializer())
print(sess.run(glorot_var))
# [[ 0.27579927 -0.6790426 -0.6128938 ]
# [-0.49439836 -0.36137486 -0.7235348 ]
# [-0.23143482 -0.3394227 -0.34756017]]
In addition, if you want to the glorot uniform initializer
with tf.Variable
you can do:
tf.reset_default_graph()
tf.set_random_seed(42)
normal_var = tf.Variable(tf.glorot_uniform_initializer()((3, 3)))
sess = tf.Session()
sess.run(tf.global_variables_initializer())
print(sess.run(normal_var))
# [[ 0.27579927 -0.6790426 -0.6128938 ]
# [-0.49439836 -0.36137486 -0.7235348 ]
# [-0.23143482 -0.3394227 -0.34756017]]