The original intention of writing this blog: Because I use the tf.variable_scope() variable domain to run the program multiple times in jupyter, I often encounter:
Variable ...... already exists, disallowed. Did you mean to set reuse=True or reuse=tf. AUTO_REUSE in VarScope? Originally defined at:...
This error, after viewing many blogs and materials on the Internet, did not clearly explain the relationship between tf.variable_scope() and the jupyter operating mechanism. Therefore, after deep research and experiments, I decided to explore how to correctly use the tensorflow variable domain in jupyter and the relationship between the two.
When we create a new jupyter document, run the following code in the document:
import tensorflow as tf
with tf.variable_scope("conv"):
# Create variable named "conv/weights".
weights = tf.get_variable("weights", [5, 5, 32, 32], initializer=tf.random_normal_initializer())
# Create variable named "conv/biases".
biases = tf.get_variable("biases", [32], initializer=tf.constant_initializer(0.0))
# tf.trainable_variables()用来获取所有的可训练变量，即之前运用jupyter运行程序时，系统内存中保存的所有变量
vs = tf.trainable_variables()
print("There are %d train_able_variables in the Graph: " % len(vs))
for v in vs:
print(v.name)output:
There are 2 train_able_variables in the Graph:
conv/weights:0
conv/biases:0It can be seen that through the above program, we have created two variables named conv/weights and conv/biases respectively. At this time, we copy the above code to the box below and run it again to see what the output is.
import tensorflow as tf
with tf.variable_scope("conv"):
# Create variable named "conv/weights".
weights = tf.get_variable("weights", [5, 5, 32, 32], initializer=tf.random_normal_initializer())
# Create variable named "conv/biases".
biases = tf.get_variable("biases", [32], initializer=tf.constant_initializer(0.0))
# tf.trainable_variables()用来获取所有的可训练变量，即之前运用jupyter运行程序时，系统内存中保存的所有变量
vs = tf.trainable_variables()
print("There are %d train_able_variables in the Graph: " % len(vs))
for v in vs:
print(v.name)output:
ValueError Traceback (most recent call last)
<ipython-input-2-aa5fff98c0f6> in <module>()
3 with tf.variable_scope("conv"):
4 # Create variable named "conv/weights".
----> 5 weights = tf.get_variable("weights", [5, 5, 32, 32], initializer=tf.random_normal_initializer())
6 # Create variable named "conv/biases".
7 biases = tf.get_variable("biases", [32], initializer=tf.constant_initializer(0.0))
……………………
ValueError: Variable conv/weights already exists, disallowed. Did you mean to set reuse=True or reuse=tf.AUTO_REUSE in VarScope? Originally defined at:
File "<ipython-input-1-aa5fff98c0f6>", line 5, in <module>
weights = tf.get_variable("weights", [5, 5, 32, 32], initializer=tf.random_normal_initializer())
File "/Users/zhuhai/anaconda3/lib/python3.6/site-packages/IPython/core/interactiveshell.py", line 2862, in run_code
exec(code_obj, self.user_global_ns, self.user_ns)
File "/Users/zhuhai/anaconda3/lib/python3.6/site-packages/IPython/core/interactiveshell.py", line 2802, in run_ast_nodes
if self.run_code(code, result):There is an error, the system prompts us that the variable conv/weights already exists, and asks if we want to set reuse=True, that is, to reuse the variable by setting the position of the flag that allows reuse to true. At the same time, the error prompt also tells us where the variable conv/weights was originally defined.
From this, we can conclude that when we ran the above program for the first time, there were no two variables weights and biases defined under the conv variable domain in the system, so the system automatically created these two variables and named them as conv/weights and conv/biases. When we ran it again, the system reported an error, telling us that the variable already exists, and gave us a solution by setting the reuse flag to true. Let's try it out and see the results of the program after adding reuse=True:
import tensorflow as tf
#在tf.variable_scope()中，已经内置了reuse标志位形参，不填，默认为False
with tf.variable_scope("conv", reuse=True):
# Create variable named "conv/weights".
weights = tf.get_variable("weights", [5, 5, 32, 32], initializer=tf.random_normal_initializer())
# Create variable named "conv/biases".
biases = tf.get_variable("biases", [32], initializer=tf.constant_initializer(0.0))
# tf.trainable_variables()用来获取所有的可训练变量，即之前运用jupyter运行程序时，系统内存中保存的所有变量
vs = tf.trainable_variables()
print("There are %d train_able_variables in the Graph: " % len(vs))
for v in vs:
print(v.name)output:
There are 2 train_able_variables in the Graph:
conv/weights:0
conv/biases:0It can be seen that after adding reuse=True, we run the program again and output the same result as running the program for the first time, indicating that we have shared variables.
Summary: The first time we run the program is to create a variable (hint: if reuse=True is added when running the program for the first time, the program will still run wrong, because the variable has not been created and cannot be reused, the reader can test it by himself); The second time to run the program, an error is reported, indicating that the variable is reused without setting the reuse flag to true; the third time to run the program, after we add reuse=True, the variable stored in the program memory is still the first run The variables created later do not recreate new variables, that is, we achieve variable sharing by using the variable scope variable field and the tf.get_variable() function.
Let's change the name of the variable domain above from conv to conv1, and remove reuse=True to see what will be output.
import tensorflow as tf
#在tf.variable_scope()中，已经内置了reuse标志位形参，不填，默认为False
with tf.variable_scope("conv1"):
# Create variable named "conv/weights".
weights = tf.get_variable("weights", [5, 5, 32, 32], initializer=tf.random_normal_initializer())
# Create variable named "conv/biases".
biases = tf.get_variable("biases", [32], initializer=tf.constant_initializer(0.0))
# tf.trainable_variables()用来获取所有的可训练变量，即之前运用jupyter运行程序时，系统内存中保存的所有变量
vs = tf.trainable_variables()
print("There are %d train_able_variables in the Graph: " % len(vs))
for v in vs:
print(v.name)output:
There are 4 train_able_variables in the Graph:
conv/weights:0
conv/biases:0
conv1/weights:0
conv1/biases:0It can be seen that four variables are output, the first two are the conv/weights and conv/biases variables we created before, and the last two are the conv1/weights and conv1/biases variables created by our program run.
Summary:
1. Every time a new variable is defined in jupyter, the system memory will open up a storage space named after the variable to store the variable. Variables are stored in memory only in a one-to-one correspondence with their names.
2. In the tf.get_variable() function, when a variable already exists in the memory, set the reuse of the variable domain to True, which can realize variable reuse, that is, variable sharing; when a variable does not exist in the memory, set the reuse to False to create a variable.
3. The formal parameter of tf.variable_scope( ) - the reuse flag, the scope of action is all variables defined in the entire variable scope.
Next, we will copy the above program to the following, use the tf.Variable() function to define the variable, and leave everything else unchanged to see what will be output.
import tensorflow as tf
#在tf.variable_scope()中，已经内置了reuse标志位形参，不填，默认为False
with tf.variable_scope("conv1"):
# Create variable named "conv/weights".
weights = tf.Variable(tf.random_normal([5, 5, 32, 32]), name="weights")
# Create variable named "conv/biases".
biases = tf.Variable(tf.zeros([32]), name="biases")
# tf.trainable_variables()用来获取所有的可训练变量，即之前运用jupyter运行程序时，系统内存中保存的所有变量
vs = tf.trainable_variables()
print("There are %d train_able_variables in the Graph: " % len(vs))
for v in vs:
print(v.name)output:
There are 6 train_able_variables in the Graph:
conv/weights:0
conv/biases:0
conv1/weights:0
conv1/biases:0
conv1_2/weights:0
conv1_2/biases:0Summary: We have not changed the variable domain and variable name, but the system has created two more variables, conv1_2/weights and conv1_2/biases. It can be seen that the function of tf.Variable() is: regardless of whether the variable has been created before, the system Will automatically assign new names to variables, open up new storage space, and create new variables.
This article is the author's original work, please indicate the source when reprinting.