Preface: tensorflow is a huge system. There are many functions in it, and many conventional operations are implemented, but there is always no way to cover all operations. Sometimes we need to define some of our own operation logic to achieve the specified functions, but found that it is not so Simple, this article is a problem that occurred when writing tf.data.DataSet, made a centralized summary, it will involve the following concepts:
EagerTensor and Tensor, tf.py_function and tf.numpy_function, dataset.map and so on .
1. Problem description
The problem that needs to be solved, there are three text files, which are stored in the files folder. The names are file1.txt, file2.txt, and file3.txt. The contents are as follows:
file1.txt
1,2,3,4,5
file2.txt
11,22,33,44,55
file3.txt
111,222,333,444,555
The tags of each file are 1, 2, and 3. Now assuming that it has been one-hot encoded, the categories are respectively
[ [1,0,0],[0,1,0],[0,0,1] ]
First, I need to read these three samples through the dataset standard pipeline, so that I can put them into the neural network for training. Obviously, I need to read each text file, and I need to use the datase.map() function. The code is as follows:
X=["file1.txt","file2.txt","file3.txt"]
Y=[[1,0,0],[0,1,0],[0,0,1]]
# 构建dataset对象
dataset = tf.data.Dataset.from_tensor_slices((X,Y)) # 第一步：构造dataset对象
# 对每一个dataset的元素，实际上就是一个example进行解析
dataset = dataset.map(read_file)
for features,label in dataset:
print(features)
print(label)
print("===========================================================")
The parsing function read_file is as follows
def read_file(filename,label):
tf.print(type(filename))
tf.print(type(label))
# filename_ = filename.numpy()
# label_ = label.numpy()
filename = "./files/" + filename
tf.print("/")
f = open(filename,mode="r")
s =f.readline()
x_ =s.split(',')
result =[]
for i in x_:
result.append(int(i))
return result,label
The code looks okay, but it actually displays the following error when running:
TypeError: expected str, bytes or os.PathLike object, not TensorThe wrong place is
f = open(filename,mode="r")The meaning is very simple, that is, the filename that reads the file should be a str, or an object representing a path, rather than a Tensor object.
Note: This problem has troubled me for 2 days. I searched for a long time on google to find a solution. Chinese search has almost no suitable answer.
then what should we do?
It seems to be very simple. Since he said that the filename and label are a Tensor, then we only need to read the value in the Tensor. Don't we get the string. In fact, it is true, tensorflow2.x It tells us that we can use t.numpy() to get the value of tensor, but when we use these two methods, we find that it is still wrong, and the following error is displayed:
filename_ = filename.numpy()
label_ = label.numpy()AttributeError: 'Tensor' object has no attribute 'numpy'How can Tensor not have numpy attributes? Don't we all get the value of tensor through t.numpy()? This actually leads to the following question.
Two, distinguish between tf.EagerTensor and tf.Tensor
2.1 Simple example
Look at a few simple examples:
In [59]: a = tf.constant([1,2,3,4,5])
In [60]: a
Out[60]: <tf.Tensor: shape=(5,), dtype=int32, numpy=array([1, 2, 3, 4, 5])>
In [61]: type(a)
Out[61]: tensorflow.python.framework.ops.EagerTensorTwo problems were found:
(1) Here a is indeed a Tensor, and it has the attribute numpy, we can get its value through a.numpy()
(2) Its type is essentially an EagerTensor,
The reason why the above Tensor has no numpy attribute is because it looks like this
<class 'tensorflow.python.framework.ops.Tensor'> # 类型
tf.Tensor([102 102 102], shape=(3,), dtype=int64) # TensorTherefore, in tensorflow 2.x, any value that can be obtained with numpy refers to EagerTensor, although it is printed out and displayed in the following form:
<tf.Tensor: ... ...>And what exactly is Tensor? It is actually a kind of Tensor in static graph. Although we are currently using a dynamic library, there is still a process of building a graph in the background. The value of Tensor cannot be obtained in time, but it needs to be obtained after data. In the tensorflow1.x static graph, we need Use the following methods to get the value of Tensor:
with tf.Session() as sess:
result = sess.run([t]) # 获取Tensor t 的值
print(result)
# 或者是
result = t.eval()2.2 Precautions for using tensorflow2.x
Some notes on the use of EagerTensor and Tensor
(1) I hope to print and see the result of the calculation, tf.print(tensor)instead of usingprint(tensor.numpy())
Use tf.print(tensor)can be able to print both in static or dynamic map drawing a tensor of content , and print(tensor.numpy())can only be used in motion picture, but can only use EagerTensor, the following is a correct demonstration:
(2) Use tf.device instead of tensor.gpu(), tensor.cpu()
When a tensor is created in the new version, it will be automatically assigned to a device with a higher priority. For example, when there is a GPU, it will be directly assigned to the GPU:
# 需要GPU版本才能看出创建的张量会直接放置到gpu上!CPU版本不行
import tensorflow as tf
print(tf.test.is_gpu_available())
# True
r = tf.random.normal((3, 4))
print(r.device)
# '/job:localhost/replica:0/task:0/device:GPU:0'For a given version of the new device, EagerTensormay directly .cpu(), .gpu()the method moves directly to the corresponding tensor device, but tf.Tensordid not , the method is compatible with both the tf.deviceoperation of the created scope. An error example of creating a tensor under gpu and moving to cpu for sin operation is:
(3) Don't traverse tensors, try to use vectorized operations
EagerTensorIt can be traversed, but tf.Tensornot, so try not to traverse the tensor, think about how to perform vectorization operations, not only the compatibility of dynamic and static images, but the speed improvement after vectorization is also very large. of.
2.3 Analysis and understanding,
We can understand it like this,
EagerTensor is real-time, and its value can be obtained at any time, that is, obtained through numpy
Tensor is non-real-time. It is a component in a static graph. Only when data is fed and the operation is completed can the value of the Tensor be obtained.
Then why datastep.map(function)
The parameters passed into the parsing function function, that is, the filename and label in the above read_file(filename, label) are Tensor?
Because for a data set dataset.map, it does not perform the function operation mapped in the map for each set of samples in advance, but only tells the dataset that you need to perform a function operation every time you take out the sample before using it. , So the function call is called every time the dataset is iterated, but the pre-parameter filename and label are just a "pit". The data is used to fill this "pit" during iteration, and during the operation, although The data is filled in, but the filename and label are still a Tensor instead of an EagerTensor, so the above problem occurs.
Note: Two issues:
(1) There is no way to directly convert between Tensor and EagerTensor
(2) Tensor cannot be used directly in the python function, because I cannot get the value of Tensor in the python function
Three, the way tensorflow interacts with python code-tf.py_function
We need to define the implementation of the function ourselves. There is no way for a function written in python to interact with Tensor directly, so what should we do?
The tensorflow 2.x version provides the function tf.py_function to implement self-defined functions.
3.1 Function prototype
tf.py_function(func, inp, Tout, name=None)Role: Wrap Python functions, so that Python mothers can interact with tensorflow
parameter:
func: the name of the python function defined by yourself
inp: Define the parameter list of the python function by yourself, written in the form of a list, each element of the [tensor1,tensor2,tensor3] list is a Tensor object,
Pay attention to matching with the defined function parameters
Tout: It corresponds to the return value of a custom python function,
- When Tout is a list, such as [tf.string,tf,int64,tf.float] means that the custom function has three return values, that is, three tensors are returned, and the element type of each tensor corresponds to it
- When Tout has only one value, such as tf.int64, it means that the custom function returns an integer list or integer tensor
- When Tout has no value, it means that the custom function does not return a value
3.2 Solutions to the above problems
(1) Define your own python function
# dataset.map函数没有直接使用它，而是先用tf.py_function来包装他
def read_file(filename,label):
tf.print(type(filename)) # 包装之后类型不再是Tensor，而是EagerTensor
tf.print(type(label))
filename_ = filename.numpy() # 因为是EagerTensor，可以使用numpy获取值，在tensorflow中，字符串以byte存储，所以它的值是 b'xxxxx' 的形式
label_ = label.numpy()
new_filename = filename_.decode() # 将byte解码得到str
new_filename = "./files/" + new_filename
# 先在的new_filename就是纯python字符串了，可以直接打开了
f = open(new_filename,mode="r")
s =f.readline()
x_ =s.split(',')
result =[]
for i in x_:
result.append(int(i))
return result,label # 返回，result是一个列表list(2) Define a function to use tf.py_function to wrap the python function defined by yourself
z 注意参数的匹配以及类型的匹配
def wrap_function(x,y):
x, y = tf.py_function(read_file, inp=[x, y], Tout=[tf.int32, tf.int32])
return x,yOf course, we can also directly use lambda expressions in one step without writing wrapper functions.
If tf.py_function() is not used to wrap the read function read_file here, the two parameters of read_file are both Tensor
After using tf.py_function() to wrap the read_file function, its parameter becomes EagerTensor,
As for why this is the case, I am not very clear yet, I hope a great god will tell you!
That is as follows:
dataset = dataset.map(lambda x, y: tf.py_function(read_file, inp=[x, y], Tout=[tf.int32, tf.int32]))
(3) Preparation of the pipeline of the dataset
X=["file1.txt","file2.txt","file3.txt"]
Y=[[1,0,0],[0,1,0],[0,0,1]]
dataset = tf.data.Dataset.from_tensor_slices((X,Y)) # 第一步：构造dataset对象
dataset = dataset.map(wrap_function)
dataset=dataset.repeat(3) # 重复三次
dataset=dataset.batch(3) # 每次3个样本一个batch
for features,label in dataset:
print(features)
print(label)
print("=================================================================")The results are as follows:
tf.Tensor(
[[ 1 2 3 4 5]
[ 11 22 33 44 55]
[111 222 333 444 555]], shape=(3, 5), dtype=int32)
tf.Tensor(
[[1 0 0]
[0 1 0]
[0 0 1]], shape=(3, 3), dtype=int32)
=======================================================================================================
tf.Tensor(
[[ 1 2 3 4 5]
[ 11 22 33 44 55]
[111 222 333 444 555]], shape=(3, 5), dtype=int32)
tf.Tensor(
[[1 0 0]
[0 1 0]
[0 0 1]], shape=(3, 3), dtype=int32)
=======================================================================================================
tf.Tensor(
[[ 1 2 3 4 5]
[ 11 22 33 44 55]
[111 222 333 444 555]], shape=(3, 5), dtype=int32)
tf.Tensor(
[[1 0 0]
[0 1 0]
[0 0 1]], shape=(3, 3), dtype=int32)
=======================================================================================================It can be found that the current results are completely consistent!
3.3 Further explanation about Tensor and EagerTensor
Note: EagerTensor can directly interact with python code, or iteratively facilitate operations. It is actually Tensor that does not support direct interaction with Python. This requires attention, as shown in the following example:
(1) The interaction between EagerTensor and python function
def iterate_tensor(tensor):
tf.print(type(tensor)) # EagerTensor
(x1, x2, x3), (x4, x5, x6) = tensor
return tf.stack([x2, x4, x6])
const = tf.constant(range(6), shape=(2, 3)) # EagerTensor
o = iterate_tensor(const)
print(o)
'''运行结果为：
<class 'tensorflow.python.framework.ops.EagerTensor'>
tf.Tensor([1 3 5], shape=(3,), dtype=int32)
'''(2) Interaction between Tensor and python function
Use tf.function to decorate the function, as follows:
@tf.function
def iterate_tensor(tensor):
tf.print(type(tensor)) # Tensor
(x1, x2, x3), (x4, x5, x6) = tensor
return tf.stack([x2, x4, x6])
const = tf.constant(range(6), shape=(2, 3)) # EagerTensor
o = iterate_tensor(const)
print(o)Because tf.function is used to decorate the Python function, it will be compiled into a static graph operation. At this time, the tensor becomes Tensor, so the above code will be wrong:
OperatorNotAllowedInGraphError: iterating over `tf.Tensor` is not allowed: AutoGraph did not convert this function. Try decorating it directly with @tf.function.It can be seen that tensor has become tensor, and iterative operation is not allowed on it, and errors will occur.
Summary: Be sure to distinguish between EagerTensor and tf.Tensor
The tensor created under the dynamic graph is EagerTensor(reference method is from tensorflow.python.framework.ops import EagerTensor), and the tensor created under the static graph is tf.Tensor. EagerTensorAnd tf.Tensoralthough very similar, but not exactly the same , if dependent on EagerTensora number of unique methods will lead to static Figure tf.Tensornone of these methods error
We often don't know whether a tensor is EagerTensor or Tensor? The easiest way is to use
tf.print(type(tensor_name))To view
Fourth, supplement-about tf.py_function and tf.numpy_function
It must be admitted that TensorFlowthere are so many len(dir(tf.raw_ops))Ops ( approximately 1,227 ) that are still not enough to fully cover numpyall functions. Therefore, in some cases, when you can't find a suitable Op (or Op combination) to express the logic of the operation, you can use The above numpyfunction is also quite good, so some people may think that it is not allowed to EagerTensorconvert to numpyand then use the numpycalculation and then convert Tensor，tf.functionto it. It is better to be honest and practical tf.numpy_function. (Of course, you can write Op Kernel yourself and compile and use, and then see if there is extra time to make a custom Op summary. At present, I still need to fill in the flag of the 2.0 summary in the early years.> <)
More about
tf.py_function
tf.numpy_function
Please refer to the following example for the use of