In the context of transformers, pooling refers to the process of summarizing the output of a transformer layer into a fixed-size vector, typically used for downstream tasks such as classification.
In the Transformer architecture, the input sequence is processed by a series of self-attention layers and feed-forward layers. Each layer produces a sequence of output vectors that encode the input sequence in a higher-level representation. Pooling involves taking output vectors from one or more of these layers and aggregating them into one vector.
Different types of pooling mechanisms are used in the Transformer architecture, including:
-
Max Pooling: Select the maximum value in the sequence of output vectors as the summary representation.
-
Mean Pooling: where the mean of the output vector is represented as a summary.
-
Last Hidden State: where the final output vector of the transformer is used as the summary representation.
-
Self-Attention Pooling: Computes a weighted sum of output vectors, with weights determined by the learned attention mechanism.
Overall, pooling is an important part of the transformer architecture, as it allows to extract a fixed-size representation of the input sequence, which can be used for various downstream tasks.
English link
Read the original English text
AI Good Book Recommendation
AI is changing with each passing day, but a high-rise building cannot be separated from a good foundation. Are you interested in learning about the principles and practice of artificial intelligence? Look no further! Our book on AI principles and practices is the perfect resource for anyone looking to gain insight into the world of AI. Written by leading experts in the field, this comprehensive guide covers everything from the basics of machine learning to advanced techniques for building intelligent systems. Whether you are a beginner or an experienced AI practitioner, this book has you covered. So why wait?