2022, which just ended, will be an incredible year for many advances in artificial intelligence. Most of the recent famous landmarks in AI have been driven by a class-specific model called a Transformer, whether it’s the incredible progress of chatGPT, which has taken the world by storm, or the steady proliferation, which brings your smartphone Here comes the science-fiction-like feature. Even Tesla's self-driving software stack, perhaps the most widely deployed deep learning system in the world, uses transformer models (pun intended) under the hood. The " neural attention mechanism " is the secret sauce that makes Transformer so successful on various tasks and datasets.
This is the first in a series of articles about the Vision Translator (ViT). In this article, we will learn about the attention mechanism and review the evolution of ideas that led to it. Next, let's take a look at it intuitively. We will implement attention mechanisms in the PyTorch framework from scratch, combining intuitive understanding with mathematical details, and finally translating this understanding into code. Although we will discuss visual transformers specifically at the end of the article, much of the discussion applies equally to large language models (LLMs) such as GPT-3 and the recently released chatG