In 2017, the authors of Google published a paper called "Attention is All You Need" in which they introduced the Transformer architecture. This new architecture achieved unrivaled success in language translation tasks, and the paper quickly became must-read for anyone in the field. Like many others, when I first read this paper, I could see the value of its innovative ideas, but I didn't realize how disruptive this paper would be to the rest of the broader field of AI. Within a few years, researchers were applying the Transformer architecture to many tasks beyond language translation, including image classification, image generation, and protein folding problems. In particular, the Transformer architecture revolutionized text generation and paved the way for GPT models and the exponential growth we are currently experiencing in AI.
Given the popularity of Transformer models in industry and academia today, understanding the details of how they work is an important skill for every AI practitioner. This article will focus primarily on the architecture of GPT models built using a subset of the original Transformer architecture, but will also introduce the original Transformer at the end. For the model code, I'll start with the cleanest written implementation I've found for the original Transformer: Annotated Transformer from Harvard University. I'll keep the parts related to the GPT transformer and delete the irrelevant parts. During this process, I will avoid making any unnecessary changes to the code so that you can easily compare the GPT-like version of the code with the original code and understand the differences.
This article is intended for experienced data scientists and machine learning engineers. In particular, I'm assuming you're fluent in tensor algebra, you've implemented a neural network from scratch, and you're familiar with Python. Also, while I have tried to make this article self-contained, it will be easier to understand if you read my previous article on how the GPT model works.
The code in this article can be found in the related project on GitHub.
https://github.com/bstollnitz/gpt-transform