ResNet (Residual Network) is a deep convolutional neural network architecture proposed by Kaiming He et al. in 2015. It has revolutionized the field of computer vision, making it possible to train deeper neural networks, beyond the limitations of traditional network architectures.
The main innovation of ResNet is the concept of residual learning . Traditional neural networks suffer from the problem of vanishing gradients, that is, as the gradient propagates through multiple layers, its value becomes exponentially smaller, thus hindering the learning process and limiting the effective training depth of the network.
ResNet solves this problem by introducing skip connections or shortcut connections . Instead of directly fitting the desired map, ResNet learns to fit a residual map, which is the difference between the desired output and the input. These skip connections allow gradients to flow directly between multiple layers and alleviate the problem of vanishing gradients.
The core building block of ResNet is the residual block, which consists of two convolutional layers, batch normalization and ReLU activation function. The residual block receives the input tensor, passes it through the layers, and then adds the input tensor and the output of the convolutional layer. The addition operation combines the original input with the learned residual, creating a shortcut path to the gradient flow .
residual block
The residual structure uses a shortcut connection method, which can also be understood as a shortcut. Let the feature matrices be added layer by layer. Note that the shapes of F(X) and
There are first two 3*3 convolutional layers with the same number of output channels in the residual block. Each convolutional layer is followed by a batch normalization layer and ReLU activation function. Then we skip these two convolution operations through the cross-layer data path, and add the input directly before the final ReLU activation function. Such a design requires that the outputs of the two convolutional layers have the same shape as the input, so that they can be added. If you want to change the number of channels, you need to introduce an additional 1*1 convolution layer to transform the input into the required shape before performing the addition operation.
import torch
import torch.nn as nn
# Residual Block (Basic Building Block of ResNet)
class ResidualBlock(nn.Module):
def __init__(self, in_channels, out_channels, stride=1):
super(ResidualBlock, self).__init__()
self.conv1 = nn.Conv2d(in_channels, out_channels, kernel_size=3, stride=stride, padding=1, bias=False)
self.bn1 = nn.BatchNorm2d(out_channels)
self.relu = nn.ReLU(inplace=True)
self.conv2 = nn.Conv2d(out_channels, out_channels, kernel_size=3, stride=1, padding=1, bias=False)
self.bn2 = nn.BatchNorm2d(out_channels)
self.shortcut = nn.Sequential()
if stride != 1 or in_channels != out_channels:
self.shortcut = nn.Sequential(
nn.Conv2d(in_channels, out_channels, kernel_size=1, stride=stride, bias=False),
nn.BatchNorm2d(out_channels)
)
def forward(self, x):
residual = x
out = self.conv1(x)
out = self.bn1(out)
out = self.relu(out)
out = self.conv2(out)
out = self.bn2(out)
out += self.shortcut(residual)
out = self.relu(out)
return out
# ResNet Architecture
class ResNet(nn.Module):
def __init__(self, num_classes=1000):
super(ResNet, self).__init__()
self.in_channels = 64
self.conv1 = nn.Conv2d(3, 64, kernel_size=7, stride=2, padding=3, bias=False)
self.bn1 = nn.BatchNorm2d(64)
self.relu = nn.ReLU(inplace=True)
self.maxpool = nn.MaxPool2d(kernel_size=3, stride=2, padding=1)
self.layer1 = self._make_layer(64, 3)
self.layer2 = self._make_layer(128, 4, stride=2)
self.layer3 = self._make_layer(256, 6, stride=2)
self.layer4 = self._make_layer(512, 3, stride=2)
self.avgpool = nn.AdaptiveAvgPool2d((1, 1))
self.fc = nn.Linear(512, num_classes)
def _make_layer(self, out_channels, num_blocks, stride=1):
layers = []
layers.append(ResidualBlock(self.in_channels, out_channels, stride))
self.in_channels = out_channels
for _ in range(1, num_blocks):
layers.append(ResidualBlock(out_channels, out_channels))
return nn.Sequential(*layers)
def forward(self, x):
out = self.conv1(x)
out = self.bn1(out)
out = self.relu(out)
out = self.maxpool(out)
out = self.layer1(out)
out = self.layer2(out)
out = self.layer3(out)
out = self.layer4(out)
out = self.avgpool(out)
out = torch.flatten(out, 1)
out = self.fc(out)
return out
# Create an instance of the ResNet model
model = ResNet(num_classes=1000)
# Print the model architecture
print(model) ResNet usually consists of multiple stacked residual blocks with gradually increasing depth. The network architecture includes different variants such as ResNet-18, ResNet-34, ResNet-50, ResNet-101, and ResNet-152, where the number indicates the total number of layers in the network. Darker variants show better performance in various computer vision tasks such as image classification, object detection and segmentation.
A significant advantage of ResNet is that very deep networks can be trained without degrading performance. It makes it possible to train networks with over 100 layers while still maintaining accuracy and convergence. Furthermore, skip connections enable easy implementation of identity mapping, meaning that shallow networks can be transformed into deeper networks without degrading performance.
ResNet has had a significant impact on the field of deep learning and has become a widely adopted architecture in various computer vision applications. Its residual learning concept has also inspired the development of other architectures using skip connections, such as DenseNet and Highway Networks.
