2023 Certification Cup Little Beauty Competition (Topic B): Industrial Surface Defect Detection | Modeling Analysis, Senior Lu Lu leads the team to guide the entire article code ideas

I am Senior Lulu, studying at Shanghai Jiao Tong University. So far, I have helped 200+ people complete modeling and idea building~
Let’s take a look. Question B of the Certification Cup!

The complete content can be obtained at the end of the article!

Problem restatement

In the field of industrial surface defect detection, the challenge we face is to automatically detect defects on the surface of metal or plastic products through deep learning models. We hope to use the defective production item image data set provided by Kolektor Group to explore the construction of a mathematical model suitable for surface defect detection. Unlike previous studies, our focus is on building a model that can be deployed on inexpensive handheld devices with limited storage and computing resources. At the same time, we hope that the model can still generalize relatively well in the face of other defect types not covered in the dataset. Specific tasks include:

1. Determine whether surface defects are present in a photo and measure the amount of computation and storage required by the model to do so.
2. Automatically mark locations or areas where surface defects occur and measure the computational effort, storage space and marking accuracy required for the model.
3. Elucidate the generalization ability of the model, i.e., the reason why the model still works when encountering defect types that are not present in the data set.

Question one

1. Data preparation:

• Dataset: Use the defective production item image data set provided by Kolektor Group. Ensure that the dataset contains both normal and defective samples and are appropriately annotated.

• Data preprocessing: Preprocess the image, including operations such as scaling, normalization, and data enhancement, to improve the robustness of the model. Make sure the data set is divided into training and test sets.

2. Model selection:

• Choose a lightweight model: Use a lightweight deep learning model suitable for mobile devices, such as MobileNetV2.

• Model fine-tuning: Use a pre-trained lightweight model and fine-tune it on the surface defect detection task to improve performance.

3. Resource optimization:

• Model compression: Use model compression technology, such as model pruning, quantization, etc., to reduce the calculation amount and storage space requirements of the model.

• Lightweight model: MobileNetV2 is a lightweight model, but you can still optimize performance by further reducing model complexity.

4. Model training:

• Define the goal: Define the task as a binary classification problem, that is, determine whether there are surface defects in the image.

• Loss function: Use an appropriate loss function, such as the binary cross-entropy loss function.

• Optimization Algorithm: Use a suitable optimization algorithm, such as Stochastic Gradient Descent (SGD) or Adam, for model training.

5. Model evaluation:

• Test set evaluation: Use the test set to evaluate the model's performance on the surface defect detection task, including accuracy, recall, and precision.

6. Calculation and storage space measurement:

• Model parameter statistics: Count the number of parameters of the model to evaluate storage space requirements.

• Estimation of computational effort: Use model analysis tools such as FLOPs (floating point operations) to estimate the computational effort of the model.

7. Result analysis and adjustment:

• Result analysis: Analyze the performance of the model on the test set and check whether it meets the handheld device resource constraints.

• Adjust the model: Adjust the model structure or parameters based on the analysis results to meet resource requirements.

8. Deploy to handheld devices:

• Convert model format: Convert the trained model to a format suitable for handheld devices, such as ONNX or TensorFlow Lite.

• Deployment verification: Deploy the model on a handheld device and verify the model's performance on actual devices.

import torch
import torch.nn as nn
import torch.optim as optim
from torchvision import transforms
from torchvision.models import mobilenet_v2
from torch.utils.data import DataLoader, Dataset

# 定义轻量级模型
class SurfaceDefectModel(nn.Module):
    def __init__(self, num_classes=2):
        super(SurfaceDefectModel, self).__init__()
        self.mobilenet = mobilenet_v2(pretrained=True)
        self.mobilenet.classifier[1] = nn.Linear(1280, num_classes)

    def forward(self, x):
        return self.mobilenet(x)

# 定义数据集和转换
class SurfaceDefectDataset(Dataset):
    # 请根据你的数据集格式进行定义
    # 需要包含图像和对应的标签信息

# 数据预处理和加载器
transform = transforms.Compose([
    transforms.Resize((224, 224)),
    transforms.ToTensor(),
])

# 准备数据
# Replace 'train_data', 'train_labels', 'test_data', 'test_labels' with your actual data
train_dataset = SurfaceDefectDataset(train_data, train_labels, transform=transform)
test_dataset = SurfaceDefectDataset(test_data, test_labels, transform=transform)

train_loader = DataLoader(train_dataset, batch_size=32, shuffle=True)
test_loader = DataLoader(test_dataset, batch_size=32, shuffle=False)

# 初始化轻量级模型
model = SurfaceDefectModel()

# 定义损失函数和优化器
criterion = nn.CrossEntropyLoss()
optimizer = optim.Adam(model.parameters(), lr=0.001)

Question 2

Task two is to automatically mark the locations or areas where surface defects occur, and measure the amount of computation, storage space and marking accuracy required by the model. The following is a general modeling idea to solve this problem:

1. Data preparation:

• Annotation data: Ensure that the training data set contains images of surface defects, and annotate each image with the location information of the defect, usually in the form of a bounding box.

• Data preprocessing: Preprocess the image, including operations such as scaling, normalization, and data enhancement, to improve the robustness of the model. Make sure the data set is divided into training and test sets.

2. Model selection:

• Select a target detection model: Use a deep learning model suitable for target detection tasks, such as Faster R-CNN, YOLO, SSD, etc.

• Model fine-tuning: Use a pre-trained object detection model and fine-tune it on the surface defect detection task to improve performance.

3. Resource optimization:

• Model compression: Use model compression technology, such as model pruning, quantization, etc., to reduce the calculation amount and storage space requirements of the model.

• Lightweight model: Consider using a lightweight object detection model to accommodate the resource constraints of handheld devices.

4. Loss function and evaluation index:

• Loss function: Use an appropriate loss function for object detection, such as classification loss and bounding box regression loss in Faster R-CNN.

• Evaluation indicators: Use the evaluation indicators of the target detection task, such as average precision (mAP), etc., to evaluate the performance of the model on the test set.

5. Calculation and storage space measurement:

• Model parameter statistics: Count the number of parameters of the model to evaluate storage space requirements.

• Estimation of computational effort: Use model analysis tools such as FLOPs (floating point operations) to estimate the computational effort of the model.

6. Model training:

• Define the goal: The goal is to detect surface defects in the image and mark the location of the defect.

• Loss function: Use an appropriate loss function that takes into account both classification and location information.

• Optimization algorithm: Use an appropriate optimization algorithm for model training.

7. Model evaluation:

• Test set evaluation: Use the test set to evaluate the model's performance on the surface defect detection task, including accuracy, recall, and precision.

8. Result analysis and adjustment:

• Result analysis: Analyze the performance of the model on the test set and check whether it meets the handheld device resource constraints.

• Adjust the model: Adjust the model structure or parameters based on the analysis results to meet resource requirements.

9. Interpretability:

• Interpretability considerations: Consider the interpretability of the model output to ensure that the marking of defect locations is understandable to the operator.

10. Deploy to handheld devices:

• Convert model format: Convert the trained model to a format suitable for handheld devices, such as ONNX or TensorFlow Lite.

• Deployment verification: Deploy the model on a handheld device and verify the model's performance on actual devices.

import torch
import torch.nn as nn
import torch.optim as optim
from torchvision import transforms
from torchvision.models.detection import fasterrcnn_resnet50_fpn
from torch.utils.data import DataLoader, Dataset
from torchvision.transforms.functional import to_tensor

# 定义 Faster R-CNN 模型
class SurfaceDefectModel(nn.Module):
    def __init__(self, num_classes=2):
        super(SurfaceDefectModel, self).__init__()
        self.model = fasterrcnn_resnet50_fpn(pretrained=True)
        # 修改输出层，适应你的类别数
        in_features = self.model.roi_heads.box_predictor.cls_score.in_features
        self.model.roi_heads.box_predictor = nn.Linear(in_features, num_classes)

    def forward(self, images, targets=None):
        if self.training and targets is None:
            raise ValueError("In training mode, targets should be passed")

        return self.model(images, targets)

# 定义数据集和转换
class SurfaceDefectDataset(Dataset):
    # 请根据你的数据集格式进行定义
    # 需要包含图像和对应的标签信息

    def __getitem__(self, index):
        # 返回图像和对应的标签
        # 使用 torchvision.transforms.ToTensor() 将图像转换为张量

    def __len__(self):
        # 返回数据集的大小

# 数据预处理和加载器
transform = transforms.Compose([
    transforms.ToTensor(),
])

# 准备数据
# Replace 'train_data', 'train_labels', 'test_data', 'test_labels' with your actual data
train_dataset = SurfaceDefectDataset(train_data, train_labels, transform=transform)
test_dataset = SurfaceDefectDataset(test_data, test_labels, transform=transform)

train_loader = DataLoader(train_dataset, batch_size=2, shuffle=True)
test_loader = DataLoader(test_dataset, batch_size=2, shuffle=False)

# 初始化 Faster R-CNN 模型
model = SurfaceDefectModel()

# 定义损失函数和优化器
criterion = nn.CrossEntropyLoss()
optimizer = optim.Adam(model.parameters(), lr=0.001)

# 训练模型
num_epochs = 10
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
model.to(device)

for epoch in range(num_epochs):
    model.train()
    for images, targets in train_loader:
        images = list(image.to(device) for image in images)
        targets = [{
    
    k: v.to(device) for k, v in t.items()} for t in targets]

        optimizer.zero_grad()
        loss_dict = model(images, targets)
        loss = sum(loss for loss in loss_dict.values())
        loss.backward()
        optimizer.step()

Question three

Task three is about the generalization ability of the model, that is, the reason why the model is still feasible when encountering defect types that are not present in the data set. The following is a general modeling idea to solve this problem:

1. Data set analysis:

• Understand the dataset: Conduct a detailed analysis of the provided defective production item image dataset to understand the defect types and sample distributions present in the dataset.

• Model training set selection: Ensure that the model’s training set contains as many defect types as possible to improve the model’s generalization ability to different defects.

2. Data expansion:

• Data augmentation: Use data augmentation techniques, such as rotation, flipping, scaling, etc., to generate more diverse training data to help the model better generalize to future Situations I’ve seen.

3. Transfer learning:

• Use a pre-trained model: Using a model pre-trained on large-scale image data can enable the model to learn common features and improve its generalization ability to new categories.

• Fine-tuning strategy: Use a smaller learning rate during training to ensure that previously learned common features are not quickly lost when fine-tuning on new data.

4. Reinforcement learning:

• Online learning: Use an online learning strategy that allows the model to continue learning and adapting to new defect types after deployment.

5. Model evaluation:

• Cross-validation: Use techniques such as cross-validation to evaluate the performance of the model on different subsets of data to ensure the robustness of generalization capabilities.

6. Uncertainty modeling:

• Uncertainty estimates: Introduce uncertainty estimates into models, such as Monte Carlo inference, to better handle unseen data.

7. Multi-task learning:

• Multi-task learning: Consider using multi-task learning to let the model learn multiple related tasks at the same time to improve its adaptability to new tasks.

8. Active learning:

• Active learning strategy: Use an active learning strategy to select and label samples that are more challenging for the model to optimize the model's performance on future data.

9. Anomaly detection:

• Anomaly detection: Introduce anomaly detection technology to help the model discover anomalies when encountering unknown defect types.

10. Model monitoring:

• Real-time monitoring: After deployment, the performance of the model is monitored in real time and regularly updated and maintained to adapt to new defect types.

import torch
import torch.nn as nn
import torch.optim as optim
from torchvision import transforms
from torch.utils.data import DataLoader, Dataset
from torchvision.models import resnet18

# 定义轻量级模型
class SurfaceDefectModel(nn.Module):
    def __init__(self, num_classes=2):
        super(SurfaceDefectModel, self).__init__()
        self.resnet = resnet18(pretrained=True)
        # 修改输出层，适应你的类别数
        in_features = self.resnet.fc.in_features
        self.resnet.fc = nn.Linear(in_features, num_classes)

    def forward(self, x):
        return self.resnet(x)

# 定义数据集和转换
class SurfaceDefectDataset(Dataset):
    # 请根据你的数据集格式进行定义
    # 需要包含图像和对应的标签信息

# 数据预处理和加载器
transform = transforms.Compose([
    transforms.Resize((224, 224)),
    transforms.ToTensor(),
])

# 准备数据
# Replace 'train_data', 'train_labels', 'test_data', 'test_labels' with your actual data
train_dataset = SurfaceDefectDataset(train_data, train_labels, transform=transform)
test_dataset = SurfaceDefectDataset(test_data, test_labels, transform=transform)

train_loader = DataLoader(train_dataset, batch_size=32, shuffle=True)
test_loader = DataLoader(test_dataset, batch_size=32, shuffle=False)

# 初始化轻量级模型
model = SurfaceDefectModel()

# 定义损失函数和优化器
criterion = nn.CrossEntropyLoss()
optimizer = optim.Adam(model.parameters(), lr=0.001)

# 训练模型
num_epochs = 10
for epoch in range(num_epochs):
    model.train()
    for inputs, labels in train_loader:
        optimizer.zero_grad()
        outputs = model(inputs)
        loss = criterion(outputs, labels)
        loss.backward()
        optimizer.step()

    # 在测试集上评估模型
    model.eval()
    with torch.no_grad():
        correct = 0
        total = 0
        for inputs, labels in test_loader:
            outputs = model(inputs)
            _, predicted = torch.max(outputs.data, 1)
            #见完整版

For more information, you can click on the business card below to learn more, and let Senior Xiaolu lead you on the road to winning the Certification Cup! Stay tuned to see what our efforts will bring! Remember to follow Senior Lulu!