2023 APMCM Asia-Pacific Competition (Question A) Image Recognition of Fruit Picking Robots | Asia-Pacific College Student Mathematical Modeling Competition Modeling Analysis, Senior Xiaolu leads the team to guide full code articles and ideas

I am senior Xiaolu, studying at Shanghai Jiao Tong University. So far, I have helped more than 200 people complete modeling and idea building~
Let’s take a look at the Asia-Pacific Question A of the competition!

The complete content can be obtained at the end of the article!

Problem restatement

Question A: Image recognition of fruit picking robots

This competition aims to establish an apple image recognition model with high recognition rate, fast speed and accuracy by analyzing and extracting marked fruit image features, and perform data analysis on the image, such as automatically calculating the number and location of apples in the image. , maturity level, and estimate quality. The specific tasks are as follows:

Question 1: Counting apples
Based on the provided harvest-ready apple image data set (see attachment 1), extract image features, establish a mathematical model, and calculate the number of apples in each image , and draw the histogram of the distribution of all apples in Appendix 1.

Question 2: Estimating the position of the apples
Based on the provided harvest-ready apple image dataset (see attachment 1), determine the position of the apple in each image, to the lower left corner of the image as the coordinate origin, and draw a two-dimensional scatter plot of all apple geometric coordinates in Appendix 1.

Question 3: Estimating the ripeness status of apples
Based on the provided harvest-ready apple image data set (see attachment 1), establish a mathematical model to calculate the ripeness of the apples in each image. , and draw a histogram of the ripeness distribution of all apples in Appendix 1.

Question 4: Estimating the quality of apples
Based on the provided harvest-ready apple image data set (see attachment 1), calculate the two-dimensional area of ​​the apples in each image, in terms of the image's The lower left corner is the origin of the coordinates. Estimate the mass of the apples and draw a histogram of the mass distribution of all apples in Appendix 1.

Question 5: Apple recognition
Based on the provided harvested fruit image data set (see attachment 2), extract image features, train an apple recognition model, and identify the apples in attachment 3 , and draw the distribution histogram of all Apple image ID numbers in Appendix 3.

Question one

First, we need to perform image preprocessing. Considering that the apples in the image may have different colors and brightness, we can simplify the process by converting the image to grayscale. Next, a threshold segmentation technique, such as Otsu’s method, can be used to binarize the grayscale image in order to separate the apples from the background.

Suppose we get the binary image$B$, inside $B_{i,j}=1$ means that a pixel in the image is judged to be an apple, $B_{i,j}=0$ means not an apple.

Next, we can use connected region analysis to find the outline of the apple. This can be achieved by labeling each connected component. Suppose we get the labeled image $L$, inside $L_{i,j}$A label that represents the connected component to which a pixel in the image belongs.

Now, we have a connected component set containing apples S = { S 1 , S 2 , . . . , S k } S = \{S_1, S_2, ..., S_k\} S={ S1​,S2​,...,Sk​}, inside $k$This is a quantitative quantity. Each$S_i$Contains the pixels of an apple in the image.

Finally, we can calculate the number of apples, that is, the number of connected components $|S|$。

The entire process can be summarized into the following steps:

1. Image preprocessing:

   • Convert original image to grayscale image.
   • Use threshold segmentation technology to generate binary images $B$。

2. Connected region analysis:

   • According to the binary image $B$ Perform connected region analysis and obtain labeled image$L$。

3. Count:

   • Calculate the number of connected components $|S|$, immediate fruit quantity.

Specific implementation can use functions provided in image processing libraries (such as OpenCV), such as grayscale conversion, threshold segmentation, connected area marking, etc. Here is the mathematical representation of the relevant steps:

1. Image preprocessing:
   $B_{i,j}=\{\begin{array}{ll}1,& \text{Young image raw score}(i,j)\text{judged apple}\\ 0,& \text{Other}\end{array}$

2. Connection area analysis:
   $L_{i,j}=\text{Label}(B_{i,j})$

3. 计数：
   $\text{Count}=|S|$

这り的 $\text{The Label}$ function labels each pixel with its connected component label.

import cv2
import numpy as np

# 读取图像
image = cv2.imread('apple_image.jpg')

# 将图像转换为灰度图
gray_image = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY)

# 使用阈值分割生成二值化图像
_, binary_image = cv2.threshold(gray_image, 150, 255, cv2.THRESH_BINARY)

# 连通区域分析
_, labels, stats, centroids = cv2.connectedComponentsWithStats(binary_image)

# 统计苹果数量
apple_count = len(stats) - 1  # 减去背景标签

# 输出结果
print(f"苹果数量：{
      
      apple_count}")

# 可视化结果（可选）
for i in range(1, len(stats)):
    x, y, w, h = stats[i]
   #关注文末名片继续获取代码

Question 2

Question 2: Specific modeling ideas for estimating Apple’s position:

In question 1, we have obtained the set of connected components of Apple S = { S 1 , S 2 , . . . , S k } S = \{S_1, S_2, ..., S_k\} S={ S1​,S2​,...,Sk​}, inside$k$ is a quantitative quantity. Each $S_i$Contains the pixels of an apple in the image.

To estimate the position of the apple, we can calculate each connected component $S_i$the geometric center. The geometric center is the average position of all pixels and can be used to represent the approximate location of the apple in the image.

设 $C_i=(x_i,{and}_i)$ Correct $S_i$The geometric center of can be calculated by the following formula:

$x_i=\frac{1}{|S_i|}{\sum }_{(p,q)\in S_i}p$

${and}_i=\frac{1}{|S_i|}{\sum }_{(p,q)\in S_i}q$

in that,$(p,q)$表示 $S_i$A pixel coordinate in , $|S_i|$ Corre$S_i$The number of pixels in it.

In this way, we have for each $S_i$ have obtained a geometric center $C_i$, indicating the position of the corresponding apple. We can represent these geometric center coordinates in the form of a two-dimensional scatter plot, where the lower left corner of the image is the coordinate origin.

The entire process can be summarized into the following steps:

1. Transfer quantity $S_i$, calculate geometric center $C_i$。
2. Express all geometric center coordinates in the form of a two-dimensional scatter plot.

In code, we can use mathematical formulas of the form:

$C_i=\left(\frac{1}{|S_i|}{\sum }_{(p,q)\in S_i}p,\frac{1}{|S_i|}{\sum }_{(p,q)\in S_i}q\right)$

import cv2
import numpy as np
import matplotlib.pyplot as plt

def preprocess_image(image_path):
    # 读取图像
    image = cv2.imread(image_path)

    # 将图像转换为灰度图
    gray_image = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY)

    # 使用高斯模糊以减少噪音
    blurred_image = cv2.GaussianBlur(gray_image, (5, 5), 0)

    # 使用自适应阈值分割生成二值化图像
    _, binary_image = cv2.threshold(blurred_image, 0, 255, cv2.THRESH_BINARY + cv2.THRESH_OTSU)

    return binary_image

def estimate_apple_positions(binary_image):
    # 连通区域分析
    _, labels, stats, centroids = cv2.connectedComponentsWithStats(binary_image)

    # 统计苹果数量
    apple_count = len(stats) - 1  # 减去背景标签

    # 获取几何中心坐标
    centers = centroids[1:]  # 去除背景的几何中心

    return apple_count, centers

def visualize_apple_positions(image, centers):
    # 绘制二维散点图
    plt.figure(figsize=(8, 8))
    plt.imshow(cv2.cvtColor(image, cv2.COLOR_BGR2RGB))  # 显示原始图像

    for center in centers:
        plt.scatter(center[0], center[1], c='red', marker='o')

    plt.title('Estimated Apple Positions')
    #关注文末名片继续获取代码

Question three

Question 3: Specific modeling ideas for estimating the ripeness of apples:

To estimate the ripeness state of an apple, we can consider using the color information of the image. Ripe apples usually have a specific color, such as red or yellow, while unripe apples may have a greener hue. In practical applications, we can use color space conversion and color thresholding to extract the regions of ripe apples in the image.

Specific steps are as follows:

1. Color space conversion: Convert the image from BGR (Blue-Green-Red) color space to HSV (Hue-Saturation-Value) color space. HSV color space is more suitable for processing color information.

2. Color threshold: By setting the threshold, the ripe apple area in the image is separated. This can be achieved by selecting specific hue and saturation ranges in the HSV image.

3. Connected region analysis: For the extracted binary image, perform connected region analysis to find the connected components of the ripe apple.

4. Estimate ripeness: By analyzing the number and distribution of ripe apples, the overall ripeness can be estimated. For example, an image with a higher number of ripe apples might indicate higher overall ripeness.

Mathematically, this can be expressed as:

$Maturity=\frac{MatureApples}{TotalApples}\times 100$

其中，$MatureApples$ 表示成熟苹果的数量，$TotalApples$ represents the total number of apples. Overall ripeness can also be obtained by calculating the ripeness of each apple and averaging it.

The entire process can be summarized into the following steps:

1. Color space conversion: Convert BGR image to HSV image.
2. Color threshold: Set the threshold to extract the binary image of ripe apples.
3. Connected region analysis: Perform connected region analysis to find the connected components of ripe apples.
4. Estimate ripeness: Calculates the number of ripe apples and their overall ripeness.

Here is a simplified Python code example that demonstrates the above idea:

import cv2
import numpy as np

def estimate_maturity(image_path):
    # 读取图像
    image = cv2.imread(image_path)

    # 将图像转换为HSV颜色空间
    hsv_image = cv2.cvtColor(image, cv2.COLOR_BGR2HSV)

    # 定义颜色阈值范围，这里以红色为例
    lower_red = np.array([0, 100, 100])
    upper_red = np.array([10, 255, 255])

    # 创建二值图像，根据颜色阈值提取成熟苹果区域
    mask = cv2.inRange(hsv_image, lower_red, upper_red)

    # 连通区域分析
    _, labels, stats, _ = cv2.connectedComponentsWithStats(mask)

    # 统计成熟苹果数量
    mature_apples = len(stats) - 1  # 减去背景标签

    # 统计总苹果数量
    total_apples = len(stats) - 1  # 减去背景标签

    # 计算整体成熟度
    maturity = (mature_apples / total_apples) * 100

Question 4

Question 4: Specific modeling ideas for estimating the quality of apples:

To estimate the mass of an apple, we can obtain the two-dimensional area of ​​each apple from the image, assuming that the density of the apples is similar, and then map the area to the mass through a certain relationship. Specific steps are as follows:

1. Calculate the two-dimensional area of ​​an apple: Use image processing technology to calculate the area of ​​each apple. This can be done through connected area analysis to get the number of pixels in the apple, and then convert it to the actual area.

2. Establish a model of the relationship between mass and area: Establish a model of the relationship between mass and area through experiments or other means. This can be a simple linear model or a more complex model, depending on the characteristics of the data.

3. Apply relational model to estimate quality: Use the established relational model to map the area of ​​each apple to the mass to obtain the estimated apple quality.

4. Draw a mass distribution histogram: Combine the estimated apple masses into a histogram to understand the distribution of apple mass.

Mathematically, this can be expressed as:

${\text{quality}}_i=f({\text{majesty}}_i)$

in that,$i$ Display number $i$ 个苹 Fruit，$f$ is the established relationship model.

Here is a simplified Python code example that demonstrates the above idea:

import cv2
import numpy as np
import matplotlib.pyplot as plt

def estimate_apple_mass(image_path):
    # 读取图像
    image = cv2.imread(image_path)

    # 将图像转换为灰度图
    gray_image = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY)

    # 使用阈值分割生成二值化图像
    _, binary_image = cv2.threshold(gray_image, 150, 255, cv2.THRESH_BINARY)

    # 连通区域分析
    _, labels, stats, _ = cv2.connectedComponentsWithStats(binary_image)

    # 建立质量和面积的关系模型（简化为线性关系，实际应用中可能需要更复杂的模型）
    def mass_area_relationship(area):
        # 假设质量和面积成线性关系，具体关系需要根据实际情况调整
        return 0.01 * area

   #关注文末名片继续获取代码

In this example,mass_area_relationship the function represents the relationship model between mass and area.

Question 5

Question 5: Specific modeling ideas for identifying apples:

To solve problem 5, which is to identify apples in images, we can use image classification methods. The following is a simplified modeling idea:

1. Dataset preparation: First, a labeled dataset containing images of different categories (apples and other fruits or objects) is required. Labeling means that for each image, we know which category it belongs to.

2. Feature Extraction: Use image processing and computer vision techniques to extract features from each image. This can include features such as color, texture, shape, etc. A common practice is to use convolutional neural networks (CNN) for end-to-end feature extraction.

3. Model training: Select an appropriate machine learning model, such as a convolutional neural network (CNN) in deep learning, and train the model using the prepared data set. During training, the model learns how to distinguish different categories from the extracted features.

4. Model evaluation: Use a separate test data set to evaluate the model's performance. This can be done by calculating metrics such as accuracy, precision, recall, etc.

5. Model application: When the model performs well on the test data, it can be applied to new images to identify apples in images.

In practical applications, due to the complexity of deep learning models, already trained models (for example, pre-trained models on ImageNet) are usually used for fine-tuning instead of training a new model from scratch.

The following is a simplified Python code example that demonstrates this idea using the deep learning framework PyTorch:

import torch
import torchvision.transforms as transforms
from torchvision.models import resnet50
from PIL import Image

# 定义数据预处理和转换
preprocess = transforms.Compose([
    transforms.Resize((224, 224)),
    transforms.ToTensor(),
    transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225]),
])

# 加载预训练的ResNet-50模型
model = resnet50(pretrained=True)
model.eval()

def classify_apple(image_path):
    # 读取图像
    image = Image.open(image_path)

    # 数据预处理
    input_tensor = preprocess(image)
    input_batch = input_tensor.unsqueeze(0)

    # 将输入传递给模型
    with torch.no_grad():
        output = model(input_batch)

    # 使用Softmax函数获取类别概率
    probabilities = torch.nn.functional.softmax(output[0], dim=0)

    # 获取最可能的类别索引
    apple_index = 924  # 在ImageNet中，苹果的类别索引为924
    apple_probability = probabilities[apple_index].item()

    return apple_probability

if __name__ == "__main__":
    image_path = 'apple_image.jpg'

The above code uses the ResNet-50 model pretrained on the ImageNet dataset, which is available in PyTorch.

For more information, you can click on the business card below to learn more, and let Senior Xiaolu lead you on the road to winning the Asia-Pacific Championship! Stay tuned to see what our efforts will bring! Remember to follow Senior Lulu Yeah