Table of contents
Autoencoder is a deep learning algorithm whose main goal is to learn to encode input data into a low-dimensional representation and then use this representation to reconstruct the original data. The key idea in this algorithm is that a good encoding should be able to retain as much of the original data information as possible through this encoding and decoding process. Its basic structure is as follows:
Data feature extraction and classification based on autoencoders mainly utilize the learning ability and feature extraction capabilities of autoencoders. Its principles and mathematical formulas will be introduced below.
1. Autoencoder principle
Autoencoders consist of two parts: encoder and decoder. The encoder compresses the input data into a low-dimensional representation, and the decoder recovers the original data from this low-dimensional representation. The learning process of the autoencoder is achieved by maximizing the minimum value of the reconstruction error, that is, minimizing the difference between the original data and the reconstructed data.
Feature extraction based on autoencoders mainly maps input data to a low-dimensional space through the encoder. This low-dimensional space can better represent the characteristics of the data. The decoder then uses this feature representation to reconstruct the data to retain more of the original data information. The classification task requires classifying the output of the encoder, which can be accomplished by training an additional classifier, such as a softmax classifier. In this case, the autoencoder can be regarded as a feature extractor, used to extract meaningful features from the raw data, which are then used to train the classifier.
Suppose our input data is X, the encoder function is f, the decoder function is g, and the reconstruction error is reconstruction_error, then the loss function of the autoencoder can be defined as:
error = E[||X - g(f(X))||_2^2]
Where E[||X - g(f(X))||_2^2] represents the expected value of the above calculation for all input data.
In order to optimize the above loss function, we can use gradient descent method or variational inference for optimization. Among them, the autoencoder based on the gradient descent method can be expressed as:
f, g = argmin_f, g E[||X - g(f(X))||_2^2]
An autoencoder optimized using variational inference can be expressed as:
f, g = argmin_f, g E[logP(X|f(X))]
where P(X|f(X)) represents the probability of reconstructing the original data given the encoded representation f(X).
In autoencoder-based classification tasks, we usually train an additional classifier, such as a softmax classifier, to classify the output of the encoder. In this case, our loss function can be defined as:
classification_error = E[C(f(X), Y)] + E[||X - g(f(X))||_2^2]
where C(f(X), Y) represents the classification error rate based on the encoded representation f(X) and the true label Y. In this way, we combine the reconstruction error and classification error rate of the autoencoder, which can be optimized through gradient descent or variational inference.
Data feature extraction and classification based on autoencoders is a very effective deep learning method, which can be widely used in various tasks, such as data dimensionality reduction, anomaly detection, data visualization, speech recognition, etc. At the same time, autoencoders can also be used in combination with other algorithms, such as convolutional neural networks, recurrent neural networks, etc., to achieve more complex functions.
2. Application of Autoencoder
Autoencoders have a wide range of applications in various fields, including:
Dimensionality reduction
Autoencoders can reduce the dimensionality of input data by learning compact and efficient representations in the latent space. This is useful for visualization, data compression, and accelerating other machine learning algorithms.
Feature learning
Autoencoders can learn meaningful features from input data that can be used in downstream machine learning tasks such as classification, clustering, or regression.
abnormal detection
By training an autoencoder on common data instances, it can learn to reconstruct those instances with low error. When presented with outlier data points, autoencoders may have higher reconstruction errors and can be used to identify outliers or outliers.
Image noise reduction
Autoencoders can be trained to reconstruct clean input data from noisy versions. Denoising autoencoders learn to remove noise and produce a clean version of the input data.
3. MATLAB program
.....................................................................
%在此处定义时期的数量
for epoch= 1:1:1500
epoch
J_epoch=[];Total_Activation = zeros(HN,1);Activated_Outputs_Outputlayer = cell(4000,1);
der_Activated_Outputs_Outputlayer = cell(4000,1);
Activated_Outputs_Hiddenlayer = cell(4000,1);
der_Activated_Outputs_Hiddenlayer = cell(4000,1);
delta_Outputlayer = cell(4000,1);
delta_Hiddenlayer = cell(4000,1);
%4000个输入进入隐藏层
%所有输入的平均激活和误差的计算
for inp=1:1:4000
Hjk = train_set(inp,1:784)';
for i=1:1:HL+1
s=Weight_Matrices{i}*[1;Hjk];%此处[1列车组]表示输入数量，外部1表示偏差
Hjk = arrayfun(Yj,s);
Activated_Outputs{i}=Hjk;
der_Activated_Outputs{i} = (1-Hjk).*Hjk;
end
..................................................................................
end
Average_Activation_Calculated = (Total_Activation)./4000;
%Kullback-leibler散度的术语定义如下
A1 = (Average_Activation_req)./(Average_Activation_Calculated);
B1 = (1-Average_Activation_req)./(1-Average_Activation_Calculated);
%这些值在一个时期内保持不变，并在每个新时期发生变化
%计算误差。必须添加稀疏性和权重衰减项
if rem(epoch,10)==0 || epoch ==1
J_mat(end+1)= sum(J_epoch);
end
%使用适当的公式添加错误项。这里有测试版术语，而
%加法以及所有权重参数都必须被平方并相加。添加
accum_del_Weights_Outputlayer = zeros(784,151);
del_Weights_Outputlayer = zeros(784,151);
for inp=1:1:4000
%计算增量值
delta_Outputs = [train_set(inp,1:784)'-Activated_Outputs_Outputlayer{inp}].*der_Activated_Outputs_Outputlayer{inp};
delta_Outputlayer{inp} = delta_Outputs;
del_Weights_Outputlayer = (ita_1.*(delta_Outputs))*[1; Activated_Outputs_Hiddenlayer{inp}]' + (alpha.*del_Weights_Outputlayer);
accum_del_Weights_Outputlayer = accum_del_Weights_Outputlayer + del_Weights_Outputlayer;
end
%Wij更改在此处完成1个输入
accum_del_Weights_Hiddenlayer = zeros(150,785);
del_Weights_Hiddenlayer = zeros(150,785);
for inp = 1:1:4000
delta_Inputs = ((Weight_Matrices{2,1}(:,2:TN(2)+1)'*(delta_Outputlayer{inp})) - beta*(B1-A1)).*(der_Activated_Outputs_Hiddenlayer{inp});
delta_Hiddenlayer{inp} = delta_Inputs;
del_Weights_Hiddenlayer = [ita_1.*(delta_Inputs)]*[1 train_set(inp,1:784)] + [alpha.*(del_Weights_Hiddenlayer)];
accum_del_Weights_Hiddenlayer = accum_del_Weights_Hiddenlayer + del_Weights_Hiddenlayer;
end
%计算的累计权重。
Weight_Matrices{1,1} = Weight_Matrices{1,1} + (accum_del_Weights_Hiddenlayer)./4000 - 4000*decay_constant*[zeros(150,1) Weight_Matrices{1,1}(:,2:TN(1)+1)];
Weight_Matrices{2,1} = Weight_Matrices{2,1} + (accum_del_Weights_Outputlayer)./4000 - 4000*decay_constant*[zeros(784,1) Weight_Matrices{2,1}(:,2:TN(2)+1)];
end
........................................................................................
up3041
4. Simulation results
