A neural network is a computing system with interconnected nodes, and its nodes work more like neurons in the human brain. These neurons process and transmit information between them. Each neural network is a series of algorithms that try to identify potential relationships in a set of data through a process that simulates the operation of the human brain.
What is the difference between deep learning algorithms and classic neural networks? The most obvious difference is that the neural network used in deep learning has more hidden layers. These layers are located between the first layer (the input layer) and the last layer (the output layer) of the neuron. In addition, it is not necessary to connect all neurons in different layers.
9 deep learning algorithms you should know
#1 Backpropagation Backpropagation
is a very popular supervised learning algorithm for training feedforward neural networks. Essentially, backpropagation calculates the expression of the derivative of the cost function, which is the derivative product from left to right between each layer, and the weight gradient between each layer is a simple modification of the partial product ("reverse Propagation error").
We provide data to the network, it produces an output, we compare the output with the expected output (using a loss function), and then readjust the weights based on the difference. Then repeat this process. The adjustment of the weight is achieved through a non-linear optimization technique called stochastic gradient descent.
Suppose, for some reason, we want to identify the tree in the image. We provide any kind of images to the network and generate output. Since we know whether the image actually has a tree, we can compare the output with the real situation and adjust the network. As we deliver more and more images, there will be fewer and fewer errors on the network. Now we can provide it with an unknown image, which will tell us whether the image contains a tree.
#2 Feedforward Neural Network (FNN)
Feedforward neural networks are usually fully connected, which means that every neuron in a layer is connected to all other neurons in the next layer. The described structure is called a "multilayer perceptron", which originated in 1958. Single-layer perceptrons can only learn linearly separable patterns, while multi-layer perceptrons can learn non-linear relationships between data.
The goal of a feedforward network is to approximate a certain function f. For example, for category, =(x) maps input x to category y. The feedforward network defines a mapping y=f(x;θ), and learns the value of the parameter θ that leads to the best function approximation.
These models are called feedforward because there is no feedback connection from x to the intermediate calculation defined by f, and finally to the output y. There is no feedback connection that feeds back the output of the model to itself. When feedforward neural networks are extended to include feedback connections, they are called recurrent neural networks.
#3 Convolutional Neural Network (CNN)
In addition to providing assistance for the vision of robots and self-driving cars, convolutional neural networks have also been successfully applied to areas such as face recognition, object monitoring and traffic sign recognition.
In mathematics, convolution is an integral measure of how much two functions overlap when one function crosses another.
The green curve represents the convolution of the blue and red curves, which is a function of t, and the position is represented by the vertical green line. The gray area represents the product g(tau)f(t-tau) as a function of t, so its area as a function of t is convolution.
The product of the overlap of these two functions at each point on the x-axis is their convolution.
To a certain extent, they try to regularize the feedforward network to avoid overfitting (when the model only learns the pre-seen data and cannot generalize), which allows them to well identify the difference between the data Spatial Relations.
#4 Recurrent Neural Network (RNN)
Recurrent neural network is very successful in many NLP tasks. In traditional neural networks, it can be understood that all inputs and outputs are independent. However, for many tasks, this is inappropriate. If you want to predict the next word in a sentence, it is best to consider the word before it.
RNNs are called loops because they perform the same task on each element of the sequence, and the output depends on previous calculations. Another explanation of RNN: These networks have "memory" and consider previous information.
For example, if the sequence is a sentence of 5 words, it consists of 5 layers, one layer for each word.
The calculation formula defined in RNN is as follows:
x_t-input at time step t. For example, x_1 can be a one-hot vector corresponding to the second word of the sentence.
s_t is the hidden state in step t. This is the "memory" of the network. s_t as a function depends on the previous state and current input x_t: s_t=f(Ux_t+Ws_{t-1}). The function f is usually non-linear, such as tanh or ReLU. The s_{-1} required to calculate the first hidden state is usually initialized to zero (zero vector).
o_t-Exit at step t. For example, if we want to predict words in a sentence, the output may be a probability vector in a dictionary. o_t=softmax(Vs_t)
Image description generation
Together with convolutional neural networks, RNNs are used as part of the model to generate descriptions of unlabeled images. The combined model combines the generated words with the features in the image:
The most commonly used type of RNN is LSTM, which captures (stores) long-term dependencies better than RNN. LSTM and RNN are essentially the same, but they have different ways of calculating hidden states.
The memory in LSTM is called cells, and you can think of it as a black box that accepts the previous state h_{t-1} and the current input parameter x_t as input. Internally, these cells decide which memory to save and delete. Then, they combine the previous state, current memory and input parameters.
These types of units are very effective in capturing (storing) long-term dependencies.
#5 Recurrent Neural Network
Recurrent Neural Network is another form of recurrent network, the difference is that they are tree structure. Therefore, they can model the hierarchical structure in the training data set.
Because of their relationship with binary trees, context, and natural language-based parsers, they are often used in NLP applications such as audio-to-text transcription and sentiment analysis. However, they tend to be much slower than recursive networks
#6 Self-encoder The
self-encoder can restore the input signal at the output. There is a hidden layer inside them. The autoencoder is designed to be unable to accurately copy the input to the output, but in order to minimize the error, the network is forced to learn to select the most important features.
Autoencoders can be used for pre-training, for example, when there are classification tasks and there are too few labeled pairs. Or reduce the dimensionality in the data for later visualization. Or, when you only need to learn to distinguish the useful properties of the input signal.
#7 Deep Belief Network and Restricted Boltzmann Machine
Restricted Boltzmann Machine is a random neural network (neural network means that we have neuron-like units whose binary activation depends on the neighbors they are connected to Unit; random means that these activations have probabilistic elements), it includes:
Visible unit layer
Hidden unit layer
Deviation unit
In addition, each visible unit is connected to all hidden units (this connection is undirected, so each hidden unit is also connected to all visible units), and the deviation unit is connected to all visible units and all hidden units.
In order to make learning easier, we restrict the network so that no visible unit is connected to any other visible unit, and any hidden unit is not connected to any other hidden unit.
Multiple RBMs can be superimposed to form a deep belief network. They look like fully connected layers, but they are trained in different ways.
#8 Generative Adversarial Networks (GAN)
GANs are becoming a popular online retail machine learning model because they can understand and reconstruct visual content with increasing accuracy. Use cases include:
Fill the image from the outline.
Generate realistic images from text.
Create a realistic description of the product prototype.
Convert black and white images to color images.
In video production, GAN can be used to:
Simulate human behavior and movement patterns within the framework.
Predict subsequent video frames.
Create deepfake
The Generative Adversarial Network (GAN) has two parts:
The generator learns to generate credible data. The generated instance becomes the negative training instance of the discriminator.
The discriminator learns to distinguish the fake data of the generator from the data. The discriminator penalizes generators that produce unbelievable results.
The first step in building a GAN is to identify the final output required and collect the initial training data set based on these parameters. These data are then randomized and input into the generator until the basic accuracy of the generated output is obtained.
Then, the generated image and the actual data points of the original concept are fed into the discriminator. The discriminator filters the information and returns a probability between 0 and 1 to indicate the authenticity of each image (1 is related to true, 0 is related to false). Then check whether these values are successful, and keep repeating until the desired result is achieved.
#9Transformers
Transformers are also very new, they are mainly used for language applications. They are based on a concept called attention, which is used to force the network to focus on specific data points.
Since the LSTM unit is too complex, the attention mechanism can be used to weigh different parts of the input according to its importance. The attention mechanism is just another layer with weights. Its sole purpose is to adjust the weights to prioritize the input part while excluding other parts.
In fact, Transformers consist of multiple stacked encoders (forming encoder layers), multiple stacked decoders (decoder layers), and a bunch of attention layers (self-attentions and encoder-decoderattentions).
Transformers are designed to handle ordered data sequences for various tasks such as machine translation and text summarization, such as natural language. Today, BERT and GPT-2 are the two most famous pre-trained natural language systems used in various NLP tasks. They are based on Transformers.
#10 Graph Neural Network
Generally speaking, unstructured data is not suitable for deep learning. In many practical applications, data is unstructured, such as social networks, compounds, knowledge graphs, spatial data, etc.
The purpose of graph neural networks is to model graph data, which means that they identify the relationships between nodes in the graph and represent them numerically. They can later be used in any other machine learning model for various tasks, such as clustering, classification, etc.