Python-L1 and L2 regularization

1. L1 and L2 regularization

L1 regularization and L2 regularization are two regularization techniques commonly used in neural networks, which are used to penalize weight parameters to reduce overfitting. They have the following connections and differences:

connect:

① Both L1 regularization and L2 regularization are extra items added to the loss function when training the neural network , which are used to punish the weight parameters, thereby limiting their numerical size.

② Both L1 regularization and L2 regularization can be used to reduce the complexity of the model, prevent overfitting, and improve the generalization ability of the model .

③ Both L1 regularization and L2 regularization introduce hyperparameters (the hyperparameter in L1 regularization is λ1, and the hyperparameter in L2 regularization is λ2), which is used to control the strength of the regularization term.

the difference:

①L1 regularization will make part of the weight parameters become zero, so as to achieve sparsity , that is, the weight parameters corresponding to some features in the model are zero, which is helpful for feature selection, that is, to select the most important features for the model. Whereas L2 regularization just penalizes the weight parameters but does not make them zero.

②L1 regularization imposes a sparsity penalty on the weight parameters , so it can be used for feature selection and model compression. Whereas L2 regularization imposes a smoothness penalty on the weight parameters and thus can be used to alleviate collinearity issues.

③L1 regularization is more suitable for sparse data sets and high-dimensional feature selection in some cases , while L2 regularization is more suitable for dealing with collinearity problems and weight parameter smoothing in some cases .

It should be noted that the choice of using L1 regularization or L2 regularization, or their combination, depends on the specific problem and data set, and the best regularization method needs to be determined through experiments and parameter adjustments.

2. Examples

2.1 L1 regularization

When applying L1 regularization, the regularization term in the loss function can be expressed as:

L1 regularization term = λ1 * Σ|Wi|

Among them, λ1 is the hyperparameter of L1 regularization, and Wi is the ith weight parameter.

In TensorFlow, tf.contrib.layers.l1_regularizerthe L1 regularization term can be created through the function and tf.contrib.layers.apply_regularizationadded to the loss function through the function, as shown in the following code

Example:

import tensorflow as tf

# 定义神经网络模型
def my_model(input_tensor):
    # 定义网络结构
    # ...

    # 添加L1正则化项
    l1_regularizer = tf.contrib.layers.l1_regularizer(scale=0.01)
    reg_term = tf.contrib.layers.apply_regularization(l1_regularizer, weights_list=tf.trainable_variables())

    # 计算损失函数
    loss = my_loss_function() + reg_term

    # 返回损失函数
    return loss

2.2 L2 regularization

When applying L2 regularization, the regularization term in the loss function can be expressed as:

L2 regularization term = λ2 * Σ(Wi^2)

Among them, λ2 is the hyperparameter of L2 regularization, and Wi is the ith weight parameter.

In TensorFlow, tf.contrib.layers.l2_regularizerthe L2 regularization term can be created through the function and tf.contrib.layers.apply_regularizationadded to the loss function through the function, as shown in the following code

Example:

import tensorflow as tf

# 定义神经网络模型
def my_model(input_tensor):
    # 定义网络结构
    # ...

    # 添加L2正则化项
    l2_regularizer = tf.contrib.layers.l2_regularizer(scale=0.01)
    reg_term = tf.contrib.layers.apply_regularization(l2_regularizer, weights_list=tf.trainable_variables())

    # 计算损失函数
    loss = my_loss_function() + reg_term

    # 返回损失函数
    return loss

It should be noted that in actual use, appropriate hyperparameter values, such as the value of λ1 or λ2, can be selected according to specific problems and data sets to control the strength of the regularization term.