REF recursive feature elimination method of random forest to filter features (python implementation does not depend on sklearn)

REF Recursive Feature Elimination for Random Forests is a model-based feature selection method. It recursively eliminates unimportant features by building a random forest model and repeatedly training the model to evaluate the importance of each feature. The REF method is implemented through the following steps:
1. Calculate the feature importance scores of the initial model on all features.
2. Remove the feature with the lowest score and retrain the model.
3. Repeat step 1 and step 2 until the number of selected features reaches the required number or reaches the predefined stopping criteria.
The advantage of this method is that feature selection can be performed without prior knowledge, and it can perform efficient feature screening on high-dimensional datasets. In addition, the REF method can also evaluate the performance of the model through cross-validation to avoid overfitting. However, the disadvantage of this method is that it is computationally expensive, especially when dealing with large datasets.

In summary, the REF recursive feature elimination method is a reliable feature selection method that can help us identify the most important features from a large number of features and improve the predictive performance of the model.

The following is a code example of the REF recursive feature elimination method for random forests using the scikit-learn library in Python:

from sklearn.ensemble import RandomForestClassifier
from sklearn.feature_selection import RFECV
from sklearn.datasets import make_classification

# 生成一组样本数据
X, y = make_classification(n_samples=1000, n_features=20, n_informative=5, n_redundant=0, random_state=42)

# 定义一个随机森林分类器
clf = RandomForestClassifier(n_estimators=100, random_state=42)

# 定义REF递归特征消除器，使用5折交叉验证
rfecv = RFECV(estimator=clf, step=1, cv=5, scoring='accuracy')

# 运行REF递归特征消除器，并返回选择的特征
selected_features = rfecv.fit_transform(X, y)

# 输出选择的特征数量和选择的特征的索引
print("Selected Features: %d" % rfecv.n_features_)
print("Feature Ranking: %s" % rfecv.ranking_)

In this example, we first use the make_classification function to generate a dataset containing 1000 samples and 20 features, of which 5 features are informative to the target variable. We then define a random forest classifier, and an RFECV object to perform REF recursive feature elimination. Finally, we pass the sample data and target variable to the fit_transform method of the RFECV object to obtain the selected features. The program outputs the number of features selected and the feature rank.

Python version, does not depend on modules

import numpy as np
from sklearn.tree import DecisionTreeClassifier

class RandomForestClassifier:
    def __init__(self, n_estimators=10, max_depth=None, random_state=None):
        self.n_estimators = n_estimators
        self.max_depth = max_depth
        self.random_state = random_state
        self.estimators = []

    def fit(self, X, y):
        np.random.seed(self.random_state)
        n_samples, n_features = X.shape

        # 构建随机森林
        for i in range(self.n_estimators):
            # 从原始数据集中随机抽样，构建一个新的子数据集
            sample_indices = np.random.choice(n_samples, n_samples, replace=True)
            X_subset, y_subset = X[sample_indices], y[sample_indices]

            # 构建决策树模型
            tree = DecisionTreeClassifier(max_depth=self.max_depth, random_state=self.random_state)

            # 训练决策树模型
            tree.fit(X_subset, y_subset)

            # 将决策树模型添加到随机森林中
            self.estimators.append(tree)

    def predict(self, X):
        # 预测分类结果
        predictions = []
        for tree in self.estimators:
            predictions.append(tree.predict(X))
        return np.mean(predictions, axis=0)

def REF(X, y, n_selected_features):
    # 定义一个随机森林分类器
    clf = RandomForestClassifier(n_estimators=100, random_state=42)

    # 定义一个包含特征索引和得分的列表
    feature_scores = []

    # 计算每个特征在随机森林模型中的重要性得分
    for i in range(X.shape[1]):
        # 选择除第i个特征之外的所有特征
        X_subset = np.delete(X, i, axis=1)

        # 训练随机森林模型并计算特征重要性得分
        clf.fit(X_subset, y)
        score = clf.estimators[0].feature_importances_

        # 将特征得分添加到列表中
        feature_scores.append(score)

    # 将得分转换为平均得分
    feature_scores = np.mean(feature_scores, axis=0)

    # 选择前n_selected_features个特征的索引
    selected_feature_indices = np.argsort(feature_scores)[::-1][:n_selected_features]

    # 返回所选特征的索引
    return selected_feature_indices

# 生成一组样本数据
X = np.random.randn(1000, 20)
y = np.random.randint(0, 2, size=1000)

# 使用REF方法选择5个特征
selected_feature_indices = REF(X, y, 5)

# 输出选择的特征的索引
print("Selected Features: %s" % selected_feature_indices)

In this example, we first define a random forest classifier implemented by ourselves, and then define a REF function to select the most important features. The implementation of the REF function is based on the following steps:

1. Define a random forest classifier, which is implemented using the RandomForestClassifier class.

2. Define a list feature_scores containing feature indices and scores.

3. Calculate the importance score of each feature in the random forest model. For each feature, we remove that feature from the original dataset, then train a random forest model and compute the feature importance score. We add the feature scores to the feature_scores list.

4. Convert the feature score to the average score, using the np.mean function.

5. Select the index of the first n_selected_features features, using np.argsort and [::-1] to achieve.

6. Return the index of the selected feature.

In this example, we used some functions from numpy and sklearn.tree, but we did not use any other external package to implement the REF function. The core idea of ​​implementing the REF function is to use the random forest model to calculate the importance score of each feature, and then select the most important features.