Characteristic (the Feature) and feature selection (Feature Selection)
Explain the concept
In machine learning, the property is called "feature (Feature)", the current learning tasks useful properties referred to as "relevant characteristics (Relevant Feature)", no use of the property called "irrelevant characteristics (Irrelevant Feature)". Selecting from a given set of process-related features of the feature subset, referred to as "feature selection (Feature Selection)"
Feature selection is an important pre-processing data. In the current machine learning, data is usually obtained after the first feature selection, after which the training learner.
Feature selection process must ensure that no loss of important features, or follow the learning process because of missing important information and can not get good performance. A given data set, if different learning tasks, the associated features will likely be different.
Further, there is a class feature called "redundant features (Redundant Feature)", the information they contain deduced from other features. Then, remove redundant features will reduce the burden on the learning process.
Wherein the selected object
In the practical application of machine learning, the number of features more often, which may not be related to the presence of features, there may be interdependencies between features, easily lead to the following consequences:
- The more the number of features, analysis features, the longer the time required to train the model.
- The more the number of features, easily lead to "curse of dimensionality" model will be more complex, its ability to promote the fall.
Feature selection can eliminate not relevant (IRRELEVANT) or feature Kang I (Redundant) and thus to decrease the number of features to improve the accuracy of the model, the purpose of reducing running time. On the other hand, select the relevant features of the real model is simplified to enable researchers to process data generated easily understood.
The main feature selection has two purposes :
- Reducing the number of features, dimension reduction, to avoid the curse of dimensionality, so make model generalization stronger, reducing over-fitting, shorten training time model.
- Enhanced understanding of the features and feature values between
Feature selection process
- Generating process (Generation Procedure): is a process of generating a search feature subset is responsible for the evaluation function provides a subset of features.
- Evaluation function (Evaluation Function): evaluation function is to evaluate how well a feature of a subset of the criterion.
- Stopping criterion (Stopping Criterion): stopping criterion is associated with the evaluation function, it is generally a threshold value, the evaluation function value when this threshold is reached the search can be stopped.
- Verification process (Validation Procedure): validate elected subset of features in the validation data set.
The method of feature selection
According to a feature selected form feature selection method is divided into three categories:
. 1) the Filter : filtration, according to divergence or relevance of the respective feature score, a threshold or a threshold number to be selected, the selected feature. Characterized in a subsequent selection process is independent of learning, a first feature by feature selection process is a "filter", and then after the filtration characteristics model training. The correlation values are sorted
2) the Wrapper : packing method, to be used according to the learning performance score, wherein each select a number, or a number of negative characteristics. Packaging Act object feature is selected for a given subset of feature learning "tailored". Greedy or exhaustive search
. 3) Embedded : embedding, an embedded feature selection is automatic feature selection training process in the learner . Sort weight coefficient characteristic weight
1 where no training, some of which may reflect direct calculation of correlation values can be sorted, 2, and 3 need to train the model, and the model training require multiple 2; 1 and 2 are characteristic of learner selection process separate training process, 3 is combined, it is carried out automatically in the learning process of training feature selection.
Select Forward (Forward Selection) and backward elimination (Backward Elimination)
Backward Selection Forward Selection and during feature selection
If you want to remove the selected feature contains a subset of all important information, if there is no field of knowledge as assumptions, then had to traverse all possible subsets of features from the original collection. Feasible method is to produce a "candidate subset" of the evaluation that it is good or bad based on the following evaluation result is a subset of the candidate, then its evaluation, ......, this process continues until it can not find a better feature subset so far. So, how to get the next candidate subset based on the evaluation results? How to evaluate the quality of the candidate feature subset?
This is a search strategy. So there will be:
- Select Forward (Forward Selection)
- After the elimination of (Backward Elimination)
- Bidirectional search (Bidirectional Selection)
- Stepwise
Backward Elimination, Forward Selection and Stepwise three feature selection method is often used.
When the number of times too many features, we can in addition dimension reduction by PCA and other methods, can also use the feature selection method of screening a few influential characteristic (feature) with the results, so the impact on the results is not the case, reduce the amount of calculation.
Backward Elimination method is very simple:
First contains all of the feature, then each feature are trying to delete, delete the test which has the largest feature to enhance the accuracy of the model, the model eventually deleted the highest lift a feature. And so on, until you delete the feature does not improve the model so far.
Forward Selection is exactly the opposite, added to the list one by one.
With respect to the Forward Selection, Backward Elimination advantage is that it allows some of the features of the low contribution to the model to be able to enter (and sometimes the low contribution of the feature to have greater contribution value combination, this ignores the Forward Selection the possibility of a combination of species), and therefore can avoid interference Backward Elimination one or two dominant features.
Stepwise method is a combination of both, when a new feature is added, stepwise tries to delete a feature, until it reaches a preset standard. The disadvantage of this method is that the default set of criteria is not good, but also easy to fall into the over-fitting them.
