Feature engineering: what is a "combined feature"? How to deal with "high-dimensional combination features"?

In order to improve the fitting ability of complex relationships, first-order discrete features are often combined in pairs in feature engineering to form high-order combined features. In practical problems, it is necessary to face a variety of high-dimensional features, and simply combining them in pairs is still prone to problems such as too many parameters and overfitting.

How to efficiently find combination features? Decision trees can be used to find combinations of features.

For example, the film and television recommendation problem has two low-level features "language" and "type". Among them, the language is divided into Chinese and English, and the type is divided into movies and TV series. Then the high-level combination features of these two features are (Chinese, movie ), (English, TV series), (English, movie), (Chinese, TV series). The data in the following table can be changed to new data:

Whether to click	language	type
0	Chinese	Movie
1	English	Movie
1	Chinese	TV drama
0	English	TV drama

Whether to click	Language = Chinese, Type = Movie	Language = English, Genre = Movie	Language = Chinese, Genre = TV Drama	Language = English, Genre = TV Series
0	1	0	0	0
1	0	1	0	0
1	0	0	0	1
0	0	0	0	1

Taking logistic regression as an example, suppose the feature vector of the data is $X=(x_1,x_2,\dots ,x_k)$，则有：
$$Y=\text{sigmoid}(\sum _i\sum _j{w}_{ij}⟨x_i,x_j⟩)$$
$⟨x_i,x_j⟩$ means$x_i$and $x_j$Combination features of $w_{ij}$The dimension of is equal to the $i$ andjjThe number of different values ​​of$j\; features.$In the above example, the feature of "language" has two choices of Chinese and English, and the feature of "genre" has two choices of movie and TV series, then$w_{ij}$The dimension is $2\times 2=4.$ When the number of different values ​​of the two features before the combination is not large, there will be no big problem in this way. But for some problems, there are user IDs and item IDs, and the number of users and items is tens of millions, tens of millions times tens of millions$m\times n$ , such a large amount of parameters, cannot be learned.

How to deal with this "high-dimensional combination feature"? Suppose the number of users and items are $m$ and$n$ , an effective method is to separate the two features with$k$ -dimensional low-dimensional vector representation ($k\ll m,k\ll n$ ), so originally$m\times The\; learning\; parameter\; of\; n$ is reduced to$m\times k+n\times k$ , which is actually equivalent to the matrix decompositionin the recommendation algorithm.

---

Reference:
[1] Zhuge Yue, Huluwa, "Hundred Faces of Machine Learning", China Industry and Information Technology Publishing Group, People's Posts and Telecommunications Publishing House