The Johnson–Lindenstrauss theorem goes like this: any 10,000 points in a one-million-dimensional space must be almost packed into a subspace of dozens of dimensions!
Strictly speaking, N points in an M-dimensional space are almost always contained in a D-dimensional subspace. Here D should intuitively be equal to the order of N, but in fact we only need to let D be the order of log(N). The exact meaning of "almost contained" here is that its projection onto this subspace is almost equidistant (an error of ε is allowed, and the constant D/log(N) depends on ε). Obviously, this matter is of great significance when reducing the dimension of high-dimensional data.
The proof of this theorem is very elementary. It relies on the basic probabilistic fact that the length of the projection of a random M-dimensional unit vector onto a random D-dimensional subspace is almost certain to be approximately D/M. The thing itself is also a bit unusual, although it can be confirmed by simple calculations. This is an example of a counter-intuitive phenomenon that often occurs in probability-theoretic calculations due to high dimensionality.
This reminds me of another paradox caused by high dimensions, which I learned about when I was studying the law of large numbers. A point is randomly selected in the M-dimensional unit cube. When M is sufficiently large, it is easy to calculate that the distance from this point to the center of the cube is almost always equal to √(M/3)/2 according to the law of large numbers. So this means that the M-dimensional solid unit cube lies almost entirely on a spherical shell of radius √(M/3)/2. There's no mischief here, in fact it is.
The Johnson–Lindenstrauss lemma states that any high-dimensional dataset can be randomly projected into a lower-dimensional Euclidean space, while controlling the distortion of pairwise distances.
theoretical boundary
The distortion introduced by a random projection P is deterministic because p defines an esp-embedding. Its probability theory is defined as follows:
u and v are arbitrary rows from a dataset of shape [n samples, n features] = [n_samples, n_features], p roommates a random Gaussian of shape [n components, n features] = [n_components, n_features] Projection of an N(0,1) matrix (or a sparse Achlioptas matrix).
The minimum number of components used to guarantee eps-embedding is given by the following formula:
