runge phenomenon and overfitting

Runge phenomenon

Runge phenomenon refers to a phenomenon: given some sample points, when polynomial fitting is performed on them, the higher the polynomial degree, the greater the gap between the actual function and the actual function.

experiment

Our real function takes $Y=\frac{1}{1+25x^2}$As an example, dig out some of these points and use numpy to fit polynomials of different degrees to see how the final function fits the real function.

Let's take a look at what this real function looks like.

x=np.linspace(-2,2,10000)
y=1/(1+25*x*x)
plt.figure(figsize=(20,8),dpi=80)
plt.plot(x,y)
plt.show()

Some points we selected on this real function are as follows.

testx=np.linspace(-2,2,7)
testy=1/(1+25*testx*testx)
plt.figure(figsize=(20,8),dpi=80)
plt.scatter(testx,testy)
plt.show()

We fit these points.

degree=[3,5,7]
f=[]
for i in range(3):
    model=np.polyfit(testx,testy,degree[i])
    f.append(np.poly1d(model))
    print(f[i])

Then visualize

colors=["r","g","orange"]
plt.figure(figsize=(20,8),dpi=80)
xs=np.linspace(-2.3,2.3,1000)
for i in range(3):
    plt.plot(xs,f[i](xs),color=colors[i],label=degree[i])
plt.scatter(testx,testy,label="o")
plt.plot(x,y,color="b",label="function")
plt.legend()
plt.show()

From the above we have seen that the 7th degree polynomial is outside the interval [-2,2] and has completely deviated from the real function. The 5th degree polynomial is the best preserved for the 3rd degree polynomial.

We extend the interval to [-10,10] to verify the closeness of the three fitting functions to the real function.

We found that the highest order has completely deviated. We continue to extend the interval to [-100,100].

Our findings are the same, the highest degree polynomial is also seriously deviated from the real function. (Note: The magnitude of the ordinate becomes $1\ast e^{11}$。）

Understanding summary

The Runge phenomenon illustrates one thing: fitting the points generated by the real distribution, the polynomial function obtained is not that the higher the degree, the closer to the real function. On the contrary, the higher the number of times in this example, the greater the deviation of the results. We need to do some a priori work to understand the curve trend of the real function, and then prescribe the right medicine. That is, under the premise of unfamiliar with the curve movement trend, do not use high-degree polynomials lightly.

Realistic significance: Even if 100 points are given, it is reasonable to solve the polynomial with the highest degree of 100 (the method of undetermined coefficients), but do not use this lightly.

This is consistent with overfitting in machine learning.

Supplement: We can consider the following practices:

1. The sample points are segmented by intervals (or in some way), and then low-order polynomial fitting is applied to each interval, and high-order can also be considered if appropriate.
2. The chebyshev node replaces the equidistant node.