Inductive bias
Inductive bias is a very subtle concept in machine learning: In machine learning, many algorithms make assumptions about some problems, and these assumptions are called inductive bias.
- Induction: Find commonality from some examples, generalize, and form a relatively general rule and procedure
- Bias: our preference for the model
Therefore, inductive bias can be understood as certain rules induced from observed phenomena in real life, and then certain constraints are placed on the model, so that it can play the role of "model selection", that is, select the model from the hypothesis space . A model that is more in line with the rules of reality.
Inductive bias is found almost everywhere in machine learning:
- Occam's Razor: Don't Multiply Entities Unnecessarily
- KNN assumes that adjacent samples in the feature space tend to belong to the same class
- The SVM classifier should maximize the class boundary distance
The same is true in deep learning. Taking neural networks as an example, various network structures, components, and mechanisms often come from inductive bias. In CNN, we assume that the features have local characteristics, that is, when we put some adjacent features together, it will be easier to get a solution; in RNN, we assume that the calculation at each moment depends on the historical calculation results; and The attention mechanism is also a rule based on human intuition and life experience.
The inductive bias of CNN is locality and spatial invariance, that is, the grid elements with similar spaces are connected but not far away, and the inductive bias of weight-sharing
RNN is sequentiality and time invariance, that is, the timesteps in the sequence order are connected and invariant to time transformation
Summarize:
In short, the inductive bias is that our tasks have these difficulties, and the model we choose can just solve these difficulties, so the ability to solve these difficulties is the inductive bias of a model