Problem Description:
No hidden layer softmax network, g is the activation function softmax
Decision boundary is linear (picture depth study of video from Andrew Ng)
First, two questions:
1, the decision on the output probabilities to any boundary softmax two are equal, i.e. for the corresponding component in the X-boundary any point, the output probability vector softmax equal. Softmax defined by the calculated apparent, zi = wi * X + bi = zj = wj * X + bj. Where wi and wj is a weight matrix of i, j row.
2, the linear boundary is equivalent to the boundary of any two points X1, X2, their linear combination X0 = t * X1 + s * X2 still on the decision boundary, where t + s = 1.
Let's start proved no hidden layer network softmax decision boundary is linear:
X1 office taken from two of the decision boundaries softmax, X2, apparent from the above description 1, w1 * X + b1 = w2 * X + b2, i.e., (w1-w2) * X = b2-b1.
Then take any point X0 = t * X1 + s * X2, s + t = 1, then the calculation of X0 softmax have z1 = w1 * X0 + b1, z2 = w2 * X0 + b2, the following proof z1 = z2.
z1-z2=w1*X0+b1-(w2*X0+b2)=t*(w1-w2)*X1+s*(w1-w2)*X2-(b1-b2)=t*(b2-b1)+s*(b2-b1)-(b1-b2)=0.
So, z1 = z2, that is also the X0 decision boundary.
NOTE: This above method may also be used to demonstrate proof of other classifiers (such as a linear regression and logistic regression classifier) is linear decision boundary, which is a general method.