13 probability distributions that must be mastered in deep learning

Author: Python, the article is excerpted from Advanced Algorithms

Introductionx

As a deep learning practitioner, you need to know about probability distributions. Here is a list of the most common basic probability distribution tutorials, mostly related to deep learning using python libraries.

1. Overview of the relationship between probability distributions

conjugate means that it has a conjugate distribution relationship.
Multi-ClassIndicates that the random variance is greater than 2.
N Timesmeans that we also consider the prior probability P(X).
To learn more about probability, I recommend reading [pattern recognition and machine learning, Bishop 2006].

In Bayesian probability theory, if the posterior distribution p(θx) and the prior probability distribution p(θ) are in the same probability distribution family, the prior and the posterior are called conjugate distributions, and the prior is called the likelihood The conjugate prior for the function. Wikipedia for the conjugate prior is here (https://en.wikipedia.org/wiki/Conjugate_prior). The lossless original image is visible: https://github.com/graykode/distribution-is-all-you-need/blob/master/overview.pptx

2. Distribution probability and characteristics

1. Uniform distribution (continuous)

A uniform distribution has the same probability value on [a,b] and is a simple probability distribution.

2. Bernoulli distribution (discrete)

The prior probability p(x) does not take into account the Bernoulli distribution. Therefore, if we optimize for maximum likelihood, we can easily be overfitted.
Binary classification using binary cross-entropy. It has the same form as the negative logarithm of the Bernoulli distribution.

3. Binomial distribution (discrete)

The binomial distribution with parameters n and p is a discrete probability distribution of the number of successes in a series of n independent experiments.
The binomial distribution is a distribution that takes into account prior probabilities by specifying the amount to pick in advance.

"""
各分布的生成代码（算法进阶 附注）：
https://github.com/graykode/distribution-is-all-you-need
Code by Tae-Hwan Hung(@graykode)
"""
import numpy as np
from matplotlib import pyplot as plt


import operator as op
from functools import reduce


def const(n, r):
    r = min(r, n-r)
    numer = reduce(op.mul, range(n, n-r, -1), 1)
    denom = reduce(op.mul, range(1, r+1), 1)
    return numer / denom


def binomial(n, p):
    q = 1 - p
    y = [const(n, k) * (p ** k) * (q ** (n-k)) for k in range(n)]
    return y, np.mean(y), np.std(y)


for ls in [(0.5, 20), (0.7, 40), (0.5, 40)]:
    p, n_experiment = ls[0], ls[1]
    x = np.arange(n_experiment)
    y, u, s = binomial(n_experiment, p)
    plt.scatter(x, y, label=r'$\mu=%.2f,\ \sigma=%.2f$' % (u, s))


plt.legend()
plt.savefig('graph/binomial.png')
plt.show()

4. Multi-Bernoulli distribution, categorical distribution (discrete)

Multi-Bernoulli is called a categorical distribution.
Cross-entropy has the same form as the multi-Bernoulli distribution that takes negative logarithms.

5. Multinomial distribution (discrete)

The multinomial distribution is related to the categorical distribution in the same way that the Bernoulli distribution is related to the binomial distribution.

6. Beta distribution (continuous)

The beta distribution is conjugated to the binomial and Bernoulli distributions.
Using conjugation, the posterior distribution can be more easily obtained using a known prior distribution.
The uniform distribution is the same when the beta distribution satisfies the special case (α=1, β=1).

7. Dirichlet distribution (continuous)

The dirichlet distribution is conjugate to the multinomial distribution.
If k=2, it is a beta distribution.

8. Gamma distribution (continuous)

If gamma(a,1)/gamma(a,1)+gamma(b,1) is the same as beta(a,b), then the gamma distribution is a beta distribution.
The exponential and chi-square distributions are special cases of the gamma distribution.

9. Exponential distribution (continuous)

The exponential distribution is a special case of the gamma distribution when α is 1.

10. Gaussian distribution (continuous)

The Gaussian distribution is a very common continuous probability distribution.

11. Normal distribution (continuous)

The normal distribution is a standard Gaussian distribution with a mean of 0 and a standard deviation of 1.

12. Chi-square distribution (continuous)

The chi-square distribution with k degrees of freedom is the distribution of the sum of squares of k independent standard normal random variables.
The chi-square distribution is a special case of the beta distribution

13. t-distribution (continuous)

The t-distribution is a symmetric bell-shaped distribution, similar to the normal distribution, but with heavier tails, which means it is more likely to produce values that are well below the mean.

Code: https://github.com/graykode/distribution-is-all-you-need/

13 probability distributions that must be mastered in deep learning

1. Overview of the relationship between probability distributions

Guess you like