[Introduction to Artificial Intelligence] Python standard library - dalib (domain adaptation)

[Introduction to Artificial Intelligence] Python standard library - dalib (domain adaptation)

---

1. Domain Discriminator

dalib.modules.domain_discriminator.DomainDiscriminator(in_feature: int, hidden_size: int)

• Function: Distinguish whether the input features come from the source domain or the target domain. The source domain label is 1 and the target domain label is 0.
• parameter:

• in_feature(int): Dimension of input feature;
• hidden_size (int): Dimension of hidden layer features.

• shape:

• inputs：（minibatch， in_feature）；
• outputs： （minibatch， 1）。

• Example:

• See DomainAdversarialLoss for an example.

2. Domain AdversarialLoss

dalib.adaptation.dann.DomainAdversarialLoss(domain_discriminator: torch.nn.modules.module.Module, reduction: Optional[str]= 'mean')

• 定义： $$Loss(D_s,D_t)=E_{x_i^s⌢D_s}log[D(f_i^s)]+E_{x_j^t⌢D_t}log[1-D(f_j^t)]$$ where D is the domain discriminator and f is the feature of the domain.
• parameter:

• domain_discriminator (nn.Module): Domain discriminator object, used to predict the domain of features;
• reduction (string, Optional): Specify the way to output the loss, 'none', 'sum', 'mean', where 'none' means direct output without any dimensionality reduction, 'sum' and 'mean' respectively calculate the loss Sum and average, the default is average.

• enter:

• f_s (tensor): Feature $f^s$；
• f_t (tensor): Features of the target domain $f^t$。

• shape:

• f_s, f_t: (N, F) F is the dimension of the input feature;
• outputs : Defaults to scalar, but if reduction is 'none' the shape of the outputs is (N,).

• Example:

from dalib.modules.domain_discriminator import DomainDiscriminator
from dalib.adaptation.dann import DomainAdversarialLoss

discriminator = DomainDiscriminator(in_feature= 1024, hidden_size= 2048)
loss = DomainAdversarialLoss(discriminator, reduction= 'mean')

f_s, f_t = torch.rand(20, 1024), torch.rand(20, 1024)
output = loss(f_s, f_t)

print(output)

3. GaussianKernel

dalib.modules.kernels.GaussianKernel(sigma: Optional[float] = None, track_running_stats: Optional[bool] = True, alpha: Optional[float] = 1.0)

• definition:

• Gaussian kernelkkDefinition of$k$$$k(x_1,x_2)=exp(-\frac{{\Vert x_1-x_2\Vert }^2}{2p^2})$$ 其中$x_1,x_2\in R^d$ is a one-dimensional tensor.
• Gaussian kernel matrix $K$ is defined at$X=(x_1,x_2,...x_m)$上：$$K(x{)}_{i,j}=k(x_i,x_j)$$
• In the operation $p^2$ There are two confirmation methods:
  the first is obtained dynamically by calculating the following formula:$$p^2=\frac{}{n^2}\sum _{i,j}{\Vert x_i-x_j\Vert }^{2The}$$
  second method is to directly give the value.

• parameter:

• sigma(float, optional)： 即$\sigma$ , default is None;
• track_running_stats(bool, optional): If it is 'True', use the previous formula to calculate $p^2$ , if 'False', use fixed$p^2$ , default is 'True';
• alpha(float, optional): When track_running_stats is 'True', calculate $p^2$ provides$a$ .

• enter:

• X(tensor): Input group X.

• shape:

• inputs: (minibatch, F), F is the dimension of the input feature;
• outputs：(minibatch, minibatch) 。

4. Multi-core maximum mean difference (MK-MMD)

dalib.adaptation.dan.MultipleKernelMaximumMeanDiscrepancy（kernels: Sequence[torch.nn.modules.module.Module], Linear: Optional[bool]= False, quadratic_program: Optional[bool]= False）

• MK-MMD：

• 源域为： D s = { ( x i s , y i s ) } i = 1 n s D_{s}= \left \{ (x_{i}^{s},y_{i}^{s}) \right \}_{i=1}^{n_{s}} Ds​={ (xis​,yis​)}i=1ns​​
• 目标域：$$D_t={\left\{x_j^t\right\}}_{j=1}^{n_t}$$
• Their respective samples are independent and identically distributed;

• Then the calculation formula of MK-MMD is: $$d_{MK-MMD}(D_s,D_t)={\Vert E_s[g(D_s)]-E_t\left[g\right(D_t)]\Vert }_{H_k}^2$$
• $H_k$Indicates having a specific kernel $k$ 's$RKHS$，$g(\ast )$ is a continuous map related to the kernel function,$E[*]$ is the expectation of a given distribution;

• It should be noted that the kernel function $k$ is defined as$The\; convex\; combination\; of\; r$ different positive semi-definite kernels has the following form:$$k(x^s,x^t)={\sum }_{i=1}^r{b}_i{k}_i(x^s,x^t)$$
• Among them: $${\sum }_i^r{b}_i=1,b_i\ge 0$$
• The so-called positive semi-definiteness is a common property of the kernel function (you can learn it by contacting related concepts in SVM). The convex combination is a linear combination. If λ i ≥ 0 is satisfied, ∑ ir λ i = 1 \lambda _{i} $$l_i\ge 0，{\sum }_i^r{l}_i=1$$则${\sum }_i^r{l}_i{x}_i$That is a convex combination;

• 使用内核技巧，MK-MMD可以简化计算为：$${\hat{D}}_k(D_s,D_t)=\frac{1}{n_s^2}{\sum }_{i=1}^{n_s}{\sum }_{j=1}^{n_s}k(D_s^i,D_s^j)+\frac{1}{n_t^2}{\sum }_{i=1}^{n_t}{\sum }_{j=1}^{n_t}k(D_t^i,D_t^j)-\frac{2}{n_s{n}_t}{\sum }_{i=1}^{n_s}{\sum }_{j=1}^{n_t}k(D_s^i,D_t^j)$$

• parameter:

• Kernel(tuple(nn.Module)): Kernel equation;
• Linear (bool): whether to use the linear version of DAN, not used by default;
• quadratic_program(bool): Whether to use quadratic programming to solve $\beta$ , not used by default.

• enter:

• d_s(tensor): Feature D s D_{s} obtained by mapping the source domain$D_s$；
• d_t(tensor): Feature D t D_{t} obtained by mapping the target domain$D_t$。
• Note that they must both have the same shape.

• shape:

• inputs: (minibatch, *) * represents any number, which is actually the incoming feature dimension;
• outputs: scalar.

• Example:

from dalib.modules.kernels import GaussianKernel
from dalib.adaptation.dan import MultipleKernelMaximumMeanDiscrepancy

feature_dim = 1024
batch_size = 10

kernels = (GaussianKernel(alpha=0.5), GaussianKernel(alpha=1.), GaussianKernel(alpha=2.))
loss = MultipleKernelMaximumMeanDiscrepancy(kernels)

# features from source domain and target domain
z_s, z_t = torch.randn(batch_size, feature_dim), torch.randn(batch_size,feature_dim)
output = loss(z_s, z_t)

print(output)