Deep learning to solve differential equation series four: a PINN solution method based on adaptive activation function - Burger equation inverse problem

Below I will introduce the embedded physics knowledge neural network (PINN) to solve differential equations. Firstly, the basic method of PINN is introduced, and the PINN solution framework based on the adaptive activation function is used to solve the inverse problem of the one-dimensional Burger equation with time term using Pytorch.
Embedded Physical Knowledge Neural Network (PINN) Introduction and Related Papers
Deep Learning to Solve Differential Equations Series 1: PINN Solution Framework (Poisson 1d)
Deep Learning to Solve Differential Equations Series 2: PINN to Solve Burger Equation Forward Problems
Deep Learning to Solve Differential Equations Series 3: PINN solves the inverse problem of the burger equation
Deep learning to solve the differential equation series IV: Based on the adaptive activation function PINN solves the inverse problem of the burger equation
Deep learning to solve the differential equation series V: PINN solves the direct and inverse problem of the Navier-Stokes equation

1. Introduction to PINN

As a powerful information processing tool, neural network has been widely used in the fields of computer vision, biomedicine, and oil and gas engineering, triggering technological changes in many fields. The deep learning network has a very strong learning ability, not only can discover physical laws, but also solve partial differential equations. In recent years, the solution of partial differential equations based on deep learning has become a new research hotspot. Embedded physics-informed neural network (PINN) is an application of scientific machines in the traditional numerical field, which can be used to solve various problems related to partial differential equations (PDE), including equation solving, parameter inversion, model discovery, control and optimization etc.

2. PINN method

The main idea of ​​PINN is shown in Figure 1, first construct an output result as $\hat{u}$ 's neural network, which is used as a proxy model for the PDE solution, and the PDE information is used as a constraint, encoded into the neural network loss function for training. The loss function mainly includes four parts: partial differential structure loss (PDE loss), boundary value condition loss (BC loss), initial value condition loss (IC loss) and real data condition loss (Data loss).

Figure 1: Schematic diagram of PINN

In particular, consider the following PDE problem, where the solution of the PDE $u(x)$在$Oh\subset {\mathbb{R}}^d$ definition, where$\mathbf{x}=\left(x_1,\dots ,x_d\right)$：
$$f\left(\mathbf{x};\frac{\partial u}{\partial x_1},\dots ,\frac{\partial u}{\partial x_d};\frac{{\partial }^2}{\partial x_1\partial x_1},\dots ,\frac{{\partial }^2}{\partial x_1\partial x_d}\right)=0,\mathbf{x}\in \Omega At$$
the same time, satisfy the following boundary
$$\mathcal{B}(u,\mathbf{x})=0\text{ on }\partial \Omega$$

The PINN solution process mainly includes:

• The first step is to define the neural network model of the fully connected layer of D layer:
  $$N_{}:=L_D\circ p\circ L_{D-1}\circ p\circ \cdots \circ p\circ L_1$$
  式中：
  $$\begin{array}{rl}{L}_1(x)& :=W_{1}x+b_1,W_1\in {\mathbb{R}}^{d_1\times d},b_1\in R^{d_1}\\ L_i\left(x\right)& :=W_{i}x+b_i,W_i\in R^{d_i\times d_{i-1}},b_i\in {\mathbb{R}}^{d_i},\forall i=2,3,\cdots D-1,\\ L_D(x)& :=W_{D}x+b_D,W_D\in R^{N\times d_{D-1}},b_D\in R^N.\end{array}$$
  and $\sigma$ is the activation function,$W$ and$b$ is the weight and bias parameters.
• The second step, in order to measure the neural network $\hat{u}$和约束之间的差异，考虑损失函数定义：
  $$\mathcal{L}\left(i\right)=w_f{L}_{PDE}\left(i;{\mathcal{T}}_f\right)+w_i{L}_{IC}\left(i;T_i\right)+w_b{L}_{BC}\left(i,;T_b\right)+w_d{L}_{Data}\left(i,;{\mathcal{T}}_{data}\right)$$
  where:
  $$\begin{array}{rl}{\mathcal{L}}_{PDE}\left(i;{\mathcal{T}}_f\right)& =\frac{1}{\left|{\mathcal{T}}_f\right|}\sum _{\mathbf{x}\in {\mathcal{T}}_f}{\Vert f\left(\mathbf{x};\frac{\partial \hat{u}}{\partial x_1},\dots ,\frac{\partial \hat{u}}{\partial x_d};\frac{{\partial }^2\hat{u}}{\partial x_1\partial x_1},\dots ,\frac{{\partial }^2\hat{u}}{\partial x_1\partial x_d}\right)\Vert }_2^2\\ L_{IC}\left(i;T_i\right)& =\frac{1}{|T_i|}\sum _{x\in T_i}\Vert \hat{u}(\mathbf{x})-u(\mathbf{x}){\Vert }_2^2\\ L_{BC}\left(i;T_b\right)& =\frac{1}{|T_b|}\sum _{x\in T_b}\Vert \mathcal{B}(\hat{u},\mathbf{x}){\Vert }_2^2\\ L_{Data}\left(i;T_{data}\right)& =\frac{1}{|T_{data}|}\sum _{x\in T_{data}}\Vert \hat{u}\left(x\right)-u\left(x\right){\Vert }_2^2\end{array}$$
  $w_f$，$w_i$、$w_b$and $w_d$is the weight. ${\mathcal{T}}_f$，$T_i$、$T_b$和$T_{data}$Represents residual points from PDE, initial value, boundary value and true value. Here $T_f\subset \Omega$ is a predefined set of points to measure the neural network output$\hat{u}$ Extent of match to PDE.
• Finally, use the gradient optimization algorithm to minimize the loss function until the network parameters that meet the prediction accuracy are found. KaTeX parse error: Undefined control sequence: \theat at position 1: \̲t̲h̲e̲a̲t̲^{*} .

It is worth noting that for inverse problems, i.e. some parameters in the equation are unknown. If only the PDE equation and boundary conditions are known, and the PDE parameters are unknown, the inverse problem is an indeterminate problem, so other information must be known, such as some observation points $the\; value\; of\; u$ . In this case, the PINN method can use the parameters in the equation as unknown variables and add them to the trainer for optimization. The loss function includes Data loss.

3. PINN based on adaptive activation function

Brown University Jagtap proposed PINN based on adaptive activation function in 19 years. Specifically, a trainable parameter is introduced in the activation function. Since the topology of the loss function involved in the optimization process will change dynamically, this parameter will be optimized during the training process to achieve the best performance of the network. Compared with the traditional PINN with fixed activation function, PINN based on adaptive activation function has better learning ability, which greatly improves the convergence speed and solution accuracy, especially in the early training.

• A neural network with a fixed activation function is represented as follows:
  $$\begin{array}{rl}& {\mathcal{L}}_k\left(x^{k-1}\right):=w^k{x}^{k-1}+b^k\\ & u_{}(x)=\left({\mathcal{L}}_k\circ p\circ {\mathcal{L}}_{k-1}\circ \dots \circ p\circ L_1\right)(x)\end{array}$$
  In the formula: use a fixed activation function.
• The neural network acquisition based on adaptive parameters adds a parameter before the output passes through the activation function:
  σ ( a L k ( xk − 1 ) ) a ∗ = arg ⁡ min ⁡ a ∈ R + \ { 0 } ( J ( a ) ) \begin{aligned} &\sigma\left(a \mathcal{L}_k\left(x^{k-1}\right)\right) \\ &a^*=\underset{a \in \mathbb {R}^{+} \backslash\{0\}}{\arg \min }(J(a)) \end{aligned}​p(aLk​(xk−1))a∗=a∈R+\{ 0}argmin​( J ( a ) )
  In the formula: variable parameter $a^{\ast }$ will be added to the neural network optimizer, and will be optimized with neural network weight parameters during the training process.

Jagtap A D, Kawaguchi K, Karniadakis G E. Adaptive activation functions accelerate convergence in deep and physics-informed neural networks[J]. Journal of Computational Physics, 2020, 404: 109136.

4. Solving problem definition - inverse problem

$\begin{array}{rl}{u}_t+u{u\; u}_x& =v{u}_{xx},x\in [-1,1],t>0\\ u(x,0)& =-\sin \left(\pi x\right)\\ u(-1,t)& =u(1,t)=0\end{array}$

In the formula: parameter $v$ is an unknown parameter, the real value is$v\in [0,0.1/\pi ]$ .$\_$$\_$The numerical solution is obtained by Hopf-Cole transformation, as shown in Figure 2.
Task requirements:

• The task is to know the boundary conditions and differential equations, but the parameters in the equations are unknown, and solve u and the equation parameters.
• This problem is a typical inverse problem, an inversion problem of optimizing equation parameters.
  ​
  

Figure 2: Burger Numerical Solution

5. Result display

The training process and the parameter change diagram are shown in Figure 3. It can be clearly seen that in the early stage of training, the PINN using the adaptive activation function can descend faster and converge to the exact value.
​

Figure 3: Variation diagram of training process problem parameters and training error

The prediction results during training are shown in Figure 4-6.

Figure 4: Prediction Error Plot

Figure 5: Prediction graph

Figure 6: Forecast results at different times ​