tensorflow2.0 uses GELU activation function

First of all, tensorflow2.0 does not have a GELU activation function, so the GELU activation function must be customized.

Secondly, the custom code is as follows:

Step one: import tensorflow as tf

Step 2: class GELU(layers.Layer):
    def __init__(self):
        super(GELU, self).__init__()

    def call(self, x):
        cdf = 0.5 * (1.0 + tf.tanh((np.sqrt(2 / np.pi) * (x + 0.044715 * tf.pow(x, 3)))))
        return x * cdf

third step:

self.act=GRLU()

Step 4: (used in call)

x = self.act(x)