BERT source code analysis

First, the overall

The entire code file as follows:

Two, tensorflow basis

1.tf.expand_dims

Usage: given tensor "input", this operation is inserted in a size Size 1 "input" size of the shape index "axis" place. Size Index "axis" from scratch; negative number if the specified "axis", starts from the end of the counting.
　　If you want to add to the batch size of a single element, this is useful. For example, if you have a shape of [[height, width, channels] ` image, it can be batch having` expand_dims (image, 0) `of an image together, which will shape [[1 , height, width, channels].

# 't' is a tensor of shape [2]
tf.shape(tf.expand_dims(t, 0))  # [1, 2]
tf.shape(tf.expand_dims(t, 1))  # [2, 1]
tf.shape(tf.expand_dims(t, -1))  # [2, 1]

# 't2' is a tensor of shape [2, 3, 5]
tf.shape(tf.expand_dims(t2, 0))  # [1, 2, 3, 5]
tf.shape(tf.expand_dims(t2, 2))  # [2, 3, 1, 5]
tf.shape(tf.expand_dims(t2, 3))  # [2, 3, 5, 1]
```

This operation requires that:

`-1-input.dims() <= dim <= input.dims()`

This operation is related to `squeeze()`, which removes dimensions of
size 1.

Args:
  input: A `Tensor`.
  axis: 0-D (scalar). Specifies the dimension index at which to
    expand the shape of `input`. Must be in the range
    `[-rank(input) - 1, rank(input)]`.
  name: The name of the output `Tensor`.
  dim: 0-D (scalar). Equivalent to `axis`, to be deprecated.

Returns:
  A `Tensor` with the same data as `input`, but its shape has an additional
  dimension of size 1 added.

Raises:
  ValueError: if both `dim` and `axis` are specified.

bert in the source code:

# The default function of the shape of the input [batch_size, seq_length, input_num]
# If the input is a 2D [batch_size, seq_length], then extended to [batch_size, seq_length, 1]
if input_ids.shape.ndims == 2:
  input_ids = tf.expand_dims(input_ids, axis=[-1])

2.tf.reshape

reshape(tensor, shape, name=None)

Role: Remodeling tensor. Given tensor, the operation returns to the shape of the shape tensor tensor having the same value. If the "shape" of a component of the special value of -1, is calculated for the size of the size, so that the total size remains constant. Specifically, [--1] "shape" of a one-dimensional flattens. "Shape" of a component may at most -1. If "shape" is a one-dimensional or higher, the operation returns to the shape of a "shape" of the tensor, in which filling value "tensor" of. In this case, the number of elements "shapes" implied must be the same number of "tensors" in the element. 

 For example:

For example:
  ```
  # tensor 't' is [1, 2, 3, 4, 5, 6, 7, 8, 9]
  # tensor 't' has shape [9]
  reshape(t, [3, 3]) ==> [[1, 2, 3],
                          [4, 5, 6],
                          [7, 8, 9]]

  # tensor 't' is [[[1, 1], [2, 2]],
  #                [[3, 3], [4, 4]]]
  # tensor 't' has shape [2, 2, 2]
  reshape(t, [2, 4]) ==> [[1, 1, 2, 2],
                          [3, 3, 4, 4]]

  # tensor 't' is [[[1, 1, 1],
  #                 [2, 2, 2]],
  #                [[3, 3, 3],
  #                 [4, 4, 4]],
  #                [[5, 5, 5],
  #                 [6, 6, 6]]]
  # tensor 't' has shape [3, 2, 3]
  # pass '[-1]' to flatten 't'
  reshape(t, [-1]) ==> [1, 1, 1, 2, 2, 2, 3, 3, 3, 4, 4, 4, 5, 5, 5, 6, 6, 6]

  # -1 can also be used to infer the shape
  # -1 is inferred to be 9:
  reshape(t, [2, -1]) ==> [[1, 1, 1, 2, 2, 2, 3, 3, 3],
                           [4, 4, 4, 5, 5, 5, 6, 6, 6]]

  # -1 is inferred to be 2:
  reshape(t, [-1, 9]) ==> [[1, 1, 1, 2, 2, 2, 3, 3, 3],
                           [4, 4, 4, 5, 5, 5, 6, 6, 6]]

  # -1 is inferred to be 3:
  reshape(t, [ 2, -1, 3]) ==> [[[1, 1, 1],
                                [2, 2, 2],
                                [3, 3, 3]],
                               [[4, 4, 4],
                                [5, 5, 5],
                                [6, 6, 6]]]

  # tensor 't' is [7]
  # shape `[]` reshapes to a scalar
  reshape(t, []) ==> 7
  ```
  Args:
    tensor: A `Tensor`.
    shape: A `Tensor`. Must be one of the following types: `int32`, `int64`.
      Defines the shape of the output tensor.
    name: A name for the operation (optional).

  Returns:
    A `Tensor`. Has the same type as `tensor`.
  """

bert in the source code:　　

# If the input is a 2D tensor of shape [batch_size, seq_length], we
# reshape to [batch_size, seq_length, 1].
if input_ids.shape.ndims == 2:
  input_ids = tf.expand_dims(input_ids, axis=[-1])

embedding_table = tf.get_variable(
      name=word_embedding_name,
      shape=[vocab_size, embedding_size],
      initializer=create_initializer(initializer_range))

flat_input_ids = tf.reshape(input_ids, [-1])

　　

 

 

 

references:

[1] BERT real (+ source code analysis stepped pit)