[nlp] Data Parallel & Model Parallel

data parallelism

Data Parallel DP

There are two types of model parallelism: pipeline parallelism and tensor parallelism

Model parallelism - pipeline parallelism :

        Put different layers on different gpus

        model.parallelize()

Model Parallelism - Tensor Parallelism :

        Split the same layer into different GPUs