How Many GPUs Are Needed?
- GPT-4 was likely trained on somewherebetween 10,000 to 25,000 A100s.
- Meta has about 21,000 A100s,
- Tesla hasabout 7,000 A100s,
- Stability Al hasabout 5,000 A100s.
- Falcon-40B was trained on 384 A100s
- Inflection used 3,500 H100s for their GPT-3.5 equivalent model