Microsoft and Nvidia unveil a 530-billion-parameter model

Microsoft and Nvidia announced Megatron-Turing NLG on 11 October 2021, a 530-billion-parameter transformer language model. It was trained with DeepSpeed and Megatron-LM on 560 Nvidia DGX A100 servers, drawing largely on the Pile and Common Crawl web data.

Why it mattered Dense language models passed GPT-3’s 175 billion parameters within about sixteen months, with the two companies’ software and hardware working together.