OpenAI publishes GPT-3, a 175-billion-parameter language model

Adapting a language model to a job, in the years before this paper, usually meant fine-tuning. A network pretrained on general text was trained further on thousands of labeled examples of one specific task, and the resulting copy did that task. Every task needed its own dataset and its own trained model.

“Language Models are Few-Shot Learners” tested a different route. OpenAI researchers led by Tom B. Brown trained a single autoregressive model with 175 billion parameters, ten times the size of the largest previous non-sparse model and built on the transformer architecture Google had published three years earlier. They then measured it with no gradient updates and no fine-tuning at all. The task was set out in the prompt itself, sometimes described in words alone, sometimes with a handful of worked examples, and the model continued the text.

It translated, answered questions, did arithmetic and unscrambled words this way. Accuracy generally rose both with the number of examples placed in the prompt and with the size of the model, which was the paper’s central claim: capability was arriving as a function of scale rather than of task-specific training.

The paper also reported that human evaluators had difficulty telling GPT-3’s news articles from ones written by people, and it discussed misuse and the biases the model reproduced from the text it was trained on.

OpenAI did not publish the weights. Access arrived two weeks later as a private API beta, and in September Microsoft licensed the model exclusively. The argument that scale alone bought general capability set the commercial direction of the years that followed. The “Stochastic Parrots” paper the following March argued that the costs of that scale were being left out of the account.