Lifelong Learning for Language Models

We present an experimental framework along with a suite of benchmarks for lifelong learning using pre-trained language models. Not only is there is a scarcity of lifelong learning benchmarks in the domain of Natural Language Processing, but also none of the available benchmarks frame the lifelong learning problem in the most general form, i.e. having multiple tasks without explicit task identifiers. To this end, we propose the Degree-of-Belief framework which can incorporate multiple tasks without giving away explicit task identifiers. In this framework, the model state its belief in the truth of a statement given a context and its past knowledge, shown in the header image. Using this experimental setup, we design a suite of benchmark data streams consisting of multiple tasks, domains and languages that can be used to investigate, evaluate and experiment with lifelong learning models. Hereby, we release a Lifelong Learning Library to download, transform and organize these datasets into data streams based on our experimental framework for general lifelong learning.