[Continuous learning] Tsinghua's latest continuous learning overview

To cope with the dynamic changes in the real world, an agent needs to acquire, update, accumulate and utilize knowledge incrementally during its life cycle. This ability, known as continuous learning, provides the basis for adaptive development of AI systems.

This article comes from https://arxiv.org/abs/2302.00487

In a general sense, continual learning is clearly limited by catastrophic forgetting, where learning new tasks often results in a sharp drop in performance on old tasks. Beyond that, there have been increasing advances in recent years that have largely expanded the understanding and application of continual learning. The growing and widespread interest in this direction demonstrates its relevance and complexity. This paper presents a comprehensive survey of continuous learning, attempting to draw connections between basic settings, theoretical foundations, representative approaches, and practical applications. Based on existing theoretical and empirical results, the general goal of continuous learning is summarized as: ensuring an appropriate stability-plasticity trade-off and sufficient intra-task/inter-task generalization capabilities in the context of resource efficiency. A state-of-the-art and detailed taxonomy is provided, with an extensive analysis of how representative strategies address continual learning and how they are adapted to specific challenges in various applications. Through in-depth discussions on current trends in continuous learning, cross-directional prospects, and interdisciplinary connections with neuroscience, it is believed that this holistic perspective can greatly facilitate subsequent explorations in this and other fields.

Introduction to Continuous Learning

Learning is the basis for an intelligent system to adapt to its environment. In order to cope with changes in the outside world, evolution has enabled humans and other organisms to have strong adaptability, and can continuously acquire, update, accumulate and utilize knowledge. Naturally, we expect artificial intelligence (AI) systems to adapt in a similar way. This has motivated the study of continual learning, where the typical setting is to learn a sequence of content one by one and behave as if it were observed simultaneously (Fig. 1, a). These can be new skills, new examples of old skills, different environments, different contexts, etc., and include specific real-world challenges. In many literatures, continuous learning is also referred to as incremental learning or lifelong learning , but there is no strict distinction.

Different from traditional machine learning models based on static data distribution, continuous learning is characterized by learning from dynamic data distribution . A major challenge is known as catastrophic forgetting , where adaptation to new distributions often results in a greatly reduced ability to capture old distributions . This dilemma is one aspect of the trade-off between learned plasticity and memory stability: too much of the former interferes with the latter, and vice versa.

Beyond simply balancing the “scale” of these two aspects, an ideal solution for continuous learning should achieve strong generalization to accommodate distributional differences within and across tasks (Fig. 1, b). As a naive baseline, retraining all old training samples (if allowed) can easily solve the above challenges, but incurs huge computational and storage overhead (and potential privacy issues). In fact, the main purpose of continuous learning is to ensure resource-efficient model updates, preferably close to learning only new training samples.

Approaches to address the above challenges can be conceptually divided into five groups (Fig. 1,c):

Add a regularization item with reference to the old model (regularization-based method);

Approximating and restoring old data distributions (replay-based methods);

Explicit manipulation of the optimizer (optimization-based methods);

Learning representations that are robust and generalize well (representation-based methods);

As well as building task-adaptive parameters with a properly designed architecture (architecture-based approach).

This taxonomy extends recent advances in commonly used taxonomies and provides refined subdirections for each category. It summarizes how these methods achieve the general goals proposed, and provides an extensive analysis of their theoretical basis and typical implementations. In particular, these methods are closely linked, e.g. regularization and replay ultimately correct the gradient direction in optimization, and are highly synergistic, e.g., the effect of replay can be improved by distilling knowledge from old models.

现实应用对持续学习提出了特殊的挑战，可以分为场景复杂性和任务特异性。对于前者，例如，在训练和测试中可能缺少任务oracle(即执行哪个任务)，训练样本可能是小批量甚至一次引入的。由于数据标记的成本和稀缺性，持续学习需要在少样本、半监督甚至无监督的场景中有效。对于后者，虽然目前的进展主要集中在视觉分类，但其他视觉领域(如目标检测、语义分割和图像生成)以及其他相关领域(如强化学习(RL)、自然语言处理(NLP)和伦理考虑)正在受到越来越多的关注，其机遇和挑战。

这篇论文主要贡献包括:

(1) 对持续学习进行了最新而全面的综述，以连接理论、方法和应用的进步;

(2) 根据现有的理论和实证结果，总结了持续学习的一般目标，并对具有代表性的策略进行了详细的分类;

(3) 将现实应用的特殊挑战分为场景复杂性和任务特殊性，并广泛分析了持续学习策略如何适应这些挑战;

(4)深入探讨了当前研究趋势和发展方向，以期为相关领域后续工作提供参考。