RMM: Reinforced Memory Management for Class-Incremental Learning

Paper Code Slides Poster Video
Reproduce in the PyCIL Toolbox

Abstract

Class-Incremental Learning (CIL) trains classifiers under a strict memory budget: in each incremental phase, learning is done for new data, most of which is abandoned to free space for the next phase. The preserved data are exemplars used for replaying. However, existing methods use a static and ad hoc strategy for memory allocation, which is often sub-optimal. In this work, we propose a dynamic memory management strategy that is optimized for the incremental phases and different object classes. We call our method reinforced memory management (RMM), leveraging reinforcement learning. RMM training is not naturally compatible with CIL as the past, and future data are strictly non-accessible during the incremental phases. We solve this by training the policy function of RMM on pseudo CIL tasks, e.g., the tasks built on the data of the zeroth phase, and then applying it to target tasks. RMM propagates two levels of actions: Level-1 determines how to split the memory between old and new classes, and Level-2 allocates memory for each specific class. In essence, it is an optimizable and general method for memory management that can be used in any replaying-based CIL method. For evaluation, we plug RMM into two top-performing baselines (LUCIR+AANets and POD+AANets) and conduct experiments on three benchmarks (CIFAR-100, ImageNet-Subset, and ImageNet-Full). Our results show clear improvements, e.g., boosting POD+AANets by 3.6%, 4.4%, and 1.9% in the 25-Phase settings of the above benchmarks, respectively.

Contributions

A hierarchical reinforcement learning algorithm called RMM to manage the memory in a way that can be conveniently modified through incremental phases and for different classes.
A pseudo task generation strategy that requires only in-domain available data (small-scale) or cross-domain datasets (large-scale), relieving the data incompatibility between reinforcement learning and class-incremental learning.
Extensive experiments, visualization, and interpretation for RMM in three CIL benchmarks and using two top models as baselines.

Presentation (subtitles available)

Framework

Fig 1. (a) Existing CIL methods allocate memory between old and new classes in an arbitrary and frozen way, causing the data imbalance between old and new classes and exacerbating the catastrophic forgetting of old knowledge in the learned model. (b) Our proposed method -- Reinforced Memory Management (RMM) -- is able to learn the optimal and class-specific memory sizes in different incremental phases. Please note we use orange, blue and green dots to denote the samples observed in the (i-1)-th, i-th, and (i+1)-th phases, respectively

Performance

Table 1. Average accuracies (%) across all phases using two state-of-the-art methods (LUCIR+AANets and POD+AANets) w/ and w/o our RMM plugged in. The upper block is for recent CIL methods. For fair comparison, we re-implement these methods using our strict memory budget based on the public code.

Reproduce in the PyCIL Toolbox

The PyCIL toolbox provides a reproduced implementation of our RMM.
Pre-training code: [Link]
Incremental learnining code: [Link]

Citation

Please cite our paper if it is helpful to your work:

@inproceedings{Liu2021RMM,
  author    = {Yaoyao Liu and
               Bernt Schiele and
               Qianru Sun},
  title     = {{RMM:} Reinforced Memory Management for Class-Incremental Learning},
  booktitle = {Advances in Neural Information Processing Systems 34: Annual Conference
               on Neural Information Processing Systems 2021, NeurIPS 2021, December
               6-14, 2021, virtual},
  pages     = {3478--3490},
  year      = {2021}
}

Contact

If you have any questions, please feel free to contact us via email: yyliu@cs.jhu.edu.