Teacher Agent: a knowledge distillation-free framework for rehearsal-based video incremental learning

Rehearsal-based video incremental learning often employs knowledge distillation to mitigate catastrophic forgetting of previously learned task. However, applying this method directly to video incremental tasks faces two major challenges. Firstly, learning video sequences alone requires substantial computing resources, and loading the network from the previous stage for experience replay further exacerbates the burden. Secondly, the knowledge review capability of the network is constrained by the performance of the teacher network from the previous stage. We revisit this issue and empirically confirm that inaccurate predictions by the teacher network for some memorized exemplars directly limit the performance of knowledge review. To address these issues, we first propose a knowledge distillation-free framework for rehearsal-based video incremental learning, which we term Teacher Agent. Instead of loading parameter-heavy teacher networks, we introduce an agent generator that is either parameter-free or uses only a few parameters to obtain accurate and reliable soft labels. This method not only greatly reduces the computing requirement but also circumvents the problem of knowledge misleading caused by inaccurate predictions of the teacher model. Moreover, we put forward a self-correction loss which provides an effective regularization signal for the review of old knowledge, which in turn alleviates the problem of catastrophic forgetting. Further, to ensure that the samples in the memory buffer are memory-efficient and representative, we introduce a unified sampler for rehearsal-based video incremental learning to mine fixed-length key video frames. Interestingly, based on the proposed strategies, the network exhibits a high level of robustness against spatial resolution reduction. Extensive experiments demonstrate the advantages of our method, yielding significant performance improvements while utilizing only half the spatial resolution of video clips as network inputs in the incremental phases.