NEO: Neuron State Dependent Mechanisms for Efficient Continual Learning

Daram, Anurag
Kudithipudi, Dhireesha
Journal Title
Journal ISSN
Volume Title
Association for Computing Machinery

Continual learning (sequential learning of tasks) is challenging for deep neural networks, mainly because of catastrophic forgetting, the tendency for accuracy on previously trained tasks to drop when new tasks are learned. Although several biologically-inspired techniques have been proposed for mitigating catastrophic forgetting, they typically require additional memory and/or computational overhead. Here, we propose a novel regularization approach that combines neuronal activation-based importance measurement with neuron state-dependent learning mechanisms to alleviate catastrophic forgetting in both task-aware and task-agnostic scenarios. We introduce a neuronal state-dependent mechanism driven by neuronal activity traces and selective learning rules, with storage requirements for regularization parameters that grow slower with network size - compared to schemes that calculate weight importance, whose storage grows quadratically. The proposed model, NEO, is able to achieve performance comparable to other state-of-the-art regularization based approaches to catastrophic forgetting, while operating with a reduced memory overhead.

catastrophic forgetting, task agnostic, task incremental learning, domain incremental learning, neuron importance
Daram, A., & Kudithipudi, D. (2023). NEO: Neuron State Dependent Mechanisms for Efficient Continual Learning. Paper presented at the 2023 Annual Neuro-Inspired Computational Elements Conference, San Antonio, TX, USA.
Electrical and Computer Engineering
Computer Science