Learns a hard attention mask for each task that selectively gates hidden units. During training on new tasks, a gradient compensation mechanism prevents updates to units that are important for previous tasks. This provides near-zero forgetting with minimal capacity overhead.
2016
Allocates a new neural network column for each new task and freezes all previous columns. Lateral connections between columns allow forward transfer of features from old tasks to new ones, while the frozen parameters guarantee zero forgetting of previously learned tasks.
2018
Iteratively assigns free parameters to new tasks through network pruning. After training on a task, PackNet prunes unimportant weights and freezes the remaining ones, freeing capacity for future tasks. This packing strategy achieves zero forgetting while efficiently utilizing network capacity.
2018
Dynamically decides whether to reuse existing neurons, add new ones, or split neurons when learning a new task. By selectively expanding the network architecture based on task requirements, DEN balances model capacity with task complexity.