Computes parameter importance online during training by accumulating the contribution of each weight to the reduction in loss along the optimization trajectory. Unlike EWC which requires a separate importance computation phase, SI tracks importance in real-time, making it more computationally efficient.
2017
Adds a quadratic penalty term to the loss function that discourages changes to parameters important for previous tasks. Importance is estimated using the diagonal of the Fisher Information Matrix, computed after training on each task. This allows the model to learn new tasks while preserving critical weights.
2017
Uses knowledge distillation to preserve the predictions of the old model on new task data. Before training on a new task, the current model's outputs on the new data are recorded as soft targets. A distillation loss then ensures the updated model stays close to these original predictions.
2018
Estimates parameter importance based on how sensitive the learned function output is to parameter changes, using the gradient of the squared L2 norm of the network output. Unlike EWC, MAS computes importance in an unsupervised manner, not requiring task-specific labels.