Combines the benefits of EWC and path integral-based methods (like SI) by computing parameter importance using both the Fisher information and the contribution of each weight along the optimization path. This unified framework provides more robust importance estimates.
2017
Adds a quadratic penalty term to the loss function that discourages changes to parameters important for previous tasks. Importance is estimated using the diagonal of the Fisher Information Matrix, computed after training on each task. This allows the model to learn new tasks while preserving critical weights.
2017
Computes parameter importance online during training by accumulating the contribution of each weight to the reduction in loss along the optimization trajectory. Unlike EWC which requires a separate importance computation phase, SI tracks importance in real-time, making it more computationally efficient.
2017
Uses knowledge distillation to preserve the predictions of the old model on new task data. Before training on a new task, the current model's outputs on the new data are recorded as soft targets. A distillation loss then ensures the updated model stays close to these original predictions.