Deep learning models are increasingly deployed on edge devices, where they must adapt to new data in dynamic environments. These edge devices are inherently resource-constrained, making model compression techniques such as pruning essential. This involves removing redundant neurons from a model, making it more efficient at the potential cost of accuracy, creating a conflict between efficiency and adaptability. We propose a novel method for training and compressing models that maintains and extends their ability to generalize to new data, enabling better online adaptation without sacrificing compression rates. By pre-training the model on additional knowledge and identifying the parts of the deep neural network that actually encode task-relevant knowledge, we can effectively prune the model by 80% and achieve accuracies that are 16% higher when adapting to a new domain.