Model/dataset compression for optimizing the efficiency of deep networks

About

Despite the unprecedented success of deep networks in recent years, the massive size of these models and the enormous data required for training them increase the burden on storage, transmission and model training as the latter requires enormous computational costs. Even a trained model requires high performance hardware to perform inference. To reduce the resources required for deep networks, two orthogonal directions have been investigated: 1) Model compression/pruning was suggested to decrease model size of a pre-trained deep network without compromising its accuracy. 2) To improve the efficiency of network training, dataset distillation was proposed to reduce a large dataset into a small synthetic set, such that training a network on the small set is expected to match the accuracy of the original network trained on the original large dataset. Those directions have been approached separately with a different set of tools. We propose to approach both directions with a unified framework. In both cases, we have a large set of items (neurons/filters in a network or examples in datasets) that we want to reduce with minimal information loss. I will show that we can learn a compressed (synthetic) set in both model compression and dataset distillation applications using the ideas inspired from the coreset algorithmic theory. I will also present a first approach for dataset distillation that generalizes to both small and large networks (all previous dataset distillation methods focused on large or even infinite width networks).

Margarita (Rita) Osadchy is a Professor in the Department of Computer Science at the University of Haifa. Her main research interests are deep learning, data science, cyber security and privacy. She was one of the pioneers of deep learning and her work on secure computation of face identification won a Best Paper Award in IEEE Symposium on Security & Privacy. Rita has been a PI on many projects, including grants from the Israeli Ministry of…