Compressing Convolutional Neural Networks in the Frequency Domain (FreshNets)

freshnets

Convolutional neural networks (CNN) are increasingly used in many areas of computer vision. They are particularly attractive because of their ability to "absorb" great quantities of labeled data through millions of parameters. However, as model sizes increase, so do the storage and memory requirements of the classifiers, hindering many applications such as image and speech recognition on mobile phones and other devices. We present a novel network architecture, Frequency-Sensitive Hashed Nets (FreshNets), which exploits inherent redundancy in both convolutional layers and fully-connected layers of a deep learning model, leading to dramatic savings in memory and storage consumption. Based on the key observation that the weights of learned convolutional filters are typically smooth and low-frequency, we first convert filter weights to the frequency domain with a discrete cosine transform (DCT) and use a low-cost hash function to randomly group frequency parameters into hash buckets. All parameters assigned the same hash bucket share a single value learned with standard back-propagation. To further reduce model size we allocate fewer hash buckets to high-frequency components, which are generally less important.

Torch Code Download

For any questions or bug reports, please contact Yixin Chen.

Reference

W. Chen, J. Wilson, S. Tyree, K. Weinberger and Y. Chen, Compressing Convolutional Neural Networks in the Frequency Domain, Proc. ACM SIGKDD Conference (KDD'16), 2016.(PDF)

Acknowledgement

The authors are supported in part by the IIS-1343896, DBI-1356669, III-1526012, IIA-1355406, IIS- 1149882 and EFRI-1137211 grants from the National Science Foundation of the United States, a Microsoft Research New Faculty Fellowship, a Washington University URSA grant, and a Barnes-Jewish Hospital Foundation grant.