Paper accepted by ICCV 2017, Venice, Italy [paper][code][poster]. This is a work to make Deep Neural Networks (DNNs) run faster after deployment. We proposed Force Regularization to train DNNs towards lower-rank space for more aggressive low-rank approximation.