Awesome Pruning

A curated list of neural network pruning and related resources. Inspired by awesome-deep-vision, awesome-adversarial-machine-learning, awesome-deep-learning-papers and Awesome-NAS.

Please feel free to pull requests or open an issue to add papers.

Type of Pruning
A Survey of Structured Pruning
2023 Venues
2022 Venues
2021 Venues
2020 Venues
2019 Venues
2018 Venues
2017 Venues
2016 Venues
2015 Venues

Type of Pruning

Type	`F`	`W`	`S`	`Other`
Explanation	Filter pruning	Weight pruning	Special Networks	other types

A Survey of Structured Pruning (arXiv version and IEEE T-PAMI version)

Please cite our paper if it's helpful:

@article{he2024structured,
  author={He, Yang and Xiao, Lingao},
  journal={IEEE Transactions on Pattern Analysis and Machine Intelligence}, 
  title={Structured Pruning for Deep Convolutional Neural Networks: A Survey}, 
  year={2024},
  volume={46},
  number={5},
  pages={2900-2919},
  doi={10.1109/TPAMI.2023.3334614}}

The related papers are categorized as below: Structured Pruning Taxonomy

2023

Title	Venue	Type	Code
Revisiting Pruning at Initialization Through the Lens of Ramanujan Graph	ICLR	`W`	PyTorch(Author)(Releasing)
Unmasking the Lottery Ticket Hypothesis: What's Encoded in a Winning Ticket's Mask?	ICLR	`W`	-
Bit-Pruning: A Sparse Multiplication-Less Dot-Product	ICLR	`W`	Code Deleted
NTK-SAP: Improving neural network pruning by aligning training dynamics	ICLR	`W`	-
A Unified Framework for Soft Threshold Pruning	ICLR	`W`	PyTorch(Author)
CrAM: A Compression-Aware Minimizer	ICLR	`W`	-
Trainability Preserving Neural Pruning	ICLR	`F`	-
DFPC: Data flow driven pruning of coupled channels without data	ICLR	`F`	PyTorch(Author)
TVSPrune - Pruning Non-discriminative filters via Total Variation separability of intermediate representations without fine tuning	ICLR	`F`	PyTorch(Author)
HomoDistil: Homotopic Task-Agnostic Distillation of Pre-trained Transformers	ICLR	`F`	-
MECTA: Memory-Economic Continual Test-Time Model Adaptation	ICLR	`F`	-
DepthFL : Depthwise Federated Learning for Heterogeneous Clients	ICLR	`F`	-
OTOv2: Automatic, Generic, User-Friendly	ICLR	`F`	PyTorch(Author)
Over-parameterized Model Optimization with Polyak-Lojasiewicz Condition	ICLR	`F`	-
Pruning Deep Neural Networks from a Sparsity Perspective	ICLR	`WF`	PyTorch(Author)
Holistic Adversarially Robust Pruning	ICLR	`WF`	-
How I Learned to Stop Worrying and Love Retraining	ICLR	`WF`	PyTorch(Author)
Symmetric Pruning in Quantum Neural Networks	ICLR	`S`	-
Rethinking Graph Lottery Tickets: Graph Sparsity Matters	ICLR	`S`	-
Joint Edge-Model Sparse Learning is Provably Efficient for Graph Neural Networks	ICLR	`S`	-
Searching Lottery Tickets in Graph Neural Networks: A Dual Perspective	ICLR	`S`	-
Diffusion Models for Causal Discovery via Topological Ordering	ICLR	`S`	-
A General Framework For Proving The Equivariant Strong Lottery Ticket Hypothesis	ICLR	`Other`	-
Sparsity May Cry: Let Us Fail (Current) Sparse Neural Networks Together!	ICLR	`Other`	-
Minimum Variance Unbiased N:M Sparsity for the Neural Gradients	ICLR	`Other`	-

2022

Title	Venue	Type	Code
Parameter-Efficient Masking Networks	NeurIPS	`W`	PyTorch(Author)
"Lossless" Compression of Deep Neural Networks: A High-dimensional Neural Tangent Kernel Approach	NeurIPS	`W`	PyTorch(Author)
Losses Can Be Blessings: Routing Self-Supervised Speech Representations Towards Efficient Multilingual and Multitask Speech Processing	NeurIPS	`W`	PyTorch(Author)
Models Out of Line: A Fourier Lens on Distribution Shift Robustness	NeurIPS	`W`	PyTorch(Author)
Robust Binary Models by Pruning Randomly-initialized Networks	NeurIPS	`W`	PyTorch(Author)
Rare Gems: Finding Lottery Tickets at Initialization	NeurIPS	`W`	PyTorch(Author)
Optimal Brain Compression: A Framework for Accurate Post-Training Quantization and Pruning	NeurIPS	`W`	PyTorch(Author)
Pruning’s Effect on Generalization Through the Lens of Training and Regularization	NeurIPS	`W`	-
Back Razor: Memory-Efficient Transfer Learning by Self-Sparsified Backpropagation	NeurIPS	`W`	PyTorch(Author)
Analyzing Lottery Ticket Hypothesis from PAC-Bayesian Theory Perspective	NeurIPS	`W`	-
Sparse Winning Tickets are Data-Efficient Image Recognizers	NeurIPS	`W`	PyTorch(Author)
Lottery Tickets on a Data Diet: Finding Initializations with Sparse Trainable Networks	NeurIPS	`W`	-
Weighted Mutual Learning with Diversity-Driven Model Compression	NeurIPS	`F`	-
SInGE: Sparsity via Integrated Gradients Estimation of Neuron Relevance	NeurIPS	`F`	-
Data-Efficient Structured Pruning via Submodular Optimization	NeurIPS	`F`	PyTorch(Author)
Structural Pruning via Latency-Saliency Knapsack	NeurIPS	`F`	PyTorch(Author)
Recall Distortion in Neural Network Pruning and the Undecayed Pruning Algorithm	NeurIPS	`WF`	-
Pruning Neural Networks via Coresets and Convex Geometry: Towards No Assumptions	NeurIPS	`WF`	-
Controlled Sparsity via Constrained Optimization or: How I Learned to Stop Tuning Penalties and Love Constraints	NeurIPS	`WF`	PyTorch(Author)
Advancing Model Pruning via Bi-level Optimization	NeurIPS	`WF`	PyTorch(Author)
Emergence of Hierarchical Layers in a Single Sheet of Self-Organizing Spiking Neurons	NeurIPS	`S`	-
CryptoGCN: Fast and Scalable Homomorphically Encrypted Graph Convolutional Network Inference	NeurIPS	`S`	PyTorch(Author)(Releasing)
Transform Once: Efficient Operator Learning in Frequency Domain	NeurIPS	`Other`	PyTorch(Author)(Releasing)
Most Activation Functions Can Win the Lottery Without Excessive Depth	NeurIPS	`Other`	PyTorch(Author)
Pruning has a disparate impact on model accuracy	NeurIPS	`Other`	-
Model Preserving Compression for Neural Networks	NeurIPS	`Other`	PyTorch(Author)
Prune Your Model Before Distill It	ECCV	`W`	PyTorch(Author)
FedLTN: Federated Learning for Sparse and Personalized Lottery Ticket Networks	ECCV	`W`	-
FairGRAPE: Fairness-Aware GRAdient Pruning mEthod for Face Attribute Classification	ECCV	`F`	PyTorch(Author)
SuperTickets: Drawing Task-Agnostic Lottery Tickets from Supernets via Jointly Architecture Searching and Parameter Pruning	ECCV	`F`	PyTorch(Author)
Ensemble Knowledge Guided Sub-network Search and Fine-Tuning for Filter Pruning	ECCV	`F`	PyTorch(Author)
[CPrune: Compiler-Informed Model Pruning for Efficient Target-Aware DNN