A PyTorch library for developing energy efficient multiplication-less models.

TorchShiftAdd Overview

Welcome to TorchShiftAdd, your go-to open-source library for crafting energy-efficient multiplication-less models and applications!

TorchShiftAdd embodies a pioneering initiative to simplify and expand the realm of multiplication-less networks within the machine learning community. Key features include:

• Ready-to-use implementation of a wide range of ShiftAdd-based multiplication-less CNNs or Transformers.
• CUDA kernels and TVM compilation support for seamless GPU deployment.
• Profiling tools to furnish FLOPs, energy, and latency breakdown data for in-depth analysis and optimization.
• Hardware accelerator simulators to estimate energy savings and latency improvements on ASICs or FPGAs.
• Flexible support for developing both algorithmic and hardware accelerator designs tailored for multiplication-less networks.

List of Implemented Papers

• ShiftAdd-based Convolutional Neural Networks
  • [NeurIPS'20] ShiftAddNet: A Hardware-Inspired Deep Network
  • [CVPR'20 Oral] AdderNet: Do We Really Need Multiplications in Deep Learning?
  • [CVPR'21 Workshop] DeepShift: Towards Multiplication-Less Neural Networks
• ShiftAdd-based Transformers
  • [NeurIPS'23] ShiftAddViT: Mixture of Multiplication Primitives Towards Efficient Vision Transformer
  • [NeurIPS'24] ShiftAddLLM: Accelerating Pretrained LLMs via Post-Training Multiplication-Less Reparameterization
• Hardware Accelerators for ShiftAdd-based Multiplication-less Networks
  • [ICCAD'22] NASA: Neural Architecture Search and Acceleration for Hardware Inspired Hybrid Networks
  • [IEEE TCAS-I] NASA+: Neural Architecture Search and Acceleration for Multiplication-Reduced Hybrid Networks

Installation

python setup.py install -e .

Qucik Start

Currently codebase supports ShiftAdd-based CNNs. To use them, check our test files:

python test/models/test_resnet20_adder.py
python test/models/test_resnet20_shift.py
python test/models/test_resnet20_shiftadd.py

Upcoming Features

We will continously develop this toolbox:

• ShiftAdd-based Convolutional Neural Networks
• ShiftAdd-based Transformers
• Hardware Accelerators for Energy & Latency Estimation

Contributing

TorchShiftAdd is released under Apache-2.0 License. Everyone is welcome to contribute to the development of TorchShiftAdd. Please refer to contributing guidelines for more details.

Acknowledgement

All co-authors of ShiftAddNet, ShiftAddNAS, ShiftAddViT, and ShiftAddLLM.