Fast Inference from Transformers via Speculative Decoding

This research paper introduces a new technique called speculative decoding that aims to accelerate inference from large autoregressive models like Transformers. The core idea is to use a smaller, more efficient model to generate potential continuations of a text sequence, which are then evaluated by the larger model in parallel. This process, called speculative sampling, can lead to significant speedups, especially when computational resources are abundant and memory bandwidth is the bottleneck. The authors demonstrate the effectiveness of their approach by applying it to T5-XXL and achieving a 2X-3X acceleration compared to standard implementations. They also provide a detailed analysis of the method's performance, including the factors influencing the speedup and the trade-off between speed and computational cost.