Comparison with synchronous speculative decoding (left) with AMUSD (right). The synchronous approach alternatives between drafting and verifying phases, meaning only model can work at a time. AMUSD uses asynchronous generation running on multiple GPUs enable continuous generation. The draft model must rollback invalid tokens that conflict with the verify model.

Contexts in source publication

Context 1

... Speculative Decoding has shown promise in accelerating language model inference, it still faces limitations due to its synchronous nature. As illustrated in Figure 1 (left), traditional speculative decoding alternates between distinct drafting and verifying phases, allowing only one model to perform computations at any given time. To overcome this constraint, we propose Asynchronous Multi-process Specu- lative Decoding (AMUSD), a novel system that decouples these phases into continuous, parallel operations. ...

Context 2

... overcome this constraint, we propose Asynchronous Multi-process Specu- lative Decoding (AMUSD), a novel system that decouples these phases into continuous, parallel operations. As shown in Figure 1 (right), AMUSD leverages multiple devices (e.g., GPUs) to enable simultaneous and independent predictions from both the draft and verify models. This asynchronous approach allows for continuous generation, with the draft model producing candidate tokens while the verify model concurrently validates previously generated sequences. ...