LLMs are getting smarter, but they still generate one token at a time. 🐢

For agentic workloads involving long reasoning, tool calls, and code generation, that sequential process results in higher latency, higher compute costs, and lower GPU efficiency.

Speculative decoding helps by letting a smaller drafter propose several tokens for the target model to verify together. DFlash went further by drafting an entire block in parallel.

The remaining challenge is coherence. Each prediction may look right on its own, but the complete sequence may not fit together. Accuracy can also fade toward the end of the block. Once verification encounters a wrong token, the remaining draft is rejected.

DFlash 2 addresses both issues without giving up parallelism. A lightweight path selector chooses a coherent sequence from multiple candidates, while local convolutions help preserve dependencies across the block.

What caught my attention is how much improvement comes from these two lightweight architectural additions.

Inco AI reports 16–25% more output per verification pass for around 1% additional cycle latency. With Qwen3.8-27B, DFlash 2 delivers 2.7–3.4× the throughput of standard autoregressive decoding at batch size 1—while remaining lossless.

Sometimes, the biggest gain comes from changing how tokens are generated. ⚡

Read more 👉 https://lnkd.in/gSkY5iFp

Model checkpoints 👉 https://lnkd.in/gJQKqaSa

#DFlash2 #SpeculativeDecoding #LLMInference #AgenticAI #OpenSourceAI