The dominance of large decoder-only language models has led to a neglect of encoder-decoder architectures, despite their inherent efficiency advantages. A recent analysis reveals that for small language models (SLMs) with 1 billion parameters or fewer, encoder-decoder architectures achieve 47% lower first-token latency and 4.7x higher throughput on edge devices compared to their decoder-only counterparts.
This research suggests that fine-tuning smaller, more efficient models may be a more practical and scalable approach for many deployment scenarios, particularly in resource-constrained environments where the computational overhead of large models is prohibitive. The encoder-decoder architecture's specialized processing and flexible resource distribution offer a compelling alternative to the prevailing trend of ever-larger decoder-only models.