Researchers at Percepta demonstrate that language models (LLMs) can internally execute programs by embedding a computer within a transformer architecture, bypassing typical reliance on external tools for computation. By implementing an efficient decoding scheme that leverages 2D attention heads, their approach performs execution traces with logarithmic-time attention lookups, enabling millions of computational steps within a single transformer run. This enables reliable, step-by-step internal computation of complex tasks like solving Sudoku or min-cost perfect matching, marking a significant advancement in LLMs’ ability to perform exact, long-horizon computations autonomously.
Can LLMs Be Computers?
