Intelligence Per Watt: Measuring Local Inference Viability, Studying 20+ Models, 8 HW Accelerators (Stanford Univ.)

Researchers at Stanford University and Together AI published a technical paper titled “Intelligence per Watt: Measuring Intelligence Efficiency of Local AI.” The paper explores the viability of local inference using small language models and accelerators like Apple M4 Max. They introduce the concept of intelligence per watt (IPW) as a metric to assess the efficiency of local inference. Through a study involving 20+ local language models and 8 accelerators, they found that local inference can accurately answer real-world queries with improved efficiency, showing potential for redistributing demand from centralized infrastructure.