Four Architectural Opportunities for LLM Inference Hardware (Google)

Google published a technical paper titled "Challenges and Research Directions for Large Language Model Inference Hardware," focusing on the difficulties of Large Language Model (LLM) inference. The paper highlights four architectural research opportunities to address challenges in memory and interconnect rather than compute for LLM inference. These opportunities include High Bandwidth Flash for increased memory capacity, Processing-Near-Memory and 3D memory-logic stacking for enhanced memory bandwidth, and low-latency interconnect to improve communication speed. The research primarily targets datacenter AI applications but also considers applicability for mobile devices.