Theme 2: Next-Generation Architecture - Research Highlights
The Hitchhiker’s Guide to Programming and Optimizing Cache Coherent Heterogeneous Systems: CXL, NVLink-C2C, and AMD Infinity Fabric
Authors:
Zixuan Wang, Suyash Mahar, Luyi Li, Jangseon Park, Jinpyo Kim, Theodore Michailidis, Yue Pan, Mingyao Shen, Tajana Rosing, Dean Tullsen, Steven Swanson, Jishen Zhao. * https://github.com/awesome-
Abstract:
This paper presents a thorough analysis of the use of modern heterogeneous systems interconnected by various cache-coherent links, including CXL, NVLink-C2C, and Infinity Fabric. The paper studied a wide range of server systems (Table 1.) that combined CPUs from different vendors and various types of coherent memory devices, including CXL memory expander (Figure 1(b)(c)), CXL pool, CXL shared memory, GH200 GPU, and AMD MI300a HBM. For this study, the team developed a heterogeneous memory benchmark suite, HEIMDALL, to profile the performance of such heterogeneous systems and present a detailed performance comparison across systems. By leveraging HEIMDALL, the paper unveiled the detailed architecture design in these systems, drew observations on optimizing performance for workloads, and pointed out directions for future development of cache coherent heterogeneous systems. HEIMDALL is open sourced at https://github.com/awesome-cxl/heimdall
Capacity-Latency Tradeoffs in CXL Memory Expander at Hyperscale
Authors:
* Haneul Park, Grant Ayers, Nam Sung Kim, Philip Levis, Brian Morris. IEEE Computer Architecture Letters (CAL), 2026.
Abstract:
DRAM dominates the cost of hyperscale datacenter servers today and AI-driven memory supply shortages are pushing costs even higher. CXL memory expanders are a new technology to reduce server costs by reusing legacy DDR DRAM modules from older servers. Memory expanders further save cost through inline compression that increases the effective capacity (Figure 2(a)). This paper investigates two critical design points of a memory expander: the compression algorithm and the compression block size. Both design points introduce tradeoffs between latency and capacity that involve complex interactions with the memory expander architecture, access patterns to the device, and the entropy of stored data. Current commercial devices use simple, fast compression and a 4 kB block as the default. Using a compressibility benchmark and the latest traces of cold memory accesses from a hyperscaler, we find that using a different compression algorithm and block size can simultaneously increase expander capacity and reduce access latency. The key insight is counter-intuitive: using a slower, more complex compression algorithm improves latency because the reduction in DRAM traffic more than offsets the additional decompression time (Figure 2(b)).
Revisiting Pebble Games for Modeling and Efficient Use of Disaggregated Memory Systems
Authors:
Anusha Devulapally, Mahantesh Halappanavar, Bruno Jacob, Vijaykrishnan Narayanan, Andres Marquez. MEMSYS 2025. https://github.com/UVA-
Abstract:
Introduced in 1981, the Red-Blue game of Hong and Kung uses red (representing limited but fast memory) and blue (representing unlimited but slow memory) pebbles to compute the I/O complexity of an application represented as a directed acyclic graph. Several variants of this game have been designed to address different aspects of I/O performance and system configurations. In contrast, disaggregated memory systems have recently emerged as a means to address poor utilization of memory resources, especially in the context of cloud computing. This position paper posits that pebble games are an effective tool to model performance, as well as design, effective disaggregated memory systems (Figure 3). The paper introduces a novel variant of the game, Dynamic red-orange-blue Pebble Game with multiple processors (DyPeG), to reflect dynamic provisioning of memory resources that is appropriate for multi-tenant cloud computing environments. Using representative real world applications, the paper also provides empirical evaluation to demonstrate the effectiveness of the proposed approach to study disaggregated memory systems.
Architectural Modeling and Benchmarking for Digital DRAM PIM
Authors:
Farzana Ahmed Siddique, Deyuan Guo, Zhenxing Fan, Mohammadhosein Gholamrezaei, Morteza Baradaran, Alif Ahmed, Hugo Abbot, Kyle Durrer, Kumaresh Nandagopal, Ethan Ermovick, Khyati Kiyawat, Beenish Gul, Abdullah Mughrabi, Ashish Venkat, Kevin Skadron. IEEE International Symposium on Workload Characterization (IISWC), 2024.
Abstract:
Processing In Memory (PIM) integrates computational logic units directly into the memory architecture, offering significant performance improvements for memory-bound applications such as matrix operations, vector operations, and database applications compared to general-purpose CPUs or GPUs. However, the lack of a standardized benchmark suite and simulation framework poses a challenge in exploring, evaluating, and designing different PIM architectures. This paper addresses this gap by introducing a comprehensive benchmark suite, PIMbench, along with a performance and energy modeling framework, PIMeval (Figure 4.), both designed to support a wide range of PIM architectures for DRAM. This paper also proposes a set of PIM APIs for writing PIM programs, enabling benchmarks to be executed across different PIM architectures, and allowing for comparing the performance of bit-serial and bit-parallel subarray-level PIM and bank-level PIM. PIMbench and PIMeval have been open-sourced and can be accessed at: https://github.com/UVA-
PIMsynth: A Unified Compiler Framework for Bit-Serial Processing-in-Memory Architectures
Authors:
Deyuan Guo, Mohammadhosein Gholamrezaei, Matthew Hofmann, Ashish Venkat, Zhiru Zhang, Kevin Skadron. IEEE Computer Architecture Letters (CAL), 2025.
Abstract:
Bit-serial processing-in-memory (PIM) architectures have been extensively studied, yet a standardized tool for generating efficient bit-serial code is lacking, hindering fair comparisons. This paper presents a fully automated compiler framework, PIMsynth (illustrated in Figure 5.), for bit-serial PIM architectures, targeting both digital and analog substrates. The compiler takes Verilog as input and generates optimized micro-operation code for programmable bit-serial PIM backends. The proposed flow integrates logic synthesis, optimization steps, instruction scheduling, and backend code generation into a unified toolchain. With the compiler, the paper provides a bit-serial compilation benchmark suite designed for efficient bit-serial code generation. To enable correctness and performance validation, the paper extends an existing PIM simulator to support compiler-generated micro-op-level workloads. Preliminary results demonstrate that the compiler generates competitive bit-serial code within 1.08× and 1.54× of hand-optimized digital and analog PIM baselines.
CHIME: Chiplet-based Heterogeneous Near-Memory Acceleration for Edge Multimodal LLM Inference
Authors:
* Yanru Chen, Runyang Tian, Yue Pan, Zheyu Li, Weihong Xu, Tajana Rosing. Design, Automation & Test in Europe Conference (DATE), 2026.
Abstract:
The proliferation of large language models (LLMs) is accelerating the integration of multimodal assistants into edge devices, where inference is executed under stringent latency and energy constraints, often exacerbated by intermittent connectivity. These challenges become particularly acute in the context of multimodal LLMs (MLLMs), as high-dimensional visual inputs are transformed into extensive token sequences, thereby inflating the key-value (KV) cache and imposing substantial data movement overheads to the LLM backbone. To address these issues, this paper presents CHIME, a chiplet-based heterogeneous near-memory acceleration for edge MLLMs inference. CHIME leverages the complementary strengths of integrated monolithic 3D (M3D) DRAM and RRAM chiplets (Figure 6(a)): DRAM supplies low-latency bandwidth for attention, while RRAM offers dense, non-volatile storage for weights (Figure 6(b)). This heterogeneous hardware is orchestrated by a co-designed mapping framework that executes fused kernels near data, minimizing cross-chiplet traffic to maximize effective bandwidth. On FastVLM (0.6B/1.7B) and MobileVLM (1.7B/3B), CHIME achieves up to 54x speedup and up to 246x better energy efficiency per inference as compared to the edge GPU NVIDIA Jetson Orin NX. It sustains 116.5-266.5 token/J compared to Jetson’s 0.7-1.1 token/J. Furthermore, it delivers up to 69.2x higher throughput than the state-of-the-art PIM accelerator FACIL. Compared to the M3D DRAM-only design, CHIME’s heterogeneous memory further improves energy efficiency by 7% and performance by 2.4x.
Near-Storage Acceleration of MoE Inference on CPU-Only Memory-Constrained Systems
Authors:
Near-Storage Acceleration of MoE Inference on CPU-Only Memory-Constrained Systems. Under submission. SRC 468637.
Abstract:
Abstract to follow, once accepted.
UPS: Unlocking On-Chip Analytics Accelerator for Memory-Efficient Filtered Proximity Graph Search.
Authors:
Jaeyoung Kang, Sujin Kim, Hyunji Kim, Houxiang Ji, Ji-Hoon Kim, Nam Sung Kim.
Abstract:
Retrieval-Augmented Generation (RAG) reduces hallucination in large language models (LLMs) by retrieving external documents from vector databases, where the documents are stored as vectors. However, when retrieval ranks documents by vector distance alone, it cannot enforce constraints on document metadata, and returns contextually irrelevant documents that degrade LLM generation quality. To address this, filtered proximity graph (PG)-based search (FPGS) integrates metadata filtering into PG-based approximate nearest neighbor search (ANNS). This integration requires auxiliary structures to preserve reachability to nodes matching the filter condition, increasing memory footprint. Compression can reduce this overhead, but decompression adds latency to the search critical path. Analytic accelerators, which support both decompression and filtering, can alleviate this overhead by decompressing only the structures a filter needs. However, existing FPGS designs mismatch these accelerators in two ways. They neither store metadata in an accelerator-friendly fixed-width layout nor leave room to overlap decompression with the search.
To bridge these mismatches, we propose UPS (Figure 7.), a hardware-aware design for FPGS. UPS stores metadata and the auxiliary structures in UPS-overlay, a compact compressed layout that converts irregular metadata into a fixed-width format. During graph traversal, UPS leverages Intel In-Memory Analytics Accelerator (IAA) in the latest Intel CPUs to retrieve only the data a query needs instead of decompressing everything in memory, and uses workload-aware synchronization to hide accelerator latency on the critical path. Across vector database datasets, UPS improves search throughput by up to 2.77× over general-purpose filtering baselines at matched recall whil
* “Addressing the complexities of disaggregated datacenters, Theme 2 introduced a hardware-agnostic auto-tuning framework to dynamically optimize ML inference pipelines, successfully increasing throughput by 48% while reducing SLA violations by over 4×”
The same publication listed above: Anusha Devulapally, Mahantesh Halappanavar, Bruno Jacob, Vijaykrishnan Narayanan, Andres Marquez, “Revisiting Pebble Games for Modeling and Efficient Use of Disaggregated Memory Systems”, MEMSYS 2025
Abstract and figure see above.
HMT: Hierarchical Memory Transformer for Long-context Language Processing
* “Theme 2 further advanced LLM generation quality by integrating a sophisticated memory management system into Hierarchical Memory Transformers, lowering perplexity on benchmark suites like ChatQA2 to 11”
Authors:
Zifan He, Zongyue Qin, Neha Prakriya, Yizhou Sun, Jason Cong. NAACL 2025.
Abstract:
Transformer-based large language models (LLM) have been widely used in language processing applications. However, due to the memory constraints of the devices, most of them restrict the context window. Even though recurrent models in previous works can memorize past tokens to enable unlimited context and maintain effectiveness, they have “flat” memory architectures. Such architectures have limitations in selecting and filtering information. Since humans are good at learning and self-adjustment, the paper shows that imitating brain memory hierarchy is beneficial for model memorization. Thus, the paper proposes the Hierarchical Memory Transformer (HMT), a novel framework that facilitates a model’s long-context processing ability by imitating human memorization behavior. Figure 8. shows an overview of HMT workflow. Leveraging memory-augmented segment-level recurrence, the design organizes the memory hierarchy by preserving tokens from early input segments, passing memory embeddings along the sequence, and recalling relevant information from history. Evaluating general language modeling, question-answering tasks, and the summarization task, the paper shows that HMT consistently improves the long-context processing ability of existing models. Furthermore, HMT achieves a comparable or superior generation quality to long-context LLMs with 2 ∼ 57× fewer parameters and 2.5 ∼ 116× less inference memory, significantly outperforming previous memory-augmented models
Stream-HLS: Towards Automatic Dataflow Acceleration
* “Simultaneously, Theme 2 introduced Stream-HLS, an automated framework for streaming dataflow architectures that outperformed prior art by a remarkable 79×, while also finalizing a memory-intensive SAT acceleration chip that leverages multiple clause learning heuristics to maximize on-chip logic utilization.”
Authors:
Suhail Basalama, Jason Cong. FPGA 2025. https://dl.acm.org/doi/10.
Abstract:
High-level synthesis (HLS) has enabled the rapid development of custom hardware circuits for many software applications. However, developing high-performance hardware circuits using HLS is still a non-trivial task requiring expertise in hardware design. Further, the hardware design space, especially for multi-kernel applications, grows exponentially. Therefore, several HLS automation and abstraction frameworks have been proposed recently, but many issues remain unresolved. These issues include: 1) relying mainly on hardware directives (pragmas) to apply hardware optimizations without exploring loop scheduling opportunities. 2) targeting single-kernel applications only. 3) lacking automatic and/or global design space exploration. 4) missing critical hardware optimizations, such as graph-level pipelining for multi-kernel applications. To address these challenges, this paper proposes a novel methodology and framework on top of the popular multi-level intermediate representation (MLIR) infrastructure called Stream-HLS. As illustrated in Figure 9, the framework takes a C/C++ or PyTorch software code and automatically generates an optimized dataflow architecture along with host code for field-programmable gate arrays (FPGAs). To achieve this, the paper presents an accurate analytical performance model for global scheduling and optimization of dataflow architectures. Stream-HLS is evaluated using various standard HLS benchmarks and real-world benchmarks from transformer models, convolution neural networks, and multilayer perceptrons. Stream-HLS designs outperform the designs of prior state-of-the-art automation frameworks and manually-optimized designs of abstraction frameworks by up to 79.43× and 10.62× geometric means respectively. Finally, the Stream-HLS framework is modularized, extensible, and open-sourced at https://github.com/UCLA-
