Compute Express Link
An open industry standard built for high speed low latency communication between processors (CPU), memory expansion devices, and accelerators (GPUs). It uses PCIe1 as the physical layer and adds 3 protocols on top of it.
- CXL.io for device discovery, configuration and non-coherent I/O.
- CXL.cache: Allows devices to access host CPU memory with cache coherency. MESI protocol.
CXL.mem allows to access memory of a device with cache coherency.
Can be used to access memory expansion devices as if it was RAM.
CXL 3.0 (introduced in 2022/23) uses PBR (Port Based Routing) to do data transfer. Each host and device gets a 12-bit port id. When a memory transaction happens, the switch looks at target memory address and maps to the destination port. It then wraps the flit2 with source and destination port and routes the packet to the correct node using a routing table.
The routing table is updated by a Fabric3 Manager which has global idea about topology of the switches and nodes. It can decide best route and update the table dynamically if any node goes offline.
This way GPUs can directly talk to other GPUs without going through host CPU. Previously, HBR (Hierarchy-based routing) required data to go through host CPU.
Footnotes:
PCIe (Peripheral Component Interconnect Express) is a general interface to connected devices in the motherboard.
The unit of data transfer in CXL is called a Flit (Flow Control Units).
Fabric: In Computer architecture and networking, a fabric is a topology where multiple devices interconnect through one or more switches. In contrast to a bus, multiple simultaneous communication can be happening.