How do Gen-Z components communicate with one another?
What are memory-semantic communications?
Why are memory-semantic communications needed?
How much memory can a Gen-Z component access?
How do Gen-Z components access data?
What topologies does Gen-Z support?
Does Gen-Z sacrifice switch fabric flexibility for performance?
Does Gen-Z require operating system or software middleware changes to work?
Does Gen-Z specify a new physical layer technology?
What is different about Gen-Z’s physical layer?
What signaling rates does Gen-Z support?
How many lanes per link does Gen-Z support?
Does Gen-Z support symmetric and asymmetric links?
How scalable is Gen-Z with respect to the number of end-points it can support?
Does Gen-Z work with existing mechanical form factors and cables?
Is Gen-Z developing a new connector and mechanical form factor?
Does Gen-Z support co-packaged solutions?
Does Gen-Z support cache coherency?
Q: How do Gen-Z components communicate with one another?
A: Gen-Z specifies a packet protocol. The protocol packets are transported across serial links that connect components. The Gen-Z protocol uses memory-semantic communications to move data between the components.
Q: What are memory-semantic communications?
A: Memory-semantic communications are used to move data between buffers located on different components with minimal overhead. For example, Gen-Z-attached memory can be mapped using a processor memory management unit (MMU) such that any processor load, store, or atomic operation is transparently translated into Gen-Z read, write, or atomic operation and transported to the destination memory component. Similarly, Gen-Z supports buffer put and get operations to move up to 232 bytes of data between buffers without any processor involvement.
Q: Why are memory-semantic communications needed?
A: Memory-semantic communications are extremely efficient and simple. Efficiency and simplicity are critical to delivering optimal performance and power consumption, which directly impact customer capital and operational costs.
Q: How much memory can a Gen-Z component access?
A: Each component can support up to 264 bytes of addressable memory. Gen-Z can scale from as few as two components to up to 4096 components per subnet. In theory, the architecture supports up to 276 bytes of memory per subnet.
Q: How do Gen-Z components access data?
A: To access data, the component and associated address ranges are advertised to the communicating components. Once advertised and mapped / configured into each communicating component’s address space, the components use memory-semantic communications to directly access each other’s data. Components use low-latency read and write operations to directly access up to 256 bytes of data request,and use a variety of advanced operations to move up to 232 bytes of data with minimal application or processor involvement.
Q: What topologies does Gen-Z support?
A: Gen-Z supports point-to-point, daisy-chain, and switch-based topologies.
Q: Does Gen-Z sacrifice switch fabric flexibility for performance?
A: Gen-Z delivers maximum performance without sacrificing flexibility. This enables solutions scalability and flexibility in choice of:
- Interconnect topology and routing algorithms
- Switch design and implementation. Gen-Z supports a wide range of switch radix, buffer capacities, virtual channels, etc. from small to rack-scale solutions.
Q: Does Gen-Z require operating system or software middleware changes to work?
A: Gen-Z components and existing solution stacks can be transparently supported by unmodified operating systems and application middleware. For example:
- A Gen-Z memory component that supports DRAM or byte-addressable NVM media is mapped into an unmodified operating system the same way that a DRAM DIMM component is mapped. Once mapped, the Gen-Z memory component is accessed by applications just as any other DRAM memory component is accessed.
- A Gen-Z I/O component is mapped into an unmodified operating system as a conventional PCI device. Once mapped, the Gen-Z I/O component can be associated with any existing, unmodified I/O stack and accessed just as a PCI device is accessed. This enables a wide range of I/O device types to be supported through the Gen-Z interconnect.
- A Gen-Z block storage component is mapped into an unmodified operating system and unmodified storage stack either as a PCI NVM device (e.g., NVM Express) or through an I/O host bus adapter (e.g., SAS). Once mapped, the unmodified storage stack is used to access the Gen-Z block storage component as it would any other block storage component.
- A software NIC (an extension to the existing vNIC software implementations) is mapped into an unmodified operating system and unmodified IP network stack to enable traditional network applications to transparently operate across a Gen-Z topology. To support high-speed, low-latency communications, a Gen-Z provider library is mapped underneath the OpenFabrics’ OFI Libfabric infrastructure. This enables a broad set of advanced messaging applications to operate across Gen-Z, e.g., MPI, SHMEM, Sockets, and many more.
All of the above enable Gen-Z components and existing solutions to be constructed using unmodified operating systems and application middleware. As components and solution stacks evolve to take advantage of advanced Gen-Z capabilities and topologies, software changes will be required. For example, rack-scale composable solutions will require new software to manage and coordinate shared memory pools.
Q: Does Gen-Z specify a new physical layer technology?
A: Gen-Z leverages the IEEE 802.3 physical layer specifications. The Gen-Z members chose the IEEE 802.3 physical layer as its foundation for multiple reasons:
- IEEE 802.3 physical layer is one of the highest-volume technologies, with broad adoption across multiple market segments and deployed throughout the world.
- IEEE 802.3 technology development is driven by hundreds of companies and thousands of developers.
- IEEE 802.3 physical layer IP blocks are readily available from multiple suppliers. Further, these IP blocks are constantly evolving to deliver new capabilities and performance levels.
- IEEE 802.3 physical layer is supported by many test equipment vendors.
- IEEE 802.3 physical layer can scale from chip-to-chip, enclosure-to-enclosure, and rack-to-rack.
- IEEE 802.3 physical layer presently scales up to 56 GT/s, and will scale to 112 GT/s and beyond.
Q: What is different about Gen-Z’s physical layer?
A: To reduce power consumption, complexity, and cost, Gen-Z optimizes the physical layer to support different loss budgets and topologies, e.g., multiple loss budgets for chip-to-chip within an enclosure and fabric-distance loss budgets to connect multiple enclosures. In low-loss solutions, up to 80% power and cost reductions can be achieved.
Q: What signaling rates does Gen-Z support?
A: Gen-Z specifies multiple signaling rates: 16 GT/s, 25 GT/s, 28 GT/s, 56 GT/s (PAM 4), and 112 GT/s (PAM 4). Further, Gen-Z specifies a physical layer abstraction that enables new physical layers and signaling rates to be specified without impacting the higher layer functional blocks. This abstraction enables co-packaged solutions, discrete electrical solutions, and discrete optical solutions.
Q: How many lanes per link does Gen-Z support?
A: Gen-Z architecture supports from 1 to 256 lanes per link. The number of provisioned lanes will vary based on solution requirements and the choice to avoid complex skew compensation technologies. In general, at 56 GT/s and higher signaling rates, the links will be narrower. Though this might limit the maximum bandwidth per link, Gen-Z supports multiple links per component, thus can scale bandwidth to hundreds of GB/s and beyond.
Q: Does Gen-Z support symmetric and asymmetric links?
A: Gen-Z supports symmetric links where the number of transmit and receive lanes are equal. Gen-Z supports asymmetric links where the number of transmit and receive lanes are not equal. Asymmetric links provide numerous advantages:
- For read-intensive applications, more receive lanes can be provisioned to increase read bandwidth. For example, many applications have a 3:1 read-write ratio, i.e., they issue three read requests for each one write request. Shifting half of the transmit lanes to be receive lanes can increase read bandwidth by
- Asymmetric links enable the same physical infrastructure to deliver greater read or write bandwidth, i.e., solutions do not need to use larger connectors, cables, etc. to deliver greater bandwidth. This enables platforms to be adapted to a multitude of application needs.
- Asymmetric links increase solution resiliency. For example, if a transmit lane fails, only transmit bandwidth is impacted. Receive bandwidth is not impacted since the number of active receive lanes does not need to be reduced to match the number of transmit lanes.
Q: How scalable is Gen-Z with respect to the number of end-points it can support?
A: Gen-Z can scale from as few as two components to enclosure to rack scale and beyond. The architecture supports up to 4096 components per subnet and up to 64K subnets. Initial products will be focused on simple, enclosure and rack-scale solutions.
Q: Does Gen-Z work with existing mechanical form factors and cables?
A: Yes. Gen-Z being designed to work with a wide range of existing, high-volume mechanical form factors and cables.
Q: Is Gen-Z developing a new connector and mechanical form factor?
A: Yes. Gen-Z will develop a new modular connector and modular mechanical form factor focused on high-volume solutions. These will provide the following benefits:
- The connector will support any component type including: memory (DRAM / NVM), FPGAs, GPUs / GPGPUs, DSPs, I/Os, NICs, etc. With sufficient volume, current estimates indicate connector cost should be similar to existing low-cost, high-volume connectors.
- The connector will be a high-density connector that minimizes board space consumption. Connector size will vary with the number of provisioned lanes, e.g., a NVM module that delivers 10+ GB/s of read and 10+ GB/s of write bandwidth would require 4 transmit and 4 receive lanes operating at 25 GT/s. In contrast, a high-speed memory or I/O module that delivers 100+ GB/s of read and 100+ GB/s of write bandwidth would require 16 transmit and 16 write lanes operating at 56 GT/s. Though these examples illustrate symmetric links, the modules could support asymmetric links to enable greater read or write bandwidth without requiring a larger connector or moving to a higher signaling rate.
- To enable aggregate bandwidth and resiliency, a connector can support multiple links. Though the number of transmit and receive lanes are provisioned for each additional link, pins associated with reference clock, I3C, etc. are shared.
- To increase mechanical interoperability, the connector will be modular to enable components with a smaller number of lanes to plug into slots with a larger number of lanes, or components with a larger number of lanes to plug into slots with fewer lanes.
- The connector will support vertical and horizontal insertion.
- The new connector will enable solutions to scale from 16 GT/s NRZ to 112 GT/s PAM 4 signaling.
- The mechanical form factor will be modular to enable solutions to scale in the x, y, and z axis.
- Gen-Z architecture enables solutions to scale to 1024 W. Similar to other technologies, a portion of the power will be delivered through the new connector and the remaining power will be delivered through a separate high-capacity power cable attached to the power supply.
Q: Does Gen-Z support co-packaged solutions?
A: Yes. Gen-Z can be used in discrete and co-packaged solutions. In the near-term, the industry will generally focus on discrete solutions using existing mechanical form factors and evolve to use the new mechanical form factor.
Q: Does Gen-Z support cache coherency?
A: Gen-Z does not specify cache coherent agent operations. Gen-Z does specify coherency protocol operations that facilitate cache coherent agents, and can transparently transport vendor-specific cache coherency protocols.