FAQs

Gen-Z specifies a packet protocol. The protocol packets are transported across serial links that connect components. The Gen-Z protocol uses memory-semantic communications to move data between the components.

Memory-semantic communications are used to move data between memories located on different components with minimal overhead. For example, Gen-Z-attached memory can be mapped using a processor memory management unit (MMU) such that any processor load, store, or atomic operation is transparently translated into Gen-Z read, write, or atomic operation and transported to the destination memory component. Similarly, Gen-Z supports buffer put and get operations to move up to 2³² bytes of data between buffers without any processor involvement.

Memory-semantic communications are extremely efficient and simple. Efficiency and simplicity are critical to delivering optimal performance and power consumption, which directly impact customer capital and operational costs.

Each component can support up to 2⁶⁴ bytes of addressable memory. Gen-Z can scale from as few as two components to up to 4096 components per subnet. In theory, the architecture supports up to 2⁷⁶ bytes of memory per subnet and up to 2¹⁶ subnets.

To access data, the component and associated address ranges are advertised to the communicating components. Once advertised and mapped / configured into each communicating component’s address space, the components use memory-semantic communications to directly access each other’s data. Components use low-latency read and write operations to directly access up to 2⁵⁶ bytes of data request,and use a variety of advanced operations to move up to 2³² bytes of data with minimal application or processor involvement.

Gen-Z supports point-to-point, daisy-chain, mesh, and switch-based topologies.

Gen-Z delivers maximum performance without sacrificing flexibility. This enables solutions scalability and flexibility in choice of:

• Interconnect topology and routing algorithms
• Switch design and implementation. Gen-Z supports a wide range of switch radix, buffer capacities, virtual channels, etc. from small to rack-scale solutions.
• Latency will depend upon switch radix. Small radix switches should deliver 10-30 ns latency and large radix switches should deliver 40-60 ns latency

Gen-Z components and existing solution stacks can be transparently supported by unmodified operating systems and application middleware. For example:

• A Gen-Z memory component that supports DRAM or byte-addressable storage class memory media is mapped into an unmodified operating system the same way that a DRAM DIMM component is mapped. Once mapped, the Gen-Z memory component is accessed by applications just as any other DRAM memory component is accessed.
• A Gen-Z I/O component is mapped into an unmodified operating system as a conventional PCI or PCI Express© device. Once mapped, the Gen-Z I/O component can be associated with any existing, unmodified I/O stack and accessed just as a PCI or PCI Express device is accessed. This enables a wide range of I/O device types to be supported through the Gen-Z interconnect.
• A Gen-Z block storage component is mapped into an unmodified operating system and unmodified storage stack either as a PCI NVM device (e.g., NVM Express) or through an I/O host bus adapter (e.g., SAS). Once mapped, the unmodified storage stack is used to access the Gen-Z block storage component as it would any other block storage component.
• A software-emulated NIC (eNIC) is mapped into an unmodified operating system and unmodified IP network stack to enable traditional network applications to transparently operate across a Gen-Z topology. One implementation option is to augment existing virtualized NIC (vNIC) implementations to operate across the topology as though the communication were local to the virtual machine or operating system.
• To support high-speed, low-latency communications, a Gen-Z provider library is mapped underneath the OpenFabrics’ OFI Libfabric infrastructure. This enables a broad set of advanced messaging applications to operate across Gen-Z, e.g., MPI, SHMEM, Sockets, and many more.

All of the above enable Gen-Z components and existing solutions to be constructed using unmodified operating systems and application middleware. As components and solution stacks evolve to take advantage of advanced Gen-Z capabilities and topologies, software changes will be required. For example, rack-scale composable solutions will require new software to manage and coordinate shared memory pools.

Gen-Z specifies a physical layer abstraction that enables it to support multiple physical layers without impacting the higher-level functional blocks. This enables faster deployment of new physical layers and signaling rates. It also enables co-packaged solutions, discrete electrical solutions, and discrete optical solutions.

Gen-Z supports the PCI Express Physical layer. This enables processors, accelerators and I/O components that currently support PCI Express to quickly and readily integrate Gen-Z functionality. Signaling rates from 2.5 GT/s to 32 GT/s can be supported.

Gen-Z builds upon the IEEE 802.3 electrical layer specification. signaling rates from 25 GT/s to 112 GT/s PAM 4 can be supported

To reduce power consumption, complexity, and cost, Gen-Z optimizes the 802.3 electrical layer to support different loss budgets and topologies, e.g., multiple loss budgets for chip-to-chip within an enclosure and fabric-distance loss budgets to connect multiple enclosures. In low-loss solutions, up to 80% power and cost reductions can be achieved.

Gen-Z architecture supports from 1-256 lanes per link. The number of provisioned lanes will vary based on solution requirements and the choice to avoid complex skew compensation technologies. In general, at 56 GT/s and higher signaling rates, the links will be narrower. Though this might limit the maximum bandwidth per link, Gen-Z supports multiple links per component, and thus can scale bandwidth to hundreds of GB/s and beyond.

Gen-Z supports symmetric links where the number of transmit and receive lanes are equal. Gen-Z supports asymmetric links where the number of transmit and receive lanes are not equal. Asymmetric links provide numerous advantages:

• For read-intensive applications, more receive lanes can be provisioned to increase read bandwidth. For example, many applications have a 3:1 read-write ratio, i.e., they issue three read requests for each one write request. Shifting half of the transmit lanes to be receive lanes can increase read bandwidth by
  50%.
• Asymmetric links enable the same physical infrastructure to deliver greater read or write bandwidth, i.e., solutions do not need to use larger connectors, cables, etc. to deliver greater bandwidth. This enables platforms to be adapted to a wider variety of application needs.
• Asymmetric links increase solution resiliency. For example, if a transmit lane fails, only transmit bandwidth is impacted. Receive bandwidth is not impacted since the number of active receive lanes does not need to be reduced to match the number of transmit lanes.

Gen-Z can scale from as few as two components to enclosure to rack scale and beyond. The architecture supports up to 4096 components per subnet and up to 64K subnets. Initial products will be focused on simple, enclosure and rack-scale solutions.

Yes. Gen-Z is designed to work with a wide range of existing, high-volume mechanical form factors and cables including u.2, PCIe CEM, etc. Gen-Z will work with the new DDIMM (Differential DIMM) mechanical form factor for high-speed memory.

Yes. Gen-Z developed a scalable connector specification.

• This scalable connector supports multiple component types and use cases, including: memory (DRAM / NVM), FPGAs, GPUs / GPGPUs, DSPs, I/Os, NICs, SSDs, etc. With sufficient volume, current estimates indicate connector cost should be similar to existing low-cost, high-volume connectors.
• The scalable connector is a high-density connector (0.6 mm pitch) that minimizes board space consumption. Connector size will vary with the number of provisioned lanes, e.g., a NVM module that delivers 10+ GB/s of read and 10+ GB/s of write bandwidth would require 4 transmit and 4 receive lanes operating at 25 GT/s. In contrast, a high-speed memory or I/O module that delivers 100+ GB/s of read and 100+ GB/s of write bandwidth would require 16 transmit and 16 write lanes operating at 56 GT/s. Though these examples illustrate symmetric links, the modules could support asymmetric links to enable greater read or write bandwidth without requiring a larger connector or moving to a higher signaling rate.
• There are three connector sizes:
  • 1C provides 80W of power, management, and 8 differential pairs. Each differential pair can be used as a transmit lane or as a receive lane.
  • 2C is an extension of the 1C, and provides a total of 16 differential pairs.
  • 4C is an extension of the 2C, and provides a total of 32 differential pairs.
• To increase mechanical interoperability, the scalable connector is modular, i.e., any form factor can be inserted into any slot independent of the connector size, e.g., a form factor that supports 32 differential pairs can be inserted into a 1C, 2C, or 4C connector.
• The scalable connector supports vertical and right-angle insertion.
• The scalable connector enables solutions to scale from 2.5 G1T/s NRZ to 112 GT/s PAM 4 signaling.
• The scalable connector supports cable solutions (which will become increasingly important as signaling rates increase and modular solutions become more pervasive).

One of the primary goals of the Gen-Z Consortium is to create an open ecosystem and standards body where all companies can come together to quickly develop new technology. For example, in just a few months, the Gen-Z Mechanical Work Group developed the Gen-Z Scalable Connector’s requirements, translated these into connector concepts, and completed a high-quality draft 0.9 specification. The Consortium also recently published its Core Specification 1.0, which is available for public download.

From its inception, the Gen-Z Consortium wanted to maximize adoption of the Gen-Z Scalable Connector across market segments and use cases. The Gen-Z Consortium published the draft 0.9 specification to the industry in the hopes that multiple industry standards bodies and organizations would reference this connector, thus ensuring that all would benefit. However, due to concerns raised by some companies that are not members of the Gen-Z Consortium, the Board of Directors concluded that in the best interests of the industry, it would support submission of portions of the draft 0.9 specification to the SNIA / SFF organization as a new SFF connector specification, and reference this new SFF connector specification in all applicable Gen-Z Consortium specifications.

Yes. Gen-Z Consortium members have determined that a new mechanical form factor is required to address evolving solution needs. The proposed attributes are:

• The mechanical form factor will be modular to enable solutions to scale in the x, y, and z axis. This will increase solution flexibility and agility while ensuring interoperability across multiple use cases.
• The mechanical form factor can support multiple Gen-Z Scalable Connectors. Each connector provides incremental power, bandwidth and connectivity. For example, a mechanical form factor could support two 4C connectors that provide a total of 160W of power, 64 differential pairs, up to 8 links, etc. Support for multiple connectors simplifies solution development and eliminates mechanical complexity not possible when using alternative technologies.
• Gen-Z architecture enables solutions to scale to 1024 W. Similar to other technologies, a portion of the power will be delivered through the scalable connector(s) and remaining power will be delivered through separate high-capacity power cable attached to the power supply.

Yes. Gen-Z can be used in discrete and co-packaged solutions. In the near-term, the industry will generally focus on discrete solutions using existing mechanical form factors and evolve to use the new mechanical form factor.

Yes. Gen-Z supports cache coherency in point-to-point, meshed, and switch-based topologies. Cache coherency can be used between processors with accelerators, accelerators with accelerators, or accelerators with memory and storage.

Yes. To improve performance and to reduce complexity, Gen-Z cache coherency enables components to selectively determine which communications are coherent and which are non-coherent. For example, communication between a processor and an accelerator could be entirely coherent while communications among a set of accelerators can be selectively coherent.

Gen-Z supports PCI and PCIe technology as follows:

• Gen-Z supports the PCIe physical layer. This enables PCIe solutions to integrate Gen-Z with minimal disruption.
• Gen-Z supports PCI and PCIe configuration space. This enables an unmodified operating system to transparently use Gen-Z I/O components.
• Gen-Z components and switches may support PCIe Compatible Ordering (PCO). This enables an unmodified device driver to be transparently used with a Gen-Z I/O component. Though PCO solutions are constrained by the PCIe architecture, e.g., a single interface and a single path between an I/O component and a host system, they can still take advantage of a subset of the Gen-Z capabilities, e.g., low-latency switching. In contrast, non-PCO solutions can take full advantage of Gen-Z’s capabilities.

Yes. I/O components can take full advantage of Gen-Z capabilities:

• Low-latency switching
• Memory-speed (multiple GB/s) processor-to-device communications
• security and fine-grain hardware-enforced isolation (any-to-any communication without compromise)
• Supports many more atomics than the three supported by PCIe
• Simplified single and multi-host I/O virtualization and sharing capabilities
• Multipath – transparent aggregation, resiliency, and robust topologies
• Very high-speed electrical and optical physical layers
• New Gen-Z Scalable Connector and modular form factors
• CPU-based data movers to enable new software paradigms
• Scale-up and scale-out connectivity and performance, e.g., enables up to 8192 devices per SoC
• Simplified hot-plug management and scaling, e.g., eliminates OS rebalancing complexity that inhibits scalability
• And many more.

Yes. Gen-Z supports multiple processor architecture Atomics including x86 / ARM / Power.

• Supports multiple data sizes – 8, 16, 32, 64, and 128-bit sizes
• Supports integer and floating-point data
• Supports arithmetic and logical operations
• Supports vector atomics, e.g., up to 32 32-bit data values

Yes. Gen-Z supports multiple collective operations including broadcast, barrier, scatter, gather, reduce, map reduce, etc. Further, these collective operations can be implemented in collective accelerators incorporated into a switch topology. Collective accelerators can improve performance, reduce the number of packets exchanged between components, and reduce the probability of congestion.

Yes. The Gen-Z Core specification describes multiple techniques to optimize load-to-use latency that can be incorporated into any design. The specifcation also describes techniques to enable media controller differentiation, solution-specific customizations, optimize power management, and much more.

Gen-Z was created to simplify and unify data access at nearly any scale. At a very high level, Gen-Z provides functionality that supports more topologies, more scalability, and more use cases than what is in the market today.

• Gen-Z supports multiple topologies – co-packaged, point-to-point, mesh, switch-based, and transparent router topologies within a single enclosure or at rack scale.
• Gen-Z supports very robust, flexible, and extensible data access protocol that can access any type and number of components including: processors, memory, I/O, accelerators, storage, FPGA, DSP, etc.
• Gen-Z supports two of the highest volume physical layers – the PCI Express physical layer (up to 16 GT/s today, and 32 GT/s in 2020) and an optimized version of the 802.3 electrical (up to 56 GT/s today, and 112 GT/s in 2020). Gen-Z supports electrical and optical physical media that can be used to support co-packaged, single enclosure, or rack scale solutions.
• Gen-Z uses a highly-optimized, memory semantic protocol that inherently supports multiple use cases including: memory (DRAM and Storage Class Memory (SCM)), I/O – Gen-Z native and logical PCI device support that scales beyond the PCI Express architectural constraints, block and hybrid SCM + block storage, (non-) coherent and hybrid accelerators, Ethernet-compatible networking, low-lateency messaging, coherency, etc.
• Gen-Z supports up to 64-way memory interleaving across a set of point-to-point or switch attached memory components. Memory interleave can be explicitly managed by a memory controller or can be transparently implimented to provide additional capabilities, e.g., multi-host support, highly-resilient memory, transparent management and mechanical servicing, tiered memory, etc.
• Gen-Z supports hardware-enforced isolation and built-in security.
• Gen-Z supports nearly any routing algorithm.
• Gen-Z supports congestion management.