1. What is InfiniBand?
2. InfiniBand Architecture
3. Key Features
4. Network Components
5. Bandwidth Evolution
6. NVIDIA Products
7. Customer Applications
8. Routing Engines Deep Dive
9. Troubleshooting Common Issues
10. Essential Commands and Tools
11. Study Tips & Practice Questions

InfiniBand is an industry-standard specification that defines a high-performance input/output architecture for interconnecting compute nodes, storage systems, and networking equipment.

• Traditional TCP/IP: Software-based transport with higher CPU overhead and potential packet loss
• InfiniBand: Hardware-based transport with dedicated pathways, no packet loss, and intelligent traffic management

• OS Independent: Works seamlessly with Linux, Windows, and ESXi
• Application-Centered: Applications communicate directly without heavy OS involvement
• Hardware-Based: Transport protocols are implemented in hardware for maximum performance

InfiniBand's architecture consists of five distinct layers, each with specific responsibilities:

Purpose: Interface between applications and InfiniBand services

Key Protocols:

• MPI (Message Passing Interface) - Standard for parallel computing in HPC
• NCCL (NVIDIA Collective Communication Library) - For AI/ML workloads
• RDMA Storage Protocols - Including SRP, iSER, NFS-RDMA
• IP over InfiniBand (IPoIB) - Runs standard IP applications over InfiniBand
• SDP (Sockets Direct Protocol) - Provides socket-like interface with RDMA benefits

Purpose: Hardware-based transport services

Key Feature: CPU offloading through RDMA (Remote Direct Memory Access)

Three Key Technologies:

1. Hardware-based transport: Network card handles protocol processing
2. Kernel bypass: Data doesn't go through operating system
3. RDMA: Direct memory-to-memory transfer between computers

Advantage: Messages go directly from one application's memory to another's, completely bypassing the CPU

Purpose: Routing between different InfiniBand subnets

Uses: Global IDs (GIDs) for addressing

Key Header: Global Routing Header (GRH) - used when packets need to cross subnet boundaries

Function: Connects multiple InfiniBand subnets via routers

Purpose: Packet switching within a subnet using Local IDs (LIDs)

Key Features:

• LID-based Switching: Packets forwarded based on destination LID
• Credit-based Flow Control: Prevents packet loss by tracking receiver buffer space
• Lossless Networking: Packets are never dropped during normal operation
• Switch Forwarding: Tables map destination LIDs to exit ports

Critical Concept: This layer ensures lossless packet delivery

Purpose: Signal transmission over physical medium

Key Responsibilities:

• Signal Integrity: Ensures reliable electrical/optical signal transmission
• Bit Encoding: How bits are placed on the wire
• Cable Specifications: Defines characteristics for copper and optical cables
• Valid Packet Detection: Signaling protocol to identify valid packets

Lane Aggregation: Multiple physical lanes combined for higher bandwidth

• Formula: Total bandwidth = single lane speed × number of lanes

Data transmission through the layers:

1. Upper Layer: Application initiates data transfer
2. Transport Layer: Automated protocol handling and RDMA processing
3. Network Layer: Route determination for inter-subnet communication
4. Link Layer: Local subnet traffic management and switching
5. Physical Layer: Physical signal transmission

Subnet Manager (SM): Centralized network management

Benefits: Plug-and-play functionality, automatic routing

Key Functions:

• LID Assignment: Assigns Local IDs to ALL nodes (servers and switches)
• Plug-and-Play: Enables automatic node discovery and configuration
• Routing Tables: Calculates and programs forwarding tables in switches
• Topology Discovery: Maps the entire subnet topology
• Redundancy: Master and standby SM for reliability
• Scope: Manages up to 48,000 nodes in a single subnet

Evolution Timeline:

• 2002: 10 Gb/s (Single Data Rate)
• 2008: 40 Gb/s (Quadruple Data Rate)
• 2011: 56 Gb/s (Fourteen Data Rate)
• 2015: 100 Gb/s (Enhanced Data Rate)
• 2018: 200 Gb/s (High Data Rate)
• 2021: 400 Gb/s (Next Data Rate)

Formula: Link rate = single lane speed × number of lanes

Performance: As low as 1 microsecond (1,000 nanoseconds)

Achievement: Hardware offloading eliminates software processing delays

The RDMA mechanism spans both Transport & Upper layers - implemented through BTH (Base Transport Header) and ETH (Extended Transport Header).

• Single subnet: Up to 48,000 nodes
• Multiple subnets: Unlimited scaling with routers

Function: Prioritizes different types of traffic

Implementation: Different applications mapped to different priority queues

Special VL: VL15 is exclusively reserved for subnet management traffic

• Standard Recovery: ~5 seconds (Subnet Manager recalculation)
• NVIDIA Self-Healing: ~1 millisecond (5000x faster!)

Function: Dynamically balances traffic across available paths

Manager: Queue Manager monitors port utilization

Purpose: Performs collective operations in network switches instead of endpoints

Benefit: Up to 10x performance improvement for MPI applications

Function: Connect hosts to InfiniBand network

Example: ConnectX-6 VPI cards (up to 200 Gb/s)

VPI Note: VPI (Virtual Protocol Interconnect) cards can operate in either InfiniBand mode or Ethernet mode

• Edge Switches: QM8700 series (40 ports, 200 Gb/s each)
• Director Switches: CS8500 series (800 ports, complete Fat-Tree in chassis)

• Gateway: Connects InfiniBand to Ethernet networks
• Router: Connects multiple InfiniBand subnets
• Example: Mellanox Skyway gateway

Key Distinction:

• Routers connect InfiniBand subnets (uses GIDs)
• Gateways connect InfiniBand to Ethernet

• DAC (Direct Attach Copper): Short distances, cost-effective
• AOC (Active Optical Cable): Longer distances, higher performance

InfiniBand supports multiple network topologies:

• Structure: Hierarchical tree with increasing bandwidth toward the top
• Benefit: Non-blocking bandwidth between any two nodes

• Structure: Nodes arranged in a grid with wraparound connections
• Benefit: Multiple paths between nodes

• Structure: Hierarchical with local and global groups
• Benefit: Efficient for large-scale deployments

• Default algorithm - invoked automatically if no routing algorithm is specified
• Finds the shortest path between nodes
• Simple and efficient for basic topologies

• Restriction: Does not allow paths that go "down" (away from the roots) and then "up" (toward the roots)
• Requirement: Needs clearly defined root nodes to establish hierarchy
• Fallback: If no root nodes are found, OpenSM falls back to Min-Hop routing

• Best for: Symmetrical or almost symmetrical fat-tree topologies
• Self-healing capability: Can reroute around multiple cable failures and complete switch failures
• Structure: Hierarchical tree with increasing bandwidth toward the top
• Benefit: Non-blocking bandwidth between any two nodes

• Advanced feature: Dynamically balances traffic and reroutes around failures
• Real-time optimization: Monitors congestion and automatically adjusts paths

• Specialized: Uses non-minimal multi-path routing (min-hop+1, min-hop+3) to achieve maximum throughput
• Purpose: Optimized for torus topologies with load balancing

• Structure: Hierarchical with local and global groups
• Benefit: Efficient for large-scale deployments

• Server reboots (topology changes)
• Switch resets (fabric reconfiguration needed)
• Any event requiring complete topology rediscovery and LID reassignment

• Periodic monitoring - runs at regular intervals
• Port status tracking - monitors port state changes
• Performance monitoring (like high transmit packet wait timer values)

Most likely cause: No running Subnet Manager in the fabric

• Physical link is good (LinkUp)
• But port can't complete initialization (Base LID: 0, SM LID: 0)
• Port needs SM to assign LID and transition to Active state

Cause: HCA port is not part of the InfiniBand fabric

• VPI cards can operate in either mode
• Port is working fine but in wrong protocol mode
• Needs reconfiguration from Ethernet to InfiniBand mode

Goal: Assure that packets are not lost within the fabric even in the presence of congestion

• Tracks receiver buffer space
• Prevents packet overflow
• Maintains lossless operation
• This is what makes InfiniBand "lossless"

# Check for NVIDIA/Mellanox HCAs
lspci | grep Mellanox

# Get detailed HCA information including PSID
ibv_devinfo

# Check port status
ibstat

# Discover fabric topology
ibnetdiscover

# Test connectivity between nodes
ibping <destination>

# Trace packet path through fabric
ibtracert <destination>

# Check local port status
ibstatus

# Query current firmware (use PSID to identify suitable firmware)
ibv_devinfo  # Gets PSID for firmware selection

# Firmware upgrade tool
flint -d <device> -i <firmware_file> burn

• OpenSM is embedded in DOCA-OFED - no separate installation needed
• Check OpenSM logs for routing issues:

cat /var/log/opensm.log | grep updn

# Check DOCA-OFED version
ofed_info

# Check MST (Mellanox Software Tools) status
mst status

1. High Performance Computing (HPC): Weather modeling, scientific simulations
2. Artificial Intelligence: Deep learning training and inference
3. Data Science: Big data analytics and processing
4. Cloud Computing: Hyperscale data centers
5. Machine Learning: Model training and deployment

• Supercomputers: Systems with 2K-8K nodes delivering 9-35 PFlops performance
• Cloud Providers: Microsoft Azure HPC/AI cloud infrastructure
• Research Institutions: Universities and national labs worldwide

Architecture Layers (Bottom to Top): "Please Link Networks To Users"

• Physical
• Link
• Network
• Transport
• Upper

• Latency: 1 microsecond (1,000 nanoseconds)
• Bandwidth: 400 Gb/s (current top speed)
• Subnet size: 48,000 nodes max (NOT 4,000!)
• Self-healing: 5000x faster than traditional (1ms vs 5s)
• Maximum packet payload: 4096 bytes (4KB)
• Management VL: VL15 exclusively for subnet management

Component Roles (Frequently Confused):

• Subnet Manager: Assigns LIDs, manages single subnet, enables plug-and-play
• Adaptive Routing: Dynamic load balancing and congestion avoidance
• InfiniBand Router: Connects different InfiniBand subnets (uses GIDs)
• InfiniBand Gateway: Connects InfiniBand to Ethernet networks
• SHARP: Optimizes collective operations (MPI performance)
• Self-Healing: Fast fault recovery (1ms vs 5s traditional)

Layer Responsibilities:

• Physical Layer: Signal integrity, cable specifications, bit transmission
• Link Layer: Flow control, LID-based switching, lossless networking
• Network Layer: Inter-subnet routing with GIDs, uses Global Routing Header (GRH)
• Transport Layer: Hardware-based protocols, RDMA, CPU offloading
• Upper Layer: Application interfaces (MPI, NCCL, SDP, IPoIB, etc.)

Which component assigns LIDs to nodes in the network?

Answer: Subnet Manager

Explanation: The SM is responsible for LID assignment to all nodes (both servers and switches), enabling the plug-and-play functionality that makes InfiniBand easy to manage.

How can traffic be routed between two different InfiniBand subnets?

Answer: By using an InfiniBand router

Explanation: While gateways connect InfiniBand to Ethernet, routers specifically connect different InfiniBand subnets using GIDs for addressing.

What upper layer protocol should be used for parallel computing in HPC?

Answer: MPI (Message Passing Interface)

Explanation: MPI is specifically designed for parallel computing and is the standard for HPC applications requiring coordination between multiple compute nodes.

A server shows "State: Initializing" and "Base LID: 0". What's the most likely cause?

Answer: No running Subnet Manager in the fabric

Explanation: The port has physical connectivity but can't complete initialization because there's no SM available to assign a LID.

❌ Don't Confuse These:

• Router vs Gateway: Router connects IB subnets; Gateway connects IB to Ethernet
• 4,000 vs 48,000: Single subnet supports 48,000 nodes, not 4,000
• 5 seconds vs 1 millisecond: Traditional recovery is 5s; Self-Healing is 1ms
• Flow Control Layer: It's Link Layer, NOT Network Layer
• LID Assignment: Done by Subnet Manager, NOT switches or routers
• Adaptive Routing vs SHARP: AR is for load balancing; SHARP is for collective operations

✅ Remember These Key Phrases:

• "Hardware-based transport services implemented by HCAs"
• "Credit-based flow control ensures lossless networking"
• "Aggregating multiple physical lanes provides higher bandwidth"
• "Subnet Manager assigns LIDs and enables plug-and-play"
• "Physical layer ensures signal integrity"

• Focus on understanding the technical concepts and their relationships
• Practice drawing the five-layer architecture from memory
• Study the bandwidth evolution timeline
• Understand the difference between edge switches and director switches
• Learn the key NVIDIA product families and their capabilities

• HCA: Host Channel Adapter
• SM: Subnet Manager
• LID: Local ID
• GID: Global ID
• RDMA: Remote Direct Memory Access
• QoS: Quality of Service
• SHARP: Scalable Hierarchical Aggregation and Reduction Protocol
• DAC: Direct Attach Copper
• AOC: Active Optical Cable
• VPI: Virtual Protocol Interconnect
• PSID: Parameter Set ID

The key to mastering InfiniBand is understanding that it's designed for speed, efficiency, and scale - everything else follows from these core principles! Whether you're troubleshooting fabric issues, selecting the right routing engine, or optimizing for specific workloads, always consider how each decision impacts these fundamental goals.

The combination of hardware-based transport, lossless networking, and intelligent management makes InfiniBand the backbone of modern HPC, AI, and high-performance computing environments. Master these concepts, and you'll be well-equipped to design, deploy, and troubleshoot even the most complex InfiniBand fabrics.