Wireshark support for NVMe over Fabrics

Version 3

    This post discusses Wireshark support for NVMe over Fabrics (NVMe-oF). Wireshark support for NVMe-oF (over RDMA) was added in January 2017 to the master branch, and can be used starting with Version 2.4.0rc1.

     

    References

     

     

    Usage

    1. Install Wireshark version  2.40rc1 or later.

     

     

    2. Load an NVMe-oF wireshark recording (*.pcap, *.pcapng) file.

     

     

    Packet Format

     

    1. Here are the expected headers seen on the wire for NVMe over RDMA over UDP/IP.

     

    • Ethernet
    • IP
    • UDP
    • InfiniBand - this header includes the BTH, and may have other sub-headers such as RETH, IETH.
    • NVMe-oF - this header encapsulates the  NVMe command capsule.
    • NVMe

     

     

     

     

    2. Once Wireshark is opened, start NVMe-oF recording. Use the Filter options to filter the traffic to nvme or nvme-rdma headers.

     

     

     

    3. Here is an NVMe read command example. Use the filter nvme.cmd.opc ==2 for NVMe read.

     

     

    4. Attached are two Wireshark recordings, one that has all the headers (NVMe over RDMA over UDP/IP), and another  with only InfiniBand headers (different setups).