Running Linux PTP with ConnectX-4

Version 15

    This is a quick overview for using Linux PTP in order to synchronize time in Linux server with ConnectX-4 HW timestamping and GM Clock in Ethernet mode. The following is a description of PTP, Linux PTP and its usages.

    In this post, the setup includes a GrandMaster clock connected to a GPS Antenna, and a server with ConnectX-4 supporting HW timestamping that will act only as a PTP slave. Both are connected to a Mellanox SX1036 switch.

     

    Note: PTP over IPoIB Interface is supported starting with MLNX_OFED 4.1 (use ib0 instead of Ethernet interface).

     

    References

     

    Setup

     

    Linux PTP Installation

    1. Download and build Linux PTP.

    # git clone git://git.code.sf.net/p/linuxptp/code linuxptp

    ...

    # Make

    ...

    # Make Install

    ...

     

    2. Make sure that PTP is enabled on the desired interface (in this example, cx4p2).

    # ethtool -T cx4p2

    Time stamping parameters for cx4p2:

    Capabilities:

            hardware-transmit     (SOF_TIMESTAMPING_TX_HARDWARE)

            software-transmit     (SOF_TIMESTAMPING_TX_SOFTWARE)

            hardware-receive      (SOF_TIMESTAMPING_RX_HARDWARE)

            software-receive      (SOF_TIMESTAMPING_RX_SOFTWARE)

            software-system-clock (SOF_TIMESTAMPING_SOFTWARE)

            hardware-raw-clock    (SOF_TIMESTAMPING_RAW_HARDWARE)

    PTP Hardware Clock: 0

    Hardware Transmit Timestamp Modes:

            off                   (HWTSTAMP_TX_OFF)

            on                    (HWTSTAMP_TX_ON)

    Hardware Receive Filter Modes:

            none                  (HWTSTAMP_FILTER_NONE)

            all                   (HWTSTAMP_FILTER_ALL)

     

    3. Make sure that GM clock is configured on the same subnet, and the host is getting PTP packets:

    # sudo tcpdump -i cx4p2 | grep -i ptp

    tcpdump: verbose output suppressed, use -v or -vv for full protocol decode

    listening on cx4p2, link-type EN10MB (Ethernet), capture size 65535 bytes

    10:55:16.871867 IP 11.11.11.2.ptp-event > ptp-primary.mcast.net.ptp-event: UDP, length 44

    10:55:16.871962 IP 10.10.10.1.ptp-general > ptp-primary.mcast.net.ptp-general: UDP, length 54

    10:55:17.295771 IP 10.10.10.1.ptp-event > ptp-primary.mcast.net.ptp-event: UDP, length 44

     

    Running Linux PTP Client

    1. Run ptp4l (PTP Client), root privileges is required.

    • -m - print messages
    • -s - act as slave only
    • -i - interface name

    # sudo ptp4l -m -s -i cx4p2

    ptp4l[2320129.544]: selected /dev/ptp0 as PTP clock

    ptp4l[2320129.545]: driver changed our HWTSTAMP options

    ptp4l[2320129.545]: tx_type   1 not 1

    ptp4l[2320129.545]: rx_filter 1 not 12

    ptp4l[2320129.545]: port 1: INITIALIZING to LISTENING on INITIALIZE

    ptp4l[2320129.545]: port 0: INITIALIZING to LISTENING on INITIALIZE

    ptp4l[2320131.390]: port 1: new foreign master 00b0ae.fffe.01d7d8-1

    ptp4l[2320135.390]: selected best master clock 00b0ae.fffe.01d7d8

    ptp4l[2320135.390]: port 1: LISTENING to UNCALIBRATED on RS_SLAVE

    ptp4l[2320137.327]: master offset  176697945 s0 freq   +2857 path delay      1200

    ptp4l[2320138.327]: master offset  176697811 s1 freq   +2723 path delay      1200

    ptp4l[2320139.327]: master offset        -54 s2 freq   +2669 path delay      1200

    ptp4l[2320139.327]: port 1: UNCALIBRATED to SLAVE on MASTER_CLOCK_SELECTED

    ptp4l[2320140.327]: master offset        -46 s2 freq   +2661 path delay      1200

    ptp4l[2320141.327]: master offset         58 s2 freq   +2751 path delay      1154

    ptp4l[2320142.327]: master offset          1 s2 freq   +2711 path delay      1132

    ptp4l[2320143.327]: master offset        -70 s2 freq   +2641 path delay      1141

    ptp4l[2320144.327]: master offset        -63 s2 freq   +2627 path delay      1141

    ptp4l[2320145.327]: master offset        -42 s2 freq   +2629 path delay      1114

    ptp4l[2320146.327]: master offset         -3 s2 freq   +2655 path delay      1114

    ptp4l[2320147.327]: master offset        -46 s2 freq   +2611 path delay      1119

    ptp4l[2320148.327]: master offset        -58 s2 freq   +2586 path delay      1118

     

    2. ptp4l does not adjust the system clock. To do that, you should run phc2sys application in parallel to ptp4l:

    • -s - The interface, in this case, is cx4p2.
    • -c - slave clock, in this case, is system clock.
    • -w - wait until ptp4l sync.
    • -m - print messages.

    # sudo phc2sys -s cx4p2 -c CLOCK_REALTIME -w -m

    phc2sys[2321419.421]: phc offset   -463341 s0 freq  -19250 delay   1401

    phc2sys[2321420.422]: phc offset   -463339 s1 freq  -19248 delay   1407

    phc2sys[2321421.422]: phc offset       -18 s2 freq  -19266 delay   1392

    phc2sys[2321422.422]: phc offset       -57 s2 freq  -19310 delay   1397

    phc2sys[2321423.422]: phc offset        12 s2 freq  -19259 delay   1403

    phc2sys[2321424.423]: phc offset       -20 s2 freq  -19287 delay   1399

    phc2sys[2321425.423]: phc offset       -11 s2 freq  -19284 delay   1425

    phc2sys[2321426.423]: phc offset        19 s2 freq  -19257 delay   1418

    phc2sys[2321427.423]: phc offset        -3 s2 freq  -19274 delay   1420

    phc2sys[2321428.423]: phc offset        43 s2 freq  -19228 delay   1400

    phc2sys[2321429.424]: phc offset       -38 s2 freq  -19297 delay   1407

    phc2sys[2321430.424]: phc offset       -36 s2 freq  -19306 delay   1416