Running PTPd with ConnectX-4

Version 4

    This is a quick overview for using Precision Time Protocol (PTP) to synchronize time in Linux servers with ConnectX-4 hardware time-stamping and with a Grand Master (GM) Clock.

    The following post is a description of PTP, PTPd, and its usages. The setup includes a Grand Master clock connected to a GPS antenna, and a server with ConnectX-4, supporting hardware time-stamping that will act only as a PTP slave. Both are connected to a Mellanox SX6036 switch.

     

    References

     

    Setup

     

    Linux PTP Installation

    1. Download and build PTPd with hardware timestamp support (currently it is not merged to master) as follows:

    # git clone -b wowczarek-2.3.2-hwtest https://github.com/wowczarek/ptpd.git

    ...

    # cd ptpd

    ...

    # autoreconf -vi

    ...

    # ./configure --enable-slave-only

    ...

    # make

    ...

    # sudo make install

    ...

     

    2. Make sure that PTP is enabled on the desired interface (in this example, cx4p1):

    # ethtool -T cx4p1

    Time stamping parameters for cx4p2:

    Capabilities:

            hardware-transmit     (SOF_TIMESTAMPING_TX_HARDWARE)

            software-transmit     (SOF_TIMESTAMPING_TX_SOFTWARE)

            hardware-receive      (SOF_TIMESTAMPING_RX_HARDWARE)

            software-receive      (SOF_TIMESTAMPING_RX_SOFTWARE)

            software-system-clock (SOF_TIMESTAMPING_SOFTWARE)

            hardware-raw-clock    (SOF_TIMESTAMPING_RAW_HARDWARE)

    PTP Hardware Clock: 0

    Hardware Transmit Timestamp Modes:

            off                   (HWTSTAMP_TX_OFF)

            on                    (HWTSTAMP_TX_ON)

    Hardware Receive Filter Modes:

            none                  (HWTSTAMP_FILTER_NONE)

            all                   (HWTSTAMP_FILTER_ALL)

     

    3. Make sure that the GM clock is configured on the same subnet, and that the host is receiving PTP packets:

    # sudo tcpdump -i cx4p1 | grep -i ptp

    tcpdump: verbose output suppressed, use -v or -vv for full protocol decode

    listening on cx4p1, link-type EN10MB (Ethernet), capture size 65535 bytes

    10:55:16.871867 IP 11.11.11.2.ptp-event > ptp-primary.mcast.net.ptp-event: UDP, length 44

    10:55:16.871962 IP 10.10.10.1.ptp-general > ptp-primary.mcast.net.ptp-general: UDP, length 54

    10:55:17.295771 IP 10.10.10.1.ptp-event > ptp-primary.mcast.net.ptp-event: UDP, length 44

     

    Running a PTPd Client

    1. When running ptpd as a client, use the ptpd.conf file (shown below). Change the ptpengine:interface parameter to the relevant interface:

    ; ######### ptpd 2.3.2-linuxphc configuration

    ; #### BEGIN Required settings - need changed

    ; Interface PTPd is running on

    ptpengine:interface=cx4p1

    ; PTP domain number

    ptpengine:domain=0

    ; #### END Required Settings

    ; Enable / disable hardware timestamping and hardware clock sync

    ptpengine:hardware_timestamping=y

    ; #### BEGIN Basic PTP settings

    ptpengine:preset=slaveonly

    ptpengine:delay_mechanism=E2E

    ; other options: hybrid, unicast

    ptpengine:ip_mode=multicast

    ; wait 15 seconds on master changes, let PTP offset stabilise

    ptpengine:calibration_delay=15

    ; #### END Basic PTP settings

    ; #### BEGIN Filter configuration

    ptpengine:sync_outlier_filter_enable=y

    ptpengine:sync_outlier_filter_autotune_enable=y

    ptpengine:sync_stat_filter_enable=y

    ptpengine:sync_outlier_filter_stepdetect_enable=y

    ptpengine:sync_outlier_filter_stepdetect_threshold = 1000000

    ptpengine:sync_outlier_filter_stepdetect_level = 200000

    ptpengine:delay_outlier_filter_enable=n

    ptpengine:delay_outlier_filter_autotune_enable=y

    ptpengine:delay_stat_filter_enable=y

    ptpengine:delay_outlier_filter_stepdetect_enable=y

    ptpengine:delay_outlier_filter_stepdetect_threshold = 1000000

    ptpengine:delay_outlier_filter_stepdetect_level = 200000

    ; #### END Filter Configuration

    ; #### BEGIN Clock stability estimates

    ; If a clock constantly fails to lock (LOCKED state), the low threshold may need lowered

    ; The current value can be observed in the status file

    ; Allan deviation estimates for LOCKED and TRACKING, hardware clocks

    servo:adev_locked_threshold_low_hw = 50.000000

    servo:adev_locked_threshold_high_hw = 500.000000

    ; Allan deviation estimates for LOCKED and TRACKING, software clocks (system clock)

    servo:adev_locked_threshold_low = 200.000000

    servo:adev_locked_threshold_high = 2000.000000

    ; Allan deviation measurement period

    servo:adev_interval = 10

    ; #### END Clock stability estimates

    ; #### BEGIN Other settings

    ; update online statistics every 5 seconds

    global:statistics_update_interval=30

    ; wait 5 statistics intervals for one-way delay to stabilise

    ptpengine:calibration_delay=5

    ; log file, event log only. if timing statistics are needed, see statistics_file

    global:log_file=/var/log/ptpd2.log

    ; log file up to 5M

    global:log_file_max_size=5000

    ; rotate logs up to 5 files

    global:log_file_max_files=5

    ; status file providing an overview of ptpd's operation and statistics

    global:log_status=y

    ; required if ip_mode is set to hybrid

    ;ptpengine:log_delayreq_interval=0

    ; uncomment this to log a timing log like in previous ptpd versions

    ;global:statistics_file=/var/log/ptpd.stats

    ; panic mode: delay when clock step detected

    ptpengine:panic_mode=y

    ptpengine:panic_mode_duration=60

    ; alarm support

    global:enable_alarms=y

    global:alarm_initial_delay=60

    global:alarm_timeout=30

    ; alarm when offset exceeds 100 us

    ptpengine:offset_alarm_threshold=100000

    ; do not allow a software clock (system clock) to be stepped backwards

    clock:allow_step_backwards = N

    ; allow a hardware clock to be stepped backwards

    ; this should be set to Y because a NIC may randomly reset the clock

    clock:allow_step_backwards_hw = Y

    ; bind to selected CPU core for improved system latency characteristics

    ; core 0 = first core

    global:cpuaffinity_cpucore = 0

    ; store best clock frequency in files

    clock:store_frequency = Y

    ; store frequency data in /etc

    clock:frequency_directory = /etc

    ; log file location

    ;global:log_file = /var/log/ptpd.log

    global:statistics_file=/var/log/ptpd2.stats

    ; PI servo parameters (gains)

    ; NOTE: If the system clock is steered by a hardware clock,

    ; software clock gains can be set to the same values as hardware clock gains

    ; software clock (system clock) control

    servo:kp = 0.7

    servo:ki = 0.3

    ; hardware clock control

    servo:kp_hardware = 0.7

    servo:ki_hardware = 0.3

    ; #### END Other settings

    ; ######### ptpd 2.3.2-linuxphc configuration END

    ; always leave a newline in the end

     

     

    2. For better accuracy use an outlier filter with a rate of 16 messages per second from the GM.

    # sudo ptpd -V -c ptpd.conf

    # Timestamp, State, Clock ID, One Way Delay, Offset From Master, Slave to Master, Master to Slave, Observed Drift, Last packet Received, Sequence ID, One Way Delay Mean, One Way Delay Std Dev, Offset From Master Mean, Offset From Master Std Dev, Observed Drift Mean, Observed Drift Std Dev, raw delayMS, raw delaySM

    .......

    2016-04-13 10:39:47.514783, slv, 00b0aefffe01d7d8(unknown)/1,  0.000001095,  0.000000010,  0.000001060,  0.000001106, -7245.271865234, S, 12330, 0.000001095, 0, -0.000000081, 67 0.000001106,  0.000001060

    2016-04-13 10:39:47.577262, slv, 00b0aefffe01d7d8(unknown)/1,  0.000001095,  0.000000010,  0.000001060,  0.000001106, -7245.309365234, S, 12331, 0.000001095, 0, -0.000000081, 67 0.000001106,  0.000001060

    2016-04-13 10:39:47.622410, slv, 00b0aefffe01d7d8(unknown)/1,  0.000001095,  0.000000010,  0.000001060,  0.000001106, -7245.346865234, D, 00943, 0.000001095, 0, -0.000000081, 67 0.000001106,  0.000001060

    2016-04-13 10:39:47.639728, slv, 00b0aefffe01d7d8(unknown)/1,  0.000001095,  0.000000011,  0.000001060,  0.000001108, -7245.346865234, S, 12332, 0.000001095, 0, -0.000000081, 67 0.000001108,  0.000001060

    2016-04-13 10:39:47.689544, slv, 00b0aefffe01d7d8(unknown)/1,  0.000001095,  0.000000011,  0.000001062,  0.000001108, -7245.388115234, D, 00944, 0.000001095, 0, -0.000000081, 67 0.000001108,  0.000001062

    2016-04-13 10:39:47.699912, slv, 00b0aefffe01d7d8(unknown)/1,  0.000001095,  0.000000011,  0.000001060,  0.000001108, -7245.388115234, D, 00945, 0.000001095, 0, -0.000000081, 67 0.000001108,  0.000001060

    2016-04-13 10:39:47.702196, slv, 00b0aefffe01d7d8(unknown)/1,  0.000001095,  0.000000011,  0.000001060,  0.000001106, -7245.388115234, S, 12333, 0.000001095, 0, -0.000000081, 67 0.000001106,  0.000001060

    2016-04-13 10:39:47.764670, slv, 00b0aefffe01d7d8(unknown)/1,  0.000001095,  0.000000007,  0.000001060,  0.000001100, -7245.429365234, S, 12334, 0.000001095, 0, -0.000000081, 67 0.000001100,  0.000001060

    2016-04-13 10:39:47.822065, slv, 00b0aefffe01d7d8(unknown)/1,  0.000001095,  0.000000007,  0.000001057,  0.000001100, -7245.455615234, D, 00946, 0.000001095, 0, -0.000000081, 67 0.000001100,  0.000001057

    2016-04-13 10:39:47.827136, slv, 00b0aefffe01d7d8(unknown)/1,  0.000001095,  0.000000008,  0.000001057,  0.000001108, -7245.455615234, S, 12335, 0.000001095, 0, -0.000000081, 67 0.000001108,  0.000001057

    2016-04-13 10:39:47.889610, slv, 00b0aefffe01d7d8(unknown)/1,  0.000001095,  0.000000009,  0.000001057,  0.000001101, -7245.485615234, S, 12336, 0.000001095, 0, -0.000000081, 67 0.000001101,  0.000001057

    2016-04-13 10:39:47.925997, slv, 00b0aefffe01d7d8(unknown)/1,  0.000001095,  0.000000009,  0.000001057,  0.000001101, -7245.519365234, D, 00947, 0.000001095, 0, -0.000000081, 67 0.000001101,  0.000001057

    2016-04-13 10:39:47.952076, slv, 00b0aefffe01d7d8(unknown)/1,  0.000001095,  0.000000005,  0.000001057,  0.000001099, -7245.519365234, S, 12337, 0.000001095, 0, -0.000000081, 67 0.000001099,  0.000001057

    2016-04-13 10:39:47.957163, slv, 00b0aefffe01d7d8(unknown)/1,  0.000001095,  0.000000005,  0.000001062,  0.000001099, -7245.538115234, D, 00948, 0.000001095, 0, -0.000000081, 67 0.000001099,  0.000001062

    2016-04-13 10:39:48.007753, slv, 00b0aefffe01d7d8(unknown)/1,  0.000001095,  0.000000005,  0.000001064,  0.000001099, -7245.538115234, D, 00949, 0.000001095, 0, -0.000000081, 67 0.000001099,  0.000001064

    2016-04-13 10:39:48.014545, slv, 00b0aefffe01d7d8(unknown)/1,  0.000001095,  0.000000005,  0.000001064,  0.000001101, -7245.538115234, S, 12338, 0.000001095, 0, -0.000000081, 67 0.000001101,  0.000001064

    2016-04-13 10:39:48.020000, slv, 00b0aefffe01d7d8(unknown)/1,  0.000001095,  0.000000005,  0.000001064,  0.000001101, -7245.556865234, D, 00950, 0.000001095, 0, -0.000000081, 67 0.000001101,  0.000001064

    2016-04-13 10:39:48.077017, slv, 00b0aefffe01d7d8(unknown)/1,  0.000001095,  0.000000007,  0.000001064,  0.000001103, -7245.556865234, S, 12339, 0.000001095, 0, -0.000000081, 67 0.000001103,  0.000001064

    2016-04-13 10:39:48.078451, slv, 00b0aefffe01d7d8(unknown)/1,  0.000001095,  0.000000007,  0.000001064,  0.000001103, -7245.583115234, D, 00951, 0.000001095, 0, -0.000000081, 67 0.000001103,  0.000001064

     

    3. If you want to get a statistics summary, run the script shown below. After a period of time, the offset becomes stable at ~20ns.

    # watch -n 1 cat /var/run/ptpd.status

    Every 1.0s: cat /var/run/ptpd.status                                                                                                                     Wed Apr 13 14:06:48 2016

     

    Host info          :  r-aa-hail5, PID 27389

    Local time         :  Wed Apr 13 14:06:48 IDT 2016

    Kernel time        :  Wed Apr 13 11:06:48 GMT 2016

    Interface          :  cx4p1

    Preset             :  slaveonly

    Transport          :  ipv4, multicast, HW PTP

    Delay mechanism    :  E2E

    Sync mode          :  ONE_STEP

    PTP domain         :  0

    Port state         :  PTP_SLAVE

    Local port ID      :  7cfe90fffe596f58(unknown)/1

    Best master ID     :  00b0aefffe01d7d8(unknown)/1

    Best master IP     :  10.10.10.1

    GM priority        :  Priority1 128, Priority2 128, clockClass 6

    Time properties    :  PTP timescale, tracbl: time Y, freq Y, src: GPS(0x20)

    UTC properties     :  UTC valid: Y, UTC offset: 36

    Offset from Master : -0.000000008 s, mean -0.000000004 s, dev  0.000000014 s

    Mean Path Delay    :  0.000001073 s, mean  0.000001073 s, dev  0.000000000 s

    PTP Clock status   :  calibrated, in control, stabilised

    Message rates      :  16/s sync, 16/s delay, 1/2s announce

    TimingService      :  current PTP0, best PTP0, pref PTP0

    TimingServices     :  total 1, avail 1, oper 1, idle 0, in_ctrl 1

    Performance        :  Message RX 46/s, TX 15/s

    Announce received  :  6134

    Sync received      :  196265

    DelayReq sent      :  195632

    DelayResp received :  195632

    State transitions  :  3

    PTP Engine resets  :  1

     

     

    Clock 1: syst      :   name:  syst         state: LOCKED    ref: cx4p1

    Clock 2: cx4p1     :  *name:  cx4p1        state: LOCKED    ref: PTP

    Clock 1: syst      :   offs:  0.000000084   adev: 33.072   freq: -15874.760

    Clock 2: cx4p1     :  *offs:  0.000000008   adev: 1.718    freq: -6789.222