Understanding Interrupt Moderation

Version 4

    This post describes interrupt moderation and its benefits in networking.

     

    References

     

    What is Interrupt Moderation?

    Interrupt moderation in networking is any action that changes the 1:1 ratio between packets and interrupts. Interrupts are used by the hardware for alerting the software (networking driver) it has packets waiting to be processed. In practice, this means that interrupt moderation will sometimes cause the hardware to wait before alerting the software about incoming packets.

     

    Interrupt Moderation Implementation

    There are two methods for creating interrupt moderation in networking. One method involves using a counter to count incoming packets. Once enough packets arrive, an interrupt is generated. The second method involves using a timer, which starts after each interrupts. When it expires, it causes an interrupt to "fire".

     

    Using only one of the methods is problematic - a counter without a timer might cause a deadlock, for example when running ping-pong traffic between to nodes. In these cases, one side waits for a response before sending more packets, and the other waits for the hardware to notify it that the packet is arriving, which would only happen if more packets would arrive. Using a timer without a counter is also risky, since high traffic loads might cause the buffers holding the incoming packets to overflow. Usually, both methods are used together in order to cover as many traffic scenarios as possible.

     

    Setting Interrupt Moderation

    Interrupt moderation parameters can be controlled by the user using ethtool. In order to view the current settings use the following:

    # ethtool -c eth4

    Coalesce parameters for eth4:

    Adaptive RX: off  TX: off

    ...

    rx-usecs: 8

    rx-frames: 32

    ...

    tx-usecs: 16

    tx-frames: 32

    Note: Adaptive moderation must be disabled in order to ensure that static values are in use.

    It is possible to change the values for ethtool as well:

    ethtool -C eth4 rx-usecs 0 rx-frames 10 tx-usecs 16 tx-frames 100

    Afterwards, the new values should show:

    # ethtool -c eth4

    Coalesce parameters for eth4:

    Adaptive RX: off  TX: off

    ...

    rx-usecs: 0

    rx-frames: 10

    ...

    tx-usecs: 16

    tx-frames: 100

     

    Interrupt Moderation Advantages

    In order to manage CPU usage efficiently, interrupt moderation proves to be very beneficial. By notifying the software via interrupts that it has pending packets, you can conserve CPU usage. For CPU-limited scenarios (for example maximum single stream packet rates) the CPU time that is freed up will be used to handle higher packet rates. On other occasions, performance is not limited by the CPU, in which case the CPU utilization as well as the operating costs will be reduced.

     

    Possible Disadvantages

    As mentioned before, it is necessary to use both a buffer and a timer in order to prevent issues with interrupt moderation. When setting a system's interrupt moderation to maximize throughput or packet rates, it comes at the expense of latency. Since these scenarios require that packets be buffered, the timer must be set long enough to allow enough packets to arrive. When measuring ping-pong latency in this configuration, the latency might suffer since the hardware must wait for the timer to expire before notifying the software a packet is ready to be processed.