User Space Tracepoints (USDT)

A tracepoint is placed in the code and provides a hook to call a function (probe) that you can provide at runtime. This helps to debug applications in real-time production environments.

Information on tracepoints can be found here.

Requirements:

To successfully build libtrace with tracepoints the systemtap-sdt-dev library is required.

systemtap-sdt-dev package is available for Debian / Ubuntu and Centos / RHEL based distros and can be installed with:

Debian / Ubuntu
apt-get install systemtap-sdt-dev

Centos / RHEL
yum install -y systemtap systemtap-devel systemtap-runtime

Libtrace can now be compiled with tracepoints by following Building from source and passing --enable-dtrace to the configure command. Tracepoint support can be confirmed with a configure output which will include:

configure: Compiled with USDT tracepoints: Yes

In order to hook into the tracepoints that are exposed tools such as perf or BPF tools can be used. The following examples use BPF tools which can be installed by following these instructions. We recommend using the development branch of BPF tools and bpftrace as this code is fast-changing and contains fixes for USDT tracepoints.

Available tracepoints:

View all available tracepoints: (replacing /path/to/libtrace.so with the path to libtrace.so)

tplist-bpfcc -vv -l /path/to/libtrace.so

DPDK:

• dpdk_received_packets - The number of packets read by DPDK in this batch.
• dpdk_read_message - During a packet read DPDK did not have any incoming packet but libtrace had a message in its message queue.
• dpdk_delay - DPDK hit a delay intended to promote reading packets in batches.

XDP:

• xdp_received_packets - The number of packets read by XDP in this batch.
• xdp_read_message - During a packet read DPDK did not have any incoming packet but libtrace had a message in its message queue.

Using tracepoints:

It is fairly easy to hook into tracepoints to view the data they expose.

To hook into tracepoints and view the average XDP batch size the following command can be used:

bpftrace -e 'usdt:xdp_received_packets { @batch = avg(arg0); }' -p PID

Or to construct a linear histogram of the number of packets in each XDP batch:

bpftrace -e 'usdt:xdp_received_packets { @packets = lhist(arg0, 0, 64, 2); }' -p PID

Anything within the -e switch is a program that is passed the arguments from the tracepoint arg0,arg1,.. argN. See the following reference guide for more information.

We recommend using the development branch of bpftrace as this code contains fixes for USDT tracepoints.

Adding new tracepoints:

The macro DTRACE_PROBEn(PROVIDER, PROBE, args..) can be used to insert tracepoints into libtrace code. This macro takes the provider name, probe name and arguments as parameters. There is a DTRACE_PROBEn() macro for the number of arguments to be passed.

E.G. DTRACE_PROBE1(libtrace, dpdk_received_packets, nb_rx) n = 1 since we are passing a single argument nb_rx. This particular tracepoint is used to count the number of packets received in each batch when using DPDK to read packets.

A complete example is:

#if ENABLE_DTRACE
    DTRACE_PROBE1(libtrace, dpdk_received_packets, nb_rx);
#enif