Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Perf event writes metric #4334

Merged
merged 3 commits into from
Dec 3, 2024

Conversation

geyslan
Copy link
Member

@geyslan geyslan commented Sep 27, 2024

1. Explain what the PR does

bae1532 chore(performance): update tracee dashboard
5a5c2e7 chore: add perfbuf metric per event (METRICS=1)
5fe36d1 chore(go.mod): bump api to latest 1f796cb

5a5c2e7 chore: add perfbuf metric per event (METRICS=1)

Enabled only when built with METRICS=1.

BPFPerfEventSubmitAttemptsCount and BPFPerfEventSubmitFailuresCount
count the number of events processed by the eBPF programs and written to
or attempted to be written to the perf buffer.

It is incremented right after the attempt of writing the event to the
perf buffer, making it possible to measure if the that event was
successfully written to the perf buffer or not.

This metric can be used to monitor the performance of individual eBPF
events and to detect potential bottlenecks.

image

2. Explain how to test it

METRICS=1 make tracee

sudo ./dist/tracee -e socket_dup,execve --metrics --pyroscope --pprof

make -f builder/Makefile.performance dashboard-start

3. Other comments

go.mod Outdated Show resolved Hide resolved
api/v1beta1/diagnostic.proto Outdated Show resolved Hide resolved
Makefile Outdated Show resolved Hide resolved
@geyslan

This comment was marked as outdated.

@geyslan geyslan force-pushed the perf-event-writes-metric branch 2 times, most recently from b3a3181 to fadf5a8 Compare October 1, 2024 20:39
@geyslan geyslan marked this pull request as draft October 7, 2024 10:49
@geyslan
Copy link
Member Author

geyslan commented Oct 7, 2024

I'm working on a stress tool on top of this branch. Gonna let you know soon.

@NDStrahilevitz
Copy link
Collaborator

After offline discussion, registering another way of approaching this: by enabling the metric with a flag instead of requiring a rebuild.

It's feasible considering an increment of program sizes, an additional branch check (metric enabled) and a dangling map.

I actually prefer having this in a debug build precisely so that we don't include these overheads in regular user runtimes.

@geyslan geyslan force-pushed the perf-event-writes-metric branch from fadf5a8 to 5d46a84 Compare October 14, 2024 21:25
@geyslan geyslan force-pushed the perf-event-writes-metric branch 2 times, most recently from fac164c to 7633d97 Compare October 15, 2024 14:35
@geyslan geyslan marked this pull request as ready for review October 15, 2024 14:36
@geyslan geyslan force-pushed the perf-event-writes-metric branch 2 times, most recently from a7412c1 to d58139d Compare October 15, 2024 14:39
@geyslan geyslan force-pushed the perf-event-writes-metric branch from d58139d to f1e3b2b Compare October 15, 2024 14:45
pkg/metrics/collector.go Outdated Show resolved Hide resolved
@geyslan geyslan force-pushed the perf-event-writes-metric branch from f1e3b2b to cbe8510 Compare October 15, 2024 20:20
@geyslan geyslan force-pushed the perf-event-writes-metric branch from cbe8510 to c033971 Compare October 16, 2024 20:47
@geyslan geyslan requested a review from yanivagman October 21, 2024 12:07
@geyslan geyslan requested a review from rscampos November 5, 2024 13:54
@geyslan
Copy link
Member Author

geyslan commented Nov 26, 2024

@NDStrahilevitz @yanivagman I would appreciate your +1.

@geyslan geyslan force-pushed the perf-event-writes-metric branch from c033971 to 9ba2407 Compare December 2, 2024 13:57
Copy link
Collaborator

@yanivagman yanivagman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, see one nit comment

pkg/ebpf/tracee.go Outdated Show resolved Hide resolved
@geyslan
Copy link
Member Author

geyslan commented Dec 2, 2024

LGTM, see one nit comment

I'll add a new commit to this to expose the new metrics via grpc. Gonna ping you soon.

Enabled only when built with METRICS=1.

BPFPerfEventSubmitAttemptsCount and BPFPerfEventSubmitFailuresCount
count the number of events processed by the eBPF programs and written to
or attempted to be written to the perf buffer.

It is incremented right after the attempt of writing the event to the
perf buffer, making it possible to measure if the that event was
successfully written to the perf buffer or not.

This metric can be used to monitor the performance of individual eBPF
events and to detect potential bottlenecks.
@geyslan geyslan force-pushed the perf-event-writes-metric branch from c8edcca to bae1532 Compare December 3, 2024 18:19
@github-actions github-actions bot removed the area/api label Dec 3, 2024
@geyslan
Copy link
Member Author

geyslan commented Dec 3, 2024

/fast-forward

@github-actions github-actions bot merged commit bae1532 into aquasecurity:main Dec 3, 2024
31 checks passed
@geyslan geyslan added candidate/v0.22.5 Candidate to be cherry-picked or backported into v0.22.5 release cherry-picked/v0.22.5 backported/v0.22.5 and removed candidate/v0.22.5 Candidate to be cherry-picked or backported into v0.22.5 release labels Dec 6, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants