[Crashtracking] Add more filtering on TypeLoadException #6539

kevingosse · 2025-01-13T15:15:33Z

Summary of changes

Flag TypeLoadException as suspicious only if it contains "datadog" in the message or the stacktrace.

Reason for change

There are too many false-positives caused by TypeLoadException. This is an exception that is relatively common in non-instrumented apps.

datadog-ddstaging · 2025-01-13T15:46:20Z

Datadog Report

Branch report: kevin/filter_typeloadexception
Commit report: f551240
Test service: dd-trace-dotnet

✅ 0 Failed, 237907 Passed, 1976 Skipped, 19h 9m 4.65s Total Time

andrewlock · 2025-01-13T17:14:10Z

Execution-Time Benchmarks Report ⏱️

Execution-time results for samples comparing the following branches/commits:

Execution-time benchmarks measure the whole time it takes to execute a program. And are intended to measure the one-off costs. Cases where the execution time results for the PR are worse than latest master results are shown in red. The following thresholds were used for comparing the execution times:

Welch test with statistical test for significance of 5%
Only results indicating a difference greater than 5% and 5 ms are considered.

Note that these results are based on a single point-in-time result for each branch. For full results, see the dashboard.

Graphs show the p99 interval based on the mean and StdDev of the test run, as well as the mean value of the run (shown as a diamond below the graph).

gantt
    title Execution time (ms) FakeDbCommand (.NET Framework 4.6.2) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6539) - mean (68ms)  : 66, 70
     .   : milestone, 68,
    master - mean (68ms)  : 66, 71
     .   : milestone, 68,

    section CallTarget+Inlining+NGEN
    This PR (6539) - mean (982ms)  : 955, 1008
     .   : milestone, 982,
    master - mean (975ms)  : 946, 1003
     .   : milestone, 975,

gantt
    title Execution time (ms) FakeDbCommand (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6539) - mean (107ms)  : 104, 111
     .   : milestone, 107,
    master - mean (107ms)  : 105, 110
     .   : milestone, 107,

    section CallTarget+Inlining+NGEN
    This PR (6539) - mean (678ms)  : 663, 693
     .   : milestone, 678,
    master - mean (677ms)  : 661, 694
     .   : milestone, 677,

gantt
    title Execution time (ms) FakeDbCommand (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6539) - mean (91ms)  : 89, 93
     .   : milestone, 91,
    master - mean (90ms)  : 89, 92
     .   : milestone, 90,

    section CallTarget+Inlining+NGEN
    This PR (6539) - mean (637ms)  : 623, 651
     .   : milestone, 637,
    master - mean (630ms)  : 615, 646
     .   : milestone, 630,

gantt
    title Execution time (ms) HttpMessageHandler (.NET Framework 4.6.2) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6539) - mean (194ms)  : 188, 199
     .   : milestone, 194,
    master - mean (194ms)  : 189, 199
     .   : milestone, 194,

    section CallTarget+Inlining+NGEN
    This PR (6539) - mean (1,104ms)  : 1075, 1132
     .   : milestone, 1104,
    master - mean (1,104ms)  : 1073, 1135
     .   : milestone, 1104,

gantt
    title Execution time (ms) HttpMessageHandler (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6539) - mean (278ms)  : 274, 283
     .   : milestone, 278,
    master - mean (280ms)  : 273, 286
     .   : milestone, 280,

    section CallTarget+Inlining+NGEN
    This PR (6539) - mean (871ms)  : 843, 898
     .   : milestone, 871,
    master - mean (869ms)  : 842, 896
     .   : milestone, 869,

gantt
    title Execution time (ms) HttpMessageHandler (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6539) - mean (268ms)  : 263, 272
     .   : milestone, 268,
    master - mean (267ms)  : 262, 271
     .   : milestone, 267,

    section CallTarget+Inlining+NGEN
    This PR (6539) - mean (853ms)  : 817, 890
     .   : milestone, 853,
    master - mean (853ms)  : 825, 880
     .   : milestone, 853,

andrewlock · 2025-01-14T12:24:13Z

Benchmarks Report for tracer 🐌

Benchmarks for #6539 compared to master:

1 benchmarks are slower, with geometric mean 1.161
All benchmarks have the same allocations

The following thresholds were used for comparing the benchmark speeds:

Mann–Whitney U test with statistical test for significance of 5%
Only results indicating a difference greater than 10% and 0.3 ns are considered.

Allocation changes below 0.5% are ignored.

Benchmark details

Benchmarks.Trace.ActivityBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Gen 2	Allocated
master	`StartStopWithChild`	net6.0	8.3μs	48.2ns	449ns	0.0137	0.00686	0	5.61 KB
master	`StartStopWithChild`	netcoreapp3.1	10.2μs	57ns	369ns	0.0205	0.0102	0	5.8 KB
master	`StartStopWithChild`	net472	16.2μs	40.1ns	155ns	1.05	0.306	0.0965	6.2 KB
#6539	`StartStopWithChild`	net6.0	7.99μs	45.2ns	320ns	0.0117	0.0039	0	5.61 KB
#6539	`StartStopWithChild`	netcoreapp3.1	9.93μs	43.6ns	163ns	0.0156	0.00521	0	5.8 KB
#6539	`StartStopWithChild`	net472	16.4μs	47.4ns	183ns	1.07	0.337	0.107	6.22 KB

Benchmarks.Trace.AgentWriterBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`WriteAndFlushEnrichedTraces`	net6.0	501μs	243ns	940ns	0	2.7 KB
master	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	669μs	684ns	2.56μs	0	2.7 KB
master	`WriteAndFlushEnrichedTraces`	net472	857μs	919ns	3.56μs	0.422	3.3 KB
#6539	`WriteAndFlushEnrichedTraces`	net6.0	490μs	222ns	832ns	0	2.7 KB
#6539	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	657μs	2.32μs	8.99μs	0	2.7 KB
#6539	`WriteAndFlushEnrichedTraces`	net472	850μs	532ns	2.06μs	0.422	3.3 KB

Benchmarks.Trace.AspNetCoreBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`SendRequest`	net6.0	164μs	1.34μs	13.3μs	0.145	14.48 KB
master	`SendRequest`	netcoreapp3.1	182μs	1.19μs	11.7μs	0.179	17.27 KB
master	`SendRequest`	net472	0.00798ns	0.00243ns	0.00908ns	0	0 b
#6539	`SendRequest`	net6.0	156μs	1.43μs	13.8μs	0.159	14.47 KB
#6539	`SendRequest`	netcoreapp3.1	175μs	1.13μs	11.1μs	0.168	17.27 KB
#6539	`SendRequest`	net472	0.00176ns	0.000989ns	0.0037ns	0	0 b

Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Gen 2	Allocated
master	`WriteAndFlushEnrichedTraces`	net6.0	610μs	3.3μs	29.5μs	0.581	0	0	41.78 KB
master	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	728μs	5.77μs	57.4μs	0.332	0	0	41.74 KB
master	`WriteAndFlushEnrichedTraces`	net472	874μs	3.54μs	13.2μs	8.08	2.55	0.425	53.34 KB
#6539	`WriteAndFlushEnrichedTraces`	net6.0	577μs	2.9μs	13.3μs	0.581	0	0	41.62 KB
#6539	`WriteAndFlushEnrichedTraces`	netcoreapp3.1	719μs	4.17μs	37.3μs	0.363	0	0	41.62 KB
#6539	`WriteAndFlushEnrichedTraces`	net472	847μs	3.68μs	13.8μs	8.36	2.39	0.398	53.29 KB

Benchmarks.Trace.DbCommandBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Allocated
master	`ExecuteNonQuery`	net6.0	1.31μs	1.55ns	5.99ns	0.0144	0	1.02 KB
master	`ExecuteNonQuery`	netcoreapp3.1	1.79μs	1.04ns	3.89ns	0.0133	0	1.02 KB
master	`ExecuteNonQuery`	net472	2.04μs	1.89ns	6.83ns	0.157	0.00102	987 B
#6539	`ExecuteNonQuery`	net6.0	1.26μs	1.79ns	6.71ns	0.0145	0	1.02 KB
#6539	`ExecuteNonQuery`	netcoreapp3.1	1.82μs	2.98ns	11.5ns	0.0136	0	1.02 KB
#6539	`ExecuteNonQuery`	net472	2.06μs	1.39ns	5.39ns	0.157	0.00103	987 B

Benchmarks.Trace.ElasticsearchBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`CallElasticsearch`	net6.0	1.26μs	0.915ns	3.43ns	0.0133	976 B
master	`CallElasticsearch`	netcoreapp3.1	1.63μs	0.725ns	2.71ns	0.013	976 B
master	`CallElasticsearch`	net472	2.64μs	3.75ns	14.5ns	0.158	995 B
master	`CallElasticsearchAsync`	net6.0	1.34μs	0.935ns	3.62ns	0.0134	952 B
master	`CallElasticsearchAsync`	netcoreapp3.1	1.64μs	0.983ns	3.55ns	0.0135	1.02 KB
master	`CallElasticsearchAsync`	net472	2.68μs	3.01ns	11.7ns	0.167	1.05 KB
#6539	`CallElasticsearch`	net6.0	1.15μs	0.375ns	1.45ns	0.0139	976 B
#6539	`CallElasticsearch`	netcoreapp3.1	1.56μs	0.623ns	2.41ns	0.0133	976 B
#6539	`CallElasticsearch`	net472	2.59μs	1.78ns	6.67ns	0.158	995 B
#6539	`CallElasticsearchAsync`	net6.0	1.33μs	1.23ns	4.59ns	0.0133	952 B
#6539	`CallElasticsearchAsync`	netcoreapp3.1	1.68μs	0.986ns	3.69ns	0.0136	1.02 KB
#6539	`CallElasticsearchAsync`	net472	2.58μs	1.42ns	5.3ns	0.166	1.05 KB

Benchmarks.Trace.GraphQLBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`ExecuteAsync`	net6.0	1.3μs	1.03ns	3.85ns	0.0131	952 B
master	`ExecuteAsync`	netcoreapp3.1	1.56μs	1.28ns	4.62ns	0.0125	952 B
master	`ExecuteAsync`	net472	1.85μs	0.853ns	3.19ns	0.145	915 B
#6539	`ExecuteAsync`	net6.0	1.31μs	0.938ns	3.51ns	0.0133	952 B
#6539	`ExecuteAsync`	netcoreapp3.1	1.62μs	0.658ns	2.46ns	0.0129	952 B
#6539	`ExecuteAsync`	net472	1.85μs	0.569ns	2.2ns	0.145	915 B

Benchmarks.Trace.HttpClientBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`SendAsync`	net6.0	4.28μs	1.45ns	5.41ns	0.0323	2.31 KB
master	`SendAsync`	netcoreapp3.1	5.25μs	0.952ns	3.69ns	0.0368	2.85 KB
master	`SendAsync`	net472	7.29μs	2.21ns	8.27ns	0.495	3.12 KB
#6539	`SendAsync`	net6.0	4.4μs	1.98ns	7.4ns	0.0309	2.31 KB
#6539	`SendAsync`	netcoreapp3.1	5.24μs	1.66ns	6.43ns	0.0367	2.85 KB
#6539	`SendAsync`	net472	7.46μs	1.38ns	5.34ns	0.496	3.12 KB

Benchmarks.Trace.ILoggerBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`EnrichedLog`	net6.0	1.52μs	0.698ns	2.7ns	0.0228	1.64 KB
master	`EnrichedLog`	netcoreapp3.1	2.28μs	1.05ns	3.93ns	0.0216	1.64 KB
master	`EnrichedLog`	net472	2.73μs	2.24ns	8.37ns	0.249	1.57 KB
#6539	`EnrichedLog`	net6.0	1.48μs	0.765ns	2.86ns	0.0229	1.64 KB
#6539	`EnrichedLog`	netcoreapp3.1	2.28μs	1.94ns	7.5ns	0.0215	1.64 KB
#6539	`EnrichedLog`	net472	2.72μs	0.968ns	3.75ns	0.249	1.57 KB

Benchmarks.Trace.Log4netBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Gen 1	Allocated
master	`EnrichedLog`	net6.0	119μs	180ns	699ns	0.0585	0	4.28 KB
master	`EnrichedLog`	netcoreapp3.1	120μs	132ns	510ns	0	0	4.28 KB
master	`EnrichedLog`	net472	151μs	167ns	645ns	0.68	0.227	4.46 KB
#6539	`EnrichedLog`	net6.0	116μs	128ns	495ns	0.0582	0	4.28 KB
#6539	`EnrichedLog`	netcoreapp3.1	120μs	142ns	548ns	0.0597	0	4.28 KB
#6539	`EnrichedLog`	net472	150μs	129ns	499ns	0.673	0.224	4.46 KB

Benchmarks.Trace.NLogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`EnrichedLog`	net6.0	2.9μs	0.932ns	3.61ns	0.0306	2.2 KB
master	`EnrichedLog`	netcoreapp3.1	4.25μs	2.47ns	9.57ns	0.0297	2.2 KB
master	`EnrichedLog`	net472	4.99μs	1.77ns	6.86ns	0.319	2.02 KB
#6539	`EnrichedLog`	net6.0	2.97μs	1.12ns	4.2ns	0.0314	2.2 KB
#6539	`EnrichedLog`	netcoreapp3.1	4.31μs	2.41ns	9.35ns	0.0278	2.2 KB
#6539	`EnrichedLog`	net472	4.78μs	0.864ns	3.35ns	0.319	2.02 KB

Benchmarks.Trace.RedisBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`SendReceive`	net6.0	1.38μs	1.35ns	5.03ns	0.0161	1.14 KB
master	`SendReceive`	netcoreapp3.1	1.76μs	0.909ns	3.52ns	0.015	1.14 KB
master	`SendReceive`	net472	2.06μs	1.62ns	6.29ns	0.183	1.16 KB
#6539	`SendReceive`	net6.0	1.35μs	1.21ns	4.7ns	0.0161	1.14 KB
#6539	`SendReceive`	netcoreapp3.1	1.75μs	2.18ns	8.43ns	0.0157	1.14 KB
#6539	`SendReceive`	net472	2.1μs	1.55ns	5.99ns	0.183	1.16 KB

Benchmarks.Trace.SerilogBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`EnrichedLog`	net6.0	2.65μs	0.845ns	3.16ns	0.0225	1.6 KB
master	`EnrichedLog`	netcoreapp3.1	3.9μs	1.33ns	5.15ns	0.0215	1.65 KB
master	`EnrichedLog`	net472	4.36μs	4.27ns	16ns	0.323	2.04 KB
#6539	`EnrichedLog`	net6.0	2.71μs	1.01ns	3.92ns	0.0218	1.6 KB
#6539	`EnrichedLog`	netcoreapp3.1	3.87μs	4.02ns	15.6ns	0.0214	1.65 KB
#6539	`EnrichedLog`	net472	4.43μs	2.95ns	11ns	0.323	2.04 KB

Benchmarks.Trace.SpanBenchmark - Slower ⚠️ Same allocations ✔️

Slower ⚠️ in #6539

Benchmark	diff/base	Base Median (ns)	Diff Median (ns)	Modality
Benchmarks.Trace.SpanBenchmark.StartFinishScope‑net6.0	1.161	465.70	540.47

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`StartFinishSpan`	net6.0	396ns	0.259ns	1ns	0.00804	576 B
master	`StartFinishSpan`	netcoreapp3.1	621ns	1.21ns	4.7ns	0.00773	576 B
master	`StartFinishSpan`	net472	712ns	0.333ns	1.29ns	0.0916	578 B
master	`StartFinishScope`	net6.0	466ns	0.212ns	0.766ns	0.00978	696 B
master	`StartFinishScope`	netcoreapp3.1	718ns	2.25ns	8.73ns	0.0094	696 B
master	`StartFinishScope`	net472	817ns	0.954ns	3.7ns	0.104	658 B
#6539	`StartFinishSpan`	net6.0	405ns	0.243ns	0.939ns	0.00815	576 B
#6539	`StartFinishSpan`	netcoreapp3.1	557ns	0.315ns	1.22ns	0.00782	576 B
#6539	`StartFinishSpan`	net472	653ns	0.509ns	1.97ns	0.0915	578 B
#6539	`StartFinishScope`	net6.0	540ns	0.236ns	0.916ns	0.00961	696 B
#6539	`StartFinishScope`	netcoreapp3.1	749ns	0.792ns	3.07ns	0.00939	696 B
#6539	`StartFinishScope`	net472	891ns	0.451ns	1.69ns	0.104	658 B

Benchmarks.Trace.TraceAnnotationsBenchmark - Same speed ✔️ Same allocations ✔️

Raw results

Branch	Method	Toolchain	Mean	StdError	StdDev	Gen 0	Allocated
master	`RunOnMethodBegin`	net6.0	609ns	0.354ns	1.37ns	0.00974	696 B
master	`RunOnMethodBegin`	netcoreapp3.1	970ns	0.624ns	2.34ns	0.00962	696 B
master	`RunOnMethodBegin`	net472	1.12μs	0.636ns	2.46ns	0.104	658 B
#6539	`RunOnMethodBegin`	net6.0	629ns	0.42ns	1.63ns	0.00983	696 B
#6539	`RunOnMethodBegin`	netcoreapp3.1	978ns	1.66ns	6.42ns	0.00923	696 B
#6539	`RunOnMethodBegin`	net472	1.14μs	0.403ns	1.56ns	0.104	658 B

kevingosse requested a review from a team as a code owner January 13, 2025 15:15

kevingosse added area:crashtracking type:enhancement Improvement to an existing feature labels Jan 13, 2025

andrewlock approved these changes Jan 13, 2025

View reviewed changes

Add more filtering on TypeLoadException

f551240

kevingosse force-pushed the kevin/filter_typeloadexception branch from b1196d6 to f551240 Compare January 14, 2025 10:46

kevingosse merged commit ee7f8d5 into master Jan 14, 2025
100 of 103 checks passed

kevingosse deleted the kevin/filter_typeloadexception branch January 14, 2025 15:43

github-actions bot added this to the vNext-v3 milestone Jan 14, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Crashtracking] Add more filtering on TypeLoadException #6539

[Crashtracking] Add more filtering on TypeLoadException #6539

kevingosse commented Jan 13, 2025

datadog-ddstaging bot commented Jan 13, 2025 •

edited

Loading

andrewlock commented Jan 13, 2025 •

edited

Loading

andrewlock commented Jan 14, 2025

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Slower ⚠️ in #6539

Raw results

Raw results

[Crashtracking] Add more filtering on TypeLoadException #6539

[Crashtracking] Add more filtering on TypeLoadException #6539

Conversation

kevingosse commented Jan 13, 2025

Summary of changes

Reason for change

datadog-ddstaging bot commented Jan 13, 2025 • edited Loading

Datadog Report

andrewlock commented Jan 13, 2025 • edited Loading

Execution-Time Benchmarks Report ⏱️

andrewlock commented Jan 14, 2025

Benchmarks Report for tracer 🐌

Benchmark details

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Raw results

Slower ⚠️ in #6539

Raw results

Raw results

datadog-ddstaging bot commented Jan 13, 2025 •

edited

Loading

andrewlock commented Jan 13, 2025 •

edited

Loading