[serverless] Upload serverless assets to Azure so we can release for a layer

[andrewlock]

Summary of changes

Compiles the serverless assets to an artifact, uploads them, and uses those when building the layer for a release

Reason for change

The serverless layer has been unable to build for a while for releases. The SSI artifacts (which similarly build in gitlab) pull the assets from the GitHub release page, but we're hesitant to add the serverless assets there, because of the risk of confusion for customers.

As a workaround, we upload the assets to a public blob store that can be pulled from later. Various other workflows already use this store, so this is just adding an additional asset, and sidesteps the issue.

Implementation details

• Remove the "aggregation" logic out of GitLab (which chooses which artifacts are required)
• Add a new stage to AzDo that does the aggregation logic for the required artifacts.
• Upload the artifact as a zip file to Azure
• In gitlab, pull either directly from azdo (for build branches) or from Azure (for tag branches)

Test coverage

• Tested the aggregation and upload here
• Tested the pulling from gitlab here (using a temporary hack to force calling the tag-related code)

Other details

Supersedes

• [serverless] upload r2r artifacts to azure #6543

[andrewlock]

As we do the aggregation in azdo, we don't need to build the list of artifacts here, we just pull a single artifact so, this code becomes almost identical the SSI code (which is handy)

[andrewlock]

IT looks like a big change, but it's mostly just deleing the for loops - hiding whitespace changes makes the changes more obvious

[datadog-ddstaging]

Datadog Report

Branch report: andrew/ci/fix-serverless-layer
Commit report: e51ed82
Test service: dd-trace-dotnet

✅ 0 Failed, 247125 Passed, 2014 Skipped, 18h 48m 34.32s Total Time

[andrewlock]

Execution-Time Benchmarks Report ⏱️

Execution-time results for samples comparing the following branches/commits:

• This PR (6561)
• master

Execution-time benchmarks measure the whole time it takes to execute a program. And are intended to measure the one-off costs. Cases where the execution time results for the PR are worse than latest master results are shown in red. The following thresholds were used for comparing the execution times:

• Welch test with statistical test for significance of 5%
• Only results indicating a difference greater than 5% and 5 ms are considered.

Note that these results are based on a single point-in-time result for each branch. For full results, see the dashboard.

Graphs show the p99 interval based on the mean and StdDev of the test run, as well as the mean value of the run (shown as a diamond below the graph).

gantt
    title Execution time (ms) FakeDbCommand (.NET Framework 4.6.2) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6561) - mean (69ms)  : 66, 71
     .   : milestone, 69,
    master - mean (69ms)  : 66, 72
     .   : milestone, 69,

    section CallTarget+Inlining+NGEN
    This PR (6561) - mean (983ms)  : 957, 1009
     .   : milestone, 983,
    master - mean (981ms)  : 959, 1004
     .   : milestone, 981,

Loading

gantt
    title Execution time (ms) FakeDbCommand (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6561) - mean (108ms)  : 106, 110
     .   : milestone, 108,
    master - mean (108ms)  : 106, 110
     .   : milestone, 108,

    section CallTarget+Inlining+NGEN
    This PR (6561) - mean (677ms)  : 663, 691
     .   : milestone, 677,
    master - mean (680ms)  : 662, 698
     .   : milestone, 680,

Loading

gantt
    title Execution time (ms) FakeDbCommand (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6561) - mean (91ms)  : 89, 93
     .   : milestone, 91,
    master - mean (91ms)  : 89, 93
     .   : milestone, 91,

    section CallTarget+Inlining+NGEN
    This PR (6561) - mean (632ms)  : 613, 650
     .   : milestone, 632,
    master - mean (637ms)  : 617, 657
     .   : milestone, 637,

Loading

gantt
    title Execution time (ms) HttpMessageHandler (.NET Framework 4.6.2) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6561) - mean (190ms)  : 186, 194
     .   : milestone, 190,
    master - mean (190ms)  : 186, 194
     .   : milestone, 190,

    section CallTarget+Inlining+NGEN
    This PR (6561) - mean (1,093ms)  : 1056, 1130
     .   : milestone, 1093,
    master - mean (1,091ms)  : 1062, 1119
     .   : milestone, 1091,

Loading

gantt
    title Execution time (ms) HttpMessageHandler (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6561) - mean (275ms)  : 271, 279
     .   : milestone, 275,
    master - mean (276ms)  : 271, 280
     .   : milestone, 276,

    section CallTarget+Inlining+NGEN
    This PR (6561) - mean (864ms)  : 842, 885
     .   : milestone, 864,
    master - mean (865ms)  : 838, 892
     .   : milestone, 865,

Loading

gantt
    title Execution time (ms) HttpMessageHandler (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6561) - mean (264ms)  : 260, 268
     .   : milestone, 264,
    master - mean (263ms)  : 259, 267
     .   : milestone, 263,

    section CallTarget+Inlining+NGEN
    This PR (6561) - mean (844ms)  : 806, 882
     .   : milestone, 844,
    master - mean (844ms)  : 813, 875
     .   : milestone, 844,

Loading