Thunderdome: Improve Experiment Quality

[iand]

What Is It?

Provide more realistic measurment conditions and more options for fine tuning experiments to improve the quality of experiments.

Deliverables

• New experiment option to warm up targets and adapt request load
• New experiment option to filter incoming requests
• New metrics to monitor performance of a target compared to live

Why Are We Doing It?

The initial development of Thunderdome fixed several factors that affect experiments for simplicity. For example requests are filtered to only allow send based requests starting with /ipfs or /ipns. Also, while Thunderdome probes targets to establish whether they are ready to accept requests, it immediately sends the full request load making metrics gathered at the start of the experiment skewed since the target has no chance to fill caches or grow queues.

We want to provide more options for users to control some of these factors so that experiments can be more representative of real environments. We also want to refine the way experiments report metrics by adapting the delivered load to available capacity and giving the test targets a chance to warm up into a stable configuration which will produce higher quality and more accurate metrics.

Notes

Tasks

• Dealgood: add more request filter options probe-lab/thunderdome#55
• Dealgood: report metric for request source ingest rate and lag probe-lab/thunderdome#52
• Dealgood reports experiment level metrics probe-lab/thunderdome#34
• Get ECS metrics from Cloudwatch into Prom probe-lab/thunderdome#58
• Enable more than 5 experiments to tail loki logs probe-lab/thunderdome#73