Auto-generate comparison benchmarks between commits

[jwoudenberg]

This is informed by real work on elm-test. Suppose I have a library containing some benchmarks. I'm making some changes on a branch and want to use those benchmarks to see their performance impact. Currently your best option is to run the benchmarks twice and manually compare the results. With #5 it would be possible to run the benchmarks on one test, export the result, go to another branch, run them again and import the results. Ideally I'd like to be able to say: compare these two commits using the benchmarks already in the repo.

This tool would need to:

• Check out both versions of the code at the same time and put at least one a different root (because Elm won't like us if we have multiple modules with the same name). The root-changing means some variables in the code would need to be renamed as well. The result can be dirty: I'd throw it away anyway after running the benchmark, since generating it was cheap.
• Take existing benchmarks and turn them into comparisons between the code versions from both commits.

Practical concerns:

• This can only work if both versions of the code have compatible interfaces, at least so far as the benchmarked code is concerned.

For elm-test I've done this manually (the result is here if you want to take a look), which was a bit of a pain.

Perhaps elm-format's parsing and code generation capability can be a starting point for such a tool.

[BrianHicks]

moved to elm-explorations/benchmark#5