Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Auto-generate comparison benchmarks between commits #5

Open
BrianHicks opened this issue May 11, 2018 · 0 comments
Open

Auto-generate comparison benchmarks between commits #5

BrianHicks opened this issue May 11, 2018 · 0 comments

Comments

@BrianHicks
Copy link
Collaborator

Issue by jwoudenberg
Monday Jun 05, 2017 at 12:49 GMT
Originally opened as BrianHicks/elm-benchmark#14


This is informed by real work on elm-test. Suppose I have a library containing some benchmarks. I'm making some changes on a branch and want to use those benchmarks to see their performance impact. Currently your best option is to run the benchmarks twice and manually compare the results. With #5 it would be possible to run the benchmarks on one test, export the result, go to another branch, run them again and import the results. Ideally I'd like to be able to say: compare these two commits using the benchmarks already in the repo.

This tool would need to:

  • Check out both versions of the code at the same time and put at least one a different root (because Elm won't like us if we have multiple modules with the same name). The root-changing means some variables in the code would need to be renamed as well. The result can be dirty: I'd throw it away anyway after running the benchmark, since generating it was cheap.
  • Take existing benchmarks and turn them into comparisons between the code versions from both commits.

Practical concerns:

  • This can only work if both versions of the code have compatible interfaces, at least so far as the benchmarked code is concerned.

For elm-test I've done this manually (the result is here if you want to take a look), which was a bit of a pain.

Perhaps elm-format's parsing and code generation capability can be a starting point for such a tool.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant