Automatic regression testing

[dirkwhoffmann]

The next version of VirtualC64 will support automatic regression testing. My first idea was to utilise the existing test framework provided by VICE which is explained in more detail here:

https://vice-emu.pokefinder.org/index.php/Testbench

At some point, I decided agains this approach, because everything is being configured via command line parameters and VirtualC64 already has a command line interface. The regression testing framework is now similar to the one which I’ve implemented for vAmiga recently. The test cases are executed by small scripts which eventually terminate the emulator with a screenshot.

Test execution is controlled by a small Makefile which collects all script files and runs them either one-by-one or in parallel. After a screenshot has been created, the Makefile compares it agains a reference image. If no reference image is present, it is created automatically based on the outcome of the current run. Overall, the test framework is pretty simple, but I think it serves our needs without adding too much complexity to the project.

The test repo is located here:

https://github.com/dirkwhoffmann/virtualC64TS

There are only 4 tests at the moment since I am still in the proof-of-concept phase.

[mrdudz]

I decided agains this approach, because everything is being configured via command line parameters and VirtualC64 already has a command line interface.

That doesnt make a lot of sense to me - when there is already a command line interface, why wouldnt you use it? :) (I couldnt find a documentation for it - but for other emulators it usually didnt take more than one or two extra options to support them)

I'd like to encourage you to think it over again - i can assist with adding support to the vice testbench scripts. I am sure this will be a LOT less work than setting up all the tests using your own framework. If only because a lot of them will not even work with just screenshots :) At least adding the "debugcart" feature will help a lot (and speed up the testing a whole lot... you dont want to add timeouts of several seconds for each of them)

[dirkwhoffmann]

I decided agains this approach, because everything is being configured via command line parameters and VirtualC64 already has a command line interface.

That doesnt make a lot of sense to me

Agreed, that doesn’t make sense. What I was trying to say was that VirtualC64 has gotten a debug console which offers all the commands needed for running a regression test (I also used the term „command line interface“ for that which turns my statement into nonsense). The debug console stems from vAmiga and is able to execute scripts. In the Amiga world, there is no such thing as a (VICE) test suite, so executing scripts in the debug console was the natural choice for me to set up an automatic regression testing environment. Once I got it working for vAmiga, it was the next natural choice for me to port the feature over to VirtualC64 (one of my top-level design goals is to bring both emulators as close together as possible).

You’re right that the command syntax is not documented, yet. I didn’t write it up, because the syntax is pretty new and surely needs some time to stabilise. However, there is some „online documentation“ built in. Whenever TAB is pressed twice in the debug console, the emulator prints all available options:

Although my script-based test suite is far from being optimal, it is already working pretty well (there are 562 scripts and reference images in the repo by now). Of course, using the VICE scripts would have the benefit that the emulator could be rated with a score and compared with other emulators. I agree that would be kind of cool and, as a side effect, would give VirtualC64 a little bit more publicity. Hence, I am not saying that support for the VICE test-bench is not coming. It’s just that it isn’t on top of my priority list at the moment.

[mithrendal]

thanks @dirkwhoffmann for outlining the current automatic regression testing approach of vAmiga and VirtualC64 and why it is based on the debug console by now ( mainly due to reasons of reusablility between the two projects but also to rethink the scripting part of it in a cool new fresh system-independent approach ... as far as I understood ... this does not lock out anybody to create/adapt an vice test script for VirtualC64 ... and yes @mrdudz I can understand your point of argumentation too ... it is also reasonable to me ... )

I am so happy and excited that there are such things in the world as a Vice, a Vice-Test-Suite and a VirtualC64 with embedded debug console / automatic regression testing... 🤓 👍

[mrdudz]

Of course, using the VICE scripts would have the benefit that the emulator could be rated with a score and compared with other emulators.

I wouldnt put too much weight on THAT :)

I agree that would be kind of cool and, as a side effect, would give VirtualC64 a little bit more publicity. Hence, I am not saying that support for the VICE test-bench is not coming. It’s just that it isn’t on top of my priority list at the moment.

OK, as said... i can assist with the scripting, so if you ever reach that point, just give me a ring :=)