Diagnosing consumer failures is really hard

[holly-cummins]

When a consumer test fails, the only error message tends to be an opaque the method annotated with @Pact did not run message. This makes diagnosing the underlying cause very hard.

There are several sub-problems.

Diagnosing absence

This is a fundamental Pact issue. If something didn't happen (the tests didn't hit the expected endpoint), it's hard to know what we can tell the user that's helpful. Monitoring all traffic and reporting what did happen would help, but it's not trivial. The wording of the error message maybe could be improved, but it's hard to see an obvious solution beyond that.

Maybe Quarkus can use its knowledge of the stack to monitor outbound traffic more easily than base Pact can.

Over-long stack traces

I don't think it's necessary to get the stack trace for the Pact failure, since it doesn't help us diagnose things.

Suppressed error messages

This is unique to continuous testing. And bad. For some errors, like a class not found, we get a helpful message with mvn verify, but not with continuous testing. Somehow in CT the error message gets swallowed.

Missing diagnostics for global issues

For some issues, breaking the tests is 'global' and seems to affect the whole runtime. For example, with #58, having one annotation will break all pact tests. In these cases, we don't get any messages at all (although in normal mode the tests run, so it's hard to compare.)

[holly-cummins]

Quoting from @markusdlugi's suggestion on #73 and , so we don't lose the idea:

For example, in our project we are using a custom Wiremock server that is started as part of Dev Services and acts as a proxy to the Pact Mock Server. This accomplishes two things for us:
We can point our application's REST Client to Wiremock, where we have more control due to Dev Services integration, so we also get rid of the programmatic RestClientBuilder.
Additionally, Wiremock is a little more verbose regarding its requests and responses than the Pact MockServer, so it's easier to spot mismatched request

And from @holomekc:

The approach in our wiremock / pact setup is, that we have a Quarkus test resource for Wiremock with a custom annotation, which makes it easy to inject Wiremock into tests etc. In a base test class, we do exactly that and provide a method, which accepts the MockServer and creates the proxy config for Wiremock using the MockServer url. Then we just need to call this method. In theory this setup can be achieved wherever the MockServer is available then.