Broker conformance tests using reconciler-test

[mgencur]

Fixes a number of issues:

• The previous conformance tests were calling t.Fatalf on a parent T which could fail the whole test suite like this:

  {Failed  === RUN   TestBrokerConsumer/3/No_upgrade_of_version
   testing.go:1576: test executed panic(nil) or runtime.Goexit: subtest may have called FailNow on a parent test
  

• The test namespace was always deleted for conformance tests, even in the event of failure. The debug info was often insufficient. The new conformance tests keep the namespace with its resources if the tests fail and also provide more info about events that were delivered or missed.
• The verification steps were running sequentially, now they're running in parallel.

Proposed Changes

• Use reconciler-test framework and re-implement TestBrokerConsumer from test/e2e/conformance/data_plane_conformance_test.go
• Remove TestBrokerIngress completely as all the tests are covered elsewhere:
  • Ingress Supports CE0.3, Ingress Supports CE1.0: covered by data_plane.go that is called as part of broker_conformance_test.go
  • Ingress Supports Structured Mode: covered by data_plane.go
  • Ingress Supports Binary Mode: covered by data_plane.go
  • Respond with 400 on bad CE: covered by data_plane.go and data_plane.go
  • Respond with 2XX on good CE: covered by data_plane.go

Release Note

Docs

[mgencur]

I'm just not sure about using ShouldNot/Should/Must in this whole section. Any suggestions which ones to use?

[pierDipi]

the test should follow the spec language, however, the way we run the tests should and should not in practice are like must / must not as we don't "skip should / should not"

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 45.51%. Comparing base (db85e46) to head (22b1d10).
Report is 2 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #4200      +/-   ##
==========================================
- Coverage   45.53%   45.51%   -0.02%     
==========================================
  Files         270      270              
  Lines       19925    19925              
==========================================
- Hits         9072     9069       -3     
- Misses      10127    10129       +2     
- Partials      726      727       +1     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[mgencur]

/test upgrade-tests

[mgencur]

/retest-required

[mgencur]

/test upgrade-tests

[pierDipi]

/lgtm
/approve

[knative-prow]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: mgencur, pierDipi

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [pierDipi]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[mgencur]

/retest-required

[mgencur]

/retest-required

[pierDipi]

   steps.go:47: sources.knative.dev/v1, Resource=kafkasources did not become ready, conversion webhook for sources.knative.dev/v1beta1, Kind=KafkaSource failed: Post "https://kafka-webhook-eventing.knative-eventing.svc:443/resource-conversion?timeout=30s": EOF

/retest-required

[knative-prow]

@mgencur: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
reconciler-tests_eventing-kafka-broker_main	22b1d10	link	true	/test reconciler-tests

Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.