feat: Add report section summarizing failing assertions from all tests in a plan #1288
Conversation
stalgiag
changed the title
There was a problem hiding this comment.
Thanks @stalgiag! Very impressive getting all this added and in a pretty maintainable way as well given where it has to be referenced.
The main issue I've come across is the handling of the priority 0 assertions but I left some inline suggestions on that. Seems this is all very close!
| return ( | ||
| <> | ||
| <p> | ||
| {metrics.assertionsFailedCount} assertions failed for{' '} |
There was a problem hiding this comment.
| {metrics.assertionsFailedCount} assertions failed for{' '} | |
| {failingAssertions.length} assertions failed for{' '} // or metrics.mustAssertionsFailedCount + metrics.shouldAssertionsFailedCount |
metrics.assertionsFailedCount would also cover the MAY assertions which isn't wanted here.
| <> | ||
| <p> | ||
| {metrics.assertionsFailedCount} assertions failed for{' '} | ||
| {metrics.commandsCount} commands in {metrics.testsFailedCount} tests. |
There was a problem hiding this comment.
Similarly, {metrics.commandsCount} should probably be another calculated value to present the uniqueness of the commands shown in the table (unless the "... failed for Y commands in Z tests" is meant to display the total number of commands and tests).
For example, the following table gives me: 13 assertions failed for 31 commands in 11 tests. But I'd have expected it to say: 13 assertions failed for 13 commands in 6 tests
| Test Name | Command | Assertion Priority | Assertion | JAWS Response |
|---|---|---|---|---|
| Navigate forwards to a menu button | Tab (virtual cursor active) | MUST | State 'collapsed' is conveyed | Actions button menu. Press space to activate the menu, then navigate with arrow keys. |
| Navigate forwards to a menu button | Tab (PC cursor active) | MUST | State 'collapsed' is conveyed | Actions button menu. Press space to activate the menu, then navigate with arrow keys. |
| Navigate backwards to a menu button | Shift+Tab (virtual cursor active) | MUST | State 'collapsed' is conveyed | Actions button menu. Press space to activate the menu, then navigate with arrow keys. |
| Navigate backwards to a menu button | Shift+Tab (PC cursor active) | MUST | State 'collapsed' is conveyed | Actions button menu. Press space to activate the menu, then navigate with arrow keys. |
| Request information about a menu button | Insert+Tab (virtual cursor active) | MUST | State 'collapsed' is conveyed | Actions button menu. |
| Request information about a menu button | Insert+Tab (PC cursor active) | MUST | State 'collapsed' is conveyed | Actions button menu. |
| Request information about a menu button | Insert+Up Arrow (PC cursor active) | MUST | State 'collapsed' is conveyed | Actions button menu. |
| Request information about a menu item | Insert+Tab (virtual cursor active) | SHOULD | Role of the focused item, 'menu item,' is conveyed | Actions menu. Action 1 menu 1 of 4. |
| Request information about a menu item | Insert+Up Arrow (virtual cursor active) | SHOULD | Role of the focused item, 'menu item,' is conveyed | Action 1 menu 1 of 4. |
| Request information about a menu item | Insert+Tab (PC cursor active) | SHOULD | Role of the focused item, 'menu item,' is conveyed | Actions menu. Action 1 menu 1 of 4. |
| Request information about a menu item | Insert+Up Arrow (PC cursor active) | SHOULD | Role of the focused item, 'menu item,' is conveyed | Action 1 menu 1 of 4. |
| Activate a menu item | Enter (PC cursor active) | MUST | State 'collapsed' is conveyed | Leaving menus. Actions button menu. Press space to activate the menu, then navigate with arrow keys. |
| Close a menu | Escape (PC cursor active) | MUST | State 'collapsed' is conveyed | Leaving menus. Actions button menu. Press space to activate the menu, then navigate with arrow keys. |
There was a problem hiding this comment.
Yeah, I am unsure on this one. I did think of it the way you are suggesting but with the template of X assertions failed for Y commands in Z tests, I believe that X and Y will always have the same value. It is also unclear what the communication goal is. The way it is currently assumes that the reader's goal is to understand the ratio of failures to successes - X failures of Y possible failures across Z total tests. After reading this do you still think I should take the approach of X assertions failed of Y commands with failing assertions of Z tests with commands that have failing assertions? I am still on the fence.
| -webkit-font-smoothing: antialiased; | ||
| font-family: 'Font Awesome 5 Free'; | ||
| font-weight: 900; | ||
| content: '\f03a'; /* List icon */ |
| @@ -61,6 +61,25 @@ const TestNavigator = ({ | |||
| aria-labelledby="test-navigator-heading" | |||
| className="test-navigator-list" | |||
| > | |||
| {isVendor && ( | |||
There was a problem hiding this comment.
Should probably conditionally show the Summary page or provide some default content (maybe some mashup of results table from the report page?) when there are no failing assertions.
Otherwise the following screenshot happens, which is just a blank content area and the heading being "Summary of Failing Assertions (0 must, 0 should)":
My preference is to hide if none exists because while I like the utility of the new "Begin Review" button, it would make more sense if this was an overall summary page rather than a page that just says "No failures exist, begin your review!". The overall summary isn't the request here and it may happen in the future but not something I'd push to have included right now. At least not before we make proper component based changes. Do you have any thoughts on that?
There was a problem hiding this comment.
I agree. I think just leaving it out entirely is sensible. Will do!
| onClick={handleNextTestClick} | ||
| disabled={isLastTest} | ||
| > | ||
| Begin Review |
There was a problem hiding this comment.
Nice, I like the inclusion of clear entry points!
| @@ -208,6 +208,39 @@ export const TestPlanReportConflictPropType = PropTypes.shape({ | |||
| ).isRequired | |||
| }); | |||
|
|
|||
| export const TestPlanReportMetricsPropType = PropTypes.shape({ | |||
Co-authored-by: Howard Edwards <[email protected]>
see #1253
This PR adds a report table summarizing all failing assertions to the Candidate Review and on Report pages.
There were a number of design and user experience decisions made here that I especially welcome feedback on. There were also a few decisions for which I was on the fence. Namely: