Validating message with FAIL level fails if test itself passes

[joaonc]

I'm trying to update the test case Page Should Contain with

Page Should Contain
    [Documentation]    Default log level does not have html output.
    ...    LOG 2:7 Current page contains text 'needle'.
    ...    LOG 4.1:14 FAIL REGEXP: .*
    Page Should Contain    needle
    Page Should Contain    This is the haystack
    Run Keyword And Expect Error
    ...    Page should have contained text 'non existing text' but did not.
    ...    Page Should Contain    non existing text

Basically, my code is to make it so that keyword does not log the DOM at INFO and that test case runs at INFO level and should check that the DOM is indeed not logged.

To check that the DOM is not logged, it needs to check line 14 of step 4.1.
So I need to check that that line has status FAIL.

LOG 4.1:14 FAIL REGEXP: .* indeed checks that that line has status FAIL (LogMessageChecker._check_msg_level(...) passes), but also uses that information to check that the test case has status FAIL (TestStatusChecker._check_status(...) fails).

Looks like TestStatusChecker._check_status(...) should not use the values from LOG to check for test case status.

[joaonc]

Note: Another way to check that the DOM is not there, would be to add
LOG 4.1:14 REGEXP: (?i)^(?!<html).*$ (this regex matches when the line doesn't start with '<html', case insensitive)

But that fails at LogMessageChecker._check_msg_level(...).

Would be good to have a NONE option to ignore log level, something like:
LOG 4.1:14 NONE REGEXP: (?i)^(?!<html).*$

[pekkaklarck]

Could you try adding PASS into the beginning of the doc to explicitly specify that the test should pass? The reminder of the doc ought to then be used for validating log messages.

[joaonc]

That didn't work b/c here we have

    def _get_status(self, doc):
        return 'FAIL' if 'FAIL' in doc else 'PASS'

Which returns FAIL if we have LOG 4.1:14 FAIL The failure message but the test case passes.

Created a fix for this in PR #5 but there's something I can't point my finger in the test cases that verify this functionality..

[pekkaklarck]

You are right, if there's FAIL anywhere the expected status is FAIL even if there would be PASS as well.

[pekkaklarck]

If I've understood this issue correctly, it only manifests itself if the test itself is expected to pass but there are log messages with FAIL level. This can happen, for example, when using 'Run Keyword And Expect Errorto validate that the expected error occurs. That's definitely a bug, but in many cases it can be avoided by not usingRun Keyword And Expect Error(i.e. letting the test fail) and adding the neededFAIL Whatever the message is` into the documentation.

[joaonc]

This issue manifested itself when the test case status was different than the log status.
Ex. Log status = FAIL, but test case is PASS or the other way around.

I think you got it, per your comment, but IMO, this should be fixed and supported, as opposed to work around the limitation in the tests.

The latest commits in the PR should make it work now.

[pekkaklarck]

There's no log level PASS so the problem only occurs if test is expected to pass but to have FAIL messages. This obviously needs to be fixed, but because there's a workaround I don't consider the fix (and the needed release) urgent.

[joaonc]

There's no log level PASS

Right, my bad on the previous comment.
These test cases were name appropriately:

• Test case PASS log INFO
• Test case FAIL log INFO

I'll try to get the PR in SeleniumLibrary working with the workaround.