Coded UI - Have a "continue on error" for asserts
Currenlty if a assert fails, the test stops
(Since the CodedUI is built on top the unit test framework)
But in a Coded UI test you may have multiple assert in a page and currently you have code a Try Catch block.
In web performance test the continue on error does exist.
(the feature pack 2 as misleading Continue on Error but that is only associated to playback)
BertrandH
shared this idea
4 comments
-
Sudhish Mathuria
commented
I think a parameter will serve the purpose. We can suggest playback engine with this parameter that it needs to fail or continue the test. We can keep this parameter as optional and by default value will be fail.
-
Grey Ham
commented
I think the concept of "Verify" semantics are required - similar to what is used in GTEST. Verify() is exactly the same as an Assert() except that Verify() does not terminate the test on a failure - it 'notes' the error but continues.
At the end of the test, if any Verify's failed, the test is failed (warnings tend to be ignored). This approach allows a lot more information to be gleamed from a failure, especially where MsTest is being used for, perhaps, functional UI testing and the test might contain many assertions.
-
BertrandH
commented
Well that would be better than what is currently, it would help in test desing and it could be easily discoverable in the assert's property windows (it using the feature pack add-in)
But I would still prefer my test to fail. Here is why, by experience, warnings are looked at as seriously, for example code analysis are often "warnings".
I think the team takes it more seriously when it "fails".
Also, I wonder what trickle effect such a new status would have on team build (would team build still be "partially succeeded" and the build parameter "fail build upon test failure" check box?)
-
Mathew Aniyan(MSFT)
commented
Will an overload to the Assert Methods which specify that an assertion failure should be a warning suffice?
By default an assertion failure results in a test case failure. If this new Assert overload method is used, a warning will be logged in the test results.
