Coded UI - Have a "continue on error" for asserts
Currenlty if a assert fails, the test stops
(Since the CodedUI is built on top the unit test framework)
But in a Coded UI test you may have multiple assert in a page and currently you have code a Try Catch block.
In web performance test the continue on error does exist.
(the feature pack 2 as misleading Continue on Error but that is only associated to playback)
Thank you for the feedback.
As pointed out in the comments, at the level of a test, if an assertion fails, then the outcome of the test should be fail. If the outcome were to be any different, it could well get treated as noise, and ignored – and that would be undesirable.
On the other hand, with UI tests, getting the application to a particular state in order to test some functionality can be time consuming, with lots of intermediate values, and vary from one application to another. The desired continue-on-error semantics are perhaps best left to each such application to provide.
Visual Studio Team
But why ContinueOnError results with warning? It is still error, so can't you just do it in such a way, that:
1) test case continues on error and is marked as error in test output log
2) overall status is failed
Why does it have to be logged as warning, not error?
Any updates on this request?
This would be really helpful. With UI tests, the process of getting the application to a particular state to test something can be time consuming. It doesn't make sense to apply the unit test methodology of test one thing and one thing only when the test can take minutes to run and there are lots of values to test.
Murali Krishna Channella commented
This would be really useful
Vinu chandran commented
It will be nice to see this feature implemented
Sudhish Mathuria commented
I think a parameter will serve the purpose. We can suggest playback engine with this parameter that it needs to fail or continue the test. We can keep this parameter as optional and by default value will be fail.
Grey Ham commented
I think the concept of "Verify" semantics are required - similar to what is used in GTEST. Verify() is exactly the same as an Assert() except that Verify() does not terminate the test on a failure - it 'notes' the error but continues.
At the end of the test, if any Verify's failed, the test is failed (warnings tend to be ignored). This approach allows a lot more information to be gleamed from a failure, especially where MsTest is being used for, perhaps, functional UI testing and the test might contain many assertions.
Well that would be better than what is currently, it would help in test desing and it could be easily discoverable in the assert's property windows (it using the feature pack add-in)
But I would still prefer my test to fail. Here is why, by experience, warnings are looked at as seriously, for example code analysis are often "warnings".
I think the team takes it more seriously when it "fails".
Also, I wonder what trickle effect such a new status would have on team build (would team build still be "partially succeeded" and the build parameter "fail build upon test failure" check box?)
Mathew Aniyan(MSFT) commented
Will an overload to the Assert Methods which specify that an assertion failure should be a warning suffice?
By default an assertion failure results in a test case failure. If this new Assert overload method is used, a warning will be logged in the test results.