Test Explorer is severely limited
I have to assume the people who developed the VS2012/2013 test explorer view never actually had to use it in production. The test explorer view is horribly limited in comparison to the one in VS2010.
1. Running tests does not open the Test Explorer window until after all the tests completed running. There's literally no indication that anything is happening. VS2010 immediately opened the window to show you that the tests were running.
2. Where are the columns? Most importantly: the error message column. if I'm dealing with multiple test failures, I almost always just need to see the error messages at a glance. Having to select each individual test just to see the result is annoying.
3. Where'd the ability to run tests in context go? There's no way to run an individual test without going into the full test list, finding it, and then selecting it just to run it. This might be okay if you have a project with 10 tests, but a solution with thousands of tests makes this arduous. This needs to be fixed badly. Either bring back the "run tests in context" button or at least add some right click menus for "Run this test" and "Run all tests in class".
4. The progress bar is terrible. A five pixel high bar might be useful to someone with an 800x600 monitor, but for the rest of us a barely noticable progress bar is not enough to know that all the tests finished passing. ****, half the time the progress bar color doesn't even change. It just stays red/orange rather than changing to green.
Please fix this. It has made working with unit tests far more of a chore than it needs to be.
Carl: To correct myself: "It just stays red/orange rather than changing to green when all the tests have passed after a previous failed run". Having said that, I think it was fixed in one of the updates.
"It just stays red/orange rather than changing to green."
This means that your tests have failed...
#23: If you hit the "Run All" button and then open the the test explorer while it's still running, the progress bar is grayed out and makes no indication of progress. This only seems to happen if it's the first time opening the Test Explorer after opening Visual Studio.
#22: Occasionally "Run All" does not actually run all the tests. It doesn't give any indication that it didn't unless you decide to scroll through all the tests to see if any of them are grayed out. The only fix is to reopen the solution. Rebuilding doesn't fix it, even though the test window will say it has found all of them.
#21: The biggest problem for me is that when I have multiple test properties with overlapping values, the Filter can't understand I want to map Trait/Property A with Value 1. It brings up all tests where either the key or the value match A or 1. I have thousands of tests with 20 different Traits, and this UI not only doesn't scale, it's basically broken.
Going back to #11, this seems to happen to me when I leave Visual Studio open overnight. Without fail, the moment I try to run tests for the first time I get hit with "Underlying provider could not be opened" errors.
#20: You can't undo the removal of tests from a playlist.
#19: Would be nice if it actually displayed the *total number of tests* being ran somewhere.
#18: When grouped by class and all of the class sections are collapsed, opening any section and then removing all of the tests in that section causes the top-most section to suddenly expand.
#17: Something as simple as hitting the delete key should remove tests from the playlists.
#16: Can't remove all tests from a playlist by selecting their grouped section. ex: I can't group tests by project and then right click the project heading and remove the tests. I have to highlight all of them first and then right click. That's a pain, especially if you want to remove multiple grouped areas(see #15 for why I even have to do this in the first place).
#15: There isn't a way to group the tests in the order that they ran. I'm currently stuck trying to figure out why 6 tests suddenly fail on me when they're run together with all the other unit tests in the project. I can't even begin to try and narrow down which tests ran before the failing tests so I can find out the cause of it.
Going back to #4, if you're running a single unit test that runs extremely fast(a few milliseconds), the Test Explorer window doesn't give you any indication that the test ever ran. The test completes before any of the progress indicators get a chance to start. And since there's nothing indicating *what time* a test started or completed, you're left wondering if the test even ran.
Jacques Bosch commented
Also have general gripes with the "downgrade" of test explorer features. Very frustrating to have had goodness, and then to have it taken away.
I also want to see the number of tests in the current run and the number of completed.
Also want columns back. (Optional if they have to be)
And the progress bar size and colors make it virtually unusable to me.
Here's to hoping the goodness we had comes back.
PS The filter feature is very cool, but can't make up for what has been lost.
#14: The "pretty" values listed for elapsed test times are useless for any test that runs longer than 1 second. Values are rounded down to the nearest whole second, minute, and I'm assuming hour and day. Case in point: Create a unit test that does nothing but Thread.Sleep(110000). That's almost 2 minutes, but it displays as if it's 1 minute.
This needs to go back to the old way of displaying times, with the actual minute, seconds, and milliseconds on display. I'm a developer, I'm also an avid user of digital clocks, I know how to read a display that's formatted as hh:mm:ss. Developer's tools shouldn't have dumbed down "user friendly" readouts like this when accurately being able to measure things is important.
Jason Evans commented
The only reason I use test explorer is because of MS Fakes. I use Resharper's test runner for all my tests, other then those using Fakes, simply because that test runner cannot work with Fakes.
Please can the Visual Studio team enhance the test explorer so that it's on par with what a professional developer would expect. Right now, it's very poor features make it a very disappointing experience to use.
#13: Test runner completely ignores the "On Run, when build or deployment errors occur" setting. I have this set to "Do not launch", yet it still prompts me about whether or not I want to run from a previous build.
#12: There's no way to *remove* tests from the test explorer. Again, seeing every single test just adds noise to the display.
#11: Testing randomly decides to throw "The underlying provider failed on Open." errors when you try to run any test. The only solution is to restart Visual Studio.
I just installed Visual Studio 2013 Update 1 and I was very disappointed. All the VS 2008/2010 good features about automatic testing are still missing.
Just to recap them:
- VS2010 separated test preparation and test execution, VS2013 mixes them in Test Explorer; sometimes it may be useful, but for me MOST of the time it's not useful at all. Instead, it's confusing. A filter filters both executed and "not executed" tests, so if I run a single test directly from my test source code, and then I open Test Explorer, I don't see the test and I have to clear the filter.
- Furthermore, previous test results are mixed with new test results in the Test Explorer. This is confusing.
- VS2010 had test results in table form, with customizable colums, allowing to see all error messages at a glance. VS2013 has not. This is a regression.
- With VS2010 I could easily tell which test was running at a given time
test results in table form, with customizable colums, allowing to see all error messages at a glance
- With VS2010, running a test automatically opened the running & test results pane. Ok, I understand some developers did not like it, but I DID like it, and an option to get back the old behaviour would be useful.
- VS2010 had an option pane to customize the automatic testing behaviour
- VS2010 had checkboxes to select tests to run (VS2013 has just clumsy CTRL+CLICK)
- Sometimes tests are not run at all, as if the test running process had a problem, but no errors are displayed, sometimes they are run OK.
In the past year, I manage to still use VS2008/VS2010 for testing while developing on VS2012.
Now I'm forced by my company to move to VS2013 for a new big project, and I'll be forced to use VS2013 for testing too. :(
People in my team miss the good interface of VS2008/2010. Please, Microsoft, bring it back.