Test Explorer is severely limited
I have to assume the people who developed the VS2012/2013 test explorer view never actually had to use it in production. The test explorer view is horribly limited in comparison to the one in VS2010.
1. Running tests does not open the Test Explorer window until after all the tests completed running. There's literally no indication that anything is happening. VS2010 immediately opened the window to show you that the tests were running.
2. Where are the columns? Most importantly: the error message column. if I'm dealing with multiple test failures, I almost always just need to see the error messages at a glance. Having to select each individual test just to see the result is annoying.
3. Where'd the ability to run tests in context go? There's no way to run an individual test without going into the full test list, finding it, and then selecting it just to run it. This might be okay if you have a project with 10 tests, but a solution with thousands of tests makes this arduous. This needs to be fixed badly. Either bring back the "run tests in context" button or at least add some right click menus for "Run this test" and "Run all tests in class".
4. The progress bar is terrible. A five pixel high bar might be useful to someone with an 800x600 monitor, but for the rest of us a barely noticable progress bar is not enough to know that all the tests finished passing. ****, half the time the progress bar color doesn't even change. It just stays red/orange rather than changing to green.
Please fix this. It has made working with unit tests far more of a chore than it needs to be.
Going back to #11 again: We have a backup script at work that deletes that TestResults folder before backing up our source code. So this ends up deleting the mdf file that VS makes in there. Why it doesn't just recreate the file(like it does after restarting VS) is beyond me.
Going back to #23 again: Here's a random sampling I get from hitting "Run All" in the Test Explorer window. For the current solution I have open, there are 4474 tests.
------ Discover test started ------
========== Discover test finished: 3838 found (0:00:03.811) ==========
------ Run test started ------
========== Run test finished: 2645 run (0:01:27.939) ==========
So it found 3838 files, not finding about 500 tests, and then didn't run another 1000+ of them.
If I rebuild the solution, it will then say it found 4474 tests, even though they were there in the first place. I'm assuming the discover tests count is only for the projects in the solution that were just built, but it doesn't explain why chunks of tests get ignored when attempting to do a Run All.
Going back to #23: At some point, "Run All" will only run all the unit tests for a single test project. It completely ignores all the other test projects in the solution.
#24: The Deploy directories in the TestResults directory do not get deleted on failed tests. That would be fine, except it's ignoring the "Limit number of old test results to" setting. I have it set to 1, I have never seen a need for keeping old test results. This is just wasting drive space for me on huge projects.
Carl: To correct myself: "It just stays red/orange rather than changing to green when all the tests have passed after a previous failed run". Having said that, I think it was fixed in one of the updates.
"It just stays red/orange rather than changing to green."
This means that your tests have failed...
#23: If you hit the "Run All" button and then open the the test explorer while it's still running, the progress bar is grayed out and makes no indication of progress. This only seems to happen if it's the first time opening the Test Explorer after opening Visual Studio.
#22: Occasionally "Run All" does not actually run all the tests. It doesn't give any indication that it didn't unless you decide to scroll through all the tests to see if any of them are grayed out. The only fix is to reopen the solution. Rebuilding doesn't fix it, even though the test window will say it has found all of them.
#21: The biggest problem for me is that when I have multiple test properties with overlapping values, the Filter can't understand I want to map Trait/Property A with Value 1. It brings up all tests where either the key or the value match A or 1. I have thousands of tests with 20 different Traits, and this UI not only doesn't scale, it's basically broken.
Going back to #11, this seems to happen to me when I leave Visual Studio open overnight. Without fail, the moment I try to run tests for the first time I get hit with "Underlying provider could not be opened" errors.
#20: You can't undo the removal of tests from a playlist.
#19: Would be nice if it actually displayed the *total number of tests* being ran somewhere.
#18: When grouped by class and all of the class sections are collapsed, opening any section and then removing all of the tests in that section causes the top-most section to suddenly expand.
#17: Something as simple as hitting the delete key should remove tests from the playlists.
#16: Can't remove all tests from a playlist by selecting their grouped section. ex: I can't group tests by project and then right click the project heading and remove the tests. I have to highlight all of them first and then right click. That's a pain, especially if you want to remove multiple grouped areas(see #15 for why I even have to do this in the first place).
#15: There isn't a way to group the tests in the order that they ran. I'm currently stuck trying to figure out why 6 tests suddenly fail on me when they're run together with all the other unit tests in the project. I can't even begin to try and narrow down which tests ran before the failing tests so I can find out the cause of it.
Going back to #4, if you're running a single unit test that runs extremely fast(a few milliseconds), the Test Explorer window doesn't give you any indication that the test ever ran. The test completes before any of the progress indicators get a chance to start. And since there's nothing indicating *what time* a test started or completed, you're left wondering if the test even ran.
Jacques Bosch commented
Also have general gripes with the "downgrade" of test explorer features. Very frustrating to have had goodness, and then to have it taken away.
I also want to see the number of tests in the current run and the number of completed.
Also want columns back. (Optional if they have to be)
And the progress bar size and colors make it virtually unusable to me.
Here's to hoping the goodness we had comes back.
PS The filter feature is very cool, but can't make up for what has been lost.
#14: The "pretty" values listed for elapsed test times are useless for any test that runs longer than 1 second. Values are rounded down to the nearest whole second, minute, and I'm assuming hour and day. Case in point: Create a unit test that does nothing but Thread.Sleep(110000). That's almost 2 minutes, but it displays as if it's 1 minute.
This needs to go back to the old way of displaying times, with the actual minute, seconds, and milliseconds on display. I'm a developer, I'm also an avid user of digital clocks, I know how to read a display that's formatted as hh:mm:ss. Developer's tools shouldn't have dumbed down "user friendly" readouts like this when accurately being able to measure things is important.
Jason Evans commented
The only reason I use test explorer is because of MS Fakes. I use Resharper's test runner for all my tests, other then those using Fakes, simply because that test runner cannot work with Fakes.
Please can the Visual Studio team enhance the test explorer so that it's on par with what a professional developer would expect. Right now, it's very poor features make it a very disappointing experience to use.