Test Explorer is severely limited
I have to assume the people who developed the VS2012/2013 test explorer view never actually had to use it in production. The test explorer view is horribly limited in comparison to the one in VS2010.
1. Running tests does not open the Test Explorer window until after all the tests completed running. There's literally no indication that anything is happening. VS2010 immediately opened the window to show you that the tests were running.
2. Where are the columns? Most importantly: the error message column. if I'm dealing with multiple test failures, I almost always just need to see the error messages at a glance. Having to select each individual test just to see the result is annoying.
3. Where'd the ability to run tests in context go? There's no way to run an individual test without going into the full test list, finding it, and then selecting it just to run it. This might be okay if you have a project with 10 tests, but a solution with thousands of tests makes this arduous. This needs to be fixed badly. Either bring back the "run tests in context" button or at least add some right click menus for "Run this test" and "Run all tests in class".
4. The progress bar is terrible. A five pixel high bar might be useful to someone with an 800x600 monitor, but for the rest of us a barely noticable progress bar is not enough to know that all the tests finished passing. ****, half the time the progress bar color doesn't even change. It just stays red/orange rather than changing to green.
Please fix this. It has made working with unit tests far more of a chore than it needs to be.
#34: The search textbox will randomly not show up in the test explorer window. It's rare(I've only seen it happen twice) but it requires a VS restart.
#35: If you have a group of tests running and you cancel it, hitting "Repeat Last Run" only runs the tests that completed running and not all the tests that were queued for the last run.
Agree with all of Ross' cases. Something has to be done here. Why didn't you guys just work to gradually improve the test window from 2010 instead of throwing it out and putting some half-assed solution in as a replacement.
#33: No way to order the tests by the order they ran.
#31: The test list does not keep the currently selected/running test scrolled into view if its position changes. This is annoying if it's jumping between failed/passed group and scrolling out of view.
#32: The constant re-arranging of tests is equally annoying. A test that was ran will be at the top of the list, but if you debug the test and abort it it gets shuffled back to the middle somewhere and is no longer highlighted as the last-ran test.
#30: There isn't a way to do a ctrl+f find in a test output window. The only way to do it is to copy/paste the entire output to a text editor.
#26: If you're running a lot of tests(hundreds, thousands) and see a lot of failures, there is no way to quickly see if they're all coming from the same test class or not. This was possible in the 2010 test viewer since it had adjustable columns with this info. You might see this and say "That's easy, just group results by class.", which leads to...
#27: Group headers are useless. Text with the number of tests, that's it? It doesn't give a break down of passed/failed/inconclusive or give any indication at all about the status of the tests included. Now I suppose I could scroll through the huge list and wait till I find a plethora of red error icons, but that's not gonna happen due to...
#28: No way to collapse/uncollapse all of the sub sections in the test explorer. I've got a solution with almost 5000 unit tests. The "Group by Class" breakdown gives me a tree with hundreds of collapsed nodes. I know I'm not gonna click through every single one of them just to open them.
#29: #28 wouldn't be a problem if there was a way to group by one thing and then sort by another. Like group by class, then sort by outcome so all classes with failed tests show up at the top.
I feel like, after updating this post for over a year, that it should be fairly obvious that using a Tree View to view test results is very cumbersome and limited. Test results are one of those areas where a typical grid makes the most sense for parsing information quickly.
Jaya Rayasam commented
I would love to have something like output window while running the test to know on what step of the test it is currently running. Can we get that feature?
#25: If you want to run all tests except for one, you can't just do ctrl+a and then unselect the offending test. You also have to unselect whatever parent tree node it's in because otherwise the test will be ran anyway. This would not be an issue if we had sortable columns instead of a treeview control.
Tudor Turcu commented
The progress bar is again visible in the Update 4 for VS 2013 (they made it much thicker again so no it's possible to see the progress without a loupe)
Going back to #11 again: We have a backup script at work that deletes that TestResults folder before backing up our source code. So this ends up deleting the mdf file that VS makes in there. Why it doesn't just recreate the file(like it does after restarting VS) is beyond me.
Going back to #23 again: Here's a random sampling I get from hitting "Run All" in the Test Explorer window. For the current solution I have open, there are 4474 tests.
------ Discover test started ------
========== Discover test finished: 3838 found (0:00:03.811) ==========
------ Run test started ------
========== Run test finished: 2645 run (0:01:27.939) ==========
So it found 3838 files, not finding about 500 tests, and then didn't run another 1000+ of them.
If I rebuild the solution, it will then say it found 4474 tests, even though they were there in the first place. I'm assuming the discover tests count is only for the projects in the solution that were just built, but it doesn't explain why chunks of tests get ignored when attempting to do a Run All.
Going back to #23: At some point, "Run All" will only run all the unit tests for a single test project. It completely ignores all the other test projects in the solution.
#24: The Deploy directories in the TestResults directory do not get deleted on failed tests. That would be fine, except it's ignoring the "Limit number of old test results to" setting. I have it set to 1, I have never seen a need for keeping old test results. This is just wasting drive space for me on huge projects.
Carl: To correct myself: "It just stays red/orange rather than changing to green when all the tests have passed after a previous failed run". Having said that, I think it was fixed in one of the updates.
"It just stays red/orange rather than changing to green."
This means that your tests have failed...
#23: If you hit the "Run All" button and then open the the test explorer while it's still running, the progress bar is grayed out and makes no indication of progress. This only seems to happen if it's the first time opening the Test Explorer after opening Visual Studio.
#22: Occasionally "Run All" does not actually run all the tests. It doesn't give any indication that it didn't unless you decide to scroll through all the tests to see if any of them are grayed out. The only fix is to reopen the solution. Rebuilding doesn't fix it, even though the test window will say it has found all of them.
#21: The biggest problem for me is that when I have multiple test properties with overlapping values, the Filter can't understand I want to map Trait/Property A with Value 1. It brings up all tests where either the key or the value match A or 1. I have thousands of tests with 20 different Traits, and this UI not only doesn't scale, it's basically broken.
Going back to #11, this seems to happen to me when I leave Visual Studio open overnight. Without fail, the moment I try to run tests for the first time I get hit with "Underlying provider could not be opened" errors.
#20: You can't undo the removal of tests from a playlist.