Allow tests to be grouped by class, category, and namespace in Test Explorer
The Visual Studio team has obviously taken the time to add grouping to the new Test Explorer since we can group by test outcome and duration. However, sometimes you need to run a group of tests such as all tests in a category or namespace and this is not currently possible with Test Explorer. It would be nice if the grouping options were expanded to include at least class, category, and namespace. Then it would be possible to run all tests in a class, category, or namespace from the Test Explorer.
Visual Studio 2012 Update 1 http://www.microsoft.com/visualstudio/eng/visual-studio-update has added enhancements to support both grouping and filtering by Project and Traits (category). Use of these features is detailed in this blog post, http://blogs.msdn.com/b/visualstudioalm/archive/2012/11/09/how-to-manage-unit-tests-in-visual-studio-2012-update-1-part-1-using-traits-in-the-unit-test-explorer.aspx
Group and filter by Class is completed and will be available in Update 2.
I just installed Visual Studio 2013 Update 1 and I was very disappointed. Microsoft here above says "COMPLETED" but all the VS 2008/2010 good features about automatic testing are still missing.
Just to recap them:
- VS2010 separated test preparation and test execution, VS2013 mixes them in Test Explorer; sometimes it may be useful, but for me MOST of the time it's not useful at all. Instead, it's confusing. A filter filters both executed and "not executed" tests, so if I run a single test directly from my test source code, and then I open Test Explorer, I don't see the test and I have to clear the filter.
- Furthermore, previous test results are mixed with new test results in the Test Explorer. This is confusing.
- VS2010 had test results in table form, with customizable colums, allowing to see all error messages at a glance. VS2013 has not. This is a regression.
- With VS2010 I could easily tell which test was running at a given time
- With VS2010, running a test automatically opened the running & test results pane. Ok, I understand some developers did not like it, but I DID like it, and an option to get back the old behaviour would be useful.
- VS2010 had an option pane to customize the automatic testing behaviour
- VS2010 had checkboxes to select tests to run (VS2013 has just clumsy CTRL+CLICK)
- Sometimes tests are not run at all, as if the test running process had a problem, but no errors are displayed, sometimes they are run OK.
I'm sad about this, and unfortunately I cannot vote again on this item.
p.s. In the past year, I manage to still use VS2008/VS2010 for testing while developing on VS2012.
Now I'm forced by my company to move to VS2013 for a new big project, and I'll be forced to use VS2013 for testing too. :(
People in my team miss the good interface of VS2008/2010. Please, Microsoft, bring it back.
New feature request - have visual studio 2012's test features not be worse that VS 2010. WTF
Microsoft says "Competed" but I tried the Update 2 and I still feel uncomfortable with the VS2012 autmatic testing infrastructure. See my post dated "November 30, 2012 11:59 a.m" below, very few things really changed in Update 2. So I still use VS 2008 and VS 2010 for many of my automatic testing chores
Please add some sort of "immediate" display so that I can see what test is running?
I can't believe this wasn't part of the update, this is such an enormous nuisance. I wish I'd never started unit testing in VS2010...
Is there any way MS could just reactivate the original test list editor so we can use it again? Or as a plugin? It seems this is just not a priority. it also seems astonishing we're still waiting, since the feature seems just so dead simple...
Microsoft promised improvements to the Unit Test interface for the Update 1, in fact now the Test Explorer has new filtering capabilities. I still cannot filter by namespace like in good old Test List Editor, but I can simulate it by filtering on Full Qualified Name.
But other things in the Unit Test world of VS 2012 leave me disappointed, they are the same things that left me disappointed before applying Update 1:
- it looks slower to me. I created two copies of the same solution and opened it with VS 2008 and 2012 on the same Win2008R2sp1 machine (a virtual machine on VMWare LabManager, not a real machine, but it's the SAME machine, so they are on an even footing). I selected a bunch of random tests and executed them on both VS 2008 and 2012. VS 2008 executed my tests in a few seconds, with VS 2012 I had to wait at least 18-20 seconds to see the results. I retries the same tests multiple times but I got the same results. It looks like the problem is the "initial sprint", not the actual test execution. Even if I select a SINGLE test, it takes at least 5 seconds to show me the results, and then Test Explorer says that the actual execution took only 129 milliseconds! Is there some VSHOST trick to activate?
- the beloved Test List Editor is STILL deprecated and cannot run selected tests
- the Test Explorer still doesn't allow you to select individual tests with checkboxes like the good old Test List Editor. I often find clicking checkboxes more convenient than CTRL+clicking the desired tests.
- there's still no test toolbar (or at least I cannot see it in the toolbar list)
- When I'm in the source code and I run a test, I'd like to be brought immediately to the window that shows my test running, it does not happen in VS 2012 (but it used to happen in previous versions). It happens later when the test finishes: but if there's a filter in Test Explorer, I cannot see the results, I have to clear the filter.
- If I click Debug or press F5 in previous versions, all tests in the current PROJECT (not solution) are executed. VS 2012 does not execute anything, it complains that this is a DLL and cannot be run.
- the Test Result windows does not work as it worked in VS 2008 and 2010, it does not show anything. Test results are shown in the Test Explorer exactly like before applying the Update 1, this view is clumsly and I don't find it useful like the good old Test Result window. To see the error messages, I have to click on single failed tests, I'd like to have a look at error messages at a glance to get the big picture. This is very important for me when I try to use a TDD-like approach.
Is there a way to customize the data shown by Test Explorer? Where? (VS 2010 has a "Test Tools" branch in the Option dialog, but VS 2012 hasn't it).
Sorry for the long list of complaints (which, by the way, I already posted on Brian Harry's blog) but I have to say what I think. I still prefer the testing interface of VS 2008 and 2010 (and I prefer 2008 when it does not force me to click Refresh in the Test List Editor)
Note: Oleg Sych's answer at http://stackoverflow.com/a/13588093, mentions that the group by class did not make VS 2012 Udate 1. This is really really important as the class is the most useful grouping of tests for many of out projects.
Mikhail Poliakov commented
In VS 2010, I often ran all tests in a test class (Ctrl-R,C), now it seems not working.
Another pain is that the TestContext's output is unusable now, since the output content is not line -wrapped, you can't scroll it, copy/paste the result or do anything at all with this.
Terje Sandstrøm commented
This is now in the CTP for Update 1 http://www.microsoft.com/en-ie/download/details.aspx?id=34818 , and Update 1 should be out real soon now. Blogged it here : http://blogs.msdn.com/b/visualstudioalm/archive/2012/11/09/how-to-manage-unit-tests-in-visual-studio-2012-update-1-part-1-using-traits-in-the-unit-test-explorer.aspx
I've never used ReSharper test window in VS 2010, I just didn't need it. VS 2010 Test Explorer had every thing I need. But now I use this plugin every day, because VS 2012 Test Explorer has literally NOTHING.
Same problem here... We have differrent TestProjects for Unittest and Intergration/database tests. In VS2010 it is possible to run tests per project. In VS2012 it is not possible.
Howard Richards commented
I used VS2012 for development, but load VS2010 for unit testing.
That kind of says it all - the testing functionality in VS2012 is primitive.
Thomas K commented
Just to make the pain that the changed UI has brought us clear:
We have a big solution with > 13000 tests. Some of the tests fail, but we still need to run > 6000 of them. We tried with test lists, but maintaining them is not practical. So we switched to TestPriority(X) and use /minpriority:20 to filter the known failures away. Worked with VS2010.
There is simply no help from the UI at this point in time and running tests is no fun. Also, because the test results UI has changed to the point where this approach is not feasible. And no, we do not want to mark 6000+ tests with another attribute in order to include them; for us, it is more practical to mark only those tests that we do not want to run.
The current UIs with their emphasis on tree views might be cool for smaller numbers of tests/files, but is impractical with large solutions.
VS2012: Two steps ahead, but one backwards and shoes lost!
Olexandr Kravets commented
I am very disappointed too.
Without previously available features, Visual Studio Team MADE huge step back.
UI, icons, and test explorer ...
Also before I was able to see, which test is currently running. Now - NO ...
Sad VS 2012 Test Explorer User commented
I wanted to migrate to all my Nunit tests to Mstest and boy was it huge disappointment. I just don't understand what MS was thinking when they made Test Explorer, it lacks even the basic features of a test runner. Might be because MS hired a fresh college graduate as a product manager for this feature.
Tommi Kaskinen commented
I can't believe what MS has done with 2012. Compared to VS 2010 test view, test explorer
is unusable. I have multiple projects in my solution and i always sort them at least by project.
I've always used test view for running and finding correct scripts.
We cannot use VS 2012 until this issue is fixed. MS development has taken a major leap backwards.
Allow for partial copy/paste from a unit test run details -- ie. I want to copy a path or exception name from a unit test error, and not the entire test run details.
Joshua Weber - msft commented
Thank you everyone for the feedback on this feature. We agree that different grouping options are essential for managing tests. Realize that the new Unit Test Explorer has been written to be test framework agnostic. To fully support 3rd party testing frameworks required a significant change to the underlying test execution architecture, hence the removal of the test list editor. It also has resulted in careful implementation of grouping options.
In the upcoming Visual Studio update we have added support to allow for grouping by project and trait. Trait being the generalized framework agnostic grouping, supporting framework specific method decorations (testcategory being one in the MSTest framework). We are also looking into further supporting class as a grouping option.
VS ALM Tools - PM
James Treworgy commented
How about just un-deprecating Test List Editor until this is resolved? It worked. It wasn't the prettiest thing in the world, but hey, you could actually find the tests you were looking for.
I'm not sure if this was just a massive oversight, or there's some technical reason why Test List Editor could no longer be supported and someone just didn't finish Test Explorer in time. But it really stinks when something you use about about as often as you hit the "enter" key is suddenly gone. Don't give us new toys then take them away!