Drop "Test Explorer" view and revert to the VS 2010 Test View and Test Results paradigm
I hate to bring forth blank statements like this but the Test Explorer in VS 2012 sucks. I waiting for the beta cycle to finish because I was under the impression serious improvements where in the works but now that I have the RC in my hands I'm very disapointed.
This is a major regression.
1) Believe it or not I can't even copy exception messages from the unit test results
2) I Can't run a test in context like I used to be able to (or i don't know how to)
3) I can't group tests by class name or any other way
4) I can't control the columns I'd like to see.
Either I'm completely missing something or you guys completely missed something. I hope its just me. For now I'd suggest you reintroduce the VS 2010 Test View as a solution.
Most of the requests from the initial idea and comments have been addressed in Visual Studio 2012 RTM, Update 1, and the upcoming Update 2. If you still feel that something is missing please open up a new more specific idea.
I agree the 2012 Test Explorer is unbelievably lame. How to make it go back to the way it worked/looked in 2010?
Hugh Proctor commented
This has to be one of those MASSIVE FAIL moments for the VS 2012 dev team. I don't think that they've heard the saying 'If it ain't broke don't fix it'.
I don't understand the Test Explorer, I would expect the most basic level of functionality to be there. For example, I've grouped my tests, I want to run the grouped tests - this is not possible by selecting the group instead I have to select (manually) all the tests within the group. I'm testing basic functionality and at its minimal this is 243 tests.
You would think that I'd know if the tests are running or what they're doing; a trace. If test hits an exception or an error then Test Explorer just breaks, stops! doesn't move on to the next test, doesn't send a result or any indication that there was an error, no log, (with code or with itself!), it simply sits... I have no idea what it's doing!??? WTF!!! I hate you guys, Microsoft! Bring back the 2010 development team!
I really don't know what to do... Resharper, NUnit.
It seems that you guys did not look at what was there before and went blindly forward with no good ideas or prior experience to know the needs of your customers. If it weren't Microsoft and a monopoly you'd go to the wall as business for not meeting the most basic of customer requirements.
I haven't got time to list all the things wrong with it... it's your primary business to capture the business needs upfront, market analysis, feature list. You should be fired and you should be asked to return the money that they paid you, as you failed to deliver anything close to a product fit for purpose.
In the test explorer ability to see what tests are running is very required.
I am using VSTS2012 Ultimate Update3. I still can NOT copy the console output from the test output window. This is really required for us to re-run the test manually by copying the text from the output window.
Ability to group the test is lost. I have 1000s of tests. It is very difficult to go through the list to locate specific tests
MSFT please listen & respond to the customer.
Richard Andersen commented
I agree with PaoloPablito and the orginal suggestion! I do not like the "Test Explorer" and I am running RTM. The thing I hate (other than crashing) is no indication that it is running...
Microsoft says "Competed" but I tried the Update 2 and I still feel uncomfortable with the VS2012 autmatic testing infrastructure. See my post dated "November 30, 2012 11:53 a.m" below, very few things really changed in Update 2. So I still use VS 2008 and VS 2010 for many of my automatic testing chores.
Could we please get "Associate Test to Case" in the right click menu back. It was extremely helpful to have the search dialog come up and to be able to search for a test case by id. It made my day much more efficient.
Daniel Meech commented
It would be nice to know what tests are currently running as we quite often experience the Test Explorer hanging. We have no idea if it is a bad test or just the test engine!
Geoffrey Rayback commented
My biggest problem is I can't tell which tests have run, which are running, and which didn't run in any given test run. It seems to me that tests that are running have a success icon (?) which is ever so slightly lighter green than test which passed? Is that right? And the tests all reorder themselves (randomly?) every time you do a test run so I can't find my ever so slightly less green test which maybe passed or is possibly still running or maybe didn't run at all? What? What was wrong with the little spinner on tests that were running and no icon at all on tests that didn't run? That was impossible to mistake. Now I end up running all my tests multiple times just to make sure they really ran. What a time saver!
Radu Crisan commented
In VS 2012 Test Explorer I do not see a unit test description anymore.(the one I add as attribute Description("my unit test description"). I cannot choose what columns to display. Why on earth you removed these things?
I am not so concerned with missing features, more with stability. If you search the Internet with the following keywords, you will see what I mean: test explorer visual studio crash
Other major bugs - TestContext doesn't work reliably when upgrading .NET 3.5 unit test projects, can't "view output" of tests (thus can't view logs anymore), frequent crashes of VSTTExecution engine. Despite opening Microsoft support ticket (where the undocumented workaround of toggling target framework from 3.5 to 4.5 to 3.5) worked in some cases, still can't realiably use TestContext in our tests. This works without issue in 2010
It would save me (and MS) so much time if they didn't cache anything. I find that Test Explorer often doesn't list all my tests and Pending Changes window often misses some pending changes. There is simply no need for cache anymore, why do people continue to write and use them? They waste far more time than they save.
Sean Yao commented
This new test explorer still sucks even in Update 1. Besides the ones mentioned above, I still missed the following:
1) It doesn't show the test case that is current running. If a broken test which runs forever, I can't find the test easily.
2) I can't sort the tests by StartTime or EndTime, which is useful to trouble shoot some test teardown bugs.
Joshua Weber - msft commented
Group by class will be added in Update 2. The Visual Studio ALM Blog http://blogs.msdn.com/b/visualstudioalm/ is a great resource for announcements and further information on Updates as they are announced.
"Most of the requests from the initial idea and comments have been addressed in Visual Studio 2012 RTM, Update 1, and the upcoming Update 2. If you still feel that something is missing please open up a new more specific idea"
We'll do it... when Update 2 is released.
What are your plans about it ?
Kevin Logan commented
I'm using update 1, which is better, but I'm missing group by class name. I'm using a BDD approach so I have test classes named When_logging_in and When_navigates_to.
Otherwise, I find my self using a combo of Resharper's unit test runner and this one. In 2010, I was only using Resharper's. So that's an improvement.
The new Test Explorer just sucks. We should be able to use the old interface if we want.
Microsoft promised improvements to the Unit Test interface for the Update 1, in fact now the Test Explorer has new filtering capabilities. I still cannot filter by namespace like in good old Test List Editor, but I can simulate it by filtering on Full Qualified Name.
But other things in the Unit Test world of VS 2012 leave me disappointed, they are the same things that left me disappointed before applying Update 1:
- it looks slower to me. I created two copies of the same solution and opened it with VS 2008 and 2012 on the same Win2008R2sp1 machine (a virtual machine on VMWare LabManager, not a real machine, but it's the SAME machine, so they are on an even footing). I selected a bunch of random tests and executed them on both VS 2008 and 2012. VS 2008 executed my tests in a few seconds, with VS 2012 I had to wait at least 18-20 seconds to see the results. I retries the same tests multiple times but I got the same results. It looks like the problem is the "initial sprint", not the actual test execution. Even if I select a SINGLE test, it takes at least 5 seconds to show me the results, and then Test Explorer says that the actual execution took only 129 milliseconds! Is there some VSHOST trick to activate?
- the beloved Test List Editor is STILL deprecated and cannot run selected tests
- the Test Explorer still doesn't allow you to select individual tests with checkboxes like the good old Test List Editor. I often find clicking checkboxes more convenient than CTRL+clicking the desired tests.
- there's still no test toolbar (or at least I cannot see it in the toolbar list)
- When I'm in the source code and I run a test, I'd like to be brought immediately to the window that shows my test running, it does not happen in VS 2012 (but it used to happen in previous versions). It happens later when the test finishes: but if there's a filter in Test Explorer, I cannot see the results, I have to clear the filter.
- If I click Debug or press F5 in previous versions, all tests in the current PROJECT (not solution) are executed. VS 2012 does not execute anything, it complains that this is a DLL and cannot be run.
- the Test Result windows does not work as it worked in VS 2008 and 2010, it does not show anything. Test results are shown in the Test Explorer exactly like before applying the Update 1, this view is clumsly and I don't find it useful like the good old Test Result window. To see the error messages, I have to click on single failed tests, I'd like to have a look at error messages at a glance to get the big picture. This is very important for me when I try to use a TDD-like approach.
Is there a way to customize the data shown by Test Explorer? Where? (VS 2010 has a "Test Tools" branch in the Option dialog, but VS 2012 hasn't it).
Sorry for the long list of complaints (which, by the way, I already posted on Brian Harry's blog) but I have to say what I think. I still prefer the testing interface of VS 2008 and 2010 (and I prefer 2008 when it does not force me to click Refresh in the Test List Editor)
I agree Test Explorer in the Pro sku is only useful to kick off the test and see the final result. As a Tester I want/need to know what is currently executing I have no view into what tests are executing or what still needs to be executed in a run, I used to get this information in the Test Results window but with the Pro sku nothing is showing in the Test Results window. Either something is broken on my machine or for this sku the window is not hooked up.
Also, how do you stop a run? For example, I kick off a run with 10 test cases and one of them fails and I want to stop executing the rest of the test cases how do I do this? I currently see no way to stop/pause a run that is executing - which kind of goes back to the fact I have no indicators that something is executing this is EXTREMLY frustrating.