enable changing the order of test cases on the web gui and let them be tested in this order for both manual & automated tests
With the MTM 2012, it is possible to change the order of test cases within one test suite and this order is considered when testing. This functionality is missing in the web environment. It is not possible to order test cases and then have somebody else test them in a specific order. If a test suite is run, the tests are ordered by ID only. Make this work for both manual & automated tests.
In Test Manager the user has the option to have an 'Order' column, however when I go to TFS 'Test'/Column Options there is no order option.
When I have manual test suites created in MTM I can designate an "Order" for the tests. If I execute (run) the tests in MTM, it runs in the order that is associated to the suite. When I have users log into the Team Web Access site to execute the manual tests, the tests do not execute in the order set in MTM, they execute according to the ID associated to the test case.
In MTM 2012 you are unable to change the order of test cases in a suite that is requirement-based. In a normal test suite it is possible, but it would be great if we could change the order also in requirement based test suites.
MTM 2012 doesn't support ordering tests that are created from Query Based or Requirement based test suite. It would be good to have this feature as I am trying to Link the Automation with MTM tests.
So far I have used Order Test from Coded UI Solution file for triggering Automation, But now when I am trying to do it from MTM I don't see an option to Order the tests for Requirement based and Query based testsuites.
Need to be able to Run Tests in Suite from Command Line in Order in MTM.
Using below command im able to execute automated test case in a test plan(MTM)
"C:\Program Files (x86)\Microsoft Visual Studio 10.0\Common7\IDE\TCM.exe" ru
n /create "/title:XXAutomation" /planid:17 /suiteid:528 /configid:3 /co
ngsname:Automation default" /builddir:C:\Share\xx\XX_20130515.3 "/test
These test cases are executing based on the test ID(ascending) not by order of test cases in MTM.
It would be good if you provide a option to run test cases in a test suite according to MTM order
When you need to update the order of a high number of test cases (20+) current workflow is tedious. If you try to update multiple order numbers at a time, it is confusing because sometimes you don't get the expected order. You should enable drag & drop of test cases to expected order.
Currently only test cases in a test suite that is not a query-based test suite or created from a requirement or user story can be reordered.
What's even worse is that in MTM in the Plan Stage it appears as if you CAN reorder them but when you move into the Test Stage every test in the Suite is reordered by ID - this is infuriating and makes a complete nonsense of test cases that must be followed in sequence.
The current Order feature in Dev11 for reordering test cases in a test suite is clunky. Instead of navigating away from the Contents page, I'd like to have the ability to reorder TCs by dragging/dropping them directly in the TC list of the suite.
If you have a test plan in MTM and you begin to automate tests within that plan you don't have the option of running those automated tests in the order in which they were originally defined. The "order" field doesn't apply when in reality it should make the tests into an "ordered" test.
Unlike Static test suites, you cannot currently specify an order for test cases created under a specification based suite. Would be useful!
We are currently evaluating this suggestion to be included in the TEST hub group in team web access. We are also wanting to scope the ordering solution to address only the manual test cases present in Static, Query and Requirement-based suites.
For automated test cases, the best practice is not to have dependencies between the automated tests. For folks who need the ordering between different test methods, we recommend using the “Ordered tests” in its current form (we are not planning to enhance the ordered tests in future though).
Shubham Srivastava commented
Also my observation is that "Order" works only for Static Test Suites and not available for Query based suites and Requirement based suites . I think Test Manager and Visual Studio Online both should have the option to do reordering for any type of suite.
I can only agree with John Public. I am curious that have the product team here ever done real automated testing before?
John Public commented
This is sorely missing from web view and rather sad that after bringing this up over two years ago and receiving that many votes all we get is an 'under review'. This feature is direly needed.
I disagree that automated tests should not have dependencies. I understand the motivation for that statement, but testing a few dozen variations of a complex report is way faster when allowing for dependencies because each test case only has to do minor tweaks to data rather than wipe everything clean and spend extensive amounts of time to enter data differently. That is a total waste of resources even when a machine executes the tests. Same with tests that hit services that charge per hit (some geolocation services do that). Preventing unnecessary service calls not only saves time, but also a lot of money!
Besides that, ordering tests in web view or anywhere else as no impact on independent automated tests. They will run in the set order, but they can run in any order and yield the same effect. So picking just one order will be perfectly fine.
The more I think about it the more it seems as if this is just some statement to have a bogus reason not to implement this feature. Please prove me wrong!
Paolo Sotgiu commented
The ordered execution of test is a must have for us in the scope of an automated execution of the regression test. I can't really understand what is the point in supporting it for manual tests and not for automated test (talking about coded UI)...the setup for some tests in our cases will require to duplicate a lot of the logic we already have in other test methods which are normally executed during the script (test the record creation, then the edit and then the removal....)....we will triplicate the lenght of tests and they already require 2 days to complete. At least give us the option to decide to use the ordering or not, as it is available already in MTM. I really miss NUNIT right now....
Gerhard Matzen commented
I do not like to use ordered tests for automated tests. This is because the ordered tests is reported as a monolithic pass / fail in MTM. I would much rather see individual test results instead of a monolithic pass / fail.
Sascha Kolb commented
If you really want to do this only for the Manual Testcases, plesae split this idea and see where the votes go. Thanks
Sascha Kolb commented
I dislike the way this Feature is going.
I prrincipal agree that,
"For automated test cases, the best practice is not to have dependencies between the automated tests"
But, there is options, where this is not feasible, or expensive havin each case Independent.
We e.g. use Testcases also ón a System Level with a mechatronic System, where bringing it to a inital state (rebooting) is expensive, thats why our AUTOMATIC testcases are dependend. But we want quick Feedback. Thats a trade off.
I would really see a Option to Order ALL Testcases in future also via web.
any workaround to this?
ahora necesito conocer cómo se llama y dónde está ese campo "order" en SQL 2008 para imprimir un reporte... alguien sabe??? gracias
Yes... but in spanish...
desde el Test Plan hay que crear una Suite, entonces adentro de la Suite vinculás el Requirement... luego, copiás los Test Cases del Requirement sobre la Suite...
Te posicionás sobre la Suite y arriba a la derecha vas a ver un botón llamado "Order". Hacés click en ese botón, y listo... configurás el orden de los casos de prueba para esa ejecución o las posteriores.
Dee Huskey commented
is there any plan to add the ability to order tests on the web, and if so when? Has a release been scheduled?
Need this now
Lucy Goddard commented
This is critical for us !
any updates on this issue, if not is there any work around to solve it
Any expected date when this Feature would be released??
It is a very important Feature. It should be implemented as soon as possible.
Is it available now?
CUIT automation requires one to point a build at a suite. The test cases were not created in order and but run in the order of the ID not the Order column. So, the tests don't run in the expected order. This creates so many issues from tests failing or having to write the whole test suit over again to get the tests to run in the correct order. This makes CUIT automation virtually useless since it ruins your burn down.
Lisa Woodhead commented
is there any update on this?
Lisa Woodhead commented