I met with a customer this week and they had an interesting question they wanted to answer. They wanted to answer the question “How much ‘code coverage’ am I getting with my manual testing efforts?” They literally were asking how they could know what assemblies/classes/methods/lines of code were executed during a manual testing run. I actually think the question inside their question really was “is my manual testing covering all my business cases?” Additionally, they wanted to be able to track this level of code coverage as multiple testers were accessing their QA systems over a period of several hours.
You may be thinking, but if they had automated tests, this would be easy! Yes, you are right, but the difficult part is creating automated test cases. If you have a large legacy system, and there are no unit tests that you can automate, simply creating unit tests can be a daunting, if not impossible task (but that’s a story for another day…). Anyway, this customer does not have an investment in automated tests of any kind, so they wanted to know if, with Visual Studio 2010 and TFS 2010, they could use the code coverage tools to gather this information during manual test runs. They do have a good set of reproducible manual test cases, so that’s a great start.
The short answer their question is unfortunately “no” – you cannot gather code coverage with manual testing, and in fact, you need to be executing the tests on the same machine you are collecting coverage information from – remote collection of coverage is not supported. That fact alone will generally throw a wrench into the works. This is documented on MSDN here: http://msdn.microsoft.com/en-us/library/dd504821.aspx
So what do we do? How can we use Visual Studio 2010 and TFS 2010 to answer this question, especially for this scenario where you don’t have an investment in automated tests?
Using Test Manager 2010, Visual Studio 2010, and TFS 2010, there is a pretty straightforward workflow for getting this done, and it ultimately involves getting invested in automated tests. I’m going to give you a high-level description here, and will drill down on some of the gritty details in later posts to come.
Here’s the basic idea:
Step #1: Using Test Manager 2010, testers create manual test cases. They may already have these in some form, but they need to be tracked in TFS 2010 as Test Cases (a specific work item type). From those test cases, they can create what are called “action recordings” – which are replay-able/re-executable UI-level recordings (ie. click this button, type in this textbox, close that window). Testers can create these action recordings when they execute a manual test case the first time, and can even replay all or part of that action recording in subsequent manual test runs. The action recordings are stored as a part of each Test Case work item.
Step #2: Testers next need to be able to flag a manual test case, that has an action recording, as “ready for automation”. This might be a simple flag set on the Test Case work item type, or perhaps a custom workflow state. The point being, testers need to be able to identify a specific test case and action recording as one that they are ready to hand off to the development team to turn into an automated test.
Step #3: Developers are informed that Test Case work items in the “ready for automation” state. This could be something as simple as just a work item query that lists test cases that are still flagged as “manual” but also “ready for automation”, or maybe a TFS Alert, or just good old team communication.
Step #4: Developers create a new Test project, and add a CodedUI test for each test case that is ready for automation. When the CodedUI test is created, you can specify an action recording from an existing Test Case. This is the “magic” of turning a manual test action recording into an automated test – once a CodedUI test is created from the action recording, we now have something we can run unattended. Developers can then check these CodedUI tests into version control, and should update the Test Case to indicate that it is now automated!
Step #5: Next the development team can create an automated build that executes the complete set of CodedUI tests. Note: I’m going to leave out some detail in this post, but will return to this topic later – this involves a little bit of work and setup to have the CodedUI tests executed in the right place so that Code Coverage can be captured. The Code Coverage reports and Excel workbooks that area available with the out-of-the-box process templates will provide a historical trend of code coverage as it increases as new CodedUI tests are added via the tester->developer workflow described in steps 1-4.
This strategy accomplishes a few things for you:
- A repeatable process for gathering code coverage metrics – not subject to manual testers doing slightly different actions during different test runs
- A workflow for establishing new test cases to be added to the battery of automated tests to measure code coverage
- A single store of code coverage that will allow reporting of trends and comparisons over time
There is an alternate route here also, and it requires configuring the Lab Management component of TFS 2010. Configuring Lab Management, however, puts the ability to execute the automated tests right back in the hands of the QA team, and does not rely on an automated build to execute. Again, that’s another post for another day. My goal is to come back to each of the individual steps described above and post more details.