In my previous article I wrote about the amazing new features of Visual Studio 2012/Microsoft Test Manager 2012 and mentioned that the guys from Microsoft were going to change the world of QA by making an automation platform that can be used by non-developers (exploratory testing automation, running automated scripts from any place inside manual test run, etc.)
Well, they are not the first company to claim that their tool will change the world. Automation tools were advertised as tools you can use without knowing any programming language since first such tools appeared. You just need to push a big red ‘record’ button and the program will do the rest. Record your perfect test; carefully make all your moves. Then, you need to push a big green ‘play’ button and the test will run without any extra action from your side, cleverly repeating everything you just recorded, and you can just relax in your big chair with a mug of coffee, enjoying the rise of the machines. Sounds cool, but usually everything happens differently.
When the recording automation tool fails to recognize some part of a system under test, and you have some rubbish as the outcome, or even just a load of errors and no outcome at all. You need either to do some refactoring of the system under test making it more testable (and these days testability is officially one the quality attributes of any big program), or rewrite the auto test’s code inserting parts that cannot be written using the record and play approach, or even both! It looks like we need to write some code.
Next up, auto tests will work correctly while the system under test remains unchanged. A change of environment can cause failures too. Upgrading the version of a test tool can spoil your tests too. You can avoid such situations if you start writing your auto-test from the very beginning in the right way. You should use an object model, all the parts describing interfaces, have constants and variables for interaction with the system under test should be separated from the test scenario, all input and output data should be stored separately and it would be great if you can use a data driven approach. By the way, sometimes you need to rewrite standard functions about working with the environment too (file system, network, memory, performance counters, maybe databases). It seems we really need developers here. Even worse, we need good developers here.
Microsoft want to make the first part (test recording) as easy as manual testing itself. In this case, you do not need to make a good framework or architecture for your tests. If anything changed, you just start regression (and well, you need to start regression anyway) and make a brand new set of auto-tests.
Why do I want to believe them? Because it much easier for them to avoid all problems with integration between their test tool and the system under test if both of them are written for the same platform. And, that’s why I think that automation from Microsoft will work well only with programs developed in Visual Studio and with programs written on other platforms all the cons mentions above would be applicable.
But if it really works well at least with programs developed in VS it would be a noticeable improvement in QA Automation. So you only need to deploy Microsoft Test Lab and enjoy brand new level of testing.
How easy, user-friendly, and painless is the process of deployment Microsoft test lab? You can read about this in our next article. Stay tuned.