Test automation is great. Test automation is awesome. Test automation allows you to control your product 24/7. You can load your product with production or production-like data and see how it works. You can emulate any activity or workflow for your product. You can make fancy analytics about the quality of your product and make team performance reports make sense. But I suppose there are more fails then success stories about automation implementation. Why?
Because there are still some risks associated with QA automation implementation. Here I want to talk about these risks, their general solutions, and how we in DataArt address them.
QA Automation is either cheap in implementation and expensive in further support, or cheap in support but very expensive in implementation
In general, this is the real situation. In order to create good automated tests, you need to create an initial structure for them, then separate the test data and interfaces and then add the tests themselves. This ensures your tests can be changed easily if the tested system changes.
If you work with a vendor who offers you a QA automation service, suggest they implement them on a fixed-price basis and further development set-up as a T&M/dedicated team solution. Pay special attention during first phase; engage some tech guys from your side to understand what is happening here. If automation implementation requires some R&D work – suggest your vendor share this investment.
In DataArt, we have a number of solutions to this problem. Firstly, we already have some frameworks (we have our own frameworks for Selenium, Virtual Lab by Microsoft, etc.) so all we need is to adapt them to the current project’s needs. This allows us to significantly reduce the time and cost of initial QA automation implementation. That means you can enjoy the first results of automation faster and at a lower price.
Secondly, if the required automation tool is new to us, it will require a research project, and we perform some research and initial work as our own investment and then add the automation only after we will be sure it will help.
Thirdly, we always try to assess whether test automation makes practical sense for a specific project. Sometimes it is easier to automate some parts of the QA process and some operations make manually. This approach allows saving significantly on automation costs without losing quality (and sometimes even increasing it).
QA Automation is a black box. A customer, who is not a skilled developer can’t understand what is happening if tests are incorrect, and what the numbers in test reports mean
This happens if you are using QA automation without any reporting tool. So discuss reporting requirements before starting the automation process. Check the reporting tools suggested on the market and choose the one that best suits your needs, or make a list of metrics you want to collect and make the vendor choose the proper reporting tool. Sometimes it is easier to write your own reporting system than to customize and integrate an existing one. Then you ensure that tests themselves are correct, check scenarios and test data, or ensure that your vendor reconciles it with you or the product owner.
The DataArt team focuses on being more than just another vendor; we are also experts and consultants for our customers. We use a number of tools that can make QA automation transparent, clear and even manageable from a non-tech standpoint (Fitnesse, Practitest, Microsoft Test Manager, etc.). Our QA managers have business analyst backgrounds and can explain every metric from the QA field in terms of business value, plus we always make some discovery phase to achieve the right types of test data and test scenarios.
QA automation is an enterprise solution. It makes sense only for big long term projects
In general, the longer and bigger the project the bigger profit from implementing QA automation. But there are still a lot of situations where automation can help small projects. There are always some parts of the manual QA process that can be automated. Test data generation, mobile and cross browser automation, basic load tests etc. Such an approach is called semi-automation testing and a vast majority of DataArt QA engineers can do it and understand where this technic can bring best results.
QA Automation only works while you are working with the current vendor
This can be true, if you consider automation as some side project or even side activity for your main project. Treat automation as a separate software project, the same as a general one (even if your vendor has another view on this).
That means you need well-commented code, a portable environment (cloud or environment on your side), wiki or another knowledge base, and a guarantee for some period after the project is completed.
In conclusion, I’d like to say that there are certain ways of keeping an eye on your product at each stage, but if you want to be sure that you have everything under control 24 hours a day and at the same time you want to live your life, then probably you should spend some time on setting up some automation now, and then be confident that whatever goes wrong, you will know about it right on time to prevent any unforeseen consequences.