Too Many options
Yes, this space is more crowded than ever before. It’s easy to get lost with most tools talking about “fancy” capabilities but failing to get the basics right. Success rate from test automation still remains way below expectations. It is important that a tool is capable of handling the complexities of a modern application landscape and at the same time, help you build sustainable and low maintenance automation.
Here is a simple framework you can use to help you maximize value of your time and focus on critical aspects you expect from tool.
Narrow your list down
Sometimes when doing comparisons, it’s easier to strike out some options based on obvious factors and work with a shorter list. It helps maximize your time on trial and evaluation and really ensure it works for your business and app landscape.
- Easy to get mislead by traditional “record and play”. It’s marketed and comes across as magically fast, this technology is decade older. You can put lipstick on a pig, but it’s still a pig J. The link to blog goes into more details of why record & play just doesn’t work in real world.
- Stay away from tools whose core expertise is not automation, like model-based testing tools that “integrate” with automation tools. Or tools that started out as desktop application testing tool and added capability for web and api.
- If the automation tool does not offer core functionality such as managing automation assets, reporting and tracking, and service automation – don’t waste your time. And don’t get into the “integration” trap. Core functionality should be native and easy to achieve in a simple manner. Breaking the core lifecycle of automation into multiple tools only brings inefficiencies.
- Codeless automation tools can be challenging. Moving all the test logic into some invisible executable is not really helping the purpose. Tools that drive test logic from excel spreadsheet suffer from obvious shortcomings.
- Stay away from the tools that claim do everything under the sky. For example, performance testing is better handled by a specialist tool.
Capability assessment framework
This framework will help you score on capabilities that are critical to making automation fast and sustainable. To start with just a simple Yes/No checklist could go long way to ensure a 360° evaluation. The capabilities are divided into technical-functional, accessibility, adoption and the ROI to do benefit analysis.
Technical & Functional
- Business process focused
- Automated test design
- Reusability / Modularity – Need of custom frameworks
- Robust element identification
- Seamless Data driven testing
- Test asset management & inbuilt traceability
- Change Impact management & Automated Change management
- Execution tracking and cycle management
- Service and API automation
Accessibility
- Ease of use for the broader test and project teams
- Centralized access to test automation assets
- Ability to execute tests and analyze results at a functional level
- Ability to set up governance
Adoption
- Easy to adopt
- Need of specific programming skills and resource dependency
- Vendor locking
- Hidden costs (like execution agents, add-on modules, protocols, accelerators costs, maintenance and support add-on costs..)
ROI
What are the key components to analyze ROI? There are many things here, but I am only listing the ones that are the usual suspects. They are very basic, but these are critical to automation that can give you returns on your investment.
- Need for framework development
- Element setup
- Data definitions
- Runtime & executions
- Change Management
Get the most from your trial and evaluation
It’s tough with our day jobs to give time and energy to evaluating a new tool. Here are some quick tips to help you thru the process:
- Don’t get caught up by exception use cases. Consider what serves 80% of or your Automation needs
- Carefully observe the learning curve
- Emulate situations for future change adaption and maintenance
- Consider how easy/difficult will it be to move away if needed in future
- Review pricing model for hidden add-on costs. You want to make sure ink does not cost more than printer!
Finally…
If I had to really pick the Top-5 things, here are critical ones that you shouldn’t compromise on:
- Ability to automate UI and API simply and in same flow.
- A tool that could potentially handle the complexities of today’s technology stack and robust element handling without any custom framework and programming overhead
- Something that can keep up with the continuous nature of changes and where maintenance doesn’t kill the ROI and outcome is more sustainable
- Don’t get locked into automation assets that would be proprietary to specific tool
- An automation tool that’s simple to adopt and has test management, reporting and CI integrations built in.
If you are thinking about going with or sticking to selenium, this is a comprehensive summary of how you can potentially leverage all the power of selenium without any of its framework and programming overheads
Do check out how accelQ scores and compares and sign up for trial to experience the difference.