To Automate or Not To Automate: Calculating ROI and setting the priorities for Test Automation
”The best time to automate your tests is when you have a regression kit, and there are repetitive steps in your test routine”. I’ve heard this statement many times from test automation (TA) developers. But, is it true for every case? How would you justify the benefits of TA for your delivery? How would you set the priorities?
In this article, I want to demonstrate my approach to measure Test Automation Return on Investment (TA ROI) and how I sort the best candidates. Often, TA ROI calculation denies the initial assumptions showing that “obvious to automate first” scenarios must have lower priority or shouldn’t be automated at all.
Let’s start with some examples. Given: we discovered 3 possible test suites to automate during the test analyzing phase, and we need to understand if it’s worth investing in the TA development and if so which suite should be automated first.
Our first step is to calculate time: Manual Execution vs TA Development.
Suite A.
The functionality for this test suite won’t change in the next 52 weeks. Hence, it’s safe to say the suite will be valid for the whole year. Our test professional is executing these test cases manually, and it takes 2 hours per week.
Based on our previous experience, we’re assuming that the scripting (if there are no “underwater rocks”) will take 20 business days or 160 hours. Also, it will require about half an hour per week to maintain the automation (check the results, report, fix bugs in the scripts, etc.). Besides, the analysis showed that about 30% of our test steps/functions should be developed from scratch (f.e., there are new components on the pages you’ve never automated before). So, we add 30% (or 0.3 out of 1) of the total time as a risk factor.
Suite B.
The test suite should be valid for 26 weeks. It takes 8 hours per week for a test professional to execute these test cases. We’re expecting to spend 15 business day for developing the testing scripts, and our maintenance estimation is 1 hour per week. Also, this suite got a bigger risk factor as almost half of the functionality is brand new, so we set 0.4 as the risk factor.
Suite C.
This one will be valid 52 weeks, our tester executes this suite 8 hours per week, and we assume to spend 20 business days on the scripting. The maintenance will also take 0.5 hours weekly, and the risk factor is 20%.
With the given information, we can draw the following table and calculate total Manual & Automation time.
Suite A: Manual execution 104 hours vs TA development 241.8 hours
Suite B: Manual execution 208 hours vs TA development 204.4 hours
Suite C: Manual execution 416 hours vs TA development 223.2 hours
Suite C is the winner, should we start automating it? Hold on, let’s move to the next step.
The second step is to calculate the costs, Manual execution vs TA development.
Expenses may vary for every project, but as an example, let’s consider only 4 items. These are: hours we spend, any licenses or tools if we need to execute tests or develop the automation, if our testers need any upskill courses and if we need to obtain any specific hardware.
Starting with Suite A, we pay $50 per hour to our tester, and our test automation developer charges $70p/h. Special software required to execute our suite manually, which is other $200. And we will spend $600 on TA development kit, as well as we will purchase an online course to make our developer more confident in some particular technology (we’re utilizing on the project).
Similar to other suites.
Suite A: $5400 (Manual) vs $17676 (Automated)
Suite B: $10600 (Manual) vs $15058 (Automated)
Suite C: $21000 (Manual) vs $16374 (Automated)
Again, Suite C is the winner. Is it the time to start automating this suite? Not really, we should calculate Return on Investment (ROI) before we make the final decision.
The third and final step is to calculate ROI
If you haven’t heard about ROI or haven’t calculated it before, I recommend reading “Measuring ROI” articles by Jon Allen.
I remember when I was doing my Masters in IT, we got 3 units of Project Management and every unit we had ROI related assessments. The three most common types of measuring ROI were Payback, Net present value (NPV) and Internal rate of return (IRR). In my opinion, NPV and IRR is a bit overkill for our purpose, and most suitable for bigger projects with greater cashflow. Therefore, to calculate TA ROI, we will use a variation of Payback approach.
The calculation is based on 3 items, namely: Benefit, Cost and Implicit Benefit.
- Benefit — all the costs for manual execution;
- Cost — all the costs to develop TA.
- Implicit Benefit is a new item was introduced to adjust the approach to TA ROI calculation and covers such hidden benefits as the time developers would save, reusable code to automate other suits, etc. To simplify things, we will use only developers time and reusable code for this item.
How should TA benefit developers? Once the test is automated, it may be included in Continuous Integration and run anytime. Also, it’s usually much faster than manual execution. Hence, the developers don’t need to adjust their daily graphic/idle waiting for the test results. Moreover, the tests are running more often so the issues can be found on early stages which is less expensive to fix.
Getting back to TA ROI calculation, we’ve performed further analysis and found that by covering Suite A we will save 1 hour of our developer time weekly (which is $70 per week); also we will reuse 20% of the code for Suite B and 5% for Suite C. Suite B automation will save 2 hours for our developer weekly and it will share the code with Suite A (20%) and with Suite C (30%). Suite C frees 1 hour per week ($70 weekly) for the next 52 weeks and shares 10% of the code only with Suite A. Below is ROI measuring for all three suites.
ROI is negative for Suite A, the payback is not happening during the 12 months, and we are not considering any benefits/costs after the first year, because there’s a risk the requirements will be changed and the test will have to be redeveloped.
ROI is healthy for Suite B, and the payback happens in time. Hence, Suite B is a good candidate. Let’s measure ROI for our favourite Suite C.
ROI is positive, and Payback happens between 6 and 12 months. So, both suites (Suite B and Suite C) are good candidates to automate.
My bet is on Suite B, and this is not only because it has greater ROI but also:
- The automation time is 15 days vs 20 days for Suite C
- It covers a significant part of Suite C, so I would reuse the code to speed up the scripting.
Once I finish Suite B (first priority) and then Suite C (second priority), I would go through these 3 steps again to re-measure ROI for Suite A. With all the reusable modules, tools, licenses, etc., it will most probably have positive ROI and Payback.
To conclude
Don’t rely on your assumption, do the maths. And remember, although tangible values from test automation are critical, reducing manual and repetitive steps may also improve your product quality by motivating your team and illuminating human errors.
Good luck with your projects!
