What is Automation Testing (The Ultimate Guide 2024)

After blogging for over fifteen years, I realized that I had never defined what is automation testing.

If you're in a rush, here is the quick answer: Automation Testing is a strategic approach to enhance software testing efficiency. It involves using tools to automate repetitive manual processes, ensuring faster and more consistent test execution. Ideal for Agile and DevOps, automation testing complements manual testing, offering extensive coverage without replacing human testers. Its success involves careful test case selection, a robust framework, and ongoing maintenance. Essential in modern software development, it provides rapid feedback, saves time, and increases test accuracy.

But for a deep dive – read on.

So, let’s go over the basics to set you on your way to creating what I like to call automation awesomeness.

INDEX
Definition for What is Automation Testing
Pitfalls of Manual Testing
Scope of Automation: Does Automation Testing Replace Testers?
Benefits of Automation Testing
Disadvantages of Test Automation
Automation Testing is a Development Activity
Other Automated Testing Problem Areas
What Test Cases Should be Automated?
Types of Automation Tests
Automation Testing ROI (What is the Cost of Test Automation)
What Not to Automate
What is an Automation Framework?
Successful Test Automation Process
Automation Testing Process
With Automation Testing – Test Size Matters
Automation Test Engineer Salary
Test Tool Selection – How To Pick
Popular Automation Testing Frameworks & Tools List
Popular Vendor-based Automation Test tools
API Automation Test Tools Open source API tools
Test Automation Conferences

Definition for What is Automation Testing

Automation testing refers to taking a repeatable manual process performed by a developer or tester and leveraging a test automation tool to automate the process.

Automation helps you to accelerate running through numerous test scenarios to check that the results produced by specific actions or lines of code match the expected results. If not, it raises an exception to inform you that something went wrong.

This approach is critical because it speeds up the time to test since you are using a scripted process that evaluates if the software developed actually meets your requirements.

We call this automation testing because you can easily use your computer to rapidly run through a ton of test cases/test scenarios in minutes.

Automation can save you a lot of time since you don’t have to perform these repetitive tasks repeatedly after changing an application under test.

In a nutshell, automation testing is a technique used to improve the execution speed of verification/checks or any other repeatable tasks in the software development lifecycle.

The term “‘automation testing” can also be controversial, with many folks preferring to use the term automated checking or automation in testing.

In any case, before we take a look at automated testing, let's touch on some issues with manual testing.

Pitfalls of Manual Testing

There are a few reasons why manual testing may be problematic:

• It uses a lot of resources
• It’s time-consuming
• It sometimes lacks proper coverage
• Due to their repetitiveness, testers may get bored and miss steps when executing manually, leading to possible inconsistencies.

Like automated testing, “manual testing” can also be controversial. In fact, Michael Bolton has gone as far as to say that manual testing does not exist.

Automation testing should be used to help with your testing efforts.

But does automation replace all your tests?

Here's the deal:

Scope of Automation: Does Automation Testing Replace Testers?

Some people assume automation means replacing human testers. In actual fact, however, it’s the opposite. Automated scripts are great for running tests precisely and quickly, but they never replace human testers.

Automation is also great for running the same steps repeatedly, but they don’t actually think.

Although we agree that automation testing does not replace other application testing activities, it is critical with today’s software development environment and continuous integration practices. With the increased speed at which we develop software, we need automation testing.

Another reason is that Agile and DevOps practices demand more automation. Practices like continuous integration and delivery require automated scenarios that run quickly and reliably. I’d actually go so far as to say that in today’s modern development environment, we can’t succeed without automation.

So, what are some reasons for using automated tests in quality assurance?

For more info on the difference between an automation testing engineer and a tester, along with typical automation testing salary info, check out SDET vs. Tester: What's the Difference?

Benefits of Automation Testing

Automation has many benefits for both testers and developers.

Developers

For developers, when we talk about automated tests, this would include unit testing, component testing, and integration testing.

Although most developers who work on modern software development have embraced writing unit tests-driven development, this is not always the case on legacy systems.

Some programmers are afraid to modify an existing code due to the unattended consequences of breaking something with their code change.

But given a choice, it is always safer to have tests around any changes that you make. When you change code, you can introduce errors, But when you cover your code with tests before you change it, you're more likely to catch any mistakes that you may have submitted with your change.

So, one of the main benefits of any automated test is that it acts as a safety net. When you run tests for every new build of your application before checking in your code, you can have some level of confidence in knowing that you have not broken anything.

Testers

During regression testing or smoke testing, a manual tester will take an existing test case procedure and execute it step by step against a web app or API. This can be time-consuming since it is a manual process done by hand.

Thus, many companies try to take their manual regression tests and convert them into automated test cases to save time. An automated test tool then executes the test steps automatically without human intervention.

This highlights two other benefits of test automation: 1.) It frees up the tester's time to focus their energy on high-value testing activities that can't be automated, like exploratory testing. 2.) It avoids missed steps that can occur when performing a test manually.

With the increased speed at which we develop software, we need automation testing. As more companies move toward Agile and DevOps, automation is more important than ever before.

Practices like continuous integration and delivery require tests that run quickly and reliably. Lots of manual verifications will stop your ability to achieve velocity with your software development.

I’d go so far as to say that in today’s modern development environment, we couldn’t succeed without automation.

Although teams try to create automation to save the company both time and money, it’s also important to give developers quick feedback so that when they check in code, they are alerted as soon as possible that the change they checked in broke something.

The theory is that an automation script will save the company both time and money. But it seems that many folks fail to factor in the amount of time and money it takes to maintain automated test suites.

Some other benefits of automated testing are:

• Verify newer versions of the software
• Free testers up to focus on more exploratory-type testing
• Automated scripts are more repeatable
• Data population
• Accurate benchmarking
• Less false failure due to human error
• Greater test coverage
• Reusability
• Quicker release of the software
• Get fast feedback to your developer on failing checked in software
• Saves time
• Ability to leverage programming capabilities
• Run for every new build of your application to act as a safety net

So are there any downsides to creating automation?

What's the real story?

Disadvantages of Test Automation

Since automated testing relies on programming languages for their creation, automation becomes a full-blown development effort.

As a matter of fact, you are developing a piece of software to test another piece of software.  Treat your automated code just like your development code. Follow the same processes and best practices you would use for any other software development project.

Automation testing is difficult and complicated, just like most other development software projects. It also presents many of the same issues other software programs do. So the same best practices for developing software also apply to automation as well.

Automation Testing is a Development Activity

Since automated suites usually rely on programming languages for their creation, automation becomes a full-blown development effort.

You are developing a piece of software to test another piece of software.

Automation testing is difficult and complicated, just like most other development software projects. It also presents many of the same issues other software programs do. Treating your automated code just like your development code is essential.

You must follow the same processes and best practices you would use for any other software development project if you are going to succeed.

If you ignore this advice, your automation project will fail.

Other Automated Testing Problem Areas

Some other automation testing pitfalls are:

• Setting unrealistic goals
• Believing that automation will find more new defects
• A false sense of security
• Underestimating the amount of time it takes to maintain your web app automation
• The creation of large, end-to-end tests should be avoided. Scripts should be atomic so that when they fail, you know why
• Focusing on UI automation types of testing only
• Not having a controlled test environment
• Ignoring failing tests
• Not having a test data strategy in place
• Not reusing automation code
• Developers not making their code automatable
• Not using proper synchronization in your tests
• Not making your automated test readable
• Creating automated tests that add no value
• Hard coding test data
• Not using a whole team’s collaborative automation efforts
• Expecting automation testing to replace manual testers
• Hard to discover all the technology needed to create an end-to-end automated solution

Because of the above reasons, teams often claim that automation testing “doesn’t work.”

Another common question I’m frequently asked is, “Which tests should be automated?”

What Test Cases Should Be Automated?

You shouldn't try to automate everything. In fact, not everything is automatable. When planning what test cases to automate, here are some things to look for when creating a testing strategy:

• Deterministic tests.
• Steps that don't need human interaction
• Tests that need to run more than once
• Any manual process that will save engineers time (not necessarily an official “testing” process)
• Test that focuses on the money areas of your application
• Test that focuses on the risk areas of your application
• Unit tests
• Tests that need to run against different data sets
• Tests that are hard to test manually.
• Focus on the critical paths of your application
• Tests that need to run against multiple builds and browsers
• Tests used for load/stress testing

The more repetitive the execution is, the better candidate a test is for automation testing. However, every situation is different.

Ultimately, you should consider using automation for any activity that saves your team time. It doesn't have to be a pure testing activity; you can leverage automation to help reduce the length of time-consuming activity throughout the entire development life cycle.

Test Types For Automation Tests

Remember, don't just focus on functional tests for your automation strategy.

Here are some other areas that could benefit from automation:

• User Experience
• Performance (Load Testing)
• Security
• Component testing
• Accessibility testing
• Exploratory testing using smart AI crawlers
• Unit test
• Integration test
• Environment Setups
• Data-driven testing

At this point, some of you may be asking, “What is the ROI of test automation?”

Automation Testing ROI (What is the Cost of Test Automation)

Determining the ROI of your automation testing efforts can be tricky. Here is a common calculation some folks use to get a rough estimate of their test automation costs. This can also help you decide whether a test case is even worth automating as opposed to running testing it manually.

Automation Cost = how much the tools cost + how much the labor cost to create an automated test + how much it costs to maintain the automated tests.

Consequently, if your automation cost calculation is lower than the manual execution cost of the test, it’s an indicator that automation is a good choice.

Moreover, ROI quickly adds up with each re-run of your automated test suite.

Because it’s critical that you get a good return on your test automation investment, there are some things you shouldn’t automate.

Why does this matter?

What Not to Automate

There are exceptions to everything, of course, but in general, you may not want to automate the following test case scenarios:

• One time test
• Ad hoc based testing
• Test that doesn't have predictable results
• Usability testing
• Application not developed to be testable

In addition to what to automate, another element of a successful automation project is having a test automation framework.

What is a Test Automation Framework?

An automation framework is a common set of tools, guidelines, and principles for your tests. An automation test tool framework helps minimize test script maintenance and helps scale your test, enabling parallel testing. In my opinion, common pieces and best practices of a framework are:

• Set your manager's and team's expectations
• Break your automation framework into abstraction layers
• Use proper synchronization methods
• Determine a strategy for training and retraining your framework users
• Use version control
• Look for existing libraries/tools before inventing your own
• Do code reviews on all automated tests
• Include reporting and logging that make debugging easy
• Follow a naming convention
• All elements should have unique IDs
• Avoid coordinate-based automation
• Create reusable methods
• Create reusable utilities
• Refactor code often
• Use Page Objects
• Make tests readable. (Your tests should read like English)
• Avoid code duplication
• Have a test data management strategy
• Handle running tests in parallel
• Support mocking and stubbing
• Include automation on your sprint teams definition of done (DOD)
• Remember that automation is a collaborative, “whole team” effort
• Separate your tests from your framework

That's not all…

Successful Test Automation Process

It’s a given that your applications are going to change over time. And since you know change is going to happen, you should start off right from the beginning using best practices or design patterns, like using page objects; doing so will make your automation more repeatable and maintainable.

Automation Testing Process:

Here are six steps to follow in an Automation Process to enable better automated software testing.

1. Prepare

First, prepare – and understand the functional testing objectives. Understand what test data is needed. Know what needs to be verified. Make sure you get your whole test team involved.

2. Write

Turn the requirements into an automated solution. Know what the start and end conditions are for each test. Tests should be completely independent of other tests. Add proper assertion checks to ensure your application behaves according to your specifications. Each comprehensive test should have a particular purpose.

3. Execute

Executing your tests should be reliable. Run each test at least three times in a row before checking in the code.

4. Evaluate

Verify that the automated test is doing what you expect it to do. Have manual tests to verify that it is working as expected. Remember — if it’s not asserted, it’s not checked.

5. Communicate

Be sure that everyone on your team is aware of the results. Flaky tests should be fixed ASAP, or you'll risk your team ignoring your test results.

6. Repeat/Refactor

if you notice a flaky test, refactor it to make it more reliable. Most importantly, delete any tests that are not reliable and haven’t been fixed within a given time frame.
<h2″>

With Automation Testing – Test Size Matters

Because tests need to run quickly, test size matters. At this point, many people visualize a traditional test pyramid, which has unit tests as its base. Integration tests are in the middle, and GUI tests are at the top.

But I think more in terms of test size.  By test size, I'm referring to tests that are faster than others. While I understand the need to run UI tests, if you have to create one, make it as fast as possible.

• Don't do all UI end-to-end journeys.

In addition, when an automated test fails, you need to know why. Furthermore, you should endeavor to get feedback to your developers as quickly as possible, and the best way to do that is with a fast, well-named test.

• Unit test, integration test tend to be faster.

If you're into Star Wars, you notice that Yoda in the image above is the smallest test size, and they tend to be more powerful too. As you move up the pyramid, UI tests can be evil and can drive you crazy, so although you need them, just be careful.

Automation Test Engineer Salary

You might have seen recent job postings in areas like Silicon Valley looking with the title of  SDET. Most of these positions require someone that knows how to create test automation in a programmatic way. The good news is that many of these jobs offer higher salaries than for a standard tester.

How much more? On average, I've seen the difference to be over $32,000! Check out my post on SDET vs. Tester: What’s the Difference? (The answer may surprise you!) for more detailed info.

Test Tool Selection – How To Pick

There is no “correct” test tool for automation testing. Ultimately, it all depends on your team’s unique needs and skill set. Also, you sometimes need a combination of different tools to get the coverage your application requires.

Due to this, there are a few things to look for when trying to select a test tool:

• Look at the product roadmap and make sure the tools you select will handle future features and technologies.
• Evaluate the cost, including maintenance.
• Use a tool that leverages the same tools and languages your developers use.
• Don't just assume a tool will work for you. Create a small POC for each tool and get team feedback before committing to anything.
• Is the tool extensible?
• How easy is it to use and get started?
• Does it provide reporting and debugging capabilities?
• Does it recognize all the objects in your application?
• Can it integrate with other tools like version control, test management tools, and continuous integration tools?
• Find out if the tool has an active user base.
• Select tools that other companies are using.
• How much training will it take to get your teams up to speed with the tool?
• Finally, determine how easy it is to hire folks that have the skills needed to create your automated tests.

Popular Automation Testing Frameworks & Tools List

Selenium

Selenium – has arguably become the de facto test tool standard for browser-based testing for automation engineers. Please remember: you cannot use selenium for non-browser applications; not everything can be an automation testing Selenium script. 

The first thing folks usually think of when they hear automation testing is automation testing selenium. Remember, Selenium is just for browser automation. Also, it's not a tool; it's a library. Check out some of the other automation tools not named Selenium.

Appium

Automation for apps. Appium seems to be the winner in the mobile testing space so far.

Playwright

Microsoft Playwright is a newer, open-source, JavaScript-based, cross-browser automation library for end-to-end testing.

The goal of Playwright Node.js is to provide a single API to developers and testers to automate their web applications across today’s three major browser engines.

For more info, check out my full tutorial on Playwright.

AutoIt

AutoIt v3 is a freeware BASIC-like scripting language designed for automating the Windows GUI and general scripting. Many teams integrate AutoIT with Selenium to work around non-browser windows that appear in an automated test workflow.

Serenity

Serenity – One of my favorite automation frameworks around.  Serenity is a great open-source tool because it acts as a wrapper over Selenium and Behavior Driven Development (BDD) tools like jBehave and Cucumber JVM. That means there's a lot of built-in functionality available to you in Serenity that takes care of many things you would normally have to code from scratch if you had to create your own BDD framework. What Serenity is really awesome at is creating unbelievable reports. Out-of-the-box Serenity creates living documentation that can be used not only to view your Selenium BDD test results but also as documentation for your application.

Gauge

Gauge – Gauge is a test automation solution that's being built by ThoughtWorks; it's cross platform, cross browser testing compatible, and open-source. It supports multiple languages, including Ruby, Java, C#, Python, and Javascript, and it has upcoming support for other languages — like Golang — as well.

Robot Framework

Robot Framework – If you want to use Python for your test automation efforts, you can’t go wrong using the Robot Framework. The Robot Framework is a mature solution that was created for testers and uses a keyword-driven approach to make tests readable and easy to create. It also has many test libraries and other tools you can use.

More Automation Tool Resources

For a full list of other automation testing options, check out:

Javascript Test Automation – Top 11 Essential Javascript Automation Frameworks

Visual Validation Tools – The Top 21 FREE Visual Validation Tools for Testers

Popular Vendor-based Automation Test tools

Applitools

Applitools – Actually, Applitools integrates with both Vendor and Open-source solutions. If anyone has tried doing any sort of visual testing using tools like Eggplant and UFT Insight, you know how hard it is to make these types of tests reliable. Sometimes the tests are so fragile you can only run them on the same machine they were developed on to avoid flaky tests. There are many reasons for this, but it's mostly due to pixels being slightly off from one browser or OS. Applitools is different in that it was developed from the ground up for visual validation, and its sophisticated algorithm was designed to handle many pixel issues that most other image-based testing tools have a hard time handling. Applitools allows you to find and automatically detect all the visual bugs to validate the visual correctness of your application.

UFT One

UFT One essentially combines the best of both the vendor-based and open-source worlds by morphing Selenium with some key functionality currently found in UFT (QTP).

TestComplete

SmartBear Testcomplete – Allows you to automate web, desktop, and mobile applications. Best of all, you can choose from script-free, drag & drop functionality or JavaScript, Python, VBScript, JScript, DelphiScript, C++Script, or C#Script as a scripting language.

IBM Rational Functional Tester

Like most companies, IBM’s test portfolio has grown with the acquisition of tools like Rational and Green Hat. It appears that much of the strength of rational functional tester comes from its support of numerous technologies, including Windows, Mac, and mobile platforms.

Tricentis

Tricentis – Self-billed as “the continuous testing company,” which is in line with Gartner’s finding that one of its strengths is its extensive efforts to support Agile testing and continuous improvement processes.

Worksoft

Worksoft is well known for its ERP business end-to-end solutions.

Ranorex

Ranorex supports a ton of technologies across all kinds of platforms — all from one tool. Noteworthy, however, is that it lacks a full, end-to-end solution and focuses mainly on functional test automation.

Progress

For those of you who may not be familiar with this company, Progress recently acquired Telerik, which is the home of the popular free debugging tool Fiddler. Also, I know a few test engineers who actually use Progress’s Test Studio as a front end for their Selenium test automation efforts. Strengths of Progress are its integration with Visual Studio and its supported languages.

Automation Anywhere

Does not support testing for packaged applications like SAP. It also lacks support for native mobile apps testing.

Katalon Studio

Katalon Studio is another popular test tool that has functionality like Record and Playback, reporting, Object Spy, Object Map, and more. You can use it to create automated tests for Web, API, and mobile apps. It even runs on both Windows and Mac.

API Automation Test Tools Open source API tools

Rest-Assured – Rest-Assured is an open-source Java Domain-specific language (DSL) that makes testing REST services simple. It simplifies things by eliminating the need to use boiler-plate code to test and validate complex responses. It also supports XML and JSON Requests/Responses.

RestSharp – Simple REST and HTTP API Client for .NET

Postman – Postman is a rest client that started off as a Chrome browser plugin but recently came out with native versions for both Mac and Windows.

SoapUI – is the world-leading Open Source Functional Testing tool for API Testing. It supports multiple protocols such as SOAP, REST, HTTP, JMS, and AMF.

Fiddler –  Fiddler is a tool that allows you to monitor, manipulate and reuse HTTP requests. Fiddler does many things that allow you to debug website issues, and with one of its many extensions you can accomplish even more. Check out my article on how to get started with Fiddler.

Karate – Since Karate is built on top of Cucumber-JVM, you can run tests and generate reports like any standard Java project. But instead of Java – you write tests in a language designed to make dealing with HTTP, JSON, or XML – simple.

Vendor API Tools

SoapUI Pro – Since the free version is open-source, you can actually gain access to the full source code and modify it as needed. The pro version is more user-friendly and has additional functionality, including a form editor, an assertion wizard for XPath, and SQL query builder.

Test Execution Report Tools

Allure Report

Allure Report is an open-source framework designed to create test execution reports clear to everyone in the team.

ReportPortal

Looking for an easy-to-install-and-use dashboard? Need to triage your automation test results as well as create awesome graphs?

Even better, what if it were free? Well, I’ve got something you should check out if you’ve been looking for an automation test results dashboard solution.

Check out my full article on ReportPortal, along with my get started video.

Zebrunner

Zebrunner tool is aimed to increase the visibility of test automation. The tool is built for QA engineers, developers, and QA Managers. It has both a free version and a paid version.

Run Your Automated Test in the Cloud or On Mobile Devices

Here are some vendors that allow you to save a ton of time by running your test in the cloud and on multiple OS, devices, and configurations. Get rid of the headache of having to maintain your own in-house lab/grid.

Sauce Labs

Perfecto

Browserstack

Automation Test Management Tools

Zephyr – Manage all aspects of software quality; integrate with JIRA and various test tools, foster collaboration, and gain real-time visibility.

Tricentis qTest is a platform for Software testing and QA tools built for Agile.

Allure TestOps is a DevOps-ready testing platform built to run and manage all the testing for the team. Dozens of native integrations with testing frameworks, CI systems, and bug trackers allow creating a unified QA environment for your team. The system features special mechanics for automated seeking for and managing bugs and errors.

Test Automation Conferences

AutomationGuild – Automation Guild is the first-ever event of its kind, a 100% online conference that is dedicated to helping YOU perfect the craft of creating automation awesomeness and accelerate your automation career. If you missed the LIVE Automation Guild event? No worries! Due to demand, I decided to keep registration open. So you can still get all pre-recorded sessions and recorded Q&A now!

PerfGuild – the first-ever online conference dedicated 100% to performance testing.

SeleniumConf – SeConf brings together Selenium developers & enthusiasts from around the world to share ideas, socialize, and work together on advancing the present and future success of the project.

STPCON – The Software Test Professionals Conference is the leading event where test leadership, management, and strategy converge