When I look back at my career in test automation, three distinct periods, or “waves,” come to mind.
(**I also originally wrote this in 2017 and updated it in 2023 to keep up with all the innovation in this space since it was first posted.**)
Test Automation: First Wave
Most of the early automation tools were proprietary and vendor-specific.
That means they were developed, maintained, and sold by specific companies, sometimes making them expensive and restrictive.
The first wave is filled with good old-fashioned vendor tools like WinRunner, Silk Test, and QTP.
Some characteristics of this era are:
- Platform-specific: These tools were often tied to specific platforms or technologies, which could limit their versatility.
- Record and Playback: Many tools offer ‘record and playback' features. This allowed testers to record manual test steps, which the tool would convert into an automated script. While useful for beginners, this approach often produced brittle scripts that weren't maintainable or scalable.
- Scripting Languages: Some of these tools had scripting languages or required knowledge of specific programming languages. For example, Winrunner had its language called TSL (Test Script language)
The proprietary nature during this time frame often meant higher costs, lesser flexibility, and sometimes a steeper learning curve. If the vendor decided to stop supporting the tool or made significant changes, companies could find themselves in a tough spot. (Once again, my heart was broken when WinRunner was killed off in favor of QTP)
Also, as the software industry evolved, there was a growing need for more flexible, open-source, and versatile tools. The limitations of the first wave of tools became more apparent, especially as web applications became more popular and diverse.
Enter the second wave.
Test Automation: Second Wave
Selenium, which started as a project to help test web applications, represented a significant shift in the world of test automation.
Instead of being proprietary, it was open-source, allowing a community of developers and testers to contribute and evolve it.
This began the second wave of test automation, which focused more on developers and programming best practices when creating automated tests.
As web apps became the norm, the need for a browser-only automation tool like Selenium became even more critical.
Other open-source tools followed Selenium leads and led to an explosion in web-first automation tools like Cypress and Playwright.
This was a giant leap forward in the automation evolution.
However, it also left a gap for enterprises that still required more complex automation of applications with a tech stack beyond the browser.
In 2017 the buzz was around AI and Machine Learning.
Companies began rushing to create tools they could pitch as “AI-driven.”
In fact, at a Google conference around that time, CEO Sundar Pichai opened the event by stating, “We’re moving from a mobile-first to an AI-first world.”
This led to where we find ourselves now – the era of AI in Automation.
Test Automation: Third Wave Tools
Here are a few of the “third-wave” automation tools I’ve seen in the market.
One of the main features of this software is that many of them are leveraging machine learning and AI-assisted technology.
- Applitools
- Testim
- Sealights
- TestResults.io
- Mabl
- LoadMill
- ReportPortal
Applitools
Applitools was one of the first tools in the third wave that I got my hands on, and it made me start believing that a new way of testing was possible.
When I first heard about visual validation testing, which uses a sophisticated algorithm to out potential bugs in your application without you explicitly calling out all the elements, I thought it must be B.S.
However, after speaking with founder Adam Carmi and checking out Applitools, I became a believer.
I discovered no visual processing settings, percentages, or configurations need to be set up to create visual testing with Applitools.
The algorithm is entirely adaptive, and I can only imagine where they’ll take the technology as AI and machine learning advances even further.
I recently saw an Applitools demo, and on their roadmap were some cool features they plan on adding on top of their existing machine-learning technology.
Possible AI Type Features:
- Leveraging ML/AI-based for automated maintenance (being able to group together similar groups of changes from different pages/browsers/devices)
- Modifying their comparison algorithms to be able to discern what changes are meaningful/noticeable
- Being able to automatically understand which changes are more likely to be bugs vs. desired changes and prioritize diffs
TestResults.io
In the recent podcast with Tobias Müller on “Next Generation Functional Visual Test Automation,” I had the opportunity to delve into the workings of TestResults.io.
This platform, as discussed, pioneers the application of AI in the realm of autonomous software testing, presenting a viable solution for some of the most common challenges faced in test automation. TestResults.io potentially speeds up your testing process threefold. Its primary allure comes from its reliability, emphasizing eliminating the unpredictable elements – or “flakiness” – often associated with automation testing.
Plus, it significantly lessens the maintenance effort, which could alleviate the stress involved in ensuring the continuous operation of your testing.
One significant aspect differentiating TestResults.io from its counterparts is removing the need for XPath, IDs, and other selectors. This implies reducing the technical complexities typically associated with the automation process, making it more approachable and efficient.
The platform is designed with a user-centric philosophy, enabling the automation of entire user journeys. This means you can perform tests across applications, browsers, devices, and platforms, irrespective of their specific technologies. If a user can interact with a system, TestResults.io can automate testing.
TestResults.io includes a low-code and no-code designer in its platform, alongside a management cockpit providing visual and detailed reporting. The aim is to offer a comprehensive test lab that is accessible and informative for users of varying expertise levels.
Finally, one of the critical features of TestResults.io is its ability to integrate with your CI/CD pipeline smoothly while offering a team workflow. This seamless integration could be beneficial in achieving efficient, continuous testing and delivery in software development.
Checkout TestResults for Yourself Now
Testim
Testim tries to leverage machine learning to speed up the authoring, execution, and, most importantly, the maintenance of automated tests. Their goal is to help you to start trusting your tests.
Testim focuses on reducing flaky tests and test maintenance, which they see as one of the most significant challenges for most organizations.
Oren Rubin, the co-founder of Testim, mentioned in a recent TestTalks interview that the firm’s main goal is to help liberate test automation from the exclusive realm of developers and make it simple enough for anyone on the team to create. After speaking with Oren, I sensed that Testim is well on its way to achieving that goal.
Sealights
Sealights is a Cloud-based platform. We all know that developers and QA–both managers and engineers–are super busy using CI and CD practices these days, where they have frequent releases and not enough time to test the entire application multiple times.
That’s one of the main reasons Sealights was created.
Their machine learning-like technology that analyzes both your code and the tests that run against it lets you know exactly what your tests are covering and what they're not. But when Sealights says “tests,” they don't only mean unit tests; they mean any kind of test, from functional, manual, performance, you name it.
“Quality Risks” is an even more exciting insight they provide, as it focuses the user's efforts on the things that matter, by letting him or her know exactly which files/methods/lines have changed in the last build that wasn't tested by a specific test type (or any test type). Once you know that, you can easily ensure that untested code will not reach production before undergoing, at the very least, minimal validation.
As we move toward CI/CD, dashboarding becomes critical.
If you're like most companies, everything today is within your CI/CD, but often this data is not visible or accessible for consumption by your teams.
Sealights makes it easy to create a quality dashboard that everyone will see. So for every build, you’ll be able to understand what was tested, what the status and coverage were, and whether it’s improving, decreasing, or has quality holes.
Mabl
Mabl is similar to Test. AI. Mabl, started by many ex-Google employees, runs functional tests against your apps or website. In Mabl terminology, you “train” your tests to interact with your applications. When you’re done recording, your trained tests will run at a predetermined time and alert you.
Their website makes three main promises:
- Eliminates flaky tests–like the other AI-based test automation tools, Mabl can automatically detect whether elements of your application have changed and dynamically updates the tests to compensate for those changes.
- Mabl can continuously compare test results to test history to quickly detect changes and regressions, resulting in more stable releases.
- Mabl helps identify and surface problems quickly, alerting you to possible impacts before they impact your customers.
ReportPortal.io
Looking for an easy-to-install-and-use dashboard? Need to triage your automation test results as well as create awesome graphs?
Even better, what if it were free?
Well, I’ve got something you should check out if you’ve been looking for an automation test results dashboard solution: ReportPortal.io.
ReportPortal just came our with a machine-learning algorithm to help you to analyze your results automatically.
The machine learning algorithms use your project's historical data already in the dashboard database. That means it can analyze your latest execution, and you can be confident about the status of your test cases.
I think the ability to analyze large amounts of data is the perfect use of machine learning and I see this type of approach really growing in the next few years.
To learn more, check out my full post on getting started with ReportPortal.io
So is AI/Machine Learning Just Hype?
AI/Machine Learning is the latest buzzword currently being used in the testing industry. But is it real, or just hype?
Only time will tell if the third wave will finally fulfill the promise of reliable, easy-to-maintain test automation for all.
Let me know your experience with these or any other tools you consider part of the third wave of AI test automation.
Also, check out the Automation Guild Online conference for some awesome sessions we will have on AI test automation. You'll also get a chance to ask question to many of the vendors mentioned in this article
What About Non-AI Automation Testing Tools?
For other non-functional AI automation testing tools check out:
- 15 Open Source API Testing Tools for REST & SOAP Services
- Top 11 Open Source Performance Testing Tools for Load & Stress Testing
- The Top 21 FREE Visual Validation Tools for Testers
- Top Accessibility Testing Tools for Automation
- 43 Function Automation Testing Tools
- 7 iOS Testing Tools You Need to Know
How to Stay Up To Speed with the Latest in Automation Testing?
If you feel overwhelmed with staying up to date with all the latest automation testing tools and best practices, check out my annual online conference dedicated 100% to just automation testing — Automation Guild.
