Before we get started looking at the top open source automation testing frameworks, it might be a good idea to talk about what an automation testing framework actually is. I think of a framework as a set of best practices, assumptions, common tools and libraries that can be used across different teams. (I also used this content in my article for TechBeacon)
What is an Automation Testing Framework
The reason you need a framework is to help make your test automation code:
- Reusable
- Maintainable
- Stable
However, I’ve seen some teams take it to the extreme by building their own, elaborate automation test framework from scratch. This drives me crazy because they could have easily made use of existing open-source tools and libraries that would have met their needs without writing any code — in most cases with better results. Many of these tools also have a proven built-in automation testing framework architecture.
Before you fall into the “build your own framework” trap, be sure to check out these ten open-source automation solutions.
1) Serenity (Formally Thucydides)
If you need an open source automation testing framework selenium based solution that integrates with behavior-driven development tools like Cucumber and JBehave, and want to keep your test scenarios at a high level with lower-level implementation details in your reports, Serenity might be the tool for you. It comes in two flavors Java and JavaScript(SerenityJS).
Serenity (formerly Thucydides) is an open-source tool designed to make writing automated acceptance and regression tests easier.
A key benefit of using Serenity is that it acts as a wrapper on top of Selenium WebDriver, Appium and BDD tools. This makes writing BDD and Selenium tests easier, because it abstracts away a lot of boilerplate code you sometimes need to write to get started.
Serenity proves a lot of built-in functionality like WebDriver management, managing state between steps, screenshots, running tests in parallel, Jira integration and more — all without having to write a single line of code.
What Serenity is really awesome at is creating unbelievable reports. Out-of-the-box, Serenity creates living documentation that can be used not only to view your Selenium BDD test results but also as documentation for your application.
It also has an extra annotation called @Step. The @Step method gives you an extra level of abstraction that, when used correctly, can make your tests more reusable and maintainable.
Serenity takes care of many things you would normally have to code from scratch if you were creating your own BDD framework. I think it’s one of the better open source test automation frameworks available, but I might be a bit biased because it’s the tool I use for my day-to-day test automation efforts.
2. Robot Framework
If you want to use a python test automation framework for your test automation efforts, you can’t go wrong using the Robot Framework.
The Robot Framework is a mature solution that was created for testers and uses a keyword-driven approach to make tests readable and easy to create. It also has many test libraries and other tools you can use.
The Selenium WebDriver library is probably the most used external test library, but Robot Framework can also test things other than just websites, like FTP, MongoDB, Android, Appium and more. In addition to all this open-source awesomeness, it has a lot of API’s to help make it as extensible as possible.
The Robot Framework is Python-based, but you can also use Jython(Java) or IronPython(.NET).
The keyword approach used by Robot Framework is great for testers who are already familiar with other vendor-based, keyword-driven test tools, making the transition to open source much easier for them.
If your team is made up of mostly testers, Robot Framework would be a great option for your open source automation testing framework python based choice.
3. Sahi
Sahi has both an open-source and a pro version and was developed for testing web applications. Sahi works as a proxy server that allows you to use it within a browser. From the Sahi dashboard, you can launch the browser you want to test.
The Sahi controller guides you through the recording and capturing of the interaction you want to perform against your application. When you hover over any element in the Sahi browser, for instance, it will show you all the actions you can perform against that particular element.
You can also use the controller to play your script back.
Sahi’s record and playback functionality makes it easy to get started with test automation of simple HTML applications, but beware of relying on record and playback; this approach has been proven to be a bad practice for creating maintainable, robust automation tests. It’s okay for helping create an initial test, but you’ll need to customize it to make it reliable and maintainable.
I’ve heard quite a bit about Sahi, but my first impression of it is that it’s not as powerful as some of the other options. However, it also comes in a pro version that lots of folks rave about.
4. Galen Framework
If your automation efforts are focused on User Experience Design (UX)/Layout testing, Galen automation testing framework might be a perfect fit for your needs.
Galen was created specifically for UX testing and has a special syntax and rules you can use to describe and verify your application’s layout. It also allows you to specify your browser size, then run tests against your application to verify your layout specifications. Galen tests also generate detailed HTML reports with screenshots and a visual image comparison with a cool heat map feature.
Tests can be written in a simple text file using the Galen syntax, JavaScript or Java.
Many testers confuse the purpose of a test automation framework. Some are designed for different purposes and excel in others. For example, a framework designed for UX design testing like Galen is much different than one that uses visual validation tools like Applitools. Check out my article on Applitools Vs Galen Framework What’s the Difference for more info.
5. Gauge
Gauge is one of the newest open source test automation frameworks for Mac, Windows, and Linux, and was developed by the folks at ThoughtWorks — the same company that created Selenium.
Practitioners of TDD and BDD will appreciate Gauge’s focus on creating living/executable documentation.
Gauge automation tests called specs are written using a simple markdown language using Java, Ruby, and C# within your existing developer IDEs like Eclipse and Visual Studio. Gauge’s functionality can also be extended with its support of plugins.
Gauge is also a tool agnostic test automation framework. It was developed from the start to be a BYOT (Bring Yout Own Tool) framework. So you can use Selenium or you can use anything else for driving your tests UI or API tests.
Gauge is now out of beta. If you want a readable non-BDD approach to automation you might want to give Gauge a try.
6. RedwoodHQ
RedwoodHQ takes a little bit of a different approach from the other tools on this list. It creates a website interface that allows multiple testers to work together and run their tests from one web-accessible location.
Tests can be written in Java/Groovy, Python and C# code to test web applications with Selenium, APIs or databases using their web IDE, which also includes built-in GIT integration.
You can leverage RedwoodHQ’s existing action keywords to get started creating and modifying tests quickly. To create a test script you just need to find the action you want to perform, drag it into your test case and enter in the parameter values it expects.
The built-in IDE also allows you to create and modify test cases and actions as well as the ability to execute your tests. You can also run your tests against predefined machines in parallel and see a running history of all your previous tests runs.
RedwoodHQ is tester friendly and has a lot of built-in features that allow you to create complete test automation and test management solutions for your teams.
7. Cypress.io
Compared to most of the other tools on this list, Cypress is a more developer-centric test automation framework that focuses on making test-driven development (TDD) a reality for developers. One of its design principles was to be able to package and bundle everything together to make the entire end-to-end testing experience pleasant and simple.
It has a different architecture than Selenium; while Selenium WebDriver runs remotely outside the browser, Cypress runs inside of it.
This approach delivers more consistent results since it’s able to understand everything that happens inside and outside the browser. It also gives you native access to every object without requiring you to deal with object serialization or over-the-wire protocols. You’re pulling your application into Cypress.
As a result, it can synchronously notify you of every single thing that happens inside the browser so that you have native access to every distributed object model (DOM) element.
Cypress also makes it easy to simply drop a debugger into your application, which in turn makes it easier to use the developer tools while you’re developing. Of all the automation testing framework tools listed here, Cypress is the one that I hear most developers prefer to use.
8. Katalon Studio
Katalon Studio to a free automation testing tool developed by KMS technologies.
It is not open source but it is free, and this tool can be used to automate on front-end web applications, mobile and web services. So in mobile automation testing frameworks, this might be a good choice for you.
The great thing about this tool is it’s a complete package and framework. So you don’t have to download or install any other components when working with it. All you need to do is download the zip package, and then you unzip it. Start the executer, and you have a very clean and good-looking GUI that you can use to start creating automation test.
Another benefit of using this testing tool is the learning curve is very small, and you don’t need any programming skills to start. You also can use it to create a keyword test automation framework which is popular with some QA teams. If you don't know a keyword-driven approach means that you write a test not with scripting (using a scripting language like JavaScript or any other scripting language), but as a sequence of keywords.
So testers who have no coding experience can, download it and start within 10 minutes to create automation tests cases.
9. TestCafe
TestCafe is an opensource tool built on Node.js by DevExpress.
If you work for a large enterprise and need to support customers that use older browsers, you should definitely give TestCafe a try.
Just like Selenium, it’s free and open source but you don’t need to install any other software or web drivers.
10. White Framework
Are you a QTP/UFT engineer who has wished there was a way to automate .NET and WPF applications using Visual Studio and C# rather than QuickTest Professional or Unified Functional Testing?
Do you need to automate a thick client application like WPF, Silverlight, Win32 or WinForms? If so, then White may be the automation framework for you.
White is an open-source library from Test Stack. Like CodedUI, it has a more narrow scope; it only targets desktop applications, not web applications. Both the Coded UI test automation framework and White are based on UI Automation Library for Microsoft.
There are not many automation testing tools that handle .NET applications so this is a valuable tool for folks that have to test such technologies. It's also easy to get started using White.
11. Selenium IDE
OK, many folks would not consider Selenium IDE an “automation framework” but with the release of the latest version in contains many testing functionalities that you normally would need a test framework for.
For example, it now handles being able to run your scripts in parallel execution mode. It was also designed to run on modern continuous integration and continuous deployment environments. So you can easily pass it all kinds of capabilities from the command line. IN essences it takes care of a lot of things for you out of the box that usually required that you use a framework for.
See my hands-on video demo of all the latest feature in my Selenium IDE Getting Started Guide.
What Open Source Automation Testing Framework Should You Choose?
The preceding automated testing framework open source solutions are just a small sample of the automation testing framework types you have available to you.
What automation testing framework should you choose? There is no right answer. Its whatever one works for you and your team.
But before you write that first line of code, make sure there isn’t a library or framework you can leverage.
Stop wasting your time reinventing the wheel and check out one of these automation testing frameworks first before investing any more time and effort in creating your own framework from scratch.
Open Source Frameworks for API and Performance Testing
Are you looking to automate some no GUI based tests?
If so check out my other post for API, Performance and Visual Validation testing tools:
