Back in the day, I was part of a large QA team that didn’t use on-demand environments.
We had a lot of test automation to run for every new build, and basically, only one or two environments where those tests could run.
This became a bottleneck for the developers because the tests took too long to run to give them confidence and quick feedback that the code they checked in didn't break the build.
Over the years, I've heard from other testers who are also struggling with tests environments that are either not meeting the proper requirements, have too many testers using them, are flaky, or are nowhere close to production environments.
If this sounds familiar, keep reading to discover how having the ability to spin environments as a service up and down on demand can help you deliver top-quality products without worrying about environmental blockers.
Using an on-demand environment solution can help you solve most of these issues.
Table of Contents
- What are on-demand environments?
- Benefits of on-demand environments
- Some concerns when using Environments as a Service
- Why environments play a critical factor in Agile development
- Non-Functional Testing Environments
- Build or Buy Test Environments?
What are on-demand environments?
An on-demand environment is one where you can spin up a temporary, fully-functioning environment and test a feature independently and without any other dependencies before it’s deployed to the production environment. And close the environment when you're done.
I learned about the need for scalable infrastructure in development from Tommy McClung, the CEO of Release, on my test automation podcast.
He reminded me that with this approach, you can run a dedicated test on your features before being merged into the main branch without blocking your team members' ongoing work.
Tommy also said it's common to see teams using a setup with a shared staging environment where many features are mashed together, often motivated by an ineffective release process.
This causes many issues.
One is not knowing which code check-in may have caused the build to break.
However, using feature environments instead enables you to have a staging environment for every short-lived feature branch you create.
This means that QA for a feature can access and test that feature separately without hindering other development works.
Cool, right?
Benefits of on-demand environments
There are many benefits to an on-demand environment, including the following:
- Quicker time to market
- Ensures the features released are tested and verified
- No dependency to release features
- Increasing product quality
- Run experiments and spikes in isolated environments
- Hotfix doesn't need to override the builds in the staging environment or block any other development works
- No single Integration Testing Environment.
- Happier teams
If you're building environments yourself, all of the things listed above are concerns that you will have to handle.
However, when you take in environments as a service system out of the box, you can create infinite environments that allow you to parallelize your software development and automation testing—no need to waste time doing it manually.
Some concerns when using Environments as a Service
When you have infinite environments, you need to pay attention to cost.
AWS gets happy when un-used environments are left up and running since this causes your bill to increase.
Using a service like Release can help you avoid this issue since it has built-in management features that allow you to have your environments automatically destroyed after no usage.
It also handles the automation of creating policies that allow you to define things like how many environments to allow; you can cap it and put a limit on it to keep your bill lower.
The idea of like-cost savings is fundamental since it's a core part of what an environment-as-a- service-platform helps you with.
Why environments play a critical factor in Agile development
When creating modern software in a faced-paced, Agile, continuous delivery team, having a solid plan for your environment is critical to success.
For example, when you're frequently delivering software to customers, having environment management features like the ability to create a snapshot in time is crucial.
It means that if something goes wrong, it's easy to see what the environment and code looked like when the issues began to occur. This helps speed up debugging problems that could also slow down your rapid delivery.
I used to work as a tester in a regulated industry.
If the FDA ever audited us, we needed to be able to show the exact environment, configuration, and tests that ran for any given released build.
The only way to do that was to use on-demand environments.
This approach also leads to:
- Cost optimization
- Enhanced productivity
- Higher resilience
- Higher quality products for our customers
Non-Functional Testing Environments
Another area that many teams struggle with due to environmental constraints is non-functional testing native-like performance testing.
In the case of performance, it’s best if you have a way to scale all the pieces of your infrastructure quickly to see how far you can put a load onto your system.
Once again, on-demand environments can help you create a template to quickly go in and say, “Okay, I want five replicas I want 10. I want it to auto-scale to start running my load testing tool of choice against that environment. How far can you push it before seeing bottlenecks in your ecosystem?
This allows you to test in isolation and keep scaling until your software breaks so you can see where your limiters are.
More importantly, you don't need to touch other environments that others on your team might use.
In my experience, just getting an environment set up for proper performance tests is a struggle.
I once worked on a team that manually had to have multiple folks manually configure their piece of the software on an environment to get it to work.
There wasn't one available person who had the knowledge to set up such a complex environment from scratch.
It was rough because it sometimes took weeks/months to get a stand-alone environment set up.
Using on-demand environments solved this issue for us by allowing us to focus on a bunch of testing activities that were previously difficult to perform, including performance testing.
Build or Buy Test Environments?
Of course, you can try to use existing tools to help you build your own environments.
For instance, you can declaratively create them in Terraform or other similar tools.
However, there's a lot of tooling required to help make a larger team effective with this approach.
Tommy also mentioned that when these environments come up, they’re usually in an 80% working state.
These environments are often lacking, because you may not know what dependencies your software or customers are using.
With a tool like Release, you can back all the knowledge into a template that all environments are created from. So you end up with a definition of your environment that becomes your gold standard.
Hopefully, by now you can see why having the ability to create on-demand environments is vital as a tester.
But just in case, read on for some other benefits.
Benefits of Having an Automated Testing Environment
- Reduction in manual efforts
- Reduction in operational costs
- Elimination of human errors
- Getting the environment as a service model for the company
- Eliminates bottlenecks that hold you back from scaling your automation testing
- Very easy for developers and testers to create their own test environments
Other benefits offered up by Tommy: “The premise here is that for every branch, every pull request an environment that is a full-stack, everything front-end, back-end databases, and cloud-native resources can be created just on the branch you're working on. Let's say you have 100 developers. You could have 100 environments all on separate branches; there's no contention among the developers. Everybody has their isolated copy where they can test, debug, run, automate, preview.”
As a tester, this is also freeing because you don't have to worry about breaking anything and impacting others.
Tommy went on to say,
“The nice part is you're testing in a throwaway environment. So if you completely screw everything up and hose everything, you're not impacting anybody else. You're just messing it up yourself and realizing, okay, we're highly dependent on AWS, or we're highly dependent on GitHub or whatever it might be. So now I can work on that problem without impacting anybody else. And usually, a lot of that can only be really done within a shared staging environment because it's the only place where all of those connections and the dependencies are created. After all, it's tough to define all of those. Within Release, if you define those dependencies now, you can go play around with shutting those off, turning them back on, see what happens to your application, and you don't impact anybody else.”
TL; DR
On-demand environments are very practical; they help ensure faster delivery and increase utilization and help developers and testers avoid the worry of deploying any untested features or cherry-picking feature tickets to test.
One can test features in isolation and release them as a whole.
- There are more testing opportunities
- It helps avoid environmental drifts
- More stakeholder engagement
- Increased developer productivity
- Shortens cross-functional feedback loops
