How To Do Virtual Augmented Reality Testing

I recently interviewed a few folks on my TestGuild Automation podcast, including BJ Aberle ,Noemi Ferrera, about Shane Evans on how to test virtual reality (VR) and augmented reality (AR) software.

Here are some key takeaways to help you with AR and VR testing.

AR and VR Testing Hype Curve

I think we can all agree that VR and AR are still in their infancy.

BJ pointed out that these technologies have been on the Gartner Hype curve for a long time.

But now we're finally starting to see it moving towards the plane of productivity. Many companies are now seeing that there are more and more uses for augmented reality.

Soon folks will realize they can use AR for much more than just overlaying video on top of a camera view.

Once that happens, there will be a lot more uses for VR and AR.

One thing that’s hard to find right now, since it is so early, is how to test virtual reality programs.

So how do you test a VR or AR app?

How to Test Virtual Reality Applications

There are several ways to approach testing a VR or AR app.

For example, BJ has a three-step process he uses for testing Virtual Realty Apps.

Step one: Freak out.

As is the case with a lot of things, the first time you try it you’ll most likely start freaking out.

What makes VR testing even more nerve-wracking is that it’s also something no one else has done before…because there's probably no other project out there that is exactly like yours.

So there's no model or recipe for you to Google. There's no stack overflow to look at. You have to roll on your own for a while.

Step two: Start testing!

Once your freak-out is over, you can start testing.

The problem with VR testing–especially augmented reality–is that it’s the blackest kind of Black Box testing you can imagine because you're taking input that you think the device should be getting and then trying to determine whether it’s the correct output. It’s not until a period of trial and error that you’re able to become familiar with the app.

Step Three: Work with your developers.

Once you’re somewhat familiar with your app, you should begin working with your developers to come up with some creative solutions for your testing problems.

It's an evolution-type process. Things are fluid, and nothing is set in stone. But following this three-step process should help guide you as you begin your testing efforts.

Tools to Use to Test your AR and VR Software

Many of the things you need to test in AR and VR are manual steps. But there are some tools and techniques you can use to help automate some of the work.

Record Your VR Session to Debug your Application

If you're developing on the Google Android Daydream platform, you can use Chromecast. Chromecast allows you to project what you see in the headset onto a smart TV. It also enables you to record your manual interactions. Or, for a more Agile pair testing approach, you can use it to let developers see what you're seeing as you're testing in real time.

Unity VR Testing Tools

Noemi mentioned that for automation testing of VR/AR, it depends on the device and technologies you're using.

In her case, she uses Unity.

Unity does have some testing possibilities. But it's not end-to-end. In reality, it’s more for unit testing, but it's a start.

BJ recommended that if your team is using Unity developing for DayDream, there is another plug-in available called Google Instant Preview.

What it does is allow you to preview your build in Unity in the editor. It’s pretty cool because instead of having to put on the glasses, you can just run your game and hold the phone and tilt it around. That way, everyone will be able to see what you're doing directly in the editor.

Shane Evans also recently developed GameDriver, which allows your developers to create automated testing functionality by using an API and an agent added to your Unity game project.

UI Test Automation Framework for Games, Apps, VR

Noemi told me about a cool open-source framework she is working on called AirTest, which is a cross-platform, automated testing framework that focuses mainly on games. But it also supports native apps.

It also supports Windows and Android; iOS support is in Open Beta.

AirTest provides cross-platform APIs (including app installation), simulated input, assertion and so forth. It also uses image recognition technology to locate UI elements. That means you can automate tests on games without injecting any code. The tests will generate HTML reports that allow you to locate failed test cases quickly.

Of course, it can also be used with VR games developed in Unity 3D as well. Check them out on their AirTest GitHub page.

Automation of VR Unit Testing

One of the things BJ's team is trying to do with their augmented reality project is to figure out how to do more automated unit-type testing.

So far, they've been able to launch their app in Xcode and inject a still image.

This helped them to test the ability to detect text. They've used this approach to put up the still image and give it a few different variations– like a black and white version or a blurry version–and test it to see if it's still interpreting the text correctly.

BJ, Noemie, and I all agree that unit tests are going to be important going forward.

For me, this is true of any automation testing effort.

Remember: breaking down a complex feature into smaller pieces and testing them separately is probably one of the better overall testing strategies you can use.

Be sure that your program is built with testability in mind right from the start.

For example, having your developers decouple your application objects from the input device will allow you to simulate test input without actually using the physical AR/VR controllers.

In the book Unity Virtual Reality Projects, author Jonathan Linowes gives an example of how to do this for simulation testing with scriptable objects.

Special Consideration for Testing Virtual and Augmented Reality

One of the critical points for testing an AR or VR application–especially AR–is that you have to have a physical environment to test in.

Think about all the things you'd have to do just to set up the tests. For example, BJ mentioned that just to get started testing his projects he has to make sure the database is set up in the right state, ensure he has the correct environment, the right user, the proper Docker container on the right access controls, etc. Just to be ready to test.

On top of all those testing pre-conditions, you have to take into consideration the physical space you’re testing in.

If you're doing augmented reality, for instance, you have to print out markers. Not only that, but they need to be printed out as if you were in the production environment.

What does the production environment look like? Are there a lot of windows? What's the ambient light like in the environment? These are things you usually wouldn't have to consider when testing a more traditional piece of software.

Testing for Motion Sickness

Another consideration is how someone feels when interacting with your virtual reality game or app.

For example, in VR, your eyes are seeing movement, but your body is not feeling it. This situation can cause virtual reality motion sickness

It's the same effect that happens to you when you are on a boat. Not everybody has the same degree of motion sickness.

Noemi has a unique what of testing for this scenario.

She uses her mom.  LOL!

Noemi is used to this experience because she is a gamer.   But her poor mom is not.  So for this test case, Noemi uses her mom as a tester because she has more sensitivity.

Actionable Virtual Reality Testing Advice

BJ's actionable piece of advice is to align yourself with your AR/VR developers. As testers, we're in an awesome position to align ourselves with the development team because the technology is so new. For some of the developers working on your projects, it might be their first time working with these APIs, so it’s a great time for us to educate ourselves as well.

The best way to learn is to get your hands dirty. Go to Unreal or Unity and create a simple AR or VR experience. Within a couple of hours, you'll be able to have something up and running.

This exercise will give you valuable perspective, in context, about how these apps are created as well as ideas on how to approach testing them.

Read More Blogs