Automation Tools for Mainframe Testing

Let me guess.

You’re thinking, “Mainframe testing? Really?”

Believe it or not, mainframes are still the systems of choice for companies that need high-transaction throughput with guaranteed reliability and consistent data. Insurance companies, banks, and the retail, and transportation industries, for example.

It’s human nature for a tester to always be on the lookout for the latest shiny testing tools that only support Web technologies. But what if you work for a company that still uses mainframes, and you need to automate some of your tests?

Mainframe Shift-left Testing

Remember: just because mainframes have been around forever doesn't mean you can't follow modern testing practices.

According to Rosalind, a distinguished engineer at IBM, mainframes run all modern languages (Java, Swift, Node, etc.). Whatever you need to write your application; you can write it in that environment and then run it on the system.

It also supports Git and CI/CD tools like Jenkins, so from a development process and practice, and it works the same any other development environment.

Green Screen Tools for Automation Testing Mainframes

I used to work for a large healthcare company that still used a mainframe application behind the scenes. Even though they had a browser-based application version, most customers refused to upgrade and use the new application because they were so proficient with data entry using the green screen application.

This situation may sound like a nightmare, but it actually turned into a blessing. Because the green screen application was no longer being modified, the UI was very stable. And since it was just a matter of keyboard manipulation to test the app, it was relatively easy to automate.

If you find yourself in the same situation, here are some green screen tools you should check out:

• • Rumba+ Desktop: Micro Focus has a “newer” tool called Rumba+ Desktop that offers a safe, efficient, cost-effective way to boost business performance with anywhere, anytime access from your desktop, Web, and mobile devices.
  • Jagacy 3270 is a 3270 mainframe screen-scraping library written entirely in Java. It can be used to create standalone applications, applets, N-tier client/server environments, and Web services. It supports SSL/TLS, TN3270E, internationalization, and over thirty languages. 
  • Unified Function Test (UFT, formally QTP) can test terminal emulator applications that support HLLAPI (High Level Language Application Programming Interface) as well as those that do not—for example, emulator sessions configured to work with the VT100 protocol (using the Text-only option). HLLAPI allows a PC application to communicate with a mainframe application with extended capabilities.
  • SmartBear’s TestComplete – I'm not 100% sure they have a dedicated add-in, but I'm pretty sure you can use it to send keystrokes to the terminal screen object.
  • Altran is an open-source test library for Robot Framework that enables you to create automated test scripts to test IBM Mainframe 3270.
  • Worksoft Certify is especially powerful for mainframe applications. It not only supports most 3270 and 5250 terminal emulation products, but it has support for directly importing screen maps from formats such as CICS/BMS and AutoDoc. 
  • Mainframe Brute – open source mainframe bruter and screen automation utility. Tool to brute force APPLIDs on a z/OS mainframe where you can connect to VTAM. Truthfully, it's just a useful way of automating mainframe screen interactions, i.e. I make copies of this to perform specific fuzzing/user enum/password bruting attacks against custom apps, test data etc.
  • IBM Rational Functional Tester is an automated functional testing and regression testing tool. It supports a range of applications, such as terminal emulator-based applications, zSeries, iSeries and pSeries.
  • ACCELQ – offers green screen test automation simplified by their codeless automation technology. It handles the following:
    • Natural language-based automation of any 3270 or 5250 terminals
    • In-built support for HLL and COM based terminal emulator
    • Seamless support for IBM PCOM Attachmate emulator clients
    • Hummingbird and extensible HLL and OLE/COM support
    • Validate true end to end flow across front end, backend, API and Mainframes

Chris Haggan a Rational Test Product Manager, mentioned in the comments that “these days we would suggest Rational Integration Tester for mainframe service testing as that supports all the IBM Generic Service Client technologies as well as CICS, DPL, IMS, PL1, Cobol Copybook, and DB2/Z. As with the Generic Service Client, these tests can all be run at scale with Rational Performance Tester (which is where most testers would encounter Generic Service Client).

This is pretty cool, but make sure not all your test cases are written just for green screen terminal emulator testing.

Once companies that have older, back-end systems start getting into more modern development, they usually start creating interfaces to interact with the backend.

That interface could be a Rest service, a Web service, a clicks transaction, an MQ call, or even a file that's being dropped into the environment then picked up and processed.

This is important because many other tools are available to help you with interface testing as well.

Tools for Mainframe Interface and Performance Testing

As I mentioned earlier, many mainframe system services can be tested by running tools outside of the mainframe.

Since many of these functions are exposed by an API and are as easy to access and test as any other API services, you might be familiar with using the same sort of testing tools for functional automation, performance, and monitoring.

• If it’s a Rest or Web service, check out my post on the Top 20 API Testing Tools that are FREE for a list of tools that can help you.
• Topaz Workbench – modern, Eclipse-based IDE for mainframe development, testing, and maintenance.
• IBM Application Performance Analyzer for z/OS – measures and reports how applications use available resources. This easy-to-use tool helps you identify system constraints and improve application performance.
• FreezeFrame from Macro4. Offers application performance analysis for z/OS.
• CA Mainframe Application Tuner – proactively identifies the root cause of app performance problems in production/pre-production.
• StarTool – comprehensive tools for file and data management, I/O optimization, and dump analysis.
• Compuware Strobe – Compuware’s modern performance management and analysis solution for mainframe applications. It lets performance analysts pinpoint application inefficiencies causing excessive CPU consumption and elongated elapsed times to reduce hardware and software costs.
• CA Brightside automates the development, testing, and maintenance of mainframe applications and systems by providing you with commands that execute tasks on the z/OS side. Brightside's commands are client side and run natively on your dev workstation or CI automation box, similar to other popular CLIs like aws-cli. This allows you to integrate Brightside commands with your scripting language, or test or build automation frameworks of choice including Shell/Bash, Python, JavaScript, Mocha.JS and Gulp.
• IBM Generic Service Client – The purpose of the generic service client is to send requests to any service that uses an HTTP, JMS, WebSphere® MQ, or Microsoft .NET transport. The generic service client also displays the response returned by the service. The generic service client is useful for debugging or testing a service when you do not have access to a dedicated client to send the request. You can set up a large variety of transport and security configurations for the service, edit the parameters of the request and send attachments. When a request is successfully invoked, its message return is added to the Request History. You can use this feature to look back at results that were produced at different times.
• REXX – If you are developing or maintaining traditional mainframe-based applications, you have heard of a programming language called REXX (Restructured EXtended eXecutor Language). It can be used to develop small programs that perform various automated steps like text file transformations.

Also, as in traditional, Web-based software development, the developer should be creating unit tests as well.

Mainframe development is no different.

If your mainframe team is just getting started with automation, unit testing is where you should start.

Unit Tests for Mainframes

• XaTester – Compuware recently acquired XaTester, which was designed to deliver essential automated mainframe unit testing. Enables developers to quickly create unit tests for both batch and CICS-based programs written in COBOL, PL/I and Assembler. CICS is the transaction processing technology of choice for many of the world’s largest enterprises—especially financial institutions and insurance companies—due to its extremely high integrity, performance and scalability.
• zUnit – The zUnit feature of Developer for z Systems® provides a code-driven unit testing framework for Enterprise COBOL and PL/I. zUnit provides an automated solution for running and verifying Enterprise COBOL and PL/I unit test cases that are written by using the zUnit framework.

Rosalind mentioned that zUnit is relatively new; even though it’s been around for a while; it's been significantly improved and continues to grow to allow you to do much more including true unit testing. Well, with the caveat that in a COBOL program, you don't have the same concept of classes and methods. You have a large COBOL program. So you're testing a COBOL program with a unit test, but it's still at the program level, which is a whole lot better than the transaction on the application level.

The takeaway is that if you are testing a COBOL application, you can still create unit tests for your program!

Overall Mainframe Testing Approach

Generally, your testing strategy for mainframe applications should be the same as any other development efforts.

The only difference is that whole mainframe systems provide you with much more data. So you can get more info about how that application is performing, which is nice, but it can also be overwhelming if you don't know what to do with it.

The other thing is that it depends on the application's language. So if a newer language like Java or Node was used to develop your software, then all of the processing you do anywhere else is precisely the same. It's simply running on a different set of hardware.

If it's COBOL, then the application was probably written differently, which means you will be testing it differently. For example, you can't do a method test.

You have to test at a program or transaction level. There are some differences from that standpoint.

The tools are going to be somewhat different from the same point of debugging the system if you want to debug something in the environment.

But most testing tools that are going to hit an API are going to hit an interface. They work with a mainframe application precisely the same.

So it depends on the tool you're using, and it depends on what you're testing as far as the differences you might see when planning your testing process.